[zeromq-dev] Load balancing REQ/REP sockets

Martin Sustrik sustrik at 250bpm.com
Thu Mar 18 18:55:02 CET 2010

Hi Brian,

> I have done some further testing with simple ZMQ_HWM on P2P sockets.
> * I am using a pair of P2P sockets, both with HWM=LWM=2.
> * No matter what sequence of sends/recv I do, messages are always sent
> immediately (I see this on wireshark).
> * I put in debug statements into pipe.cpp.  I can see the various
> calls that are managing hwm/lwm so I know something is happening.  I
> dont' yet understand the reader_t/writer_t logic in pipe.cpp or the
> desired behavior enough to really debug this.

It works a bit different. Think of it this way: you have a sender and a 
recevier connected by P2P sockets. Recevier gets a message and starts 
processing it. Say the processing will take several hours. In the 
meantime sender is sending messages. Messages get to the receiver's I/O 
thread and are put into queue. Say HWM on the receiver side is set to 
10. When there are 10 messages in the queue, I/O thread stops reading 
from the underlying BSD socket. Still, there's a RECVBUF in the kernel. 
Thus more data keep flowing in till the TCP RECVBUF (128kB by default) 
fills up. At that point TCP stops receiving more data and start dropping 
packets. At that point packet queue at the switch starts filling up, 
once it's full the switch starts dropping packets. At that moment 
sender's TCP SENBUF starts filling up with undeliverable data (128kB by 
default). Once it's full, sender's underlying BSD socket stops accepting 
more data. Thus 0MQ I/O thread stops reading from the sender's queue and 
writing to the BSD socket. The sender's queue start filling up. Say HWM 
is set to 10. As soon as there's 10 messages in the sender's queue is 
stops accepting more messages and zmq_send blocks (or returns EAGAIN if 
in non-blocking mode).


More information about the zeromq-dev mailing list