[zeromq-dev] Load balancing REQ/REP sockets

Brian Granger ellisonbg at gmail.com
Thu Mar 18 19:14:39 CET 2010


Thanks for the clarification.  This is quite different from how I was
thinking about it.

* Is the process lossy?  Do any messages get lost when the SENDBUF/RECVBUF
fill up?

* Could we add this description to the docs?

* Can I think of the sender stopping sending messages at HWM +

But back to my original question of having a more dynamic load
balancing.  In my case I have small messages that take different
amounts of time to process.  Often, the total number of these small
messages won't come close to filling up the default sized
SENDBUF/RECVBUF.  Obviously, the next step is to reduce the size of
SENDBUF/RECVBUF.  What is the effect of doing that on
latency/throughput?  Can you think of any other way of handling a this
type of load balancing?

Cheers and many thanks,


> It works a bit different. Think of it this way: you have a sender and a
> recevier connected by P2P sockets. Recevier gets a message and starts
> processing it. Say the processing will take several hours. In the meantime
> sender is sending messages. Messages get to the receiver's I/O thread and
> are put into queue. Say HWM on the receiver side is set to 10. When there
> are 10 messages in the queue, I/O thread stops reading from the underlying
> BSD socket. Still, there's a RECVBUF in the kernel. Thus more data keep
> flowing in till the TCP RECVBUF (128kB by default) fills up. At that point
> TCP stops receiving more data and start dropping packets. At that point
> packet queue at the switch starts filling up, once it's full the switch
> starts dropping packets. At that moment sender's TCP SENBUF starts filling
> up with undeliverable data (128kB by default). Once it's full, sender's
> underlying BSD socket stops accepting more data. Thus 0MQ I/O thread stops
> reading from the sender's queue and writing to the BSD socket. The sender's
> queue start filling up. Say HWM is set to 10. As soon as there's 10 messages
> in the sender's queue is stops accepting more messages and zmq_send blocks
> (or returns EAGAIN if in non-blocking mode).
> Martin

Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com

More information about the zeromq-dev mailing list