[zeromq-dev] PUSH/PULL HWM documentation

[James Cipar]

Hi.  I found something that may be an error in the ZMQ documentation regarding "load balancing" with PUSH/PULL sockets.  At best, it's pretty confusing.  There are two issues.  The first is that the documentation keeps saying that when a pull socket has multiple connections, messages are "load balanced" between them.  It's probably more accurate (or at least more specific) to say that they are "round-robined".  If the verbing of "round-robin" seems too informal, maybe say that they are "scheduled with a round-robin policy".  "Load balanced" is vague, but it makes me think that there is an active effort to distribute more work to faster consumers and less work to slower consumers.  It's probably best to be specific, just to clear up any confusion.

The second problem refers to the documentation of the high water mark for push/pull sockets.  http://api.zeromq.org/2-1:zmq-socket seems to suggest that setting the HWM for the push socket is sufficient to cause the system to block on pushes, and that setting the HWM for the pull socket has no effect.

I was trying to use the HWM to fix the first problem (round-robin instead of true load balancing) by having the producer block until there is a consumer actually ready to handle work.  I found that to get the desired effect I had to set a HWM for both the push socket and the pull socket.  Setting either one would revert to round-robin scheduling.  I've put up my example code below.  Sorry it's so long, the boost threading primitives are pretty ... primitive.  I'm using the C++ interface for 0MQ 2.1.7, with inproc sockets.  Disabling either of the "setsockopt" calls will make the program take 10 seconds because it is waiting for the slow consumer.  However, with both of the calls enabled it will only take 2 seconds.

Ideally I would be able to set the HWM for the push socket to 0 to allow it to enqueue any number of messages without blocking, and set the HWM for the pull sockets to 1 so that they will only have 1 message queued at a time.  That way my producer thread would not have to block on the consumers, but the messages would still be load-balanced between the consumers.  Even better, but probably breaking the 0MQ model, would be to have no queue at the consumer side, so that the entire program would complete almost instantly.  That is, while the slow consumer is sleeping the fast consumer would eat up all the message (none being queued up on the slow consumer's queue).

http://pastebin.com/zvPWYmFF
compile with "g++ pushpull.cpp -o pp -lzmq -lboost_thread"