[zeromq-dev] Why is this Queue device blocking?

[Chuck Remes]

On Oct 19, 2011, at 12:18 PM, James Thornton wrote:

> On Wed, Oct 19, 2011 at 9:17 AM, Ian Barber <ian.barber at gmail.com> wrote:
>> 
>> Code looks all right - but I notice there's a solved notice there now
>> so I guess it got fixed :)
> 
> Hi Ian -
> 
> Yes, Chuck was on IRC last night and helped me out.
> 
> One other thing -- what's the right way to change the load-balancing
> algorithm when pushing data to the workers?
> 
> I threw in a conditional sleep() for all the workers except one to see
> how it handled it, and looks like a simple  round robin because it
> blocked on the sleep rather than using the first available worker. I'm
> going to read through the guide again today, but any insight you can
> provide would be appreciated.

What you are probably seeing is a side effect of not setting a high water mark on your producer's PUSH socket and how messages get buffered in various places.

For example, I have a PUSH/PULL setup where there is 1 producer with a HWM set to 1 and 3 workers. When I start it up, I make sure to wait until all workers have started and successfully connected to the PUSH socket (they indicate this via another socket). When I start sending tasks via the producer, it sends *9* tasks out before it blocks.

You might think that the HWM of 1 would only allow it to send out 3 tasks, but that doesn't take other buffers into account. Probably what is happening here is this:

1. Each PULL socket receives a single message and begins work. (3 tasks consumed)

2. Each PULL socket enqueues 1 more tasks that has not yet been read by the consumer. It's in queue. (3 more tasks sent for a total of 6)

3. Each PULL socket indicates to the upstream PUSH socket that the queue depth is 1.

4. Before the update from #3 is seen by the PUSH socket, it sends another set of tasks out. (3 more for a total of 9)

5. PUSH socket gets back-pressure from PULL sockets; blocks on send.


If you are testing the load balancing just by using a random sleep, then each worker will get about the same number of messages. To really verify that it is sending messages only to available workers, make all but 1 of your workers block forever. You will see that new tasks sent out will only be handled by the worker that is not blocked.

Also, sent the HWM on your device's PUSH socket to 1.

Corrections welcome. :)

cr