[zeromq-dev] Order by which PUB/SUB requests are sent/processed

[Joegen Baclor]

On 01/29/2012 08:47 AM, Gary Wright wrote:
> On Jan 24, 2012, at 10:29 PM, Joegen Baclor wrote:
>> Interesting read but it requires that the clients have a birds eye view
>> of the hash ring which IMO is an over kill for what we want to achieve.
>> I think this approach is better done if the server is doing the
>> spreading.  I would gladly settle for X_REQ/X_REP to achieve that but
>> unfortunately the dealer/router model does not fit the bill for this
>> particular app.  I do have control over the queue where the tasks are
>> popped and i think this is the best place to spread the load evenly and
>> the hash ring approach might just be what we need to achieve it.   I
>> really appreciate the input.
> Maybe I'm missing a detail, but in your use case it would seem like a
> simple hash is all that is needed.
>
> If all three workers know their own identity (0, 1, 2) a simple hash
> of the message mod 3 should let them determine who should pop the
> message off the queue and who should go do something else.
>
> Gary Wright
>

Not quite that simple.  If it happens that one or two of them is not 
available, restarted.  dead etc, the hash will break.   Thus there 
should be real time monitoring of the hash in such a manner that no 
single event is missed or drop becuase of any sort of failure.   
Although it does the work, the hash approach is an overkill for us.  
What we did right now is to simply yield the workers n% of the time.  In 
the case of 3 participants, each would yield the processor time-slice 
2/3 of a time.  This is somehow enough to spread the share.  Since its a 
yield instead of a back-off, the data still has the same number of 
participants and without any danger of being dropped.   This has its 
draw back as well because there is no guaranty that NOT all 3 
participants are yielding in which case it would slow down the 
processing of events.  This is acceptable for now because the purpose of 
the application is not time sensitive nor mission critical.

Joegen