[zeromq-dev] How to properly close a socket and HWMRECV

[Antonio Teixeira]

Hello Brian.

Without going to deep how do you keep the Workers "Waiting" at some time you
do  socket.send(GRAB ME WORK) and at the Broker side you check if you have
work  ( IN DB / Memory , etc ) then write back something like  send('NO
WORK') or send('HERE IS WORK') and if there is no work you go to sleep on
the "worker side"

P.S I think this topic is of great importance to everyone developing
scalable apps and distributed systems , im my use case i'm wiritng an ASYNC
layer for my company telephony system ( Crappy Asterisk Boxes) , to do stuff
like moving call records , around.
This type of topics also allow real life application of ZMQ and some of
strong and weak points.

Thanks
António


2011/7/11 Brian Knox <taotetek at gmail.com>

> For what it's worth -
>
> I started out going down the path you are on with push / pull.  However I
> ran into issues with HWM not doing quite what you think it should be doing.
> You'll end up with far more tasks than your HWM in your pipeline due to
> messages being buffered in other buffers before you ever hit the HWM. With
> around 200 byte messages we've seen tens of thousands of messages end up in
> the pipeline, even with a HWM of 1.
>
> I experimented with two ways of handling this:
>
> 1. Use push / pull queue, but have a task sink process counting all
> completed tasks and acknowledging "batches". For example, I specify only 100
> tasks are allowed to be "in flight" at a time.  The ventilator then sends
> 100 tasks, and waits for an ACK that all 100 are processed before sending
> 100 more.
>
> This of course has issues if you have tasks that take different amounts of
> processing time to handle.  It can also get complex as you need to handle
> the error case of a task getting lost (and therefore your full batch not
> getting acknowledged).
>
> 2. Use a req / rep pattern, with workers requesting a new task when they
> finish one.  This is the model I'm currently using.  This works well, even
> in cases where tasks are in the millisecond range to process.  In prototypes
> I've easily distributed 50k to 100k tasks a second to a pool of workers
> using req / rep.  Since a worker signals it's done and wants another task,
> work gets distributed correctly, and you will have 1 task per worker "in
> flight" at a time automatically.
>
> Brian
>
> On Mon, Jul 11, 2011 at 7:29 AM, Antonio Teixeira <eagle.antonio at gmail.com
> > wrote:
>
>> Yes really depends on the workload but we could also have work that only
>> takes a few , seconds, or just one second.
>>
>>
>> 2011/7/11 Martin Sustrik <sustrik at 250bpm.com>
>>
>>> On 07/11/2011 01:21 PM, Antonio Teixeira wrote:
>>>
>>>  You are constantly exchanging traffic , this impacts the network Level ,
>>>> Machine link will be occupied with both "actual work" but also with
>>>> load  in order to get Work.
>>>>
>>>
>>> IIRC you said that processing a task takes 30 mins. Thus, the network
>>> overhead is negligible.
>>>
>>> Martin
>>>
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110711/163a29cc/attachment.htm>