[zeromq-dev] HWM behaviour & blocking

Viet Hoang (Quant Edge) viet.hoang at quant-edge.com
Mon May 14 17:36:08 CEST 2012

The Major Domo pattern may suits this well. Receiver to ACK for each message. If you don't receive, just resend. However you sacrifice the beauty of ZeroMQ - SPEED. We applied Major Domo into our demo platform and so far so good (better than our old C# raw socket implementation). What we will do is to have async ACK to improve performance.

On May 11, 2012, at 6:44 AM, Michel Pelletier wrote:

> On Thu, May 10, 2012 at 1:53 PM, Pieter Hintjens <ph at imatix.com> wrote:
>> On Thu, May 10, 2012 at 3:44 PM, Paul Colomiets <paul at colomiets.name> wrote:
>>> Can you be more specific, why setting HWM to 1 is a bad thing? Do you
>>> mean, that it smells bad to set HWM to 1 for reliability? Or do you
>>> think that setting it will have other consequences? (low performance?)
>> it's bad because you're trying to force a synchronous model on an
>> asynchronous system, and doing it at the wrong level. If you really
>> want synchronization you MUST get some upstream data from the
>> receiver. Just throttling the sender cannot work reliably.
> Agreed.  Here's my take on what trips a lot of people up with 0mq:  we
> are used to controlling how and when something is sent at the point
> that we call "send()", or at least knowing in advance what will happen
> if we try, but in an async model you have to let that go.  send() is
> going to return immediately (if you haven't hit a blocking case) and
> your message is now on its own, free as a bird, to live in various
> queues and buffers before it ends up at its destination.  You have no
> control or visibility of its fate after you send it unless your
> receiver acknowledges it, or acknowledges it didn't receive it after a
> period of time (nack).
> The blocking case isn't really an exception, you sent when your
> application wasn't ready to receive, either because your buffers were
> full or your receivers weren't ready.  Senders and receivers should
> synchronize this application level state with each other, possibly via
> some out-of-band channel, either by indicating they are ready, or
> connected, or that they are busy and can't do anymore, or by
> exchanging some kind of flow control information so that the sender
> doesn't fill the buffers because the receiver can't keep up.
> To use an analogy, all 0mq provides are the pipes.  The pipes can't
> tell you that the tap is running and the sink is overflowing or the
> drain is clogged.  If you want to have a reservoir at some point to
> regulate flow, an inline "device" can store a certain capacity of
> messages.  If your pipe is delivering to a downstream reservoir which
> is near full capacity, someone at the downstream end needs to pickup
> the phone (yet another pipe of sorts) and tell the upstream to turn
> down the flow.  If that doesn't happen, the reservoir is full, and the
> flow stops (blocked) or maybe spills over (discards) depending on the
> design of the *application*, not the pipe.  In either case it's not
> the pipe's fault, it did its job, it can't solve your design problems
> any more than a pipe can become a city water system all by itself.
> Adding all these application level flow semantics to the "pipe" is not
> right, and would horribly complicate the library.
> -Michel
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20120514/1e6b98e7/attachment.htm>

More information about the zeromq-dev mailing list