[zeromq-dev] HWM behaviour & blocking

[Michel Pelletier]

On Thu, May 10, 2012 at 1:53 PM, Pieter Hintjens <ph at imatix.com> wrote:
> On Thu, May 10, 2012 at 3:44 PM, Paul Colomiets <paul at colomiets.name> wrote:
>
>> Can you be more specific, why setting HWM to 1 is a bad thing? Do you
>> mean, that it smells bad to set HWM to 1 for reliability? Or do you
>> think that setting it will have other consequences? (low performance?)
>
> it's bad because you're trying to force a synchronous model on an
> asynchronous system, and doing it at the wrong level. If you really
> want synchronization you MUST get some upstream data from the
> receiver. Just throttling the sender cannot work reliably.

Agreed.  Here's my take on what trips a lot of people up with 0mq:  we
are used to controlling how and when something is sent at the point
that we call "send()", or at least knowing in advance what will happen
if we try, but in an async model you have to let that go.  send() is
going to return immediately (if you haven't hit a blocking case) and
your message is now on its own, free as a bird, to live in various
queues and buffers before it ends up at its destination.  You have no
control or visibility of its fate after you send it unless your
receiver acknowledges it, or acknowledges it didn't receive it after a
period of time (nack).

The blocking case isn't really an exception, you sent when your
application wasn't ready to receive, either because your buffers were
full or your receivers weren't ready.  Senders and receivers should
synchronize this application level state with each other, possibly via
some out-of-band channel, either by indicating they are ready, or
connected, or that they are busy and can't do anymore, or by
exchanging some kind of flow control information so that the sender
doesn't fill the buffers because the receiver can't keep up.

To use an analogy, all 0mq provides are the pipes.  The pipes can't
tell you that the tap is running and the sink is overflowing or the
drain is clogged.  If you want to have a reservoir at some point to
regulate flow, an inline "device" can store a certain capacity of
messages.  If your pipe is delivering to a downstream reservoir which
is near full capacity, someone at the downstream end needs to pickup
the phone (yet another pipe of sorts) and tell the upstream to turn
down the flow.  If that doesn't happen, the reservoir is full, and the
flow stops (blocked) or maybe spills over (discards) depending on the
design of the *application*, not the pipe.  In either case it's not
the pipe's fault, it did its job, it can't solve your design problems
any more than a pipe can become a city water system all by itself.
Adding all these application level flow semantics to the "pipe" is not
right, and would horribly complicate the library.

-Michel