[zeromq-dev] A "reliable" PUSH/PULL pattern?

[Pieter Hintjens]

Hi Randall,

I never got around to building a reliable pipeline pattern, but I
think it's pretty straight forward.

The general model is you send out X pieces of work to W workers, who
send replies back to a sink. The potential failure is a worker dying
and losing one or more messages. The recovery is to resend the entire
batch. If the failure is rare enough, this is a sensible approach. If
the failure is common, you probably want to switch to a more
synchronous request-reply model like Majordomo.

-Pieter

On Sat, Jun 29, 2013 at 11:37 PM, Randall Nortman <rnzmq at wonderclown.net> wrote:
> My apologies if this is somewhere in the guide and I've missed it.
> Most of the "reliable" patterns in the guide seem to be about
> request/response patterns.  My desired behavior is:
>
> - A producer binds a socket
>
> - A consumer connects to the producer (one consumer, one producer)
>
> - The producer queues up to N messages for a consumer.  As long as the
>   queue has not reached N, messages are only removed from the queue
>   once the producer is certain that the consumer has received the
>   message.  Re-delivery of the same message is acceptable.  Once the
>   queue reaches N messages, the producer discards the OLDEST message,
>   so that always the most recent N messages are queued.  When the
>   consumer comes back, the N most recent messages are transmitted
>   (with confirmation of receipt before discarding, subject to queue
>   size, as usual).
>
> Based on my reading of the guide, ZMQ doesn't have anything out of the
> box that looks like this.  ZMQ's buffers and HWM simply don't work
> that way; if anything, they make it slightly harder to reliably do
> what I need to do.  The closest I seem to be able to come up with is a
> PUSH with the high water mark set to 1 and configured for non-blocking
> sends.  If a send would block, then I instead put the message in a
> circular queue that I maintain myself.  And I am pretty sure that even
> that solution may occasionally drop a message when the consumer
> disconnects, because the producer won't notice right away, and
> whatever message was in flight at the time will be lost.
>
> Beyond that, it seems like I need to use one of the reliable
> request/response patterns, using the responses as confirmation of
> receipt (upon which I remove the confirmed message from my queue).  It
> would be nice to reduce network traffic by allowing the consumer to
> confirm multiple messages in one response (rather than one
> confirmation for every message).  That's obviously no longer simple
> REQ/REP, but probably something involving more than one socket on each
> end.
>
> It seems like the only benefit I get from ZMQ if I want this kind of
> behavior is that it handles reconnecting automatically.  I'm kind of
> feeling like I might as well use raw UDP, because I'm going to have to
> do everything myself anyway.  (I am not knocking ZMQ.  I'm using it
> for lots of other things that it's well-suited for.  It just seems
> this isn't one of those things.)
>
> Am I missing an easy way to do it?
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev