[zeromq-dev] A "reliable" PUSH/PULL pattern?

[Randall Nortman]

So by "[workers] send replies back to a sink", you are talking about a
separate socket for the workers (consumers in my model) to communicate
back to the producer?  So use a PUSH/PULL (or PUB/SUB) socket to send
messages to the consumer, and have a separate socket (another
PUSH/PULL or PUB/SUB, but with roles reversed) by which the consumers
confirms receipt of each message (or each batch of messages)?  If the
producer receives an out-of-order confirmation, that indicates that
either some messages were lost, or the confirmation was lost.  Either
way, the producer resends the lost messages and the consumer sorts it
out.

My main gripe with that is that it sets up two TCP streams for what
could be one, which is not very efficient.  I guess you could use a
DEALER socket on each end.  The guide has little to say on
DEALER-DEALER connections, but it seems to me like this would work,
right?  (Keeping in mind that my application is one-to-one, not
many-many or one-many, but in principle I think DEALER-DEALER would
support one-many.  I think many-many would require ROUTER, yeah?)  The
producer sends messages, and occasionally the consumer sends back
receipt confirmation messages over the same socket.  Credit-based flow
control could be used.

FYI, my expectation for this application is that short-duration
failures (in the form of long delays if not complete disconnection)
are fairly frequent, because this communication is happening over a
slow, unreliable wide area network.  (Specifically, the producers are
connected via cellular modems.  Egad!  The consumers sit in comfy data
centers.) But big/long failures should be rare.  I expect to queue
messages for up to 24 hours before discarding.  The total data volume
is low enough for this to be feasible.  But I also have to keep an eye
on bandwidth efficiency due to data limits on the cell modems.

On Sat, Jun 29, 2013 at 11:59:35PM +0200, Pieter Hintjens wrote:
> Hi Randall,
> 
> I never got around to building a reliable pipeline pattern, but I
> think it's pretty straight forward.
> 
> The general model is you send out X pieces of work to W workers, who
> send replies back to a sink. The potential failure is a worker dying
> and losing one or more messages. The recovery is to resend the entire
> batch. If the failure is rare enough, this is a sensible approach. If
> the failure is common, you probably want to switch to a more
> synchronous request-reply model like Majordomo.
> 
> -Pieter
> 
> On Sat, Jun 29, 2013 at 11:37 PM, Randall Nortman <rnzmq at wonderclown.net> wrote:
> > My apologies if this is somewhere in the guide and I've missed it.
> > Most of the "reliable" patterns in the guide seem to be about
> > request/response patterns.  My desired behavior is:
> >
> > - A producer binds a socket
> >
> > - A consumer connects to the producer (one consumer, one producer)
> >
> > - The producer queues up to N messages for a consumer.  As long as the
> >   queue has not reached N, messages are only removed from the queue
> >   once the producer is certain that the consumer has received the
> >   message.  Re-delivery of the same message is acceptable.  Once the
> >   queue reaches N messages, the producer discards the OLDEST message,
> >   so that always the most recent N messages are queued.  When the
> >   consumer comes back, the N most recent messages are transmitted
> >   (with confirmation of receipt before discarding, subject to queue
> >   size, as usual).
> >
> > Based on my reading of the guide, ZMQ doesn't have anything out of the
> > box that looks like this.  ZMQ's buffers and HWM simply don't work
> > that way; if anything, they make it slightly harder to reliably do
> > what I need to do.  The closest I seem to be able to come up with is a
> > PUSH with the high water mark set to 1 and configured for non-blocking
> > sends.  If a send would block, then I instead put the message in a
> > circular queue that I maintain myself.  And I am pretty sure that even
> > that solution may occasionally drop a message when the consumer
> > disconnects, because the producer won't notice right away, and
> > whatever message was in flight at the time will be lost.
> >
> > Beyond that, it seems like I need to use one of the reliable
> > request/response patterns, using the responses as confirmation of
> > receipt (upon which I remove the confirmed message from my queue).  It
> > would be nice to reduce network traffic by allowing the consumer to
> > confirm multiple messages in one response (rather than one
> > confirmation for every message).  That's obviously no longer simple
> > REQ/REP, but probably something involving more than one socket on each
> > end.
> >
> > It seems like the only benefit I get from ZMQ if I want this kind of
> > behavior is that it handles reconnecting automatically.  I'm kind of
> > feeling like I might as well use raw UDP, because I'm going to have to
> > do everything myself anyway.  (I am not knocking ZMQ.  I'm using it
> > for lots of other things that it's well-suited for.  It just seems
> > this isn't one of those things.)
> >
> > Am I missing an easy way to do it?
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>