[zeromq-dev] Requirements for reliability

Pieter Hintjens ph at imatix.com
Tue Apr 6 12:23:36 CEST 2010


John,

OK, it's true that a blog will be better for capturing the discussion.
 I've posted this to
http://www.zeromq.org/blog:requirements-for-reliability.

-Pieter

On Tue, Apr 6, 2010 at 12:09 PM, Apps, John <john.apps at hp.com> wrote:
> Pieter,
>  May I suggest you start a blog entry on this? The list is interesting, but a clumsy medium for discussions of this nature.
>
> -- John.Apps at hp.com | +491718691813 | http://twitter.com/johnapps --
>
>
> -----Original Message-----
> From: zeromq-dev-bounces at lists.zeromq.org [mailto:zeromq-dev-bounces at lists.zeromq.org] On Behalf Of Pieter Hintjens
> Sent: Tuesday, April 06, 2010 11:06
> To: zeromq-dev at lists.zeromq.org
> Subject: [zeromq-dev] Requirements for reliability
>
> Hi,
>
> One of the main questions that AMQP users have when coming to 0MQ is
> how we do reliability.
>
> So I'm collecting requirements for reliability on top of 0MQ.  My idea
> is to build this as an application layer on top of 0MQ, using a
> different API and a separate protocol that sits on top of the 0MQ SPB
> framing.
>
> Here is a sketch of what seems to be the simplest design for
> fire-and-forget reliability:
>
> * It covers only one-to-one delivery of messages (e.g. for trade
> execution or request-response).
> * The publisher delivers a message to a local dispatcher service
> (perhaps based on a 0MQ device).
> * The dispatcher queues the message on persistent storage and the
> publisher continues.
> * The dispatcher in a second thread delivers messages to their destination.
> * Recipients reply to messages with an acknowledgment.
> * Recipients store received messages and if they get a duplicate, they
> discard it but resend the acknowledgement.
> * The dispatcher marks acknowledged messages, and ignores duplicate
> acknowledgments.
>
> In the most brutal implementation, both sides use a light database to
> hold messages, and store everything.  There are more intelligent ways
> to hold persistent queues but that's an implementation detail.
>
> The above design covers crashes in the publisher, consumer, and
> network afaics.  Throughput would depend on message size, since
> messages can easily be batched and acknowledged in blocks.
>
> Has anyone been implementing such reliability on top of 0MQ already?
>
> -Pieter
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>



More information about the zeromq-dev mailing list