[zeromq-dev] How to implement lossless pub-sub with ZeroMQ?

Pieter Hintjens ph at imatix.com
Tue Nov 3 16:06:51 CET 2015

You need something in the middle. The zeromq/malamute project is
designed for this (it does not yet implement persistent pubsub

On Tue, Nov 3, 2015 at 11:35 AM, Antti Karanta
<Antti.Karanta at neromsoft.com> wrote:
> I was wondering about loss of messages I observed with PUB-SUB although
> I had set send and recieve high water marks to 0. I reread the
> documentation and sure enough, PUB-SUB is designed to lose messages in
> some circumstances:
> "The ZeroMQ pub-sub pattern will lose messages arbitrarily when a
> subscriber is connecting, when a network failure occurs, or just if the
> subscriber or network can't keep up with the publisher."
> http://zguide.zeromq.org/php:chapter5#Pros-and-Cons-of-Pub-Sub
> Thus PUB-SUB is not really suited for my use case as loss of messages,
> while not fatal, is highly inconvenient. Detecting loss of messages will
> force the client to disconnect, after which a user must reopen his work
> (= load a lot of data) and this is annoying to say the least. The
> detection of message loss is done simply by numbering the messages.
> Are there any suggestions on how to implement "lossless" (to a
> reasonable extent, allowing at least timeouts, i.e. a message needs not
> be delivered if the recipient can't be reached for given amount of time)
> multicast with ZeroMQ?
> I've been thinking of implementing this with dealer and router sockets
> but am not sure whether this is the best way.
> Here are some further details of my scenario. If more are needed to
> provide an answer to my question I will be happy to provide it.
>    * the basic idea that there are a number of clients editing the same
> data with a server relaying the changes each of them makes to the others
> (cf. a document in google docs) and saving the changes to a persistent
> storage
>    * the clients request the present state of the data they are
> interested in (implemented with async server pattern using dealer-router
> sockets) and simultaneously (actually before) subscribe to changes to
> that data using pub-sub and a filter specific to that set of data
>      ** appropriate care is taken to tell which change messages are
> already included in the state received as a reply to the request and
> they are discarded
>    * the client must not miss any of the change messages in the
> broadcast or it will be in a corrupt state (its local data not
> reflecting what the data looks like at the moment)
>    * if a client restarts it does not require the change messages
> broadcast in between
>    * clients will detect if server is restarted and also in this case
> force reloading of the data
>    * the amount of data broadcast is not too big compared to the amount
> of available memory
>    * the messages are quickly read on the client and passed to other
> threads to be dealt with (i.e. they won't stay in ZeroMQ's buffers very
> long)
>    * the server broadcasts heartbeats roughly every 1500ms
> environment: cljzmq on the server, clrzmr on the client, zmq version
> 4.0.4 (x64) on both
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

More information about the zeromq-dev mailing list