[zeromq-dev] How to implement lossless pub-sub with ZeroMQ?

[Doron Somech]

You can use malamute. Also take a look at the clone example from ZeroMQ
guide.

Anyway what I did in event driven system is to save the event to db prior
publishing. All events are part of specific stream and numbered. When
subscriber recognize a gap it goes directly to database to get missing
events. If in your case subscriber doesn't have access to database you can
have a service for that.
On Nov 3, 2015 12:35 PM, "Antti Karanta" <Antti.Karanta at neromsoft.com>
wrote:

>
> I was wondering about loss of messages I observed with PUB-SUB although
> I had set send and recieve high water marks to 0. I reread the
> documentation and sure enough, PUB-SUB is designed to lose messages in
> some circumstances:
>
> "The ZeroMQ pub-sub pattern will lose messages arbitrarily when a
> subscriber is connecting, when a network failure occurs, or just if the
> subscriber or network can't keep up with the publisher."
> http://zguide.zeromq.org/php:chapter5#Pros-and-Cons-of-Pub-Sub
>
> Thus PUB-SUB is not really suited for my use case as loss of messages,
> while not fatal, is highly inconvenient. Detecting loss of messages will
> force the client to disconnect, after which a user must reopen his work
> (= load a lot of data) and this is annoying to say the least. The
> detection of message loss is done simply by numbering the messages.
>
>
> Are there any suggestions on how to implement "lossless" (to a
> reasonable extent, allowing at least timeouts, i.e. a message needs not
> be delivered if the recipient can't be reached for given amount of time)
> multicast with ZeroMQ?
> I've been thinking of implementing this with dealer and router sockets
> but am not sure whether this is the best way.
>
> Here are some further details of my scenario. If more are needed to
> provide an answer to my question I will be happy to provide it.
>
>    * the basic idea that there are a number of clients editing the same
> data with a server relaying the changes each of them makes to the others
> (cf. a document in google docs) and saving the changes to a persistent
> storage
>    * the clients request the present state of the data they are
> interested in (implemented with async server pattern using dealer-router
> sockets) and simultaneously (actually before) subscribe to changes to
> that data using pub-sub and a filter specific to that set of data
>      ** appropriate care is taken to tell which change messages are
> already included in the state received as a reply to the request and
> they are discarded
>    * the client must not miss any of the change messages in the
> broadcast or it will be in a corrupt state (its local data not
> reflecting what the data looks like at the moment)
>    * if a client restarts it does not require the change messages
> broadcast in between
>    * clients will detect if server is restarted and also in this case
> force reloading of the data
>    * the amount of data broadcast is not too big compared to the amount
> of available memory
>    * the messages are quickly read on the client and passed to other
> threads to be dealt with (i.e. they won't stay in ZeroMQ's buffers very
> long)
>    * the server broadcasts heartbeats roughly every 1500ms
>
> environment: cljzmq on the server, clrzmr on the client, zmq version
> 4.0.4 (x64) on both
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20151103/205a01b3/attachment.htm>