[zeromq-dev] How to implement lossless pub-sub with ZeroMQ?

[Antti Karanta]

            Hi,

   Sorry for the late response. Got busy in between, now I finally had 
the time to get back to this topic.


   With a quick glance malamute looks great! I don't need the pub-sub to 
be persistent (in the sense that it would survive publisher process 
restart) so that is not a problem. However, I was left with some other 
questions.

   On https://github.com/zeromq/malamute/blob/master/MALAMUTE.md it 
says: "The broadcast model is unreliable by design.". I did not quite 
grasp how to use malamute for reliable pub-sub. Maybe this is in the 
examples?

   My target platform is Windows and all I could find about malamute and 
windows was a question of how to build it (and no answers) 
(http://comments.gmane.org/gmane.network.zeromq.devel/27242). I take it 
that there is no downloadable windows build or instructions on how to 
build on windows?

   I take it there are no Java/C# bindings for malamute yet?

   Are there any tutorials / examples beyond tutorial.c in the git repo?

   On the malamute doc page 
(https://github.com/zeromq/malamute/blob/master/MALAMUTE.md) it says:

/> "this is a discussion whitepaper, it does not describe an actual 
system, and does not constitute a promise to build such a system/?"

   Is there any documentation reflecting what malamute can do and how to 
use it as per today?

   Are there any official releases or is it advisable just to build the 
library yourself from the latest sources?

   Is Malamute a fresh project or already used extensively in production 
or is it somewhere in between?

   Are these and other questions already addressed in some sort of FAQ? :)



        ::Antti::



3.11.2015, 17:06, Pieter Hintjens kirjoitti:
> You need something in the middle. The zeromq/malamute project is
> designed for this (it does not yet implement persistent pubsub
> though).
>
> On Tue, Nov 3, 2015 at 11:35 AM, Antti Karanta
> <Antti.Karanta at neromsoft.com>  wrote:
>> I was wondering about loss of messages I observed with PUB-SUB although
>> I had set send and recieve high water marks to 0. I reread the
>> documentation and sure enough, PUB-SUB is designed to lose messages in
>> some circumstances:
>>
>> "The ZeroMQ pub-sub pattern will lose messages arbitrarily when a
>> subscriber is connecting, when a network failure occurs, or just if the
>> subscriber or network can't keep up with the publisher." http://zguide.zeromq.org/php:chapter5#Pros-and-Cons-of-Pub-Sub Thus 
>> PUB-SUB is not really suited for my use case as loss of messages, 
>> while not fatal, is highly inconvenient. Detecting loss of messages 
>> will force the client to disconnect, after which a user must reopen 
>> his work (= load a lot of data) and this is annoying to say the 
>> least. The detection of message loss is done simply by numbering the 
>> messages. Are there any suggestions on how to implement "lossless" (to a
>> reasonable extent, allowing at least timeouts, i.e. a message needs not
>> be delivered if the recipient can't be reached for given amount of time)
>> multicast with ZeroMQ?
>> I've been thinking of implementing this with dealer and router sockets
>> but am not sure whether this is the best way.
>>
>> Here are some further details of my scenario. If more are needed to
>> provide an answer to my question I will be happy to provide it.
>>
>>     * the basic idea that there are a number of clients editing the same
>> data with a server relaying the changes each of them makes to the others
>> (cf. a document in google docs) and saving the changes to a persistent
>> storage
>>     * the clients request the present state of the data they are
>> interested in (implemented with async server pattern using dealer-router
>> sockets) and simultaneously (actually before) subscribe to changes to
>> that data using pub-sub and a filter specific to that set of data
>>       ** appropriate care is taken to tell which change messages are
>> already included in the state received as a reply to the request and
>> they are discarded
>>     * the client must not miss any of the change messages in the
>> broadcast or it will be in a corrupt state (its local data not
>> reflecting what the data looks like at the moment)
>>     * if a client restarts it does not require the change messages
>> broadcast in between
>>     * clients will detect if server is restarted and also in this case
>> force reloading of the data
>>     * the amount of data broadcast is not too big compared to the amount
>> of available memory
>>     * the messages are quickly read on the client and passed to other
>> threads to be dealt with (i.e. they won't stay in ZeroMQ's buffers very
>> long)
>>     * the server broadcasts heartbeats roughly every 1500ms
>>
>> environment: cljzmq on the server, clrzmr on the client, zmq version
>> 4.0.4 (x64) on both
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20151130/b41dce19/attachment.htm>