[zeromq-dev] Distributed logging

[Thomas Johnson]

That's very true regarding 'log messages not lost' as being hard. I guess
the standard I have is "if the log messages would be lost if the program
generating logs was just writing to a local disk, then its okay if they're
lost in the zmq system"

I assumed that if a peer joins late, it can catch up from a disk-based
record that the master archiver is writing. The master archiver can't
disconnect and reconnect, because as soon as it disconnects the logging
application the logger knows this and starts writing to local disk, and
hopefully the user notices and shuts down the app.

What are the situations where PUB/SUB silently drops messages?

If PUB/SUB is unreliable, should I use PUSH/PULL for the master archiver or
do I need to use REQ/REP?




On Mon, Aug 19, 2013 at 4:48 PM, Randall Nortman <rnzmq at wonderclown.net>wrote:

> You will find that the requirement "that log messages aren't lost" is
> going to be tricky.  Impossible, actually.  The question is what
> message-dropping scenarios are acceptable and which do you want to try
> to prevent?  For example, if the sending system spontaneously combusts
> while peers are disconnected, you are going to lose locally queued
> messages.  (The most important of those lost messages would probably
> be "System on fire".)
>
> Define what you mean by reliable delivery, and your design will follow
> from there.  ZMQ's PUB/SUB sockets are actually really bad for
> reliable delivery -- there are a host of situations in which they will
> silently drop messages.  If you use PUB/SUB as your main delivery
> route, you need a secondary mechanism where when a peer reconnects, it
> requests (via a REQ/REP or similar mechanism) redelivery of anything
> it missed.  This requires you to have something like a monotonically
> increasing message ID or timestamp, so that the client can request all
> messages after a particular ID/timestamp.
>
> On Mon, Aug 19, 2013 at 04:19:33PM -0500, Thomas Johnson wrote:
> >    I'm a total 0MQ newbie who is working on writing a distributed logging
> >    framework using 0MQ and Cython. My design goals are that it be fast
> for
> >    the system generating the logs, and that log messages aren't lost.
> >    The log messages will be consumed by a variety of systems, including
> an
> >    archiver that writes the logs to disk, as well as a Logstash
> installation.
> >    Because of the multiple consumers it seems like I want to use PubSub
> to
> >    send the logs out.
> >    Of course I need to deal with the case where the archiver dies or
> can't
> >    keep up.�
> >    I've read the guide, and my idea is for the publisher to periodically
> send
> >    REQ messages over a different socket to one "master" archiver asking
> about
> >    the latest message that the archiver received. If the master doesn't
> reply
> >    or gets behind, the publisher drops the master, writes to local disk
> any
> >    log messages not received by the master, and notifies the user of a
> fault.
> >    Does this sound reasonable or is there a much better way to do this?�
>
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20130819/5a224c31/attachment.htm>