[zeromq-dev] Distributed logging

[Matt Connolly]

I came to the conclusion of using a PUSH socket too, although I’m still in development.

Morten, what ruby bindings did you use? When I first looked at using ZeroMQ with ruby, I tried out a few and “rbczmq” seemed to be the most efficient, up to date, and ruby like in its interface. I even added a ruby logging class to it recently, and wrote about it here:

http://mattconnolly.wordpress.com/2013/07/10/zeromq-logging-for-ruby-apps/

Thanks for the link to your article.

In a C program, is it safe to close a socket during a signal handler? I thought best practice was to set a flag and use that to exit the main loop naturally.


Regards,
Matt


On 20 Aug 2013, at 12:52 pm, Morten Møller Riis <mortenmoellerriis at gmail.com> wrote:

> Hi Thomas
> 
> We are doing something similar with ZMQ for centralised Apache logging. It has been working very well.
> 
> Each web server is running a C program that sends piped apache access logs via ZMQ to a log server. This log server runs a C server which receives the logs, ignores requests from our Varnish proxy and logs to the disk (SSD since there are A LOT of requests). Similarly our Varnish proxy runs the same C program as the apache servers (piped from varnishnsca).
> 
> If the log server is down, requests are buffered on the web servers and when it's up again it catches up pretty quickly.
> 
> I've written a bit about it here:
> 
> http://whatastruggle.com/apache-logging-via-zeromq-part-2
> 
> I know it's not exactly your use-case, but maybe there's some inspiration.
> 
> Best regards
> Morten Møller Riis
> 
> 
> 
> On Aug 20, 2013, at 8:00 AM, Thomas Johnson <thomas.j.johnson at gmail.com> wrote:
> 
>> That's very true regarding 'log messages not lost' as being hard. I guess the standard I have is "if the log messages would be lost if the program generating logs was just writing to a local disk, then its okay if they're lost in the zmq system"
>> 
>> I assumed that if a peer joins late, it can catch up from a disk-based record that the master archiver is writing. The master archiver can't disconnect and reconnect, because as soon as it disconnects the logging application the logger knows this and starts writing to local disk, and hopefully the user notices and shuts down the app.
>> 
>> What are the situations where PUB/SUB silently drops messages?
>> 
>> If PUB/SUB is unreliable, should I use PUSH/PULL for the master archiver or do I need to use REQ/REP? 
>> 
>> 
>> 
>> 
>> On Mon, Aug 19, 2013 at 4:48 PM, Randall Nortman <rnzmq at wonderclown.net> wrote:
>> You will find that the requirement "that log messages aren't lost" is
>> going to be tricky.  Impossible, actually.  The question is what
>> message-dropping scenarios are acceptable and which do you want to try
>> to prevent?  For example, if the sending system spontaneously combusts
>> while peers are disconnected, you are going to lose locally queued
>> messages.  (The most important of those lost messages would probably
>> be "System on fire".)
>> 
>> Define what you mean by reliable delivery, and your design will follow
>> from there.  ZMQ's PUB/SUB sockets are actually really bad for
>> reliable delivery -- there are a host of situations in which they will
>> silently drop messages.  If you use PUB/SUB as your main delivery
>> route, you need a secondary mechanism where when a peer reconnects, it
>> requests (via a REQ/REP or similar mechanism) redelivery of anything
>> it missed.  This requires you to have something like a monotonically
>> increasing message ID or timestamp, so that the client can request all
>> messages after a particular ID/timestamp.
>> 
>> On Mon, Aug 19, 2013 at 04:19:33PM -0500, Thomas Johnson wrote:
>> >    I'm a total 0MQ newbie who is working on writing a distributed logging
>> >    framework using 0MQ and Cython. My design goals are that it be fast for
>> >    the system generating the logs, and that log messages aren't lost.
>> >    The log messages will be consumed by a variety of systems, including an
>> >    archiver that writes the logs to disk, as well as a Logstash installation.
>> >    Because of the multiple consumers it seems like I want to use PubSub to
>> >    send the logs out.
>> >    Of course I need to deal with the case where the archiver dies or can't
>> >    keep up.�
>> >    I've read the guide, and my idea is for the publisher to periodically send
>> >    REQ messages over a different socket to one "master" archiver asking about
>> >    the latest message that the archiver received. If the master doesn't reply
>> >    or gets behind, the publisher drops the master, writes to local disk any
>> >    log messages not received by the master, and notifies the user of a fault.
>> >    Does this sound reasonable or is there a much better way to do this?�
>> 
>> > _______________________________________________
>> > zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> 
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> 
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20130820/301d1f55/attachment.htm>