[zeromq-dev] Distributed logging
Thomas Johnson
thomas.j.johnson at gmail.com
Tue Aug 20 07:59:06 CEST 2013
Sean, initially I thought that I was going to be serializing the data in a
binary format, sending it over the wire, and then in another process
deserializing it and doing whatever post-processing was necessary. I had
read that syslog couldn't handle binary data, so I figured it was out.
However, I found that for small objects (i.e., structured log messages)
json encoding with ujson was as fast as anything else I tested. So now I'm
reconsidering using rsyslog with RELP. After skimming the docs, the rsyslog
configuration learning curve seems a little steep compared to zmq.
On Mon, Aug 19, 2013 at 11:40 PM, Sean Ochoa <sean.m.ochoa at gmail.com> wrote:
> Sorry for playing devil's advocate here, but I have to ask. Oh, and I
> apologize if this question has already been addressed in this thread. I
> only skimmed the last few replies.
>
> Why not use syslog forwarding?
>
> -- Sean
>
> On Aug 19, 2013, at 8:22 PM, Matt Connolly <matt.connolly at me.com> wrote:
>
> I came to the conclusion of using a PUSH socket too, although I’m still in
> development.
>
> Morten, what ruby bindings did you use? When I first looked at using
> ZeroMQ with ruby, I tried out a few and “rbczmq” seemed to be the most
> efficient, up to date, and ruby like in its interface. I even added a ruby
> logging class to it recently, and wrote about it here:
>
> http://mattconnolly.wordpress.com/2013/07/10/zeromq-logging-for-ruby-apps/
>
> Thanks for the link to your article.
>
> In a C program, is it safe to close a socket during a signal handler? I
> thought best practice was to set a flag and use that to exit the main loop
> naturally.
>
>
> Regards,
> Matt
>
>
> On 20 Aug 2013, at 12:52 pm, Morten Møller Riis <
> mortenmoellerriis at gmail.com> wrote:
>
> Hi Thomas
>
> We are doing something similar with ZMQ for centralised Apache logging. It
> has been working very well.
>
> Each web server is running a C program that sends piped apache access logs
> via ZMQ to a log server. This log server runs a C server which receives the
> logs, ignores requests from our Varnish proxy and logs to the disk (SSD
> since there are A LOT of requests). Similarly our Varnish proxy runs the
> same C program as the apache servers (piped from varnishnsca).
>
> If the log server is down, requests are buffered on the web servers and
> when it's up again it catches up pretty quickly.
>
> I've written a bit about it here:
>
> http://whatastruggle.com/apache-logging-via-zeromq-part-2
>
> I know it's not exactly your use-case, but maybe there's some inspiration.
>
> Best regards
> Morten Møller Riis
>
>
>
> On Aug 20, 2013, at 8:00 AM, Thomas Johnson <thomas.j.johnson at gmail.com>
> wrote:
>
> That's very true regarding 'log messages not lost' as being hard. I guess
> the standard I have is "if the log messages would be lost if the program
> generating logs was just writing to a local disk, then its okay if they're
> lost in the zmq system"
>
> I assumed that if a peer joins late, it can catch up from a disk-based
> record that the master archiver is writing. The master archiver can't
> disconnect and reconnect, because as soon as it disconnects the logging
> application the logger knows this and starts writing to local disk, and
> hopefully the user notices and shuts down the app.
>
> What are the situations where PUB/SUB silently drops messages?
>
> If PUB/SUB is unreliable, should I use PUSH/PULL for the master archiver
> or do I need to use REQ/REP?
>
>
>
>
> On Mon, Aug 19, 2013 at 4:48 PM, Randall Nortman <rnzmq at wonderclown.net>wrote:
>
>> You will find that the requirement "that log messages aren't lost" is
>> going to be tricky. Impossible, actually. The question is what
>> message-dropping scenarios are acceptable and which do you want to try
>> to prevent? For example, if the sending system spontaneously combusts
>> while peers are disconnected, you are going to lose locally queued
>> messages. (The most important of those lost messages would probably
>> be "System on fire".)
>>
>> Define what you mean by reliable delivery, and your design will follow
>> from there. ZMQ's PUB/SUB sockets are actually really bad for
>> reliable delivery -- there are a host of situations in which they will
>> silently drop messages. If you use PUB/SUB as your main delivery
>> route, you need a secondary mechanism where when a peer reconnects, it
>> requests (via a REQ/REP or similar mechanism) redelivery of anything
>> it missed. This requires you to have something like a monotonically
>> increasing message ID or timestamp, so that the client can request all
>> messages after a particular ID/timestamp.
>>
>> On Mon, Aug 19, 2013 at 04:19:33PM -0500, Thomas Johnson wrote:
>> > I'm a total 0MQ newbie who is working on writing a distributed
>> logging
>> > framework using 0MQ and Cython. My design goals are that it be fast
>> for
>> > the system generating the logs, and that log messages aren't lost.
>> > The log messages will be consumed by a variety of systems, including
>> an
>> > archiver that writes the logs to disk, as well as a Logstash
>> installation.
>> > Because of the multiple consumers it seems like I want to use PubSub
>> to
>> > send the logs out.
>> > Of course I need to deal with the case where the archiver dies or
>> can't
>> > keep up.�
>> > I've read the guide, and my idea is for the publisher to
>> periodically send
>> > REQ messages over a different socket to one "master" archiver asking
>> about
>> > the latest message that the archiver received. If the master doesn't
>> reply
>> > or gets behind, the publisher drops the master, writes to local disk
>> any
>> > log messages not received by the master, and notifies the user of a
>> fault.
>> > Does this sound reasonable or is there a much better way to do this?�
>>
>> > _______________________________________________
>> > zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20130820/c0223ebc/attachment.htm>
More information about the zeromq-dev
mailing list