[zeromq-dev] Distributed logging

Tue Aug 20 08:22:16 CEST 2013

Hmm, the syslog limit of 1024 bytes (or 2048 on rsyslog) seems to make it a
no-go for JSON-based logging.

On Tue, Aug 20, 2013 at 12:59 AM, Thomas Johnson <thomas.j.johnson at gmail.com
> wrote:

> Sean, initially I thought that I was going to be serializing the data in a
> binary format, sending it over the wire, and then in another process
> deserializing it and doing whatever post-processing was necessary. I had
> read that syslog couldn't handle binary data, so I figured it was out.
>
> However, I found that for small objects (i.e., structured log messages)
> json encoding with ujson was as fast as anything else I tested. So now I'm
> reconsidering using rsyslog with RELP. After skimming the docs, the rsyslog
> configuration learning curve seems a little steep compared to zmq.
>
>
>
> On Mon, Aug 19, 2013 at 11:40 PM, Sean Ochoa <sean.m.ochoa at gmail.com>wrote:
>
>> Sorry for playing devil's advocate here, but I have to ask.  Oh, and I
>> apologize if this question has already been addressed in this thread.  I
>> only skimmed the last few replies.
>>
>> Why not use syslog forwarding?
>>
>> -- Sean
>>
>> On Aug 19, 2013, at 8:22 PM, Matt Connolly <matt.connolly at me.com> wrote:
>>
>> I came to the conclusion of using a PUSH socket too, although I’m still
>> in development.
>>
>> Morten, what ruby bindings did you use? When I first looked at using
>> ZeroMQ with ruby, I tried out a few and “rbczmq” seemed to be the most
>> efficient, up to date, and ruby like in its interface. I even added a ruby
>> logging class to it recently, and wrote about it here:
>>
>> http://mattconnolly.wordpress.com/2013/07/10/zeromq-logging-for-ruby-apps/
>>
>>  Thanks for the link to your article.
>>
>> In a C program, is it safe to close a socket during a signal handler? I
>> thought best practice was to set a flag and use that to exit the main loop
>> naturally.
>>
>>
>> Regards,
>> Matt
>>
>>
>> On 20 Aug 2013, at 12:52 pm, Morten Møller Riis <
>> mortenmoellerriis at gmail.com> wrote:
>>
>> Hi Thomas
>>
>> We are doing something similar with ZMQ for centralised Apache logging.
>> It has been working very well.
>>
>> Each web server is running a C program that sends piped apache access
>> logs via ZMQ to a log server. This log server runs a C server which
>> receives the logs, ignores requests from our Varnish proxy and logs to the
>> disk (SSD since there are A LOT of requests). Similarly our Varnish proxy
>> runs the same C program as the apache servers (piped from varnishnsca).
>>
>> If the log server is down, requests are buffered on the web servers and
>> when it's up again it catches up pretty quickly.
>>
>> I've written a bit about it here:
>>
>> http://whatastruggle.com/apache-logging-via-zeromq-part-2
>>
>> I know it's not exactly your use-case, but maybe there's some inspiration.
>>
>> Best regards
>> Morten Møller Riis
>>
>>
>>
>> On Aug 20, 2013, at 8:00 AM, Thomas Johnson <thomas.j.johnson at gmail.com>
>> wrote:
>>
>> That's very true regarding 'log messages not lost' as being hard. I guess
>> the standard I have is "if the log messages would be lost if the program
>> generating logs was just writing to a local disk, then its okay if they're
>> lost in the zmq system"
>>
>> I assumed that if a peer joins late, it can catch up from a disk-based
>> record that the master archiver is writing. The master archiver can't
>> disconnect and reconnect, because as soon as it disconnects the logging
>> application the logger knows this and starts writing to local disk, and
>> hopefully the user notices and shuts down the app.
>>
>> What are the situations where PUB/SUB silently drops messages?
>>
>> If PUB/SUB is unreliable, should I use PUSH/PULL for the master archiver
>> or do I need to use REQ/REP?
>>
>>
>>
>>
>> On Mon, Aug 19, 2013 at 4:48 PM, Randall Nortman <rnzmq at wonderclown.net>wrote:
>>
>>> You will find that the requirement "that log messages aren't lost" is
>>> going to be tricky.  Impossible, actually.  The question is what
>>> message-dropping scenarios are acceptable and which do you want to try
>>> to prevent?  For example, if the sending system spontaneously combusts
>>> while peers are disconnected, you are going to lose locally queued
>>> messages.  (The most important of those lost messages would probably
>>> be "System on fire".)
>>>
>>> Define what you mean by reliable delivery, and your design will follow
>>> from there.  ZMQ's PUB/SUB sockets are actually really bad for
>>> reliable delivery -- there are a host of situations in which they will
>>> silently drop messages.  If you use PUB/SUB as your main delivery
>>> route, you need a secondary mechanism where when a peer reconnects, it
>>> requests (via a REQ/REP or similar mechanism) redelivery of anything
>>> it missed.  This requires you to have something like a monotonically
>>> increasing message ID or timestamp, so that the client can request all
>>> messages after a particular ID/timestamp.
>>>
>>> On Mon, Aug 19, 2013 at 04:19:33PM -0500, Thomas Johnson wrote:
>>> >    I'm a total 0MQ newbie who is working on writing a distributed
>>> logging
>>> >    framework using 0MQ and Cython. My design goals are that it be fast
>>> for
>>> >    the system generating the logs, and that log messages aren't lost.
>>> >    The log messages will be consumed by a variety of systems,
>>> including an
>>> >    archiver that writes the logs to disk, as well as a Logstash
>>> installation.
>>> >    Because of the multiple consumers it seems like I want to use
>>> PubSub to
>>> >    send the logs out.
>>> >    Of course I need to deal with the case where the archiver dies or
>>> can't
>>> >    keep up.�
>>> >    I've read the guide, and my idea is for the publisher to
>>> periodically send
>>> >    REQ messages over a different socket to one "master" archiver
>>> asking about
>>> >    the latest message that the archiver received. If the master
>>> doesn't reply
>>> >    or gets behind, the publisher drops the master, writes to local
>>> disk any
>>> >    log messages not received by the master, and notifies the user of a
>>> fault.
>>> >    Does this sound reasonable or is there a much better way to do
>>> this?�
>>>
>>> > _______________________________________________
>>> > zeromq-dev mailing list
>>> > zeromq-dev at lists.zeromq.org
>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20130820/b2ccf20c/attachment.htm>