[zeromq-dev] Distributed logging

Pieter Hintjens pieterh at gmail.com
Tue Aug 20 21:35:07 CEST 2013 

+1 Like
On Aug 20, 2013 12:51 PM, "Brian Knox" <brian.knox at neomailbox.net> wrote:

>  Just a point of information -
>
> Rsyslog does not have a 2048 bye limit for messages.  The default is 2048
> bytes, but it is a configuration directive.  I've set it as high as 64k.
>
> Just a heads up - rsyslog (as of v7 stable) also has zeromq input and
> output plugins (shameless plug - I designed them and helped write them
> ;)).  I've heard on the rsyslog list there may be a bug compiling against
> the latest czmq release that I need look into after I finish getting ready
> for a conference in a couple of weeks, but other than that I've used them
> pretty heavily.  They currently support SUB and PULL on the input side, and
> PUSH and PUB on the output side.
>
> Brian
>
>
>
>
>
> On 8/20/2013 2:22 AM, Thomas Johnson wrote:
>
> Hmm, the syslog limit of 1024 bytes (or 2048 on rsyslog) seems to make it
> a no-go for JSON-based logging.
>
>
> On Tue, Aug 20, 2013 at 12:59 AM, Thomas Johnson <
> thomas.j.johnson at gmail.com> wrote:
>
>> Sean, initially I thought that I was going to be serializing the data in
>> a binary format, sending it over the wire, and then in another process
>> deserializing it and doing whatever post-processing was necessary. I had
>> read that syslog couldn't handle binary data, so I figured it was out.
>>
>>  However, I found that for small objects (i.e., structured log messages)
>> json encoding with ujson was as fast as anything else I tested. So now I'm
>> reconsidering using rsyslog with RELP. After skimming the docs, the rsyslog
>> configuration learning curve seems a little steep compared to zmq.
>>
>>
>>
>> On Mon, Aug 19, 2013 at 11:40 PM, Sean Ochoa <sean.m.ochoa at gmail.com>wrote:
>>
>>> Sorry for playing devil's advocate here, but I have to ask.  Oh, and I
>>> apologize if this question has already been addressed in this thread.  I
>>> only skimmed the last few replies.
>>>
>>>  Why not use syslog forwarding?
>>>
>>>  -- Sean
>>>
>>>  On Aug 19, 2013, at 8:22 PM, Matt Connolly <matt.connolly at me.com>
>>> wrote:
>>>
>>>  I came to the conclusion of using a PUSH socket too, although I’m
>>> still in development.
>>>
>>>  Morten, what ruby bindings did you use? When I first looked at using
>>> ZeroMQ with ruby, I tried out a few and “rbczmq” seemed to be the most
>>> efficient, up to date, and ruby like in its interface. I even added a ruby
>>> logging class to it recently, and wrote about it here:
>>>
>>>
>>> http://mattconnolly.wordpress.com/2013/07/10/zeromq-logging-for-ruby-apps/
>>>
>>>  Thanks for the link to your article.
>>>
>>>  In a C program, is it safe to close a socket during a signal handler?
>>> I thought best practice was to set a flag and use that to exit the main
>>> loop naturally.
>>>
>>>
>>>  Regards,
>>> Matt
>>>
>>>
>>>  On 20 Aug 2013, at 12:52 pm, Morten Møller Riis <
>>> mortenmoellerriis at gmail.com> wrote:
>>>
>>>  Hi Thomas
>>>
>>>  We are doing something similar with ZMQ for centralised Apache
>>> logging. It has been working very well.
>>>
>>>  Each web server is running a C program that sends piped apache access
>>> logs via ZMQ to a log server. This log server runs a C server which
>>> receives the logs, ignores requests from our Varnish proxy and logs to the
>>> disk (SSD since there are A LOT of requests). Similarly our Varnish proxy
>>> runs the same C program as the apache servers (piped from varnishnsca).
>>>
>>>  If the log server is down, requests are buffered on the web servers
>>> and when it's up again it catches up pretty quickly.
>>>
>>>  I've written a bit about it here:
>>>
>>>  http://whatastruggle.com/apache-logging-via-zeromq-part-2
>>>
>>>  I know it's not exactly your use-case, but maybe there's some
>>> inspiration.
>>>
>>>  Best regards
>>> Morten Møller Riis
>>>
>>>
>>>
>>>  On Aug 20, 2013, at 8:00 AM, Thomas Johnson <thomas.j.johnson at gmail.com>
>>> wrote:
>>>
>>>  That's very true regarding 'log messages not lost' as being hard. I
>>> guess the standard I have is "if the log messages would be lost if the
>>> program generating logs was just writing to a local disk, then its okay if
>>> they're lost in the zmq system"
>>>
>>>  I assumed that if a peer joins late, it can catch up from a disk-based
>>> record that the master archiver is writing. The master archiver can't
>>> disconnect and reconnect, because as soon as it disconnects the logging
>>> application the logger knows this and starts writing to local disk, and
>>> hopefully the user notices and shuts down the app.
>>>
>>>  What are the situations where PUB/SUB silently drops messages?
>>>
>>>  If PUB/SUB is unreliable, should I use PUSH/PULL for the master
>>> archiver or do I need to use REQ/REP?
>>>
>>>
>>>
>>>
>>> On Mon, Aug 19, 2013 at 4:48 PM, Randall Nortman <rnzmq at wonderclown.net>wrote:
>>>
>>>> You will find that the requirement "that log messages aren't lost" is
>>>> going to be tricky.  Impossible, actually.  The question is what
>>>> message-dropping scenarios are acceptable and which do you want to try
>>>> to prevent?  For example, if the sending system spontaneously combusts
>>>> while peers are disconnected, you are going to lose locally queued
>>>> messages.  (The most important of those lost messages would probably
>>>> be "System on fire".)
>>>>
>>>> Define what you mean by reliable delivery, and your design will follow
>>>> from there.  ZMQ's PUB/SUB sockets are actually really bad for
>>>> reliable delivery -- there are a host of situations in which they will
>>>> silently drop messages.  If you use PUB/SUB as your main delivery
>>>> route, you need a secondary mechanism where when a peer reconnects, it
>>>> requests (via a REQ/REP or similar mechanism) redelivery of anything
>>>> it missed.  This requires you to have something like a monotonically
>>>> increasing message ID or timestamp, so that the client can request all
>>>> messages after a particular ID/timestamp.
>>>>
>>>> On Mon, Aug 19, 2013 at 04:19:33PM -0500, Thomas Johnson wrote:
>>>> >    I'm a total 0MQ newbie who is working on writing a distributed
>>>> logging
>>>> >    framework using 0MQ and Cython. My design goals are that it be
>>>> fast for
>>>> >    the system generating the logs, and that log messages aren't lost.
>>>> >    The log messages will be consumed by a variety of systems,
>>>> including an
>>>> >    archiver that writes the logs to disk, as well as a Logstash
>>>> installation.
>>>> >    Because of the multiple consumers it seems like I want to use
>>>> PubSub to
>>>> >    send the logs out.
>>>> >    Of course I need to deal with the case where the archiver dies or
>>>> can't
>>>> >    keep up.�
>>>> >    I've read the guide, and my idea is for the publisher to
>>>> periodically send
>>>> >    REQ messages over a different socket to one "master" archiver
>>>> asking about
>>>> >    the latest message that the archiver received. If the master
>>>> doesn't reply
>>>> >    or gets behind, the publisher drops the master, writes to local
>>>> disk any
>>>> >    log messages not received by the master, and notifies the user of
>>>> a fault.
>>>>  >    Does this sound reasonable or is there a much better way to do
>>>> this?�
>>>>
>>>> > _______________________________________________
>>>> > zeromq-dev mailing list
>>>> > zeromq-dev at lists.zeromq.org
>>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>
>>>> _______________________________________________
>>>> zeromq-dev mailing list
>>>> zeromq-dev at lists.zeromq.org
>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>
>>>
>>>  _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>>>  _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>>>  _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>>
>
>
> _______________________________________________
> zeromq-dev mailing listzeromq-dev at lists.zeromq.orghttp://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20130820/8b854e27/attachment.htm>

More information about the zeromq-dev mailing list