[zeromq-dev] Distributed logging
Brian Knox
brian.knox at neomailbox.net
Tue Aug 20 12:51:02 CEST 2013
Just a point of information -
Rsyslog does not have a 2048 bye limit for messages. The default is
2048 bytes, but it is a configuration directive. I've set it as high as
64k.
Just a heads up - rsyslog (as of v7 stable) also has zeromq input and
output plugins (shameless plug - I designed them and helped write them
;)). I've heard on the rsyslog list there may be a bug compiling
against the latest czmq release that I need look into after I finish
getting ready for a conference in a couple of weeks, but other than that
I've used them pretty heavily. They currently support SUB and PULL on
the input side, and PUSH and PUB on the output side.
Brian
On 8/20/2013 2:22 AM, Thomas Johnson wrote:
> Hmm, the syslog limit of 1024 bytes (or 2048 on rsyslog) seems to make
> it a no-go for JSON-based logging.
>
>
> On Tue, Aug 20, 2013 at 12:59 AM, Thomas Johnson
> <thomas.j.johnson at gmail.com <mailto:thomas.j.johnson at gmail.com>> wrote:
>
> Sean, initially I thought that I was going to be serializing the
> data in a binary format, sending it over the wire, and then in
> another process deserializing it and doing whatever
> post-processing was necessary. I had read that syslog couldn't
> handle binary data, so I figured it was out.
>
> However, I found that for small objects (i.e., structured log
> messages) json encoding with ujson was as fast as anything else I
> tested. So now I'm reconsidering using rsyslog with RELP. After
> skimming the docs, the rsyslog configuration learning curve seems
> a little steep compared to zmq.
>
>
>
> On Mon, Aug 19, 2013 at 11:40 PM, Sean Ochoa
> <sean.m.ochoa at gmail.com <mailto:sean.m.ochoa at gmail.com>> wrote:
>
> Sorry for playing devil's advocate here, but I have to ask.
> Oh, and I apologize if this question has already been
> addressed in this thread. I only skimmed the last few replies.
>
> Why not use syslog forwarding?
>
> -- Sean
>
> On Aug 19, 2013, at 8:22 PM, Matt Connolly
> <matt.connolly at me.com <mailto:matt.connolly at me.com>> wrote:
>
>> I came to the conclusion of using a PUSH socket too, although
>> I'm still in development.
>>
>> Morten, what ruby bindings did you use? When I first looked
>> at using ZeroMQ with ruby, I tried out a few and "rbczmq"
>> seemed to be the most efficient, up to date, and ruby like in
>> its interface. I even added a ruby logging class to it
>> recently, and wrote about it here:
>>
>> http://mattconnolly.wordpress.com/2013/07/10/zeromq-logging-for-ruby-apps/
>>
>> Thanks for the link to your article.
>>
>> In a C program, is it safe to close a socket during a signal
>> handler? I thought best practice was to set a flag and use
>> that to exit the main loop naturally.
>>
>>
>> Regards,
>> Matt
>>
>>
>> On 20 Aug 2013, at 12:52 pm, Morten Møller Riis
>> <mortenmoellerriis at gmail.com
>> <mailto:mortenmoellerriis at gmail.com>> wrote:
>>
>>> Hi Thomas
>>>
>>> We are doing something similar with ZMQ for centralised
>>> Apache logging. It has been working very well.
>>>
>>> Each web server is running a C program that sends piped
>>> apache access logs via ZMQ to a log server. This log server
>>> runs a C server which receives the logs, ignores requests
>>> from our Varnish proxy and logs to the disk (SSD since there
>>> are A LOT of requests). Similarly our Varnish proxy runs the
>>> same C program as the apache servers (piped from varnishnsca).
>>>
>>> If the log server is down, requests are buffered on the web
>>> servers and when it's up again it catches up pretty quickly.
>>>
>>> I've written a bit about it here:
>>>
>>> http://whatastruggle.com/apache-logging-via-zeromq-part-2
>>>
>>> I know it's not exactly your use-case, but maybe there's
>>> some inspiration.
>>>
>>> Best regards
>>> Morten Møller Riis
>>>
>>>
>>>
>>> On Aug 20, 2013, at 8:00 AM, Thomas Johnson
>>> <thomas.j.johnson at gmail.com
>>> <mailto:thomas.j.johnson at gmail.com>> wrote:
>>>
>>>> That's very true regarding 'log messages not lost' as being
>>>> hard. I guess the standard I have is "if the log messages
>>>> would be lost if the program generating logs was just
>>>> writing to a local disk, then its okay if they're lost in
>>>> the zmq system"
>>>>
>>>> I assumed that if a peer joins late, it can catch up from a
>>>> disk-based record that the master archiver is writing. The
>>>> master archiver can't disconnect and reconnect, because as
>>>> soon as it disconnects the logging application the logger
>>>> knows this and starts writing to local disk, and hopefully
>>>> the user notices and shuts down the app.
>>>>
>>>> What are the situations where PUB/SUB silently drops messages?
>>>>
>>>> If PUB/SUB is unreliable, should I use PUSH/PULL for the
>>>> master archiver or do I need to use REQ/REP?
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Aug 19, 2013 at 4:48 PM, Randall Nortman
>>>> <rnzmq at wonderclown.net <mailto:rnzmq at wonderclown.net>> wrote:
>>>>
>>>> You will find that the requirement "that log messages
>>>> aren't lost" is
>>>> going to be tricky. Impossible, actually. The
>>>> question is what
>>>> message-dropping scenarios are acceptable and which do
>>>> you want to try
>>>> to prevent? For example, if the sending system
>>>> spontaneously combusts
>>>> while peers are disconnected, you are going to lose
>>>> locally queued
>>>> messages. (The most important of those lost messages
>>>> would probably
>>>> be "System on fire".)
>>>>
>>>> Define what you mean by reliable delivery, and your
>>>> design will follow
>>>> from there. ZMQ's PUB/SUB sockets are actually really
>>>> bad for
>>>> reliable delivery -- there are a host of situations in
>>>> which they will
>>>> silently drop messages. If you use PUB/SUB as your
>>>> main delivery
>>>> route, you need a secondary mechanism where when a peer
>>>> reconnects, it
>>>> requests (via a REQ/REP or similar mechanism)
>>>> redelivery of anything
>>>> it missed. This requires you to have something like a
>>>> monotonically
>>>> increasing message ID or timestamp, so that the client
>>>> can request all
>>>> messages after a particular ID/timestamp.
>>>>
>>>> On Mon, Aug 19, 2013 at 04:19:33PM -0500, Thomas
>>>> Johnson wrote:
>>>> > I'm a total 0MQ newbie who is working on writing a
>>>> distributed logging
>>>> > framework using 0MQ and Cython. My design goals are
>>>> that it be fast for
>>>> > the system generating the logs, and that log
>>>> messages aren't lost.
>>>> > The log messages will be consumed by a variety of
>>>> systems, including an
>>>> > archiver that writes the logs to disk, as well as a
>>>> Logstash installation.
>>>> > Because of the multiple consumers it seems like I
>>>> want to use PubSub to
>>>> > send the logs out.
>>>> > Of course I need to deal with the case where the
>>>> archiver dies or can't
>>>> > keep up.?
>>>> > I've read the guide, and my idea is for the
>>>> publisher to periodically send
>>>> > REQ messages over a different socket to one
>>>> "master" archiver asking about
>>>> > the latest message that the archiver received. If
>>>> the master doesn't reply
>>>> > or gets behind, the publisher drops the master,
>>>> writes to local disk any
>>>> > log messages not received by the master, and
>>>> notifies the user of a fault.
>>>> > Does this sound reasonable or is there a much
>>>> better way to do this??
>>>>
>>>> > _______________________________________________
>>>> > zeromq-dev mailing list
>>>> > zeromq-dev at lists.zeromq.org
>>>> <mailto:zeromq-dev at lists.zeromq.org>
>>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>
>>>> _______________________________________________
>>>> zeromq-dev mailing list
>>>> zeromq-dev at lists.zeromq.org
>>>> <mailto:zeromq-dev at lists.zeromq.org>
>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>
>>>>
>>>> _______________________________________________
>>>> zeromq-dev mailing list
>>>> zeromq-dev at lists.zeromq.org
>>>> <mailto:zeromq-dev at lists.zeromq.org>
>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org <mailto:zeromq-dev at lists.zeromq.org>
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org <mailto:zeromq-dev at lists.zeromq.org>
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org <mailto:zeromq-dev at lists.zeromq.org>
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20130820/d447802e/attachment.htm>
More information about the zeromq-dev
mailing list