[zeromq-dev] Statistics reporting using zeromq/crossroads

[Schmurfy]

What I am trying to do is to have a common infrastructure for application
(my applications) and system (disk io, memory used, disk used, ...)
statistics, I shown what the collectd protocol could do but I am not
entirely satisfied with it either, here are my problems with collectd:

- I want to have statistics available as soon as possible: collectd has a
cycle time (default is 10s) and plugins have no way to know when the cycle
starts or end, they are notified when data is received or ready to be sent
but that's all. One problem I had with my zeromq attempt (my original
branch is on github: https://github.com/schmurfy/collectd/tree/zeromq but
based on collectd 4.x) is that you cannot send one "frame" per cycle which
is what I wanted to do with it (the original network plugins fills a buffer
and when the buffer reach a threshold it sends the packet so your values
can be sent now or at now+cycle time which means 10s later my a 10s cycle).

- I never used encryption, I consider it can be better handled at lower
level and bonus point is that you don't cripple your code with it
- I don't like the need to predefine the "types" used, it is a pain when
using multiple collectd servers since they need to all have the same
configuration file to understand each other, I prefer a more open way. That
said you don't need to know the types definitions to actually parse the
data stream, the number of parts is in the packet itself, you need the
definitions to match each cell with its label.
- I am not really fond of the plugin/plugin_instance/type/type_instance,
most of the time I don't remember which one is supposed to be what and many
existing collectd plugins do not use all fields so it is more annoying than
anything else.

I don't really like text protocols, sure they may seem easier to parse (and
debug since a human can read them) but for statistics which could be sent
at a high rate you waste a lot of space (binary parsers are not that hard
to write). For reference here is my ruby parser:
https://github.com/schmurfy/rrd-grapher/blob/master/lib/rrd-grapher/notifier/parsers/ruby_parser.rb
I was thinking about rewriting it in C but it works really well for my
current load (rrd-grapher has a subpart handling notifications so my daemon
receives the traffic from distant collectd nodes, raise/stop alerts and
then sends the data to the local collectd which saves the data to the disk
in rrd files).

I am currently doing some experiments using a hash like structure
serialized with messagepack and udp to transport, since messagepack support
a lot of languages it virtually means any language could
serialize/deserialize very easily the packets but I have not much to show
currently since I am in the early phases of the project.

I am currently using https://github.com/schmurfy/rrd-grapher to graph the
data read directly from the rrd files (I have a rrdcached server to help
keep my I/O from destroying the machine xD), unlike many tools out there
the server do not generate any images but sends the data to the clients
which then create the graph.
I tested some of the existing solutions but I don't like the idea of
generating images server side, today's browsers are more than capable of
rendering one of more graphs and it allows to shift some work to the client
which is a good thing for me since they don't do much ;)


The good thing with collectd is the large number of existing plugins to
collect nearly any statistic you could desire from your running servers, I
even made an attempt to interface a ruby daemon directly with the plugins
wich was not a real success ^^
(the plugins exports functions but also use exported functions from
collectd itself so interfacing with them is a bit tricky)

On 28 May 2012 21:03, Paul Colomiets <paul at colomiets.name> wrote:

> Hi Schmurfy,
>
> On Mon, May 28, 2012 at 4:00 PM, Schmurfy <schmurfy at gmail.com> wrote:
> > I am really skeptical about finding the graal of the statistic reporting
> > protocol, I made many attempts and everytime there is some things that
> works
> > in one case but not in another.
> > Anyway I like the idea :)
> >
>
> Please share use cases. I personally want to make nicer statistics
> for zerogw and implement statistics for paperjam, and add some
> statistics utility for zorro framework. We can also contribute to
> mongrel2 and to collectd. I also have few commercial projects that
> would migrate into the system when it's settled.
>
> > The network protocol used by collectd is the cleanest thing I have seen
> > until now and it is well described which is a really good thing:
> > http://collectd.org/wiki/index.php/Binary_protocol
> >
>
> Oh, thanks. I've not found this page. The coments are following.
>
> > I have implemented parsers for it in maybe 3 languages and each time they
> > were really easy to write.
> >
> > The collectd protocol allows:
>
> Will comment on the features:
>
> > - multiple statistics per network packet
> > - encryption
>
> Not particularly interesting. I'd say gathering and encryption of packets
> can
> be done by a generic device implementation (e.g. paperjam), suitable
> for all pub/sub applications (with splitting and decryption being
> transparent
> for collectd/statistics_server).
>
> > - multiple values per statistics (uptime for example could have 3 for
> 1min,
> > 5min, 15min)
>
> May probably be supported. But the way it's supported in collectd is
> a bit awkward. You must define a separate "type" for the tuple of
> values and register that type at the central collectd server. Its
> quite inconvenient.
>
> > - support for doubles and integers (64bits)
> > - extensible
>
> Nice.
>
> > - transport of notifications (severity + string)
> >
>
> Not particularly interesting at the moment.
>
>
> > I started working on zeromq collectd plugin which allowed external
> > applications to send statistics to the daemon but never finished it, I
> think
> > there is another one somewhere.
> >
>
> Is this your branch?
>
> http://git.verplant.org/?p=collectd.git;a=shortlog;h=refs/heads/ja/zeromq
>
> I'm interested in integrating zeromq/libxs to collectd too. But I'd discuss
> the protocol a bit more. I'm not sure the collectd protocol is best.
>
> The collectd protocol is bad for proposed use case in the following:
>
> 1. It doesn't fall well into subscription model because of:
>    a) allowing bulks
>    b) having non-fixed order of parts
>    c) having length-prefixed strings
>
> 2. The order of the parts is not fixed, so it may be misinterpreted
> easily when writing custom parser
>
> 3. Types must be predefined in configuration
>
>
> So I propose to define a text protocol which:
>
> 1. Plays nice with subscriptions
>
> 2. Maps into collectd types or provides its own type db (any other
> systems to consider?)
>
> 3. Is mostly a textual, human-readable format, so:
>    a) can be inspected without any dependencies
>    b) can be submitted with command-line tools
>
> > For the graphing part there are some solutions based on rrd which can be
> > written by collectd.
> >
> > Overall Collectd is the best thing I have seen to
> collect/store statistics,
> > I am not fully satisfied with it but nothing else come closer to it, just
> > look at the plugin list
>
> Similar experience. BTW, what do you use for drawing graphs?
>
> --
> Paul
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20120529/50023ae0/attachment.htm>