[zeromq-dev] Statistics reporting using zeromq/crossroads

[Schmurfy]

I am really skeptical about finding the graal of the statistic reporting
protocol, I made many attempts and everytime there is some things that
works in one case but not in another.
Anyway I like the idea :)

The network protocol used by collectd is the cleanest thing I have seen
until now and it is well described which is a really good thing:
http://collectd.org/wiki/index.php/Binary_protocol

I have implemented parsers for it in maybe 3 languages and each time they
were really easy to write.

The collectd protocol allows:
- multiple statistics per network packet
- multiple values per statistics (uptime for example could have 3 for 1min,
5min, 15min)
- support for doubles and integers (64bits)
- extensible
- transport of notifications (severity + string)
- encryption

I started working on zeromq collectd plugin which allowed external
applications to send statistics to the daemon but never finished it, I
think there is another one somewhere.

For the graphing part there are some solutions based on rrd which can be
written by collectd.

Overall Collectd is the best thing I have seen to collect/store statistics,
I am not fully satisfied with it but nothing else come closer to it, just
look at the plugin list

On 28 May 2012 10:17, Pieter Hintjens <ph at imatix.com> wrote:

> Hi Paul,
>
> This is a great idea. My advice is start simple and add functionality
> as needed over time. If you want an easy reusable legal template for
> protocols, see the rfc.zeromq.org site.
>
> -Pieter
>
> On Sun, May 27, 2012 at 4:38 PM, Paul Colomiets <paul at colomiets.name>
> wrote:
> > Hi,
> >
> > Every time I develop something using zeromq or libxs I come up with a
> > problem of how to report statistics. Pub/Sub pattern is suited for
> > statistics reporting very well, but the format of messages is unclear.
> > The result is ugly and diverse solutions for reporting real-time
> > statistics.
> >
> > I think the community would benefit from having common format for
> > reporting statistics, so that all open-source software could use the
> > format. And various plugins can be developed for existing statistics
> > and monitoring systems.
> >
> >
> > The protocol should meet the following requirements:
> >
> > 1. Simple compact format. Probably textual or fixed-size binary. It
> > should be easy to craft messages without any dependencies.
> >
> > 2. Each message should bring compact representation of the type
> > information in itself (so that centralized system doesn't need to be
> > reconfigured for each new reported value and each software upgrade on
> > remote system)
> >
> > 3. Subscriber should be able to filter statistics data by pub/sub
> subscriptions
> >
> > 4. There should be solution to collect that statistics (draw graphs,
> etc.)
> >
> >
> > So I have the following questions:
> >
> > 1. What formats are available to (re)use?
> >
> > 2. What protocols and software do you use for your projects?
> >
> > 3. Would you like to participate in standardization of the protocol?
> >
> >
> >
> > At our company we use collectd. And as zeromq plugin is not in the
> > mainstream for collectd, and it has complex binary format, we report
> > using some custom formats and a special application that translates
> > that into collectd protocol for unix sockets (which in turn nice
> > textual protocol). Same is done in zerogw (see zerogwstat utility).
> > This kind of proxying, and the point #2 from the requirements makes me
> > incomfortable.
> >
> > --
> > Paul
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20120528/7b890055/attachment.htm>