[zeromq-dev] Statistics reporting using zeromq/crossroads

[Paul Colomiets]

Hi Schmurfy,

On Mon, May 28, 2012 at 4:00 PM, Schmurfy <schmurfy at gmail.com> wrote:
> I am really skeptical about finding the graal of the statistic reporting
> protocol, I made many attempts and everytime there is some things that works
> in one case but not in another.
> Anyway I like the idea :)
>

Please share use cases. I personally want to make nicer statistics
for zerogw and implement statistics for paperjam, and add some
statistics utility for zorro framework. We can also contribute to
mongrel2 and to collectd. I also have few commercial projects that
would migrate into the system when it's settled.

> The network protocol used by collectd is the cleanest thing I have seen
> until now and it is well described which is a really good thing:
> http://collectd.org/wiki/index.php/Binary_protocol
>

Oh, thanks. I've not found this page. The coments are following.

> I have implemented parsers for it in maybe 3 languages and each time they
> were really easy to write.
>
> The collectd protocol allows:

Will comment on the features:

> - multiple statistics per network packet
> - encryption

Not particularly interesting. I'd say gathering and encryption of packets can
be done by a generic device implementation (e.g. paperjam), suitable
for all pub/sub applications (with splitting and decryption being transparent
for collectd/statistics_server).

> - multiple values per statistics (uptime for example could have 3 for 1min,
> 5min, 15min)

May probably be supported. But the way it's supported in collectd is
a bit awkward. You must define a separate "type" for the tuple of
values and register that type at the central collectd server. Its
quite inconvenient.

> - support for doubles and integers (64bits)
> - extensible

Nice.

> - transport of notifications (severity + string)
>

Not particularly interesting at the moment.


> I started working on zeromq collectd plugin which allowed external
> applications to send statistics to the daemon but never finished it, I think
> there is another one somewhere.
>

Is this your branch?

http://git.verplant.org/?p=collectd.git;a=shortlog;h=refs/heads/ja/zeromq

I'm interested in integrating zeromq/libxs to collectd too. But I'd discuss
the protocol a bit more. I'm not sure the collectd protocol is best.

The collectd protocol is bad for proposed use case in the following:

1. It doesn't fall well into subscription model because of:
    a) allowing bulks
    b) having non-fixed order of parts
    c) having length-prefixed strings

2. The order of the parts is not fixed, so it may be misinterpreted
easily when writing custom parser

3. Types must be predefined in configuration


So I propose to define a text protocol which:

1. Plays nice with subscriptions

2. Maps into collectd types or provides its own type db (any other
systems to consider?)

3. Is mostly a textual, human-readable format, so:
    a) can be inspected without any dependencies
    b) can be submitted with command-line tools

> For the graphing part there are some solutions based on rrd which can be
> written by collectd.
>
> Overall Collectd is the best thing I have seen to collect/store statistics,
> I am not fully satisfied with it but nothing else come closer to it, just
> look at the plugin list

Similar experience. BTW, what do you use for drawing graphs?

-- 
Paul