[zeromq-dev] Statistics reporting using zeromq/crossroads

[Paul Colomiets]

Hi Schmurfy,

On Wed, May 30, 2012 at 4:44 PM, Schmurfy <schmurfy at gmail.com> wrote:
> I not really sure about using subscriptions, why do you want to use this
> schema ?
> Currently what I have is many collectd sending their data to another
> collectd, the applications sends their statistics to the master collectd for
> the most part (we have embedded systems each with its own collectd
> instance). So for my use the closest pattern would be PUSH/PULL, what is
> your use case for subscriptions ?
>

1. If you would have multiple processors of statistics data (or
multiple servers of collectd) it's much nicer to have data duplicated,
instead of going to random one at each packet. So it's matches pub/sub
much better.

2. I have a use case where I do business logic based on statistics
(basically load balancing). So I would like to subscribe only for data
needed to execute the business logic.

> After digging into the core of collectd when I implemented my zeromq plugin
> (and a lua plugin too) I really feel like it was designed to handle a really
> large amount of data and/or short cycles and it makes sacrifices toward that
> goal.
>
> I like libxs/zeromq but I don't think this would fit all need, if you don't
> already use these libraries into your application you may reconsider adding
> them only to provide statistics, in my experiments I tend to prefer allowing
> multiple entry points (that's one reason I like collectd btw). As an example
> sending an udp packet should be fairly easy in any language out there and
> you don't have to maintain a connection, if the application and the collectd
> daemon (or equivalent service) are "close" or if you consider some loss
> acceptable this is a this is a perfectly valid way to send your statistics
> :)
>

Changing the underlying protocol, is OK. But what I expect in perfect
world is if I change my webserver from mongrel to nginx, is that I can
get the same value "example.org/http.requests" from the statistics
even if transport protocol would change from zeromq to websocket, http
or udp.

> One other project you could look at is Graphite
> (http://graphite.wikidot.com/)
>
> For the naming I usually go with [hostname, application_name, metric_name],
> the metric_name could itself include anything:   [example.org, disk,
> sda1/write_operations].
>

Makes sense, will try something similar for start.

>> Messagepack is great. But it's not very easy to parse in C (without
>> dependencies)
>
> nothing is easy in C :p
> (I like the language but everything takes time with it)
>

I'm thinking about something that can be parsed by sscanf. Will
publish more concrete proposal soon.

>
> PS: Do you have a screenshot of jarred ?
>

I've just put them here:

https://github.com/tailhook/jarred/wiki

-- 
Paul