[zeromq-dev] Statistics protocol v0.1

[Paul Colomiets]

Hi Gregg,

On Fri, Jun 1, 2012 at 5:56 PM, Gregg Irwin <gregg at pointillistic.com> wrote:
> How tightly do you want to couple sources and sinks?

I want to decouple them as much as possible

> e.g., do you need type1 and type2 as part of the protocol?

Yes, in some form proably yes.

> I haven't used RRD, but that's the model, correct?
>

I don't know what you mean by "model". For rrd "counter" type is
basically a growing counter sent each time (e.g. messages passed
through a gateway). rrdtool subtracts previous value from the current
value to know rate (messages per second), and stores that rate. Its
very convenient way to count several kinds of statistics.

> Starting from a minimum spec, I sometimes write a statement answering
> the WHY question for things included beyond the minimum. For example:
>

Sure. I've just wanted to have a quick review to know if I'm not
terribly wrong. I will add more explanation in future versions.

>  Given a minimum of [name timestamp value]:
>
>  NAME has a trailing delimiter (colon) because name segments can
>  contain spaces.
>

No, segments can't contain spaces. The colon is there to have a way to
subscribe to whole value. E.g. subscribing to "example.org:cpu" will
also hit "example.org:cpu0", which isn't what intended. Having to
subscribe to space-terminated topic is ugly. And there is a tab that
is also a whitespace.

>  HOST is there for ...
>

It's nice to know what host this statistics is for. Having monotonic
sequence of dot-separated items, like in graphite/carbon
"example.org.cpu" means we don't know where hostname ends. The host
name is essential parameter to group data in a GUI.


>  TYPE1 is needed because ... and values must be one of ...
>
>  TYPE2 is used to ... must be ... and indicates ...
>

Those types mostly come from collectd, and which in turn inherited
types from RRD. Here is the reference:
http://collectd.org/wiki/index.php/Data_source

Actually type1 is only needed when type2 is counter. And it seems that
collectd mostly uses derive and gauge. There are also minimum and
maximum value, which can also influence right interpretation.

Even if we want to force counting of derivatives at sources instead of
at sink. There is a big difference between counter/derive/absolute
(which are mostly similar) and a gauge. When scaling graph with gauge
(say CPU usage) to have per-hour values, you would still have 1 to 100
percent of CPU usage (probably average). When scaling graph with
messages per second (which can be submitted using counter, derive or
absolute), you probably want to see messages per hour instead (the
value is a sum over period and is probably 3600 times bigger).

So I'm a bit confused on the types so far.

>
> I like ISO8601 timestamps

Me too. It will be in the next version.

> and path syntax (slash as separator) for names.
>

I want to be able to have a path as a component in the name. e.g.:

ESTP:example.org:disk:system/root:bytes_written:

Where the "system/root" is lvm partition (/dev/system/root aka
/dev/mapper/system-root).
This is also the reason I've not used dot, like in graphite, as dot is
used in hostnames.


Thanks for feedback!

-- 
Paul