[zeromq-dev] Statistics protocol (ESTP) v0.2

[Schmurfy]

On 8 June 2012 23:54, Paul Colomiets <paul at colomiets.name> wrote:

> Hi Schmurfy, Marten,
>
> On Fri, Jun 8, 2012 at 1:21 PM, Schmurfy <schmurfy at gmail.com> wrote:
> > In the intent of building a protocol which is easy to read as a human why
> > using obscure symbols to denote the metric "type" ?
> > I am talking about <^>,<'> and <+>
> >
>
> Yes. I've just realized that too. The original intention was to make
> types parametric:
>
> 1000'-10:10 -- a derive type with limits from -10 to 10
> 1234^32bit -- counter with wrap point at 32bit
>
> This way it looks more natural -- as a separator. Then I looked at
> collectd and realized that counter that wraps is not used there,
> because it's safer to use derive with minimum of 0. And type with
> minimum and maximum is coded inside configuration there, so it's hard
> to integrate custom limits in collectd, so I've decided to leave this
> to the later extensions of the protocol.
>
> Does it make sense now?
>

Yes it makes a little more sense but I think a letter clearly describing
the type is better, see below.


>
> > I kinda like the statsd approach on this (https://github.com/etsy/statsd
> ),
> > why not using something like this (for the value field):
> >
> > gauge: 32.45|g
> > counter: 129800|c
> > derive: 56|d
> >
>
> Hm, statd uses semi-readable types. But uses cryptic "|@" to denote
> interval :)
>

I did not said I like all the protocol :)


> > always having the type would also be nicer as opposite that having gauge
> > considered as default.
> >
>
> I think gauge is most simple and understandable type, so it seems
> obvious that it's default. See more reasoning below.
>
I am really not sure on this, while I like sensible defaults when writing
code to speed things up
I prefer protocols to clearly say what they try to express, if there is a
"default" I agree that gauge
would be the most common type but it introduces one problem for me: you
have to deal
with the presence or absence of this field and as you mentioned the number
of fields
will change between a gauge value and a counter value. How do you handle
multiple flags scenario without separator ?
"43btgh" would really be ugly to parse.

If you know the value will always be something like:

23.4|ghty
(with "hty" being hypothetical flags and g the type)

then you can just split the string with "|" and look for known flags in the
second part which
becomes a little harder is the flags are directly added at the end of the
value ("34.56ght") since
in this later case you need to do some magic to separate what is two
different informations for me.
If there is a separator (the pipe here) then having gauge as default bother
me less because a quick
split will reveals there is no flags defined without further operations.

I agree mostly on the extensibility but if you wanted to add more options
to the value the logical choice
would be to add other one letters flags in this case.

Until now I never needed to add much to the value itself and its type,
knowing counters size may be
useful to detect overflow though but I never worked with counters.
I think the real question is what informations we could want linked to a
value, allowing anything and everything
to be added leter may not be the more efficient way ;)
(that's funny because some parts of the discussion start to look like low
level concerns for zeromq /libxs)


>
> > There is also the option to have the type as a proper field:
> > ESTP:org.example:sys::cpu: 2012-06-02T09:36:45 10 7.2 g
> >
>
> I'm trying to avoid that, because it is possible that some future
> revision of the protocol would allow several values in the single
> packet (if single value per packet will be a lot of pain in practice).
> So I'm trying to keep the protocol extensible to adding more values at
> the end. Moving type to the place before the value, might help, but
> will look a bit ugly if several values would have different types.
>
> Also parsing a field would encourage to compare whole word:
>
>    if type == "c"
>
> But the current check is expected to be done like the following:
>
> COUNTER_MARKER = "^"
> ...
>    if COUNTER_MARKER in value:
>
> So adding more specific typing would be easier (see possible typing
> extension above). Similar reasoning applies to something like
> "1234|c", turning it into: "1234|c:32bit" would be a pain.
>
> Similar thing applies to making gauge default. It's nice to have fixed
> number of fields, as that makes parsing easier, but if "1234|d" will
> be extended to "1234|d|-10|10" to have same field separator for the
> words, that doesn't help. And if "1234|d" will be extended to
> "1234|d:-10:10" in the future, it doesn't help either, as a lot of
> implementations will check "d:-10:10" == "d" so will treat the type as
> undefined instead of normal derived type.
>
> How about using just a single latin letter without a separator, and
> leave gauge as default?
>
> Any follow up thoughts?
>

see above.


> > The resource name is explained but not present in the fields description
> for
> > the example (above), is it the part after the "::" in the application
> name ?
> >
>
> It's between the colons. As description says it's empty for simple
> applications/subsystems. Added few more examples. Will try to make
> text more clear later.
>

I reread the document and if I understand weel this "sys::cpu", "sys" would
be the application, there would be no resource name and "cpu" would be the
metric name itself, is that right ?

>
> > Other that these remarks this is starting to look like something I could
> use
> > :)
> >
>
> Nice to hear, and thanks for feedback.
>

That nice to see you chose iso8601 as time format, I am using it for more
than 3 years now at work in our communication protocol and it proved to be
the right choice, it can express any date/time and most language already
have libraries to parse/generate it.
I also like the time to be UTC, in fact I never used anything other than
UTC on our servers and finding the GMT time on desktop application should
not be a challenge.


>
> > Ps: I sent you a pull request for a typo
> >
>
> Thanks. Applied.
>
>
>
> --
> Paul
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20120610/48953a4b/attachment.htm>