[zeromq-dev] Statistics protocol v0.1

[Andrew Hume]

comments interpolated below.

On Jun 1, 2012, at 1:40 AM, Marten Feldtmann wrote:

> Ah,
> 
> I was not precise enough ...
> 
>> Why you don't use pub/sub?
> 
> We use pub/sub .... via multicast in our system ... sorry, that I did 
> not make that clear.
> 
>> 
>> What it consists of?
>> 
>>> ->  location (computer) of the process sending this information
>>> (ip-number - no name)
>> 
>> Why do you use IP number instead names? I believe
>> it's internal internal policy in your company or somesuch.
>> Or is it just debugging info, along with node name?
>> 
>>> ->  name of the process
>>> ->  process id
>> 
>> Is PID just a debugging info, or is it meaningful? (do you aggregate
>> info from several processes?)
> 
> Well I have to differentiate, where the statistic informations come from 
> and due to the fact, that several instances of the same system can run 
> on one computer I have to add an additional information, which makes the 
> definition of the running program more or less possible.

in my system, i have arranged things so that i can run a cluster of several nodes
all on the one server. these instances are differentiated by the root of their
directory trees. we have a standardised way of munging the process name, instance number,
dir tree root and server into a single string; that is what we use for a name.

> 
>>> ->  start-time of statistic interval (in ASCII to make it more readable
>>> in a format like: 01.06.2012T21:00:00.000+12 .. in that well known
>>> format) and including timezone information.
>> 
>> It's interesting from two points:
>> 
>> 1. Textual data format may be nicer than unix timestamp. But
>> I'd prefer UTC only timestamp. As it's not intended to be presented
>> to user as is (except for debugging), its easier to deal with UTC
>> timestamps than with diverse timezones.
> 
> Well I like to have an overview over that info and a string is more 
> readable and to get a Unix Timestamp is not that portable and overall 
> easy possible to retrieve.

in my current project we simply use numeric milliseconds since the unix/posix epoch.
in my previous project, we used a pair of routines that converted between unix time
and several string formats. both schemes are equally plausible. but for heaven's sake,
don't reinvent this wheel.

> 
>> 2. We used to send timestamp at the time of sending, not
>> the start of interval. Its easier  to produce, and it's more logical
>> when we send a counter, instead of a rate value (see below).

ugh. what is a statistic? this needs to be defined if these things are to have semantics.
generally, i consider a statistic to be a sequence of events.
a measurement then is putting a cursor in this sequence (specified by a timestamp)
and then representing the events between this timestamp and the previous one.
this is not the only scheme/model, but it avoids worrying about discontinuities
introduced by specifying intervals.

>> 
>>> ->  duration length of the statistic interval in milliseconds
>> 
>> This one I've obviously missed. Will add shortly.

can you give an example or two? i've never felt a need for this.
if i ever needed to know the intervals, i use an FFT and compute the correct answer.

>> 
>>> ->  symbolic name of the statistic producer (0MQ node)
>>> ->  sub-symbolic name of the statistic producer (0MQ node)

why not just a name? why is a two level hierarchy exposed at this level?

>> 
>> So, according to the text below, I think that symbolic name
>> it's DNS name of the node, and sub-symbolic name it's name
>> of the subsystem, inside of the process. Am I right?
> 
> Well, we do not do script programming here. We have some Smalltalk 
> systems and within each running Smalltalk system (each is a process 
> within the operating system) ) we have some 0MQ sockets (or node) doing 
> its work/task. After all I would like to find out, what traffic each of 
> these processes/tasks are doing.
> 
> As an example (remember: all of these nodes may run within one Smalltalk 
> image - one OS process, but many, many Smalltalk light processes).

if you care about an entity's name, then it needs to have a name.
then just use it. this is easy. it only gets confusing when people want to
reftrofit some external model onto these names; just treat them as opaque blobs.

> 
> * we have a 0MQ statistic collector node (subscribe) (which itselfs 
> produces statistics also - but only receiving traffic ones). This one 
> collects the statistics info published. Its node type-name is "statsub". 
> We have only up to one statistic collector node in one running process, 
> so there is no need to give this type a subtype name. This node runs 
> only when needed.
> 
> * each process has one 0MQ statistic sending node (publisher). (which 
> itselfs produces statistics also - but only sending traffic ones). Its 
> node type-name is "statpub". We have only one statistic sender node in 
> one running process, so there is no need to give this type a subtype name.
> 
> The same happens for logging information. publish/subscribe and only 
> singletons are within a running process. Type names are "logpub" and 
> "logsub". No sub types needed
> 
> * then we have a domain working node (producing PDF files). (request and 
> reply). This one received commands and sends results - therefore another 
> source for statistics information. There may be several of instances 
> running within ONE process - therefore a subtype is useful (which may be 
> a unique number).
> 
> * Then we have service publishers. Each application (pub/sub) can offer 
> its services to the local network.
> 
> 
> Marten
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev


------------------
Andrew Hume  (best -> Telework) +1 623-551-2845
andrew at research.att.com  (Work) +1 973-236-2014
AT&T Labs - Research; member of USENIX and LOPSA




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20120601/6bb81f7f/attachment.htm>