[zeromq-dev] Time-reordering queue

Francesco francesco.montorsi at gmail.com
Fri Nov 11 00:18:56 CET 2022


Hi Bill,

> At the risk of telling you something you already know, have you thought
about how these timestamps are being generated?  I assume that the multiple
PUBs are, or at least can be, located on different machines.  In that
scenario the problem of keeping clocks in sync between the machines becomes
non-trivial.

Yes, you're right the producers will be (not always) on different servers.
These microservices are actually running in a Kubernetes cluster and we
ensure that all the worker nodes have a valid NTP configuration (typically
referring to stratum 1 clock sources)... I know that the time sync across
the servers will be mostly likely in the 500 usec-5msec range.... and
honestly when we are in msec-range, the accuracy is very bad considering
that the microservices exchange up to 300-500 kmsg/sec, taking about 1Gbps
of bandwidth for each link...
However, that's the best we have currently.

>  The best explanation I’ve seen is this:
https://queue.acm.org/detail.cfm?id=2878574, which in turn references
Lamport’s seminal work:
http://lamport.azurewebsites.net/pubs/time-clocks.pdf.

Thanks - I read the first one about NTP and PTP and it's a good basic
introduction to these 2 protocols.
On the same page I suggest another not-so-technical reading:
https://www.newyorker.com/tech/annals-of-technology/the-thorny-problem-of-keeping-the-internets-time
That article goes over the history of NTP and its creator and long-time
maintainer...

Thanks!

Francesco



Il giorno gio 10 nov 2022 alle ore 15:04 Bill Torpey <wallstprog at gmail.com>
ha scritto:

> Hi Francesco:
>
> At the risk of telling you something you already know, have you thought
> about how these timestamps are being generated?  I assume that the multiple
> PUBs are, or at least can be, located on different machines.  In that
> scenario the problem of keeping clocks in sync between the machines becomes
> non-trivial.
>
> The best explanation I’ve seen is this:
> https://queue.acm.org/detail.cfm?id=2878574, which in turn references
> Lamport’s seminal work:
> http://lamport.azurewebsites.net/pubs/time-clocks.pdf.
>
> Hope this helps.
>
> Bill Torpey
>
>
> On Nov 9, 2022, at 5:19 PM, Francesco <francesco.montorsi at gmail.com>
> wrote:
>
> Hi all,
>
> I have written two applications using ZMQ PUB-SUB pattern (over TCP
> transport).
> The subscriber application has its SUB socket connected to multiple PUBs
> (multiple tcp endpoints). Each message sent by the PUB encodes the
> timestamp (as obtained from clock_gettime() syscall at TX side using
> monotonically increasing clock) of the event described by the ZMQ message.
>
> The subscriber needs to process the data stream _strictly_ in order.
> However the multiple publishers have no coordination and they will emit
> messages at different rates, each with its own timestamp. The only
> guarantee that I have, according to ZMQ docs, is that the SUB socket will
> perform "fair dequeueing", but that's not enough to guarantee that every
> zmq_msg_t received from the SUB socket will have a monotonically increasing
> timestamp: it depends on the filling level of the TCP rx/tx kernel buffers,
> the zmq HWMs, etc.
>
> For this reason I'm looking for some algorithm that
> * allows me to push zmq_msg_t pulled out of the SUB socket (without strict
> time ordering)
> * allows me to pull out zmq_msg_t that have a timestamp monotonically
> increasing
> * introduces a fixed max latency of N msecs (configurable)
>
> Do you have any pointer for such kind of problem?
> Anybody already hit a similar issue?
>
> Thanks for any help,
>
> Francesco Montorsi
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20221111/4742ef6c/attachment.htm>


More information about the zeromq-dev mailing list