[zeromq-dev] Time-reordering queue

[Bill Torpey]

Hi Francesco:

At the risk of telling you something you already know, have you thought about how these timestamps are being generated?  I assume that the multiple PUBs are, or at least can be, located on different machines.  In that scenario the problem of keeping clocks in sync between the machines becomes non-trivial.

The best explanation I’ve seen is this: https://queue.acm.org/detail.cfm?id=2878574 <https://queue.acm.org/detail.cfm?id=2878574>, which in turn references Lamport’s seminal work: http://lamport.azurewebsites.net/pubs/time-clocks.pdf <http://lamport.azurewebsites.net/pubs/time-clocks.pdf>.

Hope this helps.

Bill Torpey


> On Nov 9, 2022, at 5:19 PM, Francesco <francesco.montorsi at gmail.com> wrote:
> 
> Hi all,
> 
> I have written two applications using ZMQ PUB-SUB pattern (over TCP transport).
> The subscriber application has its SUB socket connected to multiple PUBs (multiple tcp endpoints). Each message sent by the PUB encodes the timestamp (as obtained from clock_gettime() syscall at TX side using monotonically increasing clock) of the event described by the ZMQ message.
> 
> The subscriber needs to process the data stream _strictly_ in order. However the multiple publishers have no coordination and they will emit messages at different rates, each with its own timestamp. The only guarantee that I have, according to ZMQ docs, is that the SUB socket will perform "fair dequeueing", but that's not enough to guarantee that every zmq_msg_t received from the SUB socket will have a monotonically increasing timestamp: it depends on the filling level of the TCP rx/tx kernel buffers, the zmq HWMs, etc.
> 
> For this reason I'm looking for some algorithm that
> * allows me to push zmq_msg_t pulled out of the SUB socket (without strict time ordering)
> * allows me to pull out zmq_msg_t that have a timestamp monotonically increasing
> * introduces a fixed max latency of N msecs (configurable)
> 
> Do you have any pointer for such kind of problem? 
> Anybody already hit a similar issue?
> 
> Thanks for any help,
> 
> Francesco Montorsi
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20221110/c1290253/attachment.htm>