[zeromq-dev] Time-reordering queue

[Francesco]

Hi all,

I have written two applications using ZMQ PUB-SUB pattern (over TCP
transport).
The subscriber application has its SUB socket connected to multiple PUBs
(multiple tcp endpoints). Each message sent by the PUB encodes the
timestamp (as obtained from clock_gettime() syscall at TX side using
monotonically increasing clock) of the event described by the ZMQ message.

The subscriber needs to process the data stream _strictly_ in order.
However the multiple publishers have no coordination and they will emit
messages at different rates, each with its own timestamp. The only
guarantee that I have, according to ZMQ docs, is that the SUB socket will
perform "fair dequeueing", but that's not enough to guarantee that every
zmq_msg_t received from the SUB socket will have a monotonically increasing
timestamp: it depends on the filling level of the TCP rx/tx kernel buffers,
the zmq HWMs, etc.

For this reason I'm looking for some algorithm that
* allows me to push zmq_msg_t pulled out of the SUB socket (without strict
time ordering)
* allows me to pull out zmq_msg_t that have a timestamp monotonically
increasing
* introduces a fixed max latency of N msecs (configurable)

Do you have any pointer for such kind of problem?
Anybody already hit a similar issue?

Thanks for any help,

Francesco Montorsi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20221109/e53b9d4a/attachment.htm>