[zeromq-dev] Delayed message reception problem.

[Divya Mohan]

While performance testing my code (which uses ZeroMQ) I’m observing *delayed
message reception*.

 *Test Scenario*

Process1: Sends messages (each 76 bytes long) using zmq PUB/SUB (IPC
Protocol) in an infinite while loop with 1msec delay.

Process2: Receives messages from Process1 using zmq PUB/SUB (IPC protocol)
in an infinite while loop.

If either of the below two conditions are true on receiving the message I
log to a file.

- If the message delay (receive time - send time) is greater than 10msec.

- If there is message loss.

*Observation*

After running the above test for 108 hours these are my observations:

 -       No message loss

 -       *Lots of message delays are seen.* Delay of more than 100msec is
common, with max delay seen = 220msec.


Delay appears to be random and not increasing incrementally or in sequence.
If one message is delayed by 100msec the next 89 consecutive messages are
also logged as they are received with delays 99, 98, 97 and so on till
10msec. From this I understand that once we see the delay for one packet
all the subsequent packets are received at the same time.

I suspected that Process2 was not getting scheduled frequently enough and
increased the priority of this process to nice value -20 but it did not
help.

*I am not able to figure out why some messages are getting delayed. Does
anyone have any ideas? Has anyone faced this sort of thing before? Any
suggestions would be really helpful. Is there anything else I can try to
get to the root cause of the message delay problem?*

Note: Process1 and Process2 call my wrapper APIs which in turn calls the
ZeroMq Apis. In the receive/send wrapper APIs there are no locks/mutex or
any blocking statements. There are only a few parameter checks and socket
validity checks before calling zmq_send/zmq_recv. Whether send/recv is
blocking or non-blocking is decided by a parameter in wrapper API.

Thanks,

Divya
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20130327/764acbe8/attachment.htm>