[zeromq-dev] Bad ZMQ latency on first 500 messages per connection

Christian.Kohrt at rohde-schwarz.com Christian.Kohrt at rohde-schwarz.com
Wed Sep 30 08:31:32 CEST 2015


We also agree that there must be a buffer somewhere in TCP stack or 
OS/memory(page allocation)/scheduling. It becomes clear, like Ben 
mentioned, when the sender sends a lot more messages than the receiver 
receives.

A time delay for the TCP connection could be a reason for that, as Peter 
asked. But in our scenario we do a zmq_connect/zmq_bind at the beginning, 
which is not part of the measured results.
We use inproc communication with ZMQ 4.0.4 in a C++ project with "zmq_send
".

All message data we have is memory aligned structures (memory blocks). 
Does this mean we could also utilize "zmq_send_const"? We could then 
eliminate at least one copy operation if I am right. 
 


> Date: Tue, 29 Sep 2015 09:44:19 +0200
> From: Pieter Hintjens <ph at imatix.com>
> Subject: Re: [zeromq-dev] Bad ZMQ latency on first 500 messages per
>                connection
> To: ZeroMQ development list <zeromq-dev at lists.zeromq.org>
> Message-ID:
> <CADL5_sgpYKi=k2doeObsrAkmGbfzv6SMcfe=q2QWP0_ZcoAwzQ at mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
> 
> Have you excluded the cost of establishing the TCP connection itself
> (something around 10 msec, depending on the setup)?
> 
> On Tue, Sep 29, 2015 at 8:49 AM, Ben Kloosterman <bklooste at gmail.com> 
wrote:
>> "As can be seen, sender and receiver throughput's differ at the 
beginning of
>> the test. This seems to suggest that there is some kind of buffering
>> involved on the lower layers of the stack"
>>
>> Nagle is turned off  but it looks like it keeps increasing some sort of
>> buffer in the tcp stack.. could also be OS / memory ( page allocation) 
/
>> Scheduling priority.
>>
>> Note in those 500 messages the sender was sending a lot more messages , 
than
>> the receiver was receiving so a buffer would be growing somewhere.
>>
>> Ben
>>
>> On Tue, Sep 29, 2015 at 4:35 PM, <Christian.Kohrt at rohde-schwarz.com> 
wrote:
>>>
>>> Hi all,
>>>
>>> when sending many messages at once there is a known latency issue: The
>>> first 500 messages are sent with a high latency shown in the graph in 
the
>>> whitepaper http://zeromq.org/whitepapers:measuring-performance before 
it
>>> gets to very high speed messaging.
>>>
>>> The author summarizes:
>>> * For latency: ... the latency for first 500 messages is quite poor 
(up to
>>> 1,500 microseconds), however, afterwards it stabilises at 
approximately 150
>>> microseconds, with occasional peaks up to 400 microseconds.
>>> * For throughput: This seems to suggest that there is some kind of
>>> buffering involved on the lower layers of the stack.
>>>
>>> In our scenario, we use a Dealer/Router connection, which is in fact 
like
>>> REQ/REP.
>>>
>>> Question:
>>> Why have the first 500 messages this latency issue? What is the reason 
for
>>> this and how could we solve this issue, because we require a high
>>> performance application even for the first messages.
>>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20150930/3193c687/attachment.htm>


More information about the zeromq-dev mailing list