[zeromq-dev] Bad ZMQ latency on first 500 messages per connection

Pieter Hintjens ph at imatix.com
Wed Sep 30 14:49:28 CEST 2015


Before tweaking the send calls, I'd advise making a *minimal* test
case that shows the effect, in C. We can then see exactly where the
latency spike is coming from. No point in trying to remove an effect
we don't understand (in fact, counter productive, we need to
understand the spike first.)

On Wed, Sep 30, 2015 at 8:31 AM,  <Christian.Kohrt at rohde-schwarz.com> wrote:
> We also agree that there must be a buffer somewhere in TCP stack or
> OS/memory(page allocation)/scheduling. It becomes clear, like Ben mentioned,
> when the sender sends a lot more messages than the receiver receives.
>
> A time delay for the TCP connection could be a reason for that, as Peter
> asked. But in our scenario we do a zmq_connect/zmq_bind at the beginning,
> which is not part of the measured results.
> We use inproc communication with ZMQ 4.0.4 in a C++ project with "zmq_send".
>
> All message data we have is memory aligned structures (memory blocks). Does
> this mean we could also utilize "zmq_send_const"? We could then eliminate at
> least one copy operation if I am right.
>
>
>
>> Date: Tue, 29 Sep 2015 09:44:19 +0200
>> From: Pieter Hintjens <ph at imatix.com>
>> Subject: Re: [zeromq-dev] Bad ZMQ latency on first 500 messages per
>>                  connection
>> To: ZeroMQ development list <zeromq-dev at lists.zeromq.org>
>> Message-ID:
>>
>> <CADL5_sgpYKi=k2doeObsrAkmGbfzv6SMcfe=q2QWP0_ZcoAwzQ at mail.gmail.com>
>> Content-Type: text/plain; charset=UTF-8
>
>>
>> Have you excluded the cost of establishing the TCP connection itself
>> (something around 10 msec, depending on the setup)?
>>
>> On Tue, Sep 29, 2015 at 8:49 AM, Ben Kloosterman <bklooste at gmail.com>
>> wrote:
>>> "As can be seen, sender and receiver throughput's differ at the beginning
>>> of
>>> the test. This seems to suggest that there is some kind of buffering
>>> involved on the lower layers of the stack"
>>>
>>> Nagle is turned off  but it looks like it keeps increasing some sort of
>>> buffer in the tcp stack.. could also be OS / memory ( page allocation)  /
>>> Scheduling priority.
>>>
>>> Note in those 500 messages the sender was sending a lot more messages ,
>>> than
>>> the receiver was receiving so a buffer would be growing somewhere.
>>>
>>> Ben
>>>
>>> On Tue, Sep 29, 2015 at 4:35 PM, <Christian.Kohrt at rohde-schwarz.com>
>>> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> when sending many messages at once there is a known latency issue: The
>>>> first 500 messages are sent with a high latency shown in the graph in
>>>> the
>>>> whitepaper
> http://zeromq.org/whitepapers:measuring-performance before it
>
>>>> gets to very high speed messaging.
>>>>
>>>> The author summarizes:
>>>> * For latency: ... the latency for first 500 messages is quite poor (up
>>>> to
>>>> 1,500 microseconds), however, afterwards it stabilises at approximately
>>>> 150
>>>> microseconds, with occasional peaks up to 400 microseconds.
>>>> * For throughput: This seems to suggest that there is some kind of
>>>> buffering involved on the lower layers of the stack.
>>>>
>>>> In our scenario, we use a Dealer/Router connection, which is in fact
>>>> like
>>>> REQ/REP.
>>>>
>>>> Question:
>>>> Why have the first 500 messages this latency issue? What is the reason
>>>> for
>>>> this and how could we solve this issue, because we require a high
>>>> performance application even for the first messages.
>>>>
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>>
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>



More information about the zeromq-dev mailing list