[zeromq-dev] Reduce latency on Pub-Sub Broker

[Pieter Hintjens]

Robert,

You're running 3 processes on a 4-core box; each 0MQ process needs at
least 2 cores for maximum performance, one for the application thread
and one for the I/O thread. This is probably the main cause for
latency building up. You should try with higher HWMs, and giving each
process at least 2 cores.

-Pieter

On Tue, May 29, 2012 at 12:07 PM, Robert Evans
<zwergzebra at googlemail.com> wrote:
> Hi Pieter,
>
> I ran your suggested C latency tests and the results seemed to be okay.
>
> local_lat and remote_lat with 10 bytes message with 100 000 roundtrips
> showed a latency of 43 microseconds, inproc_lat 16 microseconds. I think the
> problem is my broker. When I used a direct connection between publisher with
> HWM=500 and subscriber without a broker the latency was about 0,3
> milliseconds. Once I added the broker again, the latency increased to ~15
> milliseconds.
>
> My CPU has 4 cores and the network is capable of 100 mbit/s. I'm currently
> running everything (pub, sub and broker) on the same machine. Maybe this
> leads to some problems with the CPU or the network. I will try moving the
> broker to an own machine and see what will happen. I once had an experience
> with AleriMQ, when the latency was significantly lower when the broker ran
> on a seperate machine - perhaps it could be the same here.
>
> Regards,
> Robert
>
> 2012/5/29 Pieter Hintjens <ph at imatix.com>
>>
>> Robert,
>>
>> First, try some tests in plain C (using the low level API or CZMQ), to
>> make sure that it's not the C# binding adding latency.
>>
>> There are command-line test programs you can use (perf/local_lat,
>> perf/remote_lat).
>>
>> Secondly, if the latency increases when you send messages faster,
>> there is something bogus happening. 0MQ will normally batch small
>> messages which reduces latency if you send larger volumes. The main
>> reason for setting a HWM is if your subscriber or network is too slow,
>> and you need to guard against memory exhaustion.
>>
>> It sounds like you have a CPU contention issue. How many cores on your
>> box? What kind of network?
>>
>> -Pieter
>>
>>
>>
>> On Tue, May 29, 2012 at 10:26 AM, Robert Evans
>> <zwergzebra at googlemail.com> wrote:
>> > Hello everyone,
>> >
>> > I am currently developing a Publish-Subscribe broker using the C#
>> > binding
>> > clrzmq v. 2.2.3. My operating system is Windows 7 Prof. and my system is
>> > a
>> > Intel i7 with 16 GByte RAM.
>> >
>> > The broker works without any problems, I have added one publisher and
>> > one
>> > subscriber and get message rates about 250 000 - 150 000 msg/second when
>> > sending 10 byte messages on localhost. The broker simply forwards the
>> > messages from the publisher to the subscriber, no big magic there.
>> >
>> > My problem is the increasing latency when sending many messages at short
>> > time. When I only send one message and sleep for one millisecond the
>> > latency
>> > is < 1ms, but the message rate drops to 9000 msg/s. When I'm sending
>> > messages straight without sleeping at ~ 200 000 msg/s, the latency
>> > increases
>> > with the number of messages sent. I set the HWM on the broker for the
>> > pub
>> > and sub socket to 500, which seems to narrow the latency to 10 - 20 ms.
>> > This
>> > is okay, but a value between 1 and 3 ms would be even better.
>> >
>> > Is there a way to reduce it even more? I don't really need the queuing
>> > of
>> > messages with the HWM; I would rather like to drop messages instantly
>> > instead of queuing as low latency is very important in my project (but I
>> > still need a message rate about 100 000 msg/s). As to say shortly:
>> > Better
>> > drop some messages for lower latency, instead of buffering them.
>> >
>> > It would be great if someone has an idea or already has some experience
>> > with
>> > tweaking the latency. Thank you!
>> >
>> > Regards,
>> > Robert
>> >
>> > _______________________________________________
>> > zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>