[zeromq-dev] pub/sub performance help

Chuck Remes cremes.devlist at mac.com
Mon Aug 29 16:19:50 CEST 2011


On Aug 29, 2011, at 8:34 AM, ZOUNG KALACK Jimy (EXT) wrote:

> Remes, 
> 
> http://pastie.org/2448540 (publisher)
> 
> http://pastie.org/2448547 (subscriber)
> 
> Jimy
> -----Original Message-----
> From: zeromq-dev-bounces at lists.zeromq.org [mailto:zeromq-dev-bounces at lists.zeromq.org] On Behalf Of Chuck Remes
> Sent: Monday,August 29,2011 2:49 PM
> To: ZeroMQ development list
> Subject: Re: [zeromq-dev] pub/sub performance help
> 
> On Aug 29, 2011, at 7:05 AM, ZOUNG KALACK Jimy (EXT) wrote:
> 
>> Thanks Pieter I will check again.
>> My technique is simple.
>> Publisher and subscriber are on the same box.
>> The publisher sends 10.000 messages.
>> The message sent is just the timestamp in the format (yyyy/mm/dd HH:mm:ss.fffff)
>> For each message the subscriber receives, the latency is calculated.
>> Ln = In-On
>> 
>> Jimy
> 
> Please put your code up for us to see. I recommend gist.github.com or pastie.org.

Good.

So, to solve your first problem (you only see a fraction of the published messages) you should add a sleep between lines 27 and 28 of your publisher. Alternately, have it wait for you to press a key or something before it starts publishing. This will give you time to start up the subscriber and let it finish its connection handshake so that no messages are dropped.

Secondly, the increased latency you are seeing may be due to a few factors.

1. Publishers are *always* faster than subscribers because they have less work to do. Since you do not set a HWM, by default the socket will queue as many messages as memory will allow. So the publisher is sending as fast as it can but your subscriber is slow.

2. I don't know how fast the Date operations are in python. It's entirely possible that converting to and from a string is too slow.

3. You are saving each latency calculation in a list. Perhaps it is slow to append a lot of objects to a python list. At minimum there will be many mallocs and garbage collections going on in the background all of which will slow down operations.


Here are my suggestions. 

1. Eliminate the current latency calculation and the list add operation. 

2. Change the publisher to send a fixed message (e.g. a 50-byte string). You want to measure latency, not string allocation time.

3. Replace the latency calc with a timer. When the subscriber receives its first message, start the timer. When it receives the "STOP" message, calculate latency by measuring the overall time and dividing by the number of messages received.

cr




More information about the zeromq-dev mailing list