[zeromq-dev] pub/sub performance help

[ZOUNG KALACK Jimy (EXT)]

Thanks Chuck

http://pastie.org/2453122   (publisher)
http://pastie.org/2453111   (subscriber)

Could someone tell why when the publisher publishes 1.000.000 messages, the subscriber starts picking them, then stops in the middle and restart picking them when the publisher finishes.

 

-----Original Message-----
From: zeromq-dev-bounces at lists.zeromq.org [mailto:zeromq-dev-bounces at lists.zeromq.org] On Behalf Of Chuck Remes
Sent: Monday,August 29,2011 4:20 PM
To: ZeroMQ development list
Subject: Re: [zeromq-dev] pub/sub performance help


On Aug 29, 2011, at 8:34 AM, ZOUNG KALACK Jimy (EXT) wrote:

> Remes, 
> 
> http://pastie.org/2448540 (publisher)
> 
> http://pastie.org/2448547 (subscriber)
> 
> Jimy
> -----Original Message-----
> From: zeromq-dev-bounces at lists.zeromq.org [mailto:zeromq-dev-bounces at lists.zeromq.org] On Behalf Of Chuck Remes
> Sent: Monday,August 29,2011 2:49 PM
> To: ZeroMQ development list
> Subject: Re: [zeromq-dev] pub/sub performance help
> 
> On Aug 29, 2011, at 7:05 AM, ZOUNG KALACK Jimy (EXT) wrote:
> 
>> Thanks Pieter I will check again.
>> My technique is simple.
>> Publisher and subscriber are on the same box.
>> The publisher sends 10.000 messages.
>> The message sent is just the timestamp in the format (yyyy/mm/dd HH:mm:ss.fffff)
>> For each message the subscriber receives, the latency is calculated.
>> Ln = In-On
>> 
>> Jimy
> 
> Please put your code up for us to see. I recommend gist.github.com or pastie.org.

Good.

So, to solve your first problem (you only see a fraction of the published messages) you should add a sleep between lines 27 and 28 of your publisher. Alternately, have it wait for you to press a key or something before it starts publishing. This will give you time to start up the subscriber and let it finish its connection handshake so that no messages are dropped.

Secondly, the increased latency you are seeing may be due to a few factors.

1. Publishers are *always* faster than subscribers because they have less work to do. Since you do not set a HWM, by default the socket will queue as many messages as memory will allow. So the publisher is sending as fast as it can but your subscriber is slow.

2. I don't know how fast the Date operations are in python. It's entirely possible that converting to and from a string is too slow.

3. You are saving each latency calculation in a list. Perhaps it is slow to append a lot of objects to a python list. At minimum there will be many mallocs and garbage collections going on in the background all of which will slow down operations.


Here are my suggestions. 

1. Eliminate the current latency calculation and the list add operation. 

2. Change the publisher to send a fixed message (e.g. a 50-byte string). You want to measure latency, not string allocation time.

3. Replace the latency calc with a timer. When the subscriber receives its first message, start the timer. When it receives the "STOP" message, calculate latency by measuring the overall time and dividing by the number of messages received.

cr

_______________________________________________
zeromq-dev mailing list
zeromq-dev at lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
*************************************************************************
This message and any attachments (the "message") are confidential, intended solely for the addressee(s), and may contain legally privileged information.
Any unauthorised use or dissemination is prohibited. E-mails are susceptible to alteration.   
Neither SOCIETE GENERALE nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or
falsified.
                              ************
Ce message et toutes les pieces jointes (ci-apres le "message") sont confidentiels et susceptibles de contenir des informations couvertes 
par le secret professionnel. 
Ce message est etabli a l'intention exclusive de ses destinataires. Toute utilisation ou diffusion non autorisee est interdite.
Tout message electronique est susceptible d'alteration. 
La SOCIETE GENERALE et ses filiales declinent toute responsabilite au titre de ce message s'il a ete altere, deforme ou falsifie.
*************************************************************************