[zeromq-dev] Performance issues in multiple queues from a single process..

[Arvind N]

Yes, sorry the ascii graphics got a messed up. But your understanding is right... its process2 that is doing a recv from process1 and forwarding it to process3 thru a send. Just to give some numbers, as in maybe someone would find it useful ...
Process1 pushes upwards of 3000 packets/sec. Process2 recv's it and does decoding on them where each packet inturn creates 10 data records(30000 datarecord/sec) and tries to push each data record i.e 30000 calls/sec to send towards process3. And this push of 30000 slows the whole thing down.... 

>> The loop which has to do both send and recv is obviously going to work at half speed when compared to loop that does just recv. Can that be the problem you are seeing?

Yes this is the problem ...  just wanted to know why should the loop doing the recv and then send slow down ?? as it this is no CPU loading issue.. ?? why should the speed comedown when CPU is not the issue... is it IO related ??

Thanks
Arvind


-----Original Message-----
From: Martin Sustrik [mailto:sustrik at 250bpm.com] 
Sent: Friday, May 27, 2011 1:28 AM
To: ZeroMQ development list
Cc: Arvind N
Subject: Re: [zeromq-dev] Performance issues in multiple queues from a single process..

Hi Arvind,

> Process1
> --------------------------------------------->
> Process2                                             --------300
> packets/sec -------------->             Process3 (pushq)
> slows down gradually from 2500                   (PullQ towards
> Process1, PushQ to process3)
> (PullQ towards Process2)

The ASCII graphic was messed up. However, I'm assuming you are sending a message from process 1 to process 2 which then forwards it to process 3.

> In the above usecase, we see performance issues ... first the moment, 
> the call to zmq_send from Process2  is done, the performance of
> Process2 to pull comes down to almost half and even less as time 
> progresses. This inturn leads to slowness in pulling. Commenting the 
> zmq_send makes the Process2 makes the while thing work fine again.

What exactly are you measuring?

The loop which has to do both send and recv is obviously going to work at half speed when compared to loop that does just recv. Can that be the problem you are seeing?

Martin