[zeromq-dev] pyzmq poller performance

[Brian Knox]

Thanks!  The 4x degradation is within an acceptable range for what we're
doing and will allow me to keep some things in python for awhile longer.
I'll patch locally and take it for a test drive today.

Brian


On Mon, Jun 24, 2013 at 11:09 PM, MinRK <benjaminrk at gmail.com> wrote:

> From my tests this evening, it looks like zmq_poll does have an
> appreciable cost (plus the overhead of being wrapped in Python), and will
> be notably slower than a simple `while True: s.recv()`.  But I did find
> some inefficiencies in the Poller implementation, which should be addressed
> by PR #381 <https://github.com/zeromq/pyzmq/pull/381>, bringing the
> difference closer to 4x from 10x.
>
> Your tests on my laptop:
>
> pure recv:
> mps: 241121
> mps: 419644
> mps: 413874
> mps: 434421
> mps: 428623
> mps: 404847
>
> poll (master):
> mps: 48544
> mps: 54143
> mps: 51642
> mps: 48888
> mps: 47591
> mps: 54218
>
> poll (after 381):
>  mps: 110436
> mps: 110876
> mps: 104220
> mps: 110690
> mps: 100544
> mps: 110922
>
> -MinRK
>
>
>
> On Mon, Jun 24, 2013 at 12:18 PM, Brian Knox <briank at talksum.com> wrote:
>
>> Thanks Min - additionally if I'm incorrect in my assumption that the poll
>> look in my example code should perform better, or if there's a better way
>> to do what I'm doing just let me know.
>>
>> The example code polls only one socket (which isn't that useful, hah) but
>> I wanted to keep the test case dirt simple.  The actual thing I was working
>> on has one socket that should receive data at a relatively high rate (say
>> 50k to 100k msgs/s)  , and one socket that receives data at a much lower
>> rate (say 1 msg/s) (for heartbeating, command and control, etc).
>>
>>
>> Brian
>>
>>
>> On Mon, Jun 24, 2013 at 12:12 PM, Min RK <benjaminrk at gmail.com> wrote:
>>
>>> Thanks for the report, I will look into whether I have introduced a
>>> performance degradation in the last few  iterations.
>>>
>>> -MinRK
>>>
>>> On Jun 24, 2013, at 11:27, Brian Knox <briank at talksum.com> wrote:
>>>
>>> > It's been awhile since I've used pyzmq, and I'm running into a
>>> performance issue using Poller.poll().
>>> >
>>> > With a simple blocking recv() in a while True loop, I get ~ 300k
>>> messages a second.
>>> >
>>> > With the simplest case of the same code using a poll on the socket, I
>>> get ~ 30k messages a second.
>>> >
>>> > (pyzeromq)taotetek at Moya:~/src/performance_example$ python
>>> ./receiver.py
>>> > mps: 226049
>>> > mps: 348582
>>> > mps: 271728
>>> > mps: 300389
>>> > mps: 411059
>>> > mps: 276749
>>> >
>>> > (pyzeromq)taotetek at Moya:~/src/performance_example$ python
>>> ./poll_receiver.py
>>> > mps: 28066
>>> > mps: 28116
>>> > mps: 29745
>>> > mps: 28912
>>> > mps: 28679
>>> >
>>> > I don't remember a 10x performance degradation last time I used a
>>> poller - but perhaps I'm remembering something wrong, or making a painfully
>>> obvious error in my code .. I've been using czmq from C mostly for the last
>>> while.
>>> >
>>> > I'm using pyzmq from git master, and the latest Cython (0.19.1) with
>>> libzmq 3.2.3.
>>> >
>>> > I've attached the test cases I'm using.
>>> >
>>> > <performance_example.tar>
>>> > _______________________________________________
>>> > zeromq-dev mailing list
>>> > zeromq-dev at lists.zeromq.org
>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20130625/a88c525c/attachment.htm>