[zeromq-dev] pyzmq poller performance

[Brian Knox]

Min - I merged the poll request on my little laptop and gave it a spin.
Before your pull request:

mps: 28066
mps: 28116
mps: 29745
mps: 28912
mps: 28679

after your pull request:

mps: 78138
mps: 78364
mps: 78411
mps: 78065
mps: 78175

Excellent.

By the way, just for fun I imported zmq.green instead of zmq without
changing anything else

mps: 24738
mps: 24779
mps: 24759
mps: 24630
mps: 24702

And it's performance is well below straight zmq Poller.


Thanks for the quick patch!

Brian


On Tue, Jun 25, 2013 at 8:45 AM, Brian Knox <briank at talksum.com> wrote:

> Thanks!  The 4x degradation is within an acceptable range for what we're
> doing and will allow me to keep some things in python for awhile longer.
> I'll patch locally and take it for a test drive today.
>
> Brian
>
>
> On Mon, Jun 24, 2013 at 11:09 PM, MinRK <benjaminrk at gmail.com> wrote:
>
>> From my tests this evening, it looks like zmq_poll does have an
>> appreciable cost (plus the overhead of being wrapped in Python), and will
>> be notably slower than a simple `while True: s.recv()`.  But I did find
>> some inefficiencies in the Poller implementation, which should be addressed
>> by PR #381 <https://github.com/zeromq/pyzmq/pull/381>, bringing the
>> difference closer to 4x from 10x.
>>
>> Your tests on my laptop:
>>
>> pure recv:
>> mps: 241121
>> mps: 419644
>> mps: 413874
>> mps: 434421
>> mps: 428623
>> mps: 404847
>>
>> poll (master):
>> mps: 48544
>> mps: 54143
>> mps: 51642
>> mps: 48888
>> mps: 47591
>> mps: 54218
>>
>> poll (after 381):
>>  mps: 110436
>> mps: 110876
>> mps: 104220
>> mps: 110690
>> mps: 100544
>> mps: 110922
>>
>> -MinRK
>>
>>
>>
>> On Mon, Jun 24, 2013 at 12:18 PM, Brian Knox <briank at talksum.com> wrote:
>>
>>> Thanks Min - additionally if I'm incorrect in my assumption that the
>>> poll look in my example code should perform better, or if there's a better
>>> way to do what I'm doing just let me know.
>>>
>>> The example code polls only one socket (which isn't that useful, hah)
>>> but I wanted to keep the test case dirt simple.  The actual thing I was
>>> working on has one socket that should receive data at a relatively high
>>> rate (say 50k to 100k msgs/s)  , and one socket that receives data at a
>>> much lower rate (say 1 msg/s) (for heartbeating, command and control, etc).
>>>
>>>
>>> Brian
>>>
>>>
>>> On Mon, Jun 24, 2013 at 12:12 PM, Min RK <benjaminrk at gmail.com> wrote:
>>>
>>>> Thanks for the report, I will look into whether I have introduced a
>>>> performance degradation in the last few  iterations.
>>>>
>>>> -MinRK
>>>>
>>>> On Jun 24, 2013, at 11:27, Brian Knox <briank at talksum.com> wrote:
>>>>
>>>> > It's been awhile since I've used pyzmq, and I'm running into a
>>>> performance issue using Poller.poll().
>>>> >
>>>> > With a simple blocking recv() in a while True loop, I get ~ 300k
>>>> messages a second.
>>>> >
>>>> > With the simplest case of the same code using a poll on the socket, I
>>>> get ~ 30k messages a second.
>>>> >
>>>> > (pyzeromq)taotetek at Moya:~/src/performance_example$ python
>>>> ./receiver.py
>>>> > mps: 226049
>>>> > mps: 348582
>>>> > mps: 271728
>>>> > mps: 300389
>>>> > mps: 411059
>>>> > mps: 276749
>>>> >
>>>> > (pyzeromq)taotetek at Moya:~/src/performance_example$ python
>>>> ./poll_receiver.py
>>>> > mps: 28066
>>>> > mps: 28116
>>>> > mps: 29745
>>>> > mps: 28912
>>>> > mps: 28679
>>>> >
>>>> > I don't remember a 10x performance degradation last time I used a
>>>> poller - but perhaps I'm remembering something wrong, or making a painfully
>>>> obvious error in my code .. I've been using czmq from C mostly for the last
>>>> while.
>>>> >
>>>> > I'm using pyzmq from git master, and the latest Cython (0.19.1) with
>>>> libzmq 3.2.3.
>>>> >
>>>> > I've attached the test cases I'm using.
>>>> >
>>>> > <performance_example.tar>
>>>> > _______________________________________________
>>>> > zeromq-dev mailing list
>>>> > zeromq-dev at lists.zeromq.org
>>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>> _______________________________________________
>>>> zeromq-dev mailing list
>>>> zeromq-dev at lists.zeromq.org
>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>
>>>
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20130625/215e9291/attachment.htm>