[zeromq-dev] pyzmq poller performance

[Brian Knox]

Min - just a quick follow up on this.  I had a little spare time today
(don't tell anyone) so I put together a single socket to single socket
poller / no poller test in C with libzmq.

The ratio was almost the same as pyzmq - I had ~ 1.37 million messages a
second with the poller, and ~ 4.7 million messages a second without it.

So, the python ration (with the updated pyzmq with your poller
improvements) is ~ 2.9x and in C it's ~ 2.7x.

Brian


On Tue, Jun 25, 2013 at 10:56 AM, Brian Knox <taotetek at gmail.com> wrote:

> Min - I merged the poll request on my little laptop and gave it a spin.
> Before your pull request:
>
>
> mps: 28066
> mps: 28116
> mps: 29745
> mps: 28912
> mps: 28679
>
> after your pull request:
>
> mps: 78138
> mps: 78364
> mps: 78411
> mps: 78065
> mps: 78175
>
> Excellent.
>
> By the way, just for fun I imported zmq.green instead of zmq without
> changing anything else
>
> mps: 24738
> mps: 24779
> mps: 24759
> mps: 24630
> mps: 24702
>
> And it's performance is well below straight zmq Poller.
>
>
> Thanks for the quick patch!
>
> Brian
>
>
> On Tue, Jun 25, 2013 at 8:45 AM, Brian Knox <briank at talksum.com> wrote:
>
>> Thanks!  The 4x degradation is within an acceptable range for what we're
>> doing and will allow me to keep some things in python for awhile longer.
>> I'll patch locally and take it for a test drive today.
>>
>> Brian
>>
>>
>> On Mon, Jun 24, 2013 at 11:09 PM, MinRK <benjaminrk at gmail.com> wrote:
>>
>>> From my tests this evening, it looks like zmq_poll does have an
>>> appreciable cost (plus the overhead of being wrapped in Python), and will
>>> be notably slower than a simple `while True: s.recv()`.  But I did find
>>> some inefficiencies in the Poller implementation, which should be addressed
>>> by PR #381 <https://github.com/zeromq/pyzmq/pull/381>, bringing the
>>> difference closer to 4x from 10x.
>>>
>>> Your tests on my laptop:
>>>
>>> pure recv:
>>> mps: 241121
>>> mps: 419644
>>> mps: 413874
>>> mps: 434421
>>> mps: 428623
>>> mps: 404847
>>>
>>> poll (master):
>>> mps: 48544
>>> mps: 54143
>>> mps: 51642
>>> mps: 48888
>>> mps: 47591
>>> mps: 54218
>>>
>>> poll (after 381):
>>>  mps: 110436
>>> mps: 110876
>>> mps: 104220
>>> mps: 110690
>>> mps: 100544
>>> mps: 110922
>>>
>>> -MinRK
>>>
>>>
>>>
>>> On Mon, Jun 24, 2013 at 12:18 PM, Brian Knox <briank at talksum.com> wrote:
>>>
>>>> Thanks Min - additionally if I'm incorrect in my assumption that the
>>>> poll look in my example code should perform better, or if there's a better
>>>> way to do what I'm doing just let me know.
>>>>
>>>> The example code polls only one socket (which isn't that useful, hah)
>>>> but I wanted to keep the test case dirt simple.  The actual thing I was
>>>> working on has one socket that should receive data at a relatively high
>>>> rate (say 50k to 100k msgs/s)  , and one socket that receives data at a
>>>> much lower rate (say 1 msg/s) (for heartbeating, command and control, etc).
>>>>
>>>>
>>>> Brian
>>>>
>>>>
>>>> On Mon, Jun 24, 2013 at 12:12 PM, Min RK <benjaminrk at gmail.com> wrote:
>>>>
>>>>> Thanks for the report, I will look into whether I have introduced a
>>>>> performance degradation in the last few  iterations.
>>>>>
>>>>> -MinRK
>>>>>
>>>>> On Jun 24, 2013, at 11:27, Brian Knox <briank at talksum.com> wrote:
>>>>>
>>>>> > It's been awhile since I've used pyzmq, and I'm running into a
>>>>> performance issue using Poller.poll().
>>>>> >
>>>>> > With a simple blocking recv() in a while True loop, I get ~ 300k
>>>>> messages a second.
>>>>> >
>>>>> > With the simplest case of the same code using a poll on the socket,
>>>>> I get ~ 30k messages a second.
>>>>> >
>>>>> > (pyzeromq)taotetek at Moya:~/src/performance_example$ python
>>>>> ./receiver.py
>>>>> > mps: 226049
>>>>> > mps: 348582
>>>>> > mps: 271728
>>>>> > mps: 300389
>>>>> > mps: 411059
>>>>> > mps: 276749
>>>>> >
>>>>> > (pyzeromq)taotetek at Moya:~/src/performance_example$ python
>>>>> ./poll_receiver.py
>>>>> > mps: 28066
>>>>> > mps: 28116
>>>>> > mps: 29745
>>>>> > mps: 28912
>>>>> > mps: 28679
>>>>> >
>>>>> > I don't remember a 10x performance degradation last time I used a
>>>>> poller - but perhaps I'm remembering something wrong, or making a painfully
>>>>> obvious error in my code .. I've been using czmq from C mostly for the last
>>>>> while.
>>>>> >
>>>>> > I'm using pyzmq from git master, and the latest Cython (0.19.1) with
>>>>> libzmq 3.2.3.
>>>>> >
>>>>> > I've attached the test cases I'm using.
>>>>> >
>>>>> > <performance_example.tar>
>>>>> > _______________________________________________
>>>>> > zeromq-dev mailing list
>>>>> > zeromq-dev at lists.zeromq.org
>>>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>> _______________________________________________
>>>>> zeromq-dev mailing list
>>>>> zeromq-dev at lists.zeromq.org
>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> zeromq-dev mailing list
>>>> zeromq-dev at lists.zeromq.org
>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20130724/5a67b8b3/attachment.htm>