[zeromq-dev] how to get it to run faster?

Michel Pelletier pelletier.michel at gmail.com
Mon Apr 2 22:25:43 CEST 2012


I know that feeling. :)

I'm not sure if the OP hasn't had the chance to follow up or not, so
maybe I'm not helping anyone continuing, but one of the other problems
with the original code posted is that the put() and get() function
declare whole new contexts and sockets on a *per message* basis, this
is a big waste of time as well, fixing the serialization problems (and
by fixing I mean, remove it) and sharing context and sockets sensibly
would make this example much more performant.

Hopefully Sean rejoins us if he has any questions.

-Michel

On Mon, Apr 2, 2012 at 1:14 PM, Asher Benjamin <asherbenj at gmail.com> wrote:
> memcache is my new hammer, so looking for nails.
>
> On Mon, Apr 2, 2012 at 12:53 PM, Michel Pelletier
> <pelletier.michel at gmail.com> wrote:
>> The problem, I believe, is serializing and unserializing an entire
>> queue on every read and write, i don't think memcache will help with
>> that as it has no queues and he asked for "persistence" which
>> generally means writing to disk.  Redis has true queues and also does
>> some persistence, so maybe that will work.
>>
>> There are lots of kinds of databases that have queues and can handle
>> various combinations of read and write throughput.  Almost any one of
>> them will handle this case better than the OP's original code.
>>
>> From a higher level I think this is a perfect case of persistence
>> adding more complexity and failure points to a simple problem than is
>> really needed.  Why does the op need persistence?  What happens when
>> the disk fills up?  What if unserializing the list exhausts memory?
>> What if the file is corrupted?
>>
>> It's better to build "reliability", whatever that means, into the
>> endpoints than into the broker.
>>
>> -Michel
>>
>> On Mon, Apr 2, 2012 at 9:57 AM, Asher Benjamin <asherbenj at gmail.com> wrote:
>>> Could you use memcache instead of database?
>>>
>>>
>>>
>>> On Mon, Apr 2, 2012 at 8:28 AM, Michel Pelletier
>>> <pelletier.michel at gmail.com> wrote:
>>>> I suspect the big issue is with your push() function, it unpickles a
>>>> list, appends a message to it, and then repickles it.  As the list
>>>> gets bigger and bigger, this operation will get slower and slower
>>>> until it consumes almost all cpu time for your program.  You'll need a
>>>> much more clever persistence strategy than this to get good
>>>> performance.  You're pickling/unpickling/repickling the same messages
>>>> over and over again.
>>>>
>>>> I suggest starting with something simple, like writing the message to
>>>> a database.  Also you don't need to unpickle your message and then
>>>> repickle it to write it, just write the raw off-the-wire message, then
>>>> later when you fetch it from storage, unpickle it at that point.  It
>>>> seems to me your program doesn't need pickle at all, it could work
>>>> entirely with raw messages.
>>>>
>>>> -Michel
>>>>
>>>> On Mon, Apr 2, 2012 at 12:34 AM, Sean Ochoa <sean.m.ochoa at gmail.com> wrote:
>>>>> Hey all.
>>>>>
>>>>> In my attempt at trying to create a message queue with a persistence layer,
>>>>> I may have slowed things down.
>>>>>
>>>>> Here's my code so far:  http://paste.pound-python.org/show/18411/
>>>>>
>>>>> I'm wondering if someone could help me tune this thing so that I could put
>>>>> the persistence layer back in and still get good performance:  20,000
>>>>> messages / sec or more.
>>>>>
>>>>> Any help is much appreciated!  I'm still learning how to use zeromq.
>>>>>
>>>>> --
>>>>> Sean
>>>>>
>>>>> _______________________________________________
>>>>> zeromq-dev mailing list
>>>>> zeromq-dev at lists.zeromq.org
>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>
>>>> _______________________________________________
>>>> zeromq-dev mailing list
>>>> zeromq-dev at lists.zeromq.org
>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev



More information about the zeromq-dev mailing list