[zeromq-dev] how to get it to run faster?

Michel Pelletier pelletier.michel at gmail.com
Mon Apr 2 19:53:45 CEST 2012


The problem, I believe, is serializing and unserializing an entire
queue on every read and write, i don't think memcache will help with
that as it has no queues and he asked for "persistence" which
generally means writing to disk.  Redis has true queues and also does
some persistence, so maybe that will work.

There are lots of kinds of databases that have queues and can handle
various combinations of read and write throughput.  Almost any one of
them will handle this case better than the OP's original code.

>From a higher level I think this is a perfect case of persistence
adding more complexity and failure points to a simple problem than is
really needed.  Why does the op need persistence?  What happens when
the disk fills up?  What if unserializing the list exhausts memory?
What if the file is corrupted?

It's better to build "reliability", whatever that means, into the
endpoints than into the broker.

-Michel

On Mon, Apr 2, 2012 at 9:57 AM, Asher Benjamin <asherbenj at gmail.com> wrote:
> Could you use memcache instead of database?
>
>
>
> On Mon, Apr 2, 2012 at 8:28 AM, Michel Pelletier
> <pelletier.michel at gmail.com> wrote:
>> I suspect the big issue is with your push() function, it unpickles a
>> list, appends a message to it, and then repickles it.  As the list
>> gets bigger and bigger, this operation will get slower and slower
>> until it consumes almost all cpu time for your program.  You'll need a
>> much more clever persistence strategy than this to get good
>> performance.  You're pickling/unpickling/repickling the same messages
>> over and over again.
>>
>> I suggest starting with something simple, like writing the message to
>> a database.  Also you don't need to unpickle your message and then
>> repickle it to write it, just write the raw off-the-wire message, then
>> later when you fetch it from storage, unpickle it at that point.  It
>> seems to me your program doesn't need pickle at all, it could work
>> entirely with raw messages.
>>
>> -Michel
>>
>> On Mon, Apr 2, 2012 at 12:34 AM, Sean Ochoa <sean.m.ochoa at gmail.com> wrote:
>>> Hey all.
>>>
>>> In my attempt at trying to create a message queue with a persistence layer,
>>> I may have slowed things down.
>>>
>>> Here's my code so far:  http://paste.pound-python.org/show/18411/
>>>
>>> I'm wondering if someone could help me tune this thing so that I could put
>>> the persistence layer back in and still get good performance:  20,000
>>> messages / sec or more.
>>>
>>> Any help is much appreciated!  I'm still learning how to use zeromq.
>>>
>>> --
>>> Sean
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev



More information about the zeromq-dev mailing list