[zeromq-dev] how to get it to run faster?

Michel Pelletier pelletier.michel at gmail.com
Mon Apr 2 15:28:00 CEST 2012


I suspect the big issue is with your push() function, it unpickles a
list, appends a message to it, and then repickles it.  As the list
gets bigger and bigger, this operation will get slower and slower
until it consumes almost all cpu time for your program.  You'll need a
much more clever persistence strategy than this to get good
performance.  You're pickling/unpickling/repickling the same messages
over and over again.

I suggest starting with something simple, like writing the message to
a database.  Also you don't need to unpickle your message and then
repickle it to write it, just write the raw off-the-wire message, then
later when you fetch it from storage, unpickle it at that point.  It
seems to me your program doesn't need pickle at all, it could work
entirely with raw messages.

-Michel

On Mon, Apr 2, 2012 at 12:34 AM, Sean Ochoa <sean.m.ochoa at gmail.com> wrote:
> Hey all.
>
> In my attempt at trying to create a message queue with a persistence layer,
> I may have slowed things down.
>
> Here's my code so far:  http://paste.pound-python.org/show/18411/
>
> I'm wondering if someone could help me tune this thing so that I could put
> the persistence layer back in and still get good performance:  20,000
> messages / sec or more.
>
> Any help is much appreciated!  I'm still learning how to use zeromq.
>
> --
> Sean
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>



More information about the zeromq-dev mailing list