[zeromq-dev] how to get it to run faster?

[Asher Benjamin]

Could you use memcache instead of database?



On Mon, Apr 2, 2012 at 8:28 AM, Michel Pelletier
<pelletier.michel at gmail.com> wrote:
> I suspect the big issue is with your push() function, it unpickles a
> list, appends a message to it, and then repickles it.  As the list
> gets bigger and bigger, this operation will get slower and slower
> until it consumes almost all cpu time for your program.  You'll need a
> much more clever persistence strategy than this to get good
> performance.  You're pickling/unpickling/repickling the same messages
> over and over again.
>
> I suggest starting with something simple, like writing the message to
> a database.  Also you don't need to unpickle your message and then
> repickle it to write it, just write the raw off-the-wire message, then
> later when you fetch it from storage, unpickle it at that point.  It
> seems to me your program doesn't need pickle at all, it could work
> entirely with raw messages.
>
> -Michel
>
> On Mon, Apr 2, 2012 at 12:34 AM, Sean Ochoa <sean.m.ochoa at gmail.com> wrote:
>> Hey all.
>>
>> In my attempt at trying to create a message queue with a persistence layer,
>> I may have slowed things down.
>>
>> Here's my code so far:  http://paste.pound-python.org/show/18411/
>>
>> I'm wondering if someone could help me tune this thing so that I could put
>> the persistence layer back in and still get good performance:  20,000
>> messages / sec or more.
>>
>> Any help is much appreciated!  I'm still learning how to use zeromq.
>>
>> --
>> Sean
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev