[zeromq-dev] how to get it to run faster?

[Asher Benjamin]

memcache is my new hammer, so looking for nails.

On Mon, Apr 2, 2012 at 12:53 PM, Michel Pelletier
<pelletier.michel at gmail.com> wrote:
> The problem, I believe, is serializing and unserializing an entire
> queue on every read and write, i don't think memcache will help with
> that as it has no queues and he asked for "persistence" which
> generally means writing to disk.  Redis has true queues and also does
> some persistence, so maybe that will work.
>
> There are lots of kinds of databases that have queues and can handle
> various combinations of read and write throughput.  Almost any one of
> them will handle this case better than the OP's original code.
>
> From a higher level I think this is a perfect case of persistence
> adding more complexity and failure points to a simple problem than is
> really needed.  Why does the op need persistence?  What happens when
> the disk fills up?  What if unserializing the list exhausts memory?
> What if the file is corrupted?
>
> It's better to build "reliability", whatever that means, into the
> endpoints than into the broker.
>
> -Michel
>
> On Mon, Apr 2, 2012 at 9:57 AM, Asher Benjamin <asherbenj at gmail.com> wrote:
>> Could you use memcache instead of database?
>>
>>
>>
>> On Mon, Apr 2, 2012 at 8:28 AM, Michel Pelletier
>> <pelletier.michel at gmail.com> wrote:
>>> I suspect the big issue is with your push() function, it unpickles a
>>> list, appends a message to it, and then repickles it.  As the list
>>> gets bigger and bigger, this operation will get slower and slower
>>> until it consumes almost all cpu time for your program.  You'll need a
>>> much more clever persistence strategy than this to get good
>>> performance.  You're pickling/unpickling/repickling the same messages
>>> over and over again.
>>>
>>> I suggest starting with something simple, like writing the message to
>>> a database.  Also you don't need to unpickle your message and then
>>> repickle it to write it, just write the raw off-the-wire message, then
>>> later when you fetch it from storage, unpickle it at that point.  It
>>> seems to me your program doesn't need pickle at all, it could work
>>> entirely with raw messages.
>>>
>>> -Michel
>>>
>>> On Mon, Apr 2, 2012 at 12:34 AM, Sean Ochoa <sean.m.ochoa at gmail.com> wrote:
>>>> Hey all.
>>>>
>>>> In my attempt at trying to create a message queue with a persistence layer,
>>>> I may have slowed things down.
>>>>
>>>> Here's my code so far:  http://paste.pound-python.org/show/18411/
>>>>
>>>> I'm wondering if someone could help me tune this thing so that I could put
>>>> the persistence layer back in and still get good performance:  20,000
>>>> messages / sec or more.
>>>>
>>>> Any help is much appreciated!  I'm still learning how to use zeromq.
>>>>
>>>> --
>>>> Sean
>>>>
>>>> _______________________________________________
>>>> zeromq-dev mailing list
>>>> zeromq-dev at lists.zeromq.org
>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev