[zeromq-dev] 0MQ 1.0.1 Python Global Interpreter Lock patch?

Ben Dyer ben.dyer at taguchimail.com
Sat Oct 17 12:48:38 CEST 2009


Aamir,

I can certainly see the advantages of a non-blocking solution to some  
classes of problem: for instance, those in which the application  
spends most of its time waiting for the network, such as in high- 
concurrency web servers. In those cases, compared with servers using  
one thread/process per client, non-blocking servers (such as nginx)  
provide a much greater ability to service large numbers of clients,  
and fairly consistent resource utilisation.

However, since my application only uses 0MQ for interprocess/LAN  
messaging with small numbers of stable clients, I don't see any  
benefit in switching from threaded to non-blocking -- particularly  
since 0MQ is so fast relative to everything else the application does  
(database access and HTTP proxied behind nginx), and the workload is  
less than 2000 messages per second.

Regards,
Ben

On 17/10/2009, at 08:16 , Aamir M wrote:

> Hi Ben,
>
> In regards to the code review that you mention, did you have a look at
> txAMQP by any chance? txAMQP is a new Python client for AMQP based  on
> the Twisted framework. There is an interesting discussion about it at
> http://www.apparatusproject.org/blog/2009/04/creating-a-simple-amqp-client-using-txamqp/
> ...
>
> Specifically the author claims that, "Being event-driven is important
> when implementing AMQP in your project, because it creates a
> concurrency model using non blocking calls." So he is against using
> blocking calls and instead prefers to use callbacks. And apparently
> the Twisted "concurrency model" consists of polling for messages and
> then going sleep for some amount of time, then polling again .... That
> sounds quite inferior to the concurrency model that we have with 0MQ
> (i.e. making 0MQ blocking calls, wrapped with Py_BEGIN_ALLOW_THREADS /
> Py_END_ALLOW_THREADS).
>
> Aamir
>
>
> On Thu, Oct 15, 2009 at 11:54 PM, Ben Dyer  
> <ben.dyer at taguchimail.com> wrote:
>> Aamir,
>>
>> I'm using a similar modification to the 0MQ/2.0 Alpha3 libpyzmq to  
>> support
>> multithreaded operation, and the patch been working as expected for  
>> me.
>> Although I don't have a complete understanding of the 0MQ threading  
>> model,
>> my review of the source didn't turn up any means by which disabling  
>> the GIL
>> could cause issues that wouldn't appear in multithreaded C/C++ apps  
>> (or,
>> conversely, issues that wouldn't appear in multithreaded Python  
>> apps using
>> network code other than 0MQ).
>>
>> Regards,
>> Ben
>>
>> On 16/10/2009, at 03:21 , Aamir M wrote:
>>
>>> Hello,
>>>
>>> I am trying to use the 0MQ 1.0.1 Python API to send / recv 0MQ
>>> messages in one Python thread while driving a GUI in another Python
>>> thread. However, whenever I use the blocking send / recv 0MQ calls,
>>> the whole Python process freezes and waits for network I/O (so the  
>>> GUI
>>> becomes non-responsive, etc). It seems that pyzmq.cpp
>>> (http://github.com/sustrik/zeromq1/blob/master/libpyzmq/pyzmq.cpp)  
>>> is
>>> not thread friendly. I was able to get around the problem by  
>>> modifying
>>> the pyZMQ_receive and pyZMQ_send functions .... specifically, line  
>>> 176
>>> in pyzmq.cpp reads:
>>>
>>> sent = self->api_thread->send (exchange, message, block);
>>>
>>> I replaced this with:
>>>
>>> Py_BEGIN_ALLOW_THREADS
>>> sent = self->api_thread->send (exchange, message, block);
>>> Py_END_ALLOW_THREADS
>>>
>>> Similarly, I changed line 192 to:
>>>
>>> int queue;
>>> Py_BEGIN_ALLOW_THREADS
>>> queue = self->api_thread->receive (&message, block);
>>> Py_END_ALLOW_THREADS
>>>
>>>
>>> In summary, I just simply the 0MQ C++ blocking calls with the Python
>>> macros Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS, which are
>>> described in http://docs.python.org/c-api/init.html ... These macros
>>> release and reacquire the CPython Global Interpreter Lock.
>>>
>>>
>>> Is this a bug in the 0MQ 1.0.01 library that should be patched? Or  
>>> is
>>> my patch somehow incorrect?
>>>
>>> Thanks,
>>> Aamir
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>




More information about the zeromq-dev mailing list