[zeromq-dev] interaction with threading.Events causing pyzmq to hang on socket instantiation

Chris Billington chrisjbillington at gmail.com
Wed Oct 3 08:01:52 CEST 2012


Ok I've realised my mistake. I put printlines all throughout pyzmq to see
where it was hanging, and it was hanging on an import (context.socket()imports
zmq.core.socket, though it's already been imported so it's cached in
sys.modules).

>From the python docs on threading<http://docs.python.org/library/threading.html>
:

"Firstly, other than in the main module, an import should not have the side
effect of spawning a new thread and then waiting for that thread in any
way. Failing to abide by this restriction can lead to a deadlock if the
spawned thread directly or indirectly attempts to import a module."

So I'm clearly not allowed to do this and will have to change my module
structure a bit (doing initialisation after import via a function call) or
live with the guilt of using one little time.sleep().

-Chris

On Wed, Oct 3, 2012 at 3:12 PM, Chris Billington <chrisjbillington at gmail.com
> wrote:

> Hi guys,
>
> I'm having a delightfully puzzling issue with pyzmq (pyzmq 2.2.0,  zmq
> 2.1.11, Ubuntu 12.04 64 bit, Python 2.7.3).
>
> This code works fine:
>
> import threading
> import time
> import zmq
>
> def f():
>     print 'thread started'
>     context = zmq.Context.instance()
>     print 'context obtained'
>     rep_sock = context.socket(zmq.REP)
>     print 'socket created'
>
> def connect():
>     event = threading.Event()
>     threading.Thread(target=f).start()
>     time.sleep(1000)
>     event.wait()
>
> connect()
>
> It gets up to the time.sleep(1000) and then waits for ages like you'd
> expect. However, if I put this code in a file b.py, and make a file a.pywhich has a single line import
> b, then running a.py causes a hang on the line rep_sock =
> context.socket(zmq.REP) in the thread.
>
> Furthermore if I remove the event.wait(), then a.py runs fine! This is
> the case even though the event.wait() clearly does not get executed.
>
> What I'm really using an event.wait() for is to wait for the thread to
> bind a socket to inproc:// transport  before the main thread proceeds to
> connect to said socket. This is necessary as inproc transport does not
> auto retry (https://zeromq.jira.com/browse/LIBZMQ-6), so
> connect-before-bind fails with an exception. Whilst there are obvious
> (though non ideal, retry with time.sleep, yuck) ways around this, it has
> made me curious. How on earth is a line that isn't even executed affecting
> pyzmq's ability to instantiate sockets? Why would it matter whether I'm
> importing this code or running it directly? As far as I can tell, imported
> code can only tell it's imported because __name__ is not '__main__', and
> the only other difference I can think of is that __builtins__ gives you a
> module if you're in __main__, whereas it gives you a dictionary in an
> imported module (I've never understood the logic behind that decision).
>
> I consider myself fairly knowledgeable regarding Pythons's internals, and
> I don't think I could write code that behaves like this if I wanted to,
> short of having it read its own source looking for the event.wait() line
> before deciding whether to hang or not. And somehow I don't think that's
> what pyzmq is doing.
>
> Anyway this is very odd, and I suspect there is some deep magic behind it
> regarding the implementation of threading.Event, and the fact that the
> function is bytecode compiled when imported instead of being interpreted
> as when run directly, at which point it can know about the event.wait()in advance, and pre-emptively modify some state somewhere in a way that
> breaks pyzmq.
>
> If anyone has some insight behind what's gong on, I'd love to hear it.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20121003/e7b357d5/attachment.htm>


More information about the zeromq-dev mailing list