[zeromq-dev] interaction with threading.Events causing pyzmq to hang on socket instantiation

[Chris Billington]

Hi guys,

I'm having a delightfully puzzling issue with pyzmq (pyzmq 2.2.0,  zmq
2.1.11, Ubuntu 12.04 64 bit, Python 2.7.3).

This code works fine:

import threading
import time
import zmq

def f():
    print 'thread started'
    context = zmq.Context.instance()
    print 'context obtained'
    rep_sock = context.socket(zmq.REP)
    print 'socket created'

def connect():
    event = threading.Event()
    threading.Thread(target=f).start()
    time.sleep(1000)
    event.wait()

connect()

It gets up to the time.sleep(1000) and then waits for ages like you'd
expect. However, if I put this code in a file b.py, and make a file
a.pywhich has a single line import
b, then running a.py causes a hang on the line rep_sock =
context.socket(zmq.REP) in the thread.

Furthermore if I remove the event.wait(), then a.py runs fine! This is the
case even though the event.wait() clearly does not get executed.

What I'm really using an event.wait() for is to wait for the thread to bind
a socket to inproc:// transport  before the main thread proceeds to connect
to said socket. This is necessary as inproc transport does not auto retry (
https://zeromq.jira.com/browse/LIBZMQ-6), so connect-before-bind fails with
an exception. Whilst there are obvious (though non ideal, retry with
time.sleep, yuck) ways around this, it has made me curious. How on earth is
a line that isn't even executed affecting pyzmq's ability to instantiate
sockets? Why would it matter whether I'm importing this code or running it
directly? As far as I can tell, imported code can only tell it's imported
because __name__ is not '__main__', and the only other difference I can
think of is that __builtins__ gives you a module if you're in __main__,
whereas it gives you a dictionary in an imported module (I've never
understood the logic behind that decision).

I consider myself fairly knowledgeable regarding Pythons's internals, and I
don't think I could write code that behaves like this if I wanted to, short
of having it read its own source looking for the event.wait() line before
deciding whether to hang or not. And somehow I don't think that's what
pyzmq is doing.

Anyway this is very odd, and I suspect there is some deep magic behind it
regarding the implementation of threading.Event, and the fact that the
function is bytecode compiled when imported instead of being interpreted as
when run directly, at which point it can know about the event.wait() in
advance, and pre-emptively modify some state somewhere in a way that breaks
pyzmq.

If anyone has some insight behind what's gong on, I'd love to hear it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20121003/c1138fa1/attachment.htm>