[zeromq-dev] Advice on integrating ZMQ with existing semi-multi-threaded application

Iverson, John Robert jrivers at sandia.gov
Mon Oct 15 21:19:11 CEST 2012


Hi all, first of all, sorry for the long message.  I'm investigating 
the feasibility of integrating ZMQ as the transport layer within
an existing application, which I say is semi-multi-threaded because
we guard against the application code running concurrently in two
threads using a global posix mutex.

My current feeling is that we'd mostly use PUB/SUB sockets, with
interfaces that need 1:1 or n:n connections.  There will be a
relatively small number of SUBs in each process - say less than 20.
Traditionally, we've handled network connections by having a
dedicated receiver thread that polls (outside our global mutex) for
incoming messages on a number of connections.  When a message
is found, the global mutex is grabbed and the application is notified.
Another likely complication - application layer code can register an
interface (== create a socket) from any thread, though it's only done
when the global mutex is held.

The guide says this:

    You MUST NOT share ØMQ sockets between threads.  ØMQ sockets
    are not threadsafe. Technically it's possible to do this, but it
    demands semaphores, locks, or mutexes. This will make your
    application slow and fragile. The only place where it's remotely
    sane to share sockets between threads are in language bindings
    that need to do magic like garbage collection on sockets.

I'm looking for some design advice here.  The way I see it, there
are tons of options available to me, though I'd be more than happy
to hear about others from the experts here!  Here are the ones that
popped into my crazy brain:

Option 1 - Create, poll, and recv all receiver sockets on one thread.
Grab the global mutex after recv.  Has a little complexity - application
code can register an interface from any thread, so I need to pass that
request over to the ZMQ thread to create the sockets.

Option 2 - Create socket on any thread; poll and recv on one ZMQ
thread.  Also has some complexity - since the socket is created on
one thread and polled on another, I probably need to either
protect the polling with the global mutex (bad) or perhaps protect
the creation with a separate mutex...

Option 3 - Each client connection is created/polled/recv'd on its
own thread.  Global mutex grabbed after recv.  Similarly to option
1, registering interfaces needs to cause this thread to be created.

Option 4 - Create, poll, recv all receiver sockets on one thread.
Incoming messages are passed via inproc to second ZMQ thread,
which grabs the global mutex and hands message to clients.  Same
complexity as #1, but is this more along the lines of the ZMQ
philosophy?

Hopefully I haven't overwhelmed everyone with poorly-explained
thoughts here.  If you can think of another option, I'd love to hear
it, and if you can think of problems with the other options, I'd love
to hear those, too.

Thanks!

Rob





More information about the zeromq-dev mailing list