[zeromq-dev] Multi-threaded ZeroMQ applications

Martin Sustrik sustrik at 250bpm.com
Thu Apr 1 08:16:35 CEST 2010


> After discussing the design-problems with a colleague, we came up with this:
> 1 thread to read and write to the ZeroMQ socket ("zmq-thread")
> N threads to do real work ("work-thread")
> They communicate via inproc-transport.
> The zmq-thread creates an inproc endpoint (bind) and all work-threads
> create their own inproc endpoints as well (connecting to the
> zmq-thread's inproc). Then in the zmq-thread I use ZMQ_POLL to
> zmq_poll() the ZeroMQ-external-socket + Inproc-socket and send/recv.
> This way I can send work to the worker-threads and receive responses
> back, but....

Yes. Correct. We call this scenario "multi-hop req/rep".

> ...there is one important remaining problem. I want each worker to
> connect to several servers; and send the replies back to the same
> server. Normally I use XREQ - REP, but now it will break down because
> it becomes N(servers) : 1(zmq-thread) : M(work-threads).
> Is there any way to make a request stick to the same "session"  (so
> the reply goes back to the correct server, and not some other server
> connected via the same socket) without using REQ-REP sync style? So I
> want "async" req-rep.
> For example: if I could somehow get access to the session where the
> request came from, I could pass this session-id to the worker-thread,
> the worker-thread does some work and then returns the result +
> session-id and then I could either manually lookup the correct session
> to use, or even let ZeroMQ library handle it.
> Another solution would be to manually keep a list of sockets inside
> the zmq-thread, one socket per server. I would then do the above work
> manually, so I know on which socket I need to send the reply. Any
> thoughts on this?

You'll have to do a bit of hacking to get this working. However, the 
functionality is under development (priority no.1) and hopefully it 
should be available shortly.

1. Don't use the trunk version. This part of functionality is being 
worked on and it's broken in trunk. Use 2.0.6 instead.

2. To implement your "message dispatcher thread" have a look at 
devices/zmq_queue/zmq_queue.cpp - it does exactly what you need.

3. On the REP side (worker threads) you'll have to do some manual 
tweaking of the messages. Each request you'll get is prefixed by 
identity of the endpoint that have originally generated it. It's in form 
of <1-byte-size><identity>. You have to get it from the message and use 
it as a prefix for the reply you are going to send.

John Dyte has this scenario working so ha may have more advice.


More information about the zeromq-dev mailing list