[zeromq-dev] Thread safe Pub/Sub and Multicast

MinRK benjaminrk at gmail.com
Mon Mar 14 16:42:40 CET 2016


On Sat, Mar 12, 2016 at 9:00 PM, Dave Kuhlman <dkuhlman at davekuhlman.org>
wrote:

For those of us who use Python and the pyzmq library for ZeroMQ, I'm
> wondering whether the solution to the problem and need for thread
> safety is to avoid threads and to use either the tornado ioloop
> support or asyncio ioloop.  Am I right that doing so would give a
> Python developer the capabilities of multiple threads without the
> worries about thread safety.
>
> One reason I'm asking is that I'm the one who added the ioloop
> examples to zguide (in examples/Python/tornado_ioloop and
> examples/Python/ascyncio_ioloop).  A number of those examples use
> (what I understand as) multiple concurrent tasks within a single
> thread.  So, I'm wondering whether that really is thread safe (or
> whatever the equivalent is for tasks), as I believe it is.
>
Yup, asyncio/tornado are single-threaded eventloops, so you won’t get
threadsafety problems if everything happens in a callback/coroutine on
those loops. You can also use these to make threadsafe actions, even when
using actual threads. In the tornado case, there is one (and only one)
threadsafe method, and it’s IOLoop.add_callback. You can use this to
schedule calls to be made on the IOLoop, from other threads. For instance:

loop = start_loop_in_background_thread()
def threadsafe_send(socket, msg):
    loop.add_callback(lambda : socket.send_multipart(msg))

In this way, you can *schedule* sends from any number of threads perfectly
safely, because the actual zmq socket is only ever operated on in the IO
thread. Threadsafe recv is similarly doable, but a bit more code.

On the other hand, you can also write coroutine-unsafe code if you have
yields in the middle of sending multipart messages, for instance:

@coroutinedef bad_coroutine_send(socket, msg):
    for frame in msg[:-1]:
        yield socket.send(frame, zmq.SNDMORE)
    yield socket.send(msg[-1])

Where each yield is an opportunity to take over and do some other operation
on the socket. I think it is a lot easier to write coroutine-safe code than
thread-safe code, since all context switching is explicit (make sure you
have no yield calls in the middle of an operation).

-MinRK


> Any enlightenment will be welcome.
>
> And, by the way, I suspect that something equivalent could be done
> in Node.js (JavaScript), which also uses a single threaded approach
> to "concurrency" (psuedo-concurrency, quasi-concurrency, whatever).
> In Node.js, you could use callbacks, but there are libraries that
> can wrap your code to make is seem more simple and parallel.  For
> example, see async.js.  Here are a few quick examples from the
> async.js Web page (https://www.npmjs.com/package/async):
>
>      async.map([`file1`,`file2`,`file3`], fs.stat, function(err, results){
>          // results is now an array of stats for each file
>      });
>
>      async.filter([`file1`,`file2`,`file3`], fs.exists, function(results){
>          // results now equals an array of the existing files
>      });
>
>      async.parallel([
>          function(){ ... },
>          function(){ ... }
>      ], callback);
>
>      async.series([
>          function(){ ... },
>          function(){ ... }
>      ]);
>
> As a bit of justification for this question, the main intension of
> the Python ioloop support in tornado and asyncio is to address
> problems of network and I/O delays and to enable our code to do
> something useful during those delays.  That seems like a central
> concern for users of ZeroMQ.
>
> Dave
>
>
> On Sat, Mar 12, 2016 at 03:00:58AM +0200, Doron Somech wrote:
> >    Hi All,
> >    During the ZeroMQ hackathon I have added the Radio/Dish pattern
> (thread
> >    safe version of pubsub) and multicast support (only for Radio/Dish
> >    sockets) to both libzmq and czmq.
> >    Radio/Dish is very similar to pub/sub with some differences:
> >      * Thread safe, you can send and receive messages from multiple
> >        threads, so for publisher you now don't need internal device to
> >        publish from multiple threads and with subscriber you can use it
> as
> >        load balancer.
> >      * Only single frame can be sent
> >      * Radio/Dish matching is exact matching (vs prefix of pubsub)
> >      * Group (Radio/Dish topic) is string and currently limited to 15
> >        chars (might be increased in the future)
> >      * Group is set as part of the zmq_msg (libzmq) or zframe (czmq).
> >      * You call join or leave on the socket (with methods instead of
> >        socket options)
> >
> >    To learn how to use it with czmq take a loot at the following gist:
> >    [1]https://gist.github.com/somdoron/9423196a228775c8f5af
> >    For libzmq take a loot at the test:
> >    [2]
> https://github.com/zeromq/libzmq/blob/master/tests/test_radio_dish.c
> >    pp#L90
> >    And the most exciting news, with radio-dish you can use multicast
> >    transport, just replace the address from following gist with udp
> >    transport and multicast address, following is a valid address for
> >    example:
> >    udp://[3]239.0.0.1:55555
> >    Regards
> >    Doron
> >
> > References
> >
> >    1. https://gist.github.com/somdoron/9423196a228775c8f5af
> >    2.
> https://github.com/zeromq/libzmq/blob/master/tests/test_radio_dish.cpp#L90
> >    3. http://239.0.0.1:55555/
>
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
> --
>
> Dave Kuhlman
> http://www.davekuhlman.org
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
​
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20160314/ea388a97/attachment.htm>


More information about the zeromq-dev mailing list