[zeromq-dev] Thread safe Pub/Sub and Multicast

MinRK benjaminrk at gmail.com
Wed Mar 16 11:09:53 CET 2016


On Wed, Mar 16, 2016 at 11:06 AM, Pieter Hintjens <ph at imatix.com> wrote:

> Off-topic, but I need to ask, how do you do the syntax highlighting in
> emails?
>

http://markdown-here.com/


>
> On Mon, Mar 14, 2016 at 4:42 PM, MinRK <benjaminrk at gmail.com> wrote:
> > On Sat, Mar 12, 2016 at 9:00 PM, Dave Kuhlman <dkuhlman at davekuhlman.org>
> > wrote:
> >>
> >> For those of us who use Python and the pyzmq library for ZeroMQ, I'm
> >> wondering whether the solution to the problem and need for thread
> >> safety is to avoid threads and to use either the tornado ioloop
> >> support or asyncio ioloop.  Am I right that doing so would give a
> >> Python developer the capabilities of multiple threads without the
> >> worries about thread safety.
> >>
> >> One reason I'm asking is that I'm the one who added the ioloop
> >> examples to zguide (in examples/Python/tornado_ioloop and
> >> examples/Python/ascyncio_ioloop).  A number of those examples use
> >> (what I understand as) multiple concurrent tasks within a single
> >> thread.  So, I'm wondering whether that really is thread safe (or
> >> whatever the equivalent is for tasks), as I believe it is.
> >
> > Yup, asyncio/tornado are single-threaded eventloops, so you won’t get
> > threadsafety problems if everything happens in a callback/coroutine on
> those
> > loops. You can also use these to make threadsafe actions, even when using
> > actual threads. In the tornado case, there is one (and only one)
> threadsafe
> > method, and it’s IOLoop.add_callback. You can use this to schedule calls
> to
> > be made on the IOLoop, from other threads. For instance:
> >
> > loop = start_loop_in_background_thread()
> >
> > def threadsafe_send(socket, msg):
> >     loop.add_callback(lambda : socket.send_multipart(msg))
> >
> > In this way, you can schedule sends from any number of threads perfectly
> > safely, because the actual zmq socket is only ever operated on in the IO
> > thread. Threadsafe recv is similarly doable, but a bit more code.
> >
> > On the other hand, you can also write coroutine-unsafe code if you have
> > yields in the middle of sending multipart messages, for instance:
> >
> > @coroutine
> > def bad_coroutine_send(socket, msg):
> >     for frame in msg[:-1]:
> >         yield socket.send(frame, zmq.SNDMORE)
> >     yield socket.send(msg[-1])
> >
> > Where each yield is an opportunity to take over and do some other
> operation
> > on the socket. I think it is a lot easier to write coroutine-safe code
> than
> > thread-safe code, since all context switching is explicit (make sure you
> > have no yield calls in the middle of an operation).
> >
> > -MinRK
> >>
> >>
> >> Any enlightenment will be welcome.
> >>
> >> And, by the way, I suspect that something equivalent could be done
> >> in Node.js (JavaScript), which also uses a single threaded approach
> >> to "concurrency" (psuedo-concurrency, quasi-concurrency, whatever).
> >> In Node.js, you could use callbacks, but there are libraries that
> >> can wrap your code to make is seem more simple and parallel.  For
> >> example, see async.js.  Here are a few quick examples from the
> >> async.js Web page (https://www.npmjs.com/package/async):
> >>
> >>      async.map([`file1`,`file2`,`file3`], fs.stat, function(err,
> results){
> >>          // results is now an array of stats for each file
> >>      });
> >>
> >>      async.filter([`file1`,`file2`,`file3`], fs.exists,
> function(results){
> >>          // results now equals an array of the existing files
> >>      });
> >>
> >>      async.parallel([
> >>          function(){ ... },
> >>          function(){ ... }
> >>      ], callback);
> >>
> >>      async.series([
> >>          function(){ ... },
> >>          function(){ ... }
> >>      ]);
> >>
> >> As a bit of justification for this question, the main intension of
> >> the Python ioloop support in tornado and asyncio is to address
> >> problems of network and I/O delays and to enable our code to do
> >> something useful during those delays.  That seems like a central
> >> concern for users of ZeroMQ.
> >>
> >> Dave
> >>
> >>
> >> On Sat, Mar 12, 2016 at 03:00:58AM +0200, Doron Somech wrote:
> >> >    Hi All,
> >> >    During the ZeroMQ hackathon I have added the Radio/Dish pattern
> >> > (thread
> >> >    safe version of pubsub) and multicast support (only for Radio/Dish
> >> >    sockets) to both libzmq and czmq.
> >> >    Radio/Dish is very similar to pub/sub with some differences:
> >> >      * Thread safe, you can send and receive messages from multiple
> >> >        threads, so for publisher you now don't need internal device to
> >> >        publish from multiple threads and with subscriber you can use
> it
> >> > as
> >> >        load balancer.
> >> >      * Only single frame can be sent
> >> >      * Radio/Dish matching is exact matching (vs prefix of pubsub)
> >> >      * Group (Radio/Dish topic) is string and currently limited to 15
> >> >        chars (might be increased in the future)
> >> >      * Group is set as part of the zmq_msg (libzmq) or zframe (czmq).
> >> >      * You call join or leave on the socket (with methods instead of
> >> >        socket options)
> >> >
> >> >    To learn how to use it with czmq take a loot at the following gist:
> >> >    [1]https://gist.github.com/somdoron/9423196a228775c8f5af
> >> >    For libzmq take a loot at the test:
> >> >
> >> > [2]
> https://github.com/zeromq/libzmq/blob/master/tests/test_radio_dish.c
> >> >    pp#L90
> >> >    And the most exciting news, with radio-dish you can use multicast
> >> >    transport, just replace the address from following gist with udp
> >> >    transport and multicast address, following is a valid address for
> >> >    example:
> >> >    udp://[3]239.0.0.1:55555
> >> >    Regards
> >> >    Doron
> >> >
> >> > References
> >> >
> >> >    1. https://gist.github.com/somdoron/9423196a228775c8f5af
> >> >    2.
> >> >
> https://github.com/zeromq/libzmq/blob/master/tests/test_radio_dish.cpp#L90
> >> >    3. http://239.0.0.1:55555/
> >>
> >> > _______________________________________________
> >> > zeromq-dev mailing list
> >> > zeromq-dev at lists.zeromq.org
> >> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >>
> >>
> >> --
> >>
> >> Dave Kuhlman
> >> http://www.davekuhlman.org
> >> _______________________________________________
> >> zeromq-dev mailing list
> >> zeromq-dev at lists.zeromq.org
> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
> >
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20160316/ccf6ef5f/attachment.htm>


More information about the zeromq-dev mailing list