[zeromq-dev] Thread safe Pub/Sub and Multicast
Pieter Hintjens
ph at imatix.com
Fri Mar 18 10:27:21 CET 2016
That's *wonderful*, thanks.
On Wed, Mar 16, 2016 at 11:09 AM, MinRK <benjaminrk at gmail.com> wrote:
>
>
> On Wed, Mar 16, 2016 at 11:06 AM, Pieter Hintjens <ph at imatix.com> wrote:
>>
>> Off-topic, but I need to ask, how do you do the syntax highlighting in
>> emails?
>
>
> http://markdown-here.com/
>
>>
>>
>> On Mon, Mar 14, 2016 at 4:42 PM, MinRK <benjaminrk at gmail.com> wrote:
>> > On Sat, Mar 12, 2016 at 9:00 PM, Dave Kuhlman <dkuhlman at davekuhlman.org>
>> > wrote:
>> >>
>> >> For those of us who use Python and the pyzmq library for ZeroMQ, I'm
>> >> wondering whether the solution to the problem and need for thread
>> >> safety is to avoid threads and to use either the tornado ioloop
>> >> support or asyncio ioloop. Am I right that doing so would give a
>> >> Python developer the capabilities of multiple threads without the
>> >> worries about thread safety.
>> >>
>> >> One reason I'm asking is that I'm the one who added the ioloop
>> >> examples to zguide (in examples/Python/tornado_ioloop and
>> >> examples/Python/ascyncio_ioloop). A number of those examples use
>> >> (what I understand as) multiple concurrent tasks within a single
>> >> thread. So, I'm wondering whether that really is thread safe (or
>> >> whatever the equivalent is for tasks), as I believe it is.
>> >
>> > Yup, asyncio/tornado are single-threaded eventloops, so you won’t get
>> > threadsafety problems if everything happens in a callback/coroutine on
>> > those
>> > loops. You can also use these to make threadsafe actions, even when
>> > using
>> > actual threads. In the tornado case, there is one (and only one)
>> > threadsafe
>> > method, and it’s IOLoop.add_callback. You can use this to schedule calls
>> > to
>> > be made on the IOLoop, from other threads. For instance:
>> >
>> > loop = start_loop_in_background_thread()
>> >
>> > def threadsafe_send(socket, msg):
>> > loop.add_callback(lambda : socket.send_multipart(msg))
>> >
>> > In this way, you can schedule sends from any number of threads perfectly
>> > safely, because the actual zmq socket is only ever operated on in the IO
>> > thread. Threadsafe recv is similarly doable, but a bit more code.
>> >
>> > On the other hand, you can also write coroutine-unsafe code if you have
>> > yields in the middle of sending multipart messages, for instance:
>> >
>> > @coroutine
>> > def bad_coroutine_send(socket, msg):
>> > for frame in msg[:-1]:
>> > yield socket.send(frame, zmq.SNDMORE)
>> > yield socket.send(msg[-1])
>> >
>> > Where each yield is an opportunity to take over and do some other
>> > operation
>> > on the socket. I think it is a lot easier to write coroutine-safe code
>> > than
>> > thread-safe code, since all context switching is explicit (make sure you
>> > have no yield calls in the middle of an operation).
>> >
>> > -MinRK
>> >>
>> >>
>> >> Any enlightenment will be welcome.
>> >>
>> >> And, by the way, I suspect that something equivalent could be done
>> >> in Node.js (JavaScript), which also uses a single threaded approach
>> >> to "concurrency" (psuedo-concurrency, quasi-concurrency, whatever).
>> >> In Node.js, you could use callbacks, but there are libraries that
>> >> can wrap your code to make is seem more simple and parallel. For
>> >> example, see async.js. Here are a few quick examples from the
>> >> async.js Web page (https://www.npmjs.com/package/async):
>> >>
>> >> async.map([`file1`,`file2`,`file3`], fs.stat, function(err,
>> >> results){
>> >> // results is now an array of stats for each file
>> >> });
>> >>
>> >> async.filter([`file1`,`file2`,`file3`], fs.exists,
>> >> function(results){
>> >> // results now equals an array of the existing files
>> >> });
>> >>
>> >> async.parallel([
>> >> function(){ ... },
>> >> function(){ ... }
>> >> ], callback);
>> >>
>> >> async.series([
>> >> function(){ ... },
>> >> function(){ ... }
>> >> ]);
>> >>
>> >> As a bit of justification for this question, the main intension of
>> >> the Python ioloop support in tornado and asyncio is to address
>> >> problems of network and I/O delays and to enable our code to do
>> >> something useful during those delays. That seems like a central
>> >> concern for users of ZeroMQ.
>> >>
>> >> Dave
>> >>
>> >>
>> >> On Sat, Mar 12, 2016 at 03:00:58AM +0200, Doron Somech wrote:
>> >> > Hi All,
>> >> > During the ZeroMQ hackathon I have added the Radio/Dish pattern
>> >> > (thread
>> >> > safe version of pubsub) and multicast support (only for Radio/Dish
>> >> > sockets) to both libzmq and czmq.
>> >> > Radio/Dish is very similar to pub/sub with some differences:
>> >> > * Thread safe, you can send and receive messages from multiple
>> >> > threads, so for publisher you now don't need internal device
>> >> > to
>> >> > publish from multiple threads and with subscriber you can use
>> >> > it
>> >> > as
>> >> > load balancer.
>> >> > * Only single frame can be sent
>> >> > * Radio/Dish matching is exact matching (vs prefix of pubsub)
>> >> > * Group (Radio/Dish topic) is string and currently limited to 15
>> >> > chars (might be increased in the future)
>> >> > * Group is set as part of the zmq_msg (libzmq) or zframe (czmq).
>> >> > * You call join or leave on the socket (with methods instead of
>> >> > socket options)
>> >> >
>> >> > To learn how to use it with czmq take a loot at the following
>> >> > gist:
>> >> > [1]https://gist.github.com/somdoron/9423196a228775c8f5af
>> >> > For libzmq take a loot at the test:
>> >> >
>> >> >
>> >> > [2]https://github.com/zeromq/libzmq/blob/master/tests/test_radio_dish.c
>> >> > pp#L90
>> >> > And the most exciting news, with radio-dish you can use multicast
>> >> > transport, just replace the address from following gist with udp
>> >> > transport and multicast address, following is a valid address for
>> >> > example:
>> >> > udp://[3]239.0.0.1:55555
>> >> > Regards
>> >> > Doron
>> >> >
>> >> > References
>> >> >
>> >> > 1. https://gist.github.com/somdoron/9423196a228775c8f5af
>> >> > 2.
>> >> >
>> >> > https://github.com/zeromq/libzmq/blob/master/tests/test_radio_dish.cpp#L90
>> >> > 3. http://239.0.0.1:55555/
>> >>
>> >> > _______________________________________________
>> >> > zeromq-dev mailing list
>> >> > zeromq-dev at lists.zeromq.org
>> >> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >>
>> >>
>> >> --
>> >>
>> >> Dave Kuhlman
>> >> http://www.davekuhlman.org
>> >> _______________________________________________
>> >> zeromq-dev mailing list
>> >> zeromq-dev at lists.zeromq.org
>> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >
>> >
>> > _______________________________________________
>> > zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
More information about the zeromq-dev
mailing list