[zeromq-dev] Async access to a single socket - send and recv without reducing throughput
Lindley French
lindleyf at gmail.com
Thu Jan 2 22:52:01 CET 2014
Here's my thoughts.....
It's a lot easier to deal with asynchronous sends and receives if we can
use a different thread for each. Even if you can't send on multiple threads
at once, and can't receive on multiple threads at once, you should be able
to use different threads for each simultaneously. This is, for instance,
the guarantee offered by Java SocketChannels.
So how can we arrange to allow this with optimal efficiency? Here's one
plan. Let me know if there's any problems with it.....
1) Have three ZMQ sockets: one the "normal" socket (probably TCP) called
commsock, and two inproc pair sockets. Call these sendsock and recvsock.
2) Have a private thread which polls sendsock and commsock. If a message is
available on sendsock, forward it to commsock. If a message is available on
commsock, forward it to recvsock.
3) One thread can now talk to the other endpoint of sendsock for sending,
while a second thread blocks on the other endpoint of recvsock for
receiving.
On Thu, Jan 2, 2014 at 4:27 PM, Lindley French <lindleyf at gmail.com> wrote:
> Did the inproc solution work well for you?
>
> I believe this is a very common problem (how do I send and receive
> asynchronously on the same zmq socket without touching it from multiple
> threads?), and there ought to be a best practice defined for it.
>
>
> On Tue, Dec 31, 2013 at 11:26 AM, Amir Taaki <zgenjix at yahoo.com> wrote:
>
>> thanks! I went with inproc sockets... hope performance is good. will
>> benchmark this.
>>
>>
>> https://github.com/spesmilo/obelisk/commit/c0882a9b74bcce3cca41e0cf6dab4cb93552ad39
>>
>>
>>
>>
>>
>> On Tuesday, December 31, 2013 12:44 PM, Bruno D. Rodrigues <
>> bruno.rodrigues at litux.org> wrote:
>> poll on an additional pair of zmq inproc:// sockets ;)
>>
>>
>> On Dec 31, 2013, at 12:41, Matt Connolly <matt.connolly at me.com> wrote:
>>
>> > Zmq poller can also wake on standard file descriptors (eg unix socket).
>> If your custom event can write to a unix socket or pipe you might be in
>> luck.
>> >
>> > Cheers
>> > Matt.
>> >
>> >> On 31 Dec 2013, at 4:35 pm, Amir Taaki <zgenjix at yahoo.com> wrote:
>> >>
>> >>
>> >>
>> >> I was reading the docs and saw:
>> >>
>> >> "How can I integrate ØMQ sockets with normal sockets? Or with a GUI
>> event loop?
>> >> You can use the zmq_poll() function to poll for events on both ØMQ and
>> normal sockets. The zmq_poll() function accepts a timeout so if you need to
>> poll and process GUI
>> >> events in the same application thread you can set a timeout and
>> >> periodically poll for GUI events. See also the reference
>> documentation."
>> >>
>> >>
>> >> How can I wake up zmq_poll() with a custom event? This seems to solve
>> my problems if possible. Then I can create a wakeup to process sends in the
>> polling loop.
>> >>
>> >>
>> >> On Tuesday, December 31, 2013 6:14 AM, Amir Taaki <zgenjix at yahoo.com>
>> wrote:
>> >>
>> >> Hi!
>> >>
>> >> I have a bit of a design issue. I want to achieve this:
>> >>
>> >> - Software receives a request on a DEALER socket from a ROUTER socket
>> (think the worker in paranoid pirate pattern).
>> >> - Request is executed asynchronously in the software. Result is ready
>> inside another thread.
>> >>
>> >> - Send it back over the same socket as the receive.
>> >>
>> >> In an ideal world, I'd be polling the socket to receive in one thread,
>> and then performing the sends from another. But we cannot use sockets from
>> multiple threads - sockets can only be used from one thread.
>> >>
>> >> Currently I have a multi-producer single-consumer lockless queue for
>> sends, and then there's an std::condition_variable that's waiting for a
>> signal to wake-up and batch process the send queue. Also there's a polling
>> loop for receives. All the options I can think of have downsides:
>> >>
>> >> * Use a separate receive and send socket. I'd need some mechanism so
>> that the broker (ppqueue) is aware of receive/send socket pairs. Maybe a
>> special message for the broker that isn't relayed, but indicates the
>> identity of the receive socket.
>> >>
>> >> * Polling then sending reduces the throughput of sends. I've
>> benchmarked performance and it's a significant hit. You're essentially
>> penalising sends by T microsecs. Sleeping for 0 is not a good idea since
>> CPU usage hits 100%.
>> >>
>> >> * Using the same socket but synchronising access from the send/recv
>> threads - ZMQ crashes infrequently when I do this because of triggered
>> asserts. It'd be great if I know how to do this safely (maybe by calling
>> some method on the socket).
>> >>
>> >> How do I achieve this scenario of 50% random receives and 50% random
>> sends? It's not like the classic scenario of a receive, followed by some
>> synchronous code path, and then a send (within the same thread). It's not
>> an option to wait for requests to finish as I dropped Thrift to use ZMQ
>> because of its async ability.
>> >>
>> >> Thanks!
>> >> _______________________________________________
>> >> zeromq-dev mailing list
>> >> zeromq-dev at lists.zeromq.org
>> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>> >> _______________________________________________
>> >> zeromq-dev mailing list
>> >> zeromq-dev at lists.zeromq.org
>> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> > _______________________________________________
>> > zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140102/5dc076ce/attachment.htm>
More information about the zeromq-dev
mailing list