[zeromq-dev] Async access to a single socket - send and recv without reducing throughput

Thu Jan 2 23:05:29 CET 2014

Actually, if the inproc sockets are push/pull rather than pair, then you
can now send and receive from arbitrary threads so long as you store the
"public" endpoints of sendsock and recvsock in thread-local storage and are
willing to construct new sockets whenever you try to send or receive from a
new thread. (It's possible this is a bad idea. I don't really know.)

On Thu, Jan 2, 2014 at 4:52 PM, Lindley French <lindleyf at gmail.com> wrote:

> Here's my thoughts.....
>
> It's a lot easier to deal with asynchronous sends and receives if we can
> use a different thread for each. Even if you can't send on multiple threads
> at once, and can't receive on multiple threads at once, you should be able
> to use different threads for each simultaneously. This is, for instance,
> the guarantee offered by Java SocketChannels.
>
> So how can we arrange to allow this with optimal efficiency? Here's one
> plan. Let me know if there's any problems with it.....
> 1) Have three ZMQ sockets: one the "normal" socket (probably TCP) called
> commsock, and two inproc pair sockets. Call these sendsock and recvsock.
> 2) Have a private thread which polls sendsock and commsock. If a message
> is available on sendsock, forward it to commsock. If a message is available
> on commsock, forward it to recvsock.
> 3) One thread can now talk to the other endpoint of sendsock for sending,
> while a second thread blocks on the other endpoint of recvsock for
> receiving.
>
>
> On Thu, Jan 2, 2014 at 4:27 PM, Lindley French <lindleyf at gmail.com> wrote:
>
>> Did the inproc solution work well for you?
>>
>> I believe this is a very common problem (how do I send and receive
>> asynchronously on the same zmq socket without touching it from multiple
>> threads?), and there ought to be a best practice defined for it.
>>
>>
>> On Tue, Dec 31, 2013 at 11:26 AM, Amir Taaki <zgenjix at yahoo.com> wrote:
>>
>>> thanks! I went with inproc sockets... hope performance is good. will
>>> benchmark this.
>>>
>>>
>>> https://github.com/spesmilo/obelisk/commit/c0882a9b74bcce3cca41e0cf6dab4cb93552ad39
>>>
>>>
>>>
>>>
>>>
>>> On Tuesday, December 31, 2013 12:44 PM, Bruno D. Rodrigues <
>>> bruno.rodrigues at litux.org> wrote:
>>> poll on an additional pair of zmq inproc:// sockets ;)
>>>
>>>
>>> On Dec 31, 2013, at 12:41, Matt Connolly <matt.connolly at me.com> wrote:
>>>
>>> > Zmq poller can also wake on standard file descriptors (eg unix
>>> socket). If your custom event can write to a unix socket or pipe you might
>>> be in luck.
>>> >
>>> > Cheers
>>> > Matt.
>>> >
>>> >> On 31 Dec 2013, at 4:35 pm, Amir Taaki <zgenjix at yahoo.com> wrote:
>>> >>
>>> >>
>>> >>
>>> >> I was reading the docs and saw:
>>> >>
>>> >> "How can I integrate ØMQ sockets with normal sockets? Or with a GUI
>>> event loop?
>>> >> You can use the zmq_poll() function to poll for events on both ØMQ
>>> and normal sockets. The zmq_poll() function accepts a timeout so if you
>>> need to poll and process GUI
>>> >> events in the same application thread you can set a timeout and
>>> >> periodically poll for GUI events. See also the reference
>>> documentation."
>>> >>
>>> >>
>>> >> How can I wake up zmq_poll() with a custom event? This seems to solve
>>> my problems if possible. Then I can create a wakeup to process sends in the
>>> polling loop.
>>> >>
>>> >>
>>> >> On Tuesday, December 31, 2013 6:14 AM, Amir Taaki <zgenjix at yahoo.com>
>>> wrote:
>>> >>
>>> >> Hi!
>>> >>
>>> >> I have a bit of a design issue. I want to achieve this:
>>> >>
>>> >> - Software receives a request on a DEALER socket from a ROUTER socket
>>> (think the worker in paranoid pirate pattern).
>>> >> - Request is executed asynchronously in the software. Result is ready
>>> inside another thread.
>>> >>
>>> >> - Send it back over the same socket as the receive.
>>> >>
>>> >> In an ideal world, I'd be polling the socket to receive in one
>>> thread, and then performing the sends from another. But we cannot use
>>> sockets from multiple threads - sockets can only be used from one thread.
>>> >>
>>> >> Currently I have a multi-producer single-consumer lockless queue for
>>> sends, and then there's an std::condition_variable that's waiting for a
>>> signal to wake-up and batch process the send queue. Also there's a polling
>>> loop for receives. All the options I can think of have downsides:
>>> >>
>>> >> * Use a separate receive and send socket. I'd need some mechanism so
>>> that the broker (ppqueue) is aware of receive/send socket pairs. Maybe a
>>> special message for the broker that isn't relayed, but indicates the
>>> identity of the receive socket.
>>> >>
>>> >> * Polling then sending reduces the throughput of sends. I've
>>> benchmarked performance and it's a significant hit. You're essentially
>>> penalising sends by T microsecs. Sleeping for 0 is not a good idea since
>>> CPU usage hits 100%.
>>> >>
>>> >> * Using the same socket but synchronising access from the send/recv
>>> threads - ZMQ crashes infrequently when I do this because of triggered
>>> asserts. It'd be great if I know how to do this safely (maybe by calling
>>> some method on the socket).
>>> >>
>>> >> How do I achieve this scenario of 50% random receives and 50% random
>>> sends? It's not like the classic scenario of a receive, followed by some
>>> synchronous code path, and then a send (within the same thread). It's not
>>> an option to wait for requests to finish as I dropped Thrift to use ZMQ
>>> because of its async ability.
>>> >>
>>> >> Thanks!
>>> >> _______________________________________________
>>> >> zeromq-dev mailing list
>>> >> zeromq-dev at lists.zeromq.org
>>> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>> >> _______________________________________________
>>> >> zeromq-dev mailing list
>>> >> zeromq-dev at lists.zeromq.org
>>> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>> > _______________________________________________
>>> > zeromq-dev mailing list
>>> > zeromq-dev at lists.zeromq.org
>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140102/01fbec35/attachment.htm>