[zeromq-dev] User-space polling of ZMQ sockets

Francesco francesco.montorsi at gmail.com
Thu Apr 12 22:47:01 CEST 2018


 [sorry I hit "send" too early! here's the complete email]

Hi Luca, Hi Bill,
thanks for the answers.
Actually I verified that the configure script of zmq seems to be using
epoll:

   configure: Choosing polling system from 'kqueue epoll devpoll pollset
poll select'...
   configure: Using 'epoll' polling system with CLOEXEC

and indeed inside platform.hpp I found:

  /* Use 'epoll' polling system */
  #define ZMQ_USE_EPOLL 1

  /* Use 'epoll' polling system with CLOEXEC */
  #define ZMQ_USE_EPOLL_CLOEXEC 1

I also read the links you sent me Bill: actually since I'm monitoring only
2 file descriptors (2 zmq sockets) I think that poll() and epoll()
difference is hardly measurable.
My feeling is just that whatever code ends up calling poll() so often will
pay a cost in performance due to the syscall overhead:

   http://arkanis.de/weblog/2017-01-05-measurements-of-
system-call-performance-and-overhead
   http://man7.org/linux/man-pages/man7/vdso.7.html

So my question is actually the following one:
   In a ZMQ application, in which thread context are the "real" TCP socket
file descriptors actually polled?
   Is that polling happening in ZMQ background threads?
   Or rather it happens inside the thread that calls the zmq_msg_recv() /
zmq_poll() /  zmq_poller_wait_all()  ?

Because from looking briefly at zmq code it looks like zmq_msg_recv()  is
just talking with the zmq background thread via some kind of inter-thread
signaler.
So in my view I _guess_ that the real polling of the TCP socket should
happen in zmq background threads and the application threads just dequeues
some kind of queue that lives in those background threads. However that
seem to be proved wrong by a simple test: I tried to put a breakpoint on
the syscall poll()... the stacktrace I get is:

#0  poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007ffff63db0ba in zmq::signaler_t::wait(int) ()
#2  0x00007ffff63bb045 in zmq::mailbox_t::recv(zmq::command_t*, int) ()
#3  0x00007ffff63dba37 in zmq::socket_base_t::process_commands(int, bool)
#4  0x00007ffff63ddbca in zmq::socket_base_t::recv(zmq::msg_t*, int) ()
#5  0x00007ffff6400620 in zmq_msg_recv ()
[my code stack frames]

so it seems like it's the application threads that ends up calling that
poll().
Note that I set the RX timeout on my socket equal to 0 (ZMQ_DONTWAIT)...


Thanks,
Francesco





2018-04-12 22:39 GMT+02:00 Francesco <francesco.montorsi at gmail.com>:

> Hi Luca, Hi Bill,
> thanks for the answers.
> Actually I verified that the configure script of zmq seems to be using
> epoll:
>
>    configure: Choosing polling system from 'kqueue epoll devpoll pollset
> poll select'...
>    configure: Using 'epoll' polling system with CLOEXEC
>
> and indeed inside platform.hpp I found:
>
>   /* Use 'epoll' polling system */
>   #define ZMQ_USE_EPOLL 1
>
>   /* Use 'epoll' polling system with CLOEXEC */
>   #define ZMQ_USE_EPOLL_CLOEXEC 1
>
> I also read the links you sent me Bill: actually since I'm monitoring only
> 2 file descriptors (2 zmq sockets) I think that poll() and epoll()
> difference is hardly measurable.
> My feeling is just that whatever code ends up calling poll() so often will
> pay a lot of
>
>    http://arkanis.de/weblog/2017-01-05-measurements-of-
> system-call-performance-and-overhead
>
>
>
>
>
>
> 2018-04-11 21:05 GMT+02:00 Bill Torpey <wallstprog at gmail.com>:
>
>> Well, a little googling found this, which is a pretty good writeup:
>> https://jvns.ca/blog/2017/06/03/async-io-on-linux--
>> select--poll--and-epoll/
>>
>>
>> On Apr 11, 2018, at 2:42 PM, Bill Torpey <wallstprog at gmail.com> wrote:
>>
>> So, are there any benchmark tests that can be used to quantify the
>> overhead of zmq_poll?  It seems like this question keeps coming up, and it
>> would certainly be nice to have some real numbers (and the code used to
>> generate them).
>>
>> Having said that, there are several mechanisms that zmq_poll can use, and
>> there are apparently significant performance differences between them.  My
>> understanding of the conventional wisdom is that epoll is preferable to
>> poll, which is preferable to select — but I don’t have any data to back
>> that up.
>>
>> In any event, you can examine the output of the libzmq build to see which
>> mechanism is being used:
>>
>> -- Looking for kqueue
>> -- Looking for kqueue - not found
>> -- Looking for epoll_create
>> -- Looking for epoll_create - found
>> -- Looking for epoll_create1
>> -- Looking for epoll_create1 - found
>> *-- Detected epoll polling method*
>>
>> I’ve found one (very old!) post that discusses these differences:
>> https://cs.uwaterloo.ca/~brecht/servers/epoll/.  If anyone can suggest
>> additional sources, please do.
>>
>>
>> On Apr 11, 2018, at 1:46 PM, Luca Boccassi <luca.boccassi at gmail.com>
>> wrote:
>>
>> On Wed, 2018-04-11 at 18:47 +0200, Francesco wrote:
>>
>> Hi all,
>>
>> I'm using zmq_poller_wait_all() API in one of my threads because I
>> need to
>> poll over 2 ZMQ sockets.
>> I'm receiving a lot of traffic on both these sockets.
>> I think the performance of my software is bad and that may be due
>> IMHO to
>> the huge amount of poll() syscalls that my thread does... I think the
>> overhead of doing the system call is what is killing me...
>>
>> so my question is: when zmq_poller_wait_all() polls zmq FDs it is
>> actually
>> polling the real socket FD or rather some ZMQ internal structure?
>> If the latter is true, do I have a way to poll more than 1 zmq socket
>> without doing a system call?
>>
>>
>> THanks,
>> Francesco
>>
>>
>> I'm not familiar with the zmq_poller API so someone else might help -
>> in general, a common optimisation is to drain the socket using non-
>> blocking receives when poll wakes up (taking care of using heuristic to
>> avoid starving the other sockets), so that you can process multiple
>> messages per poll
>>
>> --
>> Kind regards,
>> Luca Boccassi_______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20180412/cb421650/attachment.htm>


More information about the zeromq-dev mailing list