[zeromq-dev] User-space polling of ZMQ sockets

Luca Boccassi luca.boccassi at gmail.com
Fri Apr 13 01:51:05 CEST 2018 

On Thu, 2018-04-12 at 22:47 +0200, Francesco wrote:
>  [sorry I hit "send" too early! here's the complete email]
> 
> Hi Luca, Hi Bill,
> thanks for the answers.
> Actually I verified that the configure script of zmq seems to be
> using
> epoll:
> 
>    configure: Choosing polling system from 'kqueue epoll devpoll
> pollset
> poll select'...
>    configure: Using 'epoll' polling system with CLOEXEC
> 
> and indeed inside platform.hpp I found:
> 
>   /* Use 'epoll' polling system */
>   #define ZMQ_USE_EPOLL 1
> 
>   /* Use 'epoll' polling system with CLOEXEC */
>   #define ZMQ_USE_EPOLL_CLOEXEC 1
> 
> I also read the links you sent me Bill: actually since I'm monitoring
> only
> 2 file descriptors (2 zmq sockets) I think that poll() and epoll()
> difference is hardly measurable.
> My feeling is just that whatever code ends up calling poll() so often
> will
> pay a cost in performance due to the syscall overhead:
> 
>    http://arkanis.de/weblog/2017-01-05-measurements-of-
> system-call-performance-and-overhead
>    http://man7.org/linux/man-pages/man7/vdso.7.html
> 
> So my question is actually the following one:
>    In a ZMQ application, in which thread context are the "real" TCP
> socket
> file descriptors actually polled?
>    Is that polling happening in ZMQ background threads?
>    Or rather it happens inside the thread that calls the
> zmq_msg_recv() /
> zmq_poll() /  zmq_poller_wait_all()  ?
> 
> Because from looking briefly at zmq code it looks like
> zmq_msg_recv()  is
> just talking with the zmq background thread via some kind of inter-
> thread
> signaler.
> So in my view I _guess_ that the real polling of the TCP socket
> should
> happen in zmq background threads and the application threads just
> dequeues
> some kind of queue that lives in those background threads. However
> that
> seem to be proved wrong by a simple test: I tried to put a breakpoint
> on
> the syscall poll()... the stacktrace I get is:
> 
> #0  poll () at ../sysdeps/unix/syscall-template.S:84
> #1  0x00007ffff63db0ba in zmq::signaler_t::wait(int) ()
> #2  0x00007ffff63bb045 in zmq::mailbox_t::recv(zmq::command_t*, int)
> ()
> #3  0x00007ffff63dba37 in zmq::socket_base_t::process_commands(int,
> bool)
> #4  0x00007ffff63ddbca in zmq::socket_base_t::recv(zmq::msg_t*, int)
> ()
> #5  0x00007ffff6400620 in zmq_msg_recv ()
> [my code stack frames]
> 
> so it seems like it's the application threads that ends up calling
> that
> poll().
> Note that I set the RX timeout on my socket equal to 0
> (ZMQ_DONTWAIT)...

TCP lives in the background I/O thread, the application threads use
poll to wait for inproc control commands, which is already staggered
etc etc to avoid having an impact on performance.

> 2018-04-12 22:39 GMT+02:00 Francesco <francesco.montorsi at gmail.com>:
> 
> > Hi Luca, Hi Bill,
> > thanks for the answers.
> > Actually I verified that the configure script of zmq seems to be
> > using
> > epoll:
> > 
> >    configure: Choosing polling system from 'kqueue epoll devpoll
> > pollset
> > poll select'...
> >    configure: Using 'epoll' polling system with CLOEXEC
> > 
> > and indeed inside platform.hpp I found:
> > 
> >   /* Use 'epoll' polling system */
> >   #define ZMQ_USE_EPOLL 1
> > 
> >   /* Use 'epoll' polling system with CLOEXEC */
> >   #define ZMQ_USE_EPOLL_CLOEXEC 1
> > 
> > I also read the links you sent me Bill: actually since I'm
> > monitoring only
> > 2 file descriptors (2 zmq sockets) I think that poll() and epoll()
> > difference is hardly measurable.
> > My feeling is just that whatever code ends up calling poll() so
> > often will
> > pay a lot of
> > 
> >    http://arkanis.de/weblog/2017-01-05-measurements-of-
> > system-call-performance-and-overhead
> > 
> > 
> > 
> > 
> > 
> > 
> > 2018-04-11 21:05 GMT+02:00 Bill Torpey <wallstprog at gmail.com>:
> > 
> > > Well, a little googling found this, which is a pretty good
> > > writeup:
> > > https://jvns.ca/blog/2017/06/03/async-io-on-linux--
> > > select--poll--and-epoll/
> > > 
> > > 
> > > On Apr 11, 2018, at 2:42 PM, Bill Torpey <wallstprog at gmail.com>
> > > wrote:
> > > 
> > > So, are there any benchmark tests that can be used to quantify
> > > the
> > > overhead of zmq_poll?  It seems like this question keeps coming
> > > up, and it
> > > would certainly be nice to have some real numbers (and the code
> > > used to
> > > generate them).
> > > 
> > > Having said that, there are several mechanisms that zmq_poll can
> > > use, and
> > > there are apparently significant performance differences between
> > > them.  My
> > > understanding of the conventional wisdom is that epoll is
> > > preferable to
> > > poll, which is preferable to select — but I don’t have any data
> > > to back
> > > that up.
> > > 
> > > In any event, you can examine the output of the libzmq build to
> > > see which
> > > mechanism is being used:
> > > 
> > > -- Looking for kqueue
> > > -- Looking for kqueue - not found
> > > -- Looking for epoll_create
> > > -- Looking for epoll_create - found
> > > -- Looking for epoll_create1
> > > -- Looking for epoll_create1 - found
> > > *-- Detected epoll polling method*
> > > 
> > > I’ve found one (very old!) post that discusses these differences:
> > > https://cs.uwaterloo.ca/~brecht/servers/epoll/.  If anyone can
> > > suggest
> > > additional sources, please do.
> > > 
> > > 
> > > On Apr 11, 2018, at 1:46 PM, Luca Boccassi <luca.boccassi at gmail.c
> > > om>
> > > wrote:
> > > 
> > > On Wed, 2018-04-11 at 18:47 +0200, Francesco wrote:
> > > 
> > > Hi all,
> > > 
> > > I'm using zmq_poller_wait_all() API in one of my threads because
> > > I
> > > need to
> > > poll over 2 ZMQ sockets.
> > > I'm receiving a lot of traffic on both these sockets.
> > > I think the performance of my software is bad and that may be due
> > > IMHO to
> > > the huge amount of poll() syscalls that my thread does... I think
> > > the
> > > overhead of doing the system call is what is killing me...
> > > 
> > > so my question is: when zmq_poller_wait_all() polls zmq FDs it is
> > > actually
> > > polling the real socket FD or rather some ZMQ internal structure?
> > > If the latter is true, do I have a way to poll more than 1 zmq
> > > socket
> > > without doing a system call?
> > > 
> > > 
> > > THanks,
> > > Francesco
> > > 
> > > 
> > > I'm not familiar with the zmq_poller API so someone else might
> > > help -
> > > in general, a common optimisation is to drain the socket using
> > > non-
> > > blocking receives when poll wakes up (taking care of using
> > > heuristic to
> > > avoid starving the other sockets), so that you can process
> > > multiple
> > > messages per poll
> > > 
> > > --
> > > Kind regards,
> > > Luca Boccassi_______________________________________________
> > > zeromq-dev mailing list
> > > zeromq-dev at lists.zeromq.org
> > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> > > 
> > > 
> > > 
> > > 
> > > _______________________________________________
> > > zeromq-dev mailing list
> > > zeromq-dev at lists.zeromq.org
> > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> > > 
> > > 
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
-- 
Kind regards,
Luca Boccassi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20180413/7d787ce3/attachment.sig>

More information about the zeromq-dev mailing list