[zeromq-dev] ZMQ I/O threads CPU usage

[Brett Viren]

Francesco <francesco.montorsi at gmail.com> writes:

> So in first scenario the zmq background thread used only 12% of cpu to fill 914Mbps ; in second
> scenario it uses 97% to fill 700Mbps...
>
> how's that possible?

This is a pure guess: are you experiencing ZeroMQ's internal Nagle's
algorithm?

The guess applies if a) your messages are "small enough" and b) you
send() them "fast enough".  In this case, libzmq will pack multiple
messages into a single TCP packet.

If this indeed applies, then your spin_loop() scenario is slowing things
down beyond "fast enough" and libzmq transitions to sending out more
(and smaller) TCP packets for the same set of messages.  This means more
work for the I/O thread and that leads to the near 100% CPU usage and
finally that becomes a throughput bottleneck.

You could test this guess by repeating your two scenarios with two
different changes:

1) rerun with progressively faster spin_loop() and measure CPU% and
throughput to "detect" the Nagel timeout.  If the guess is right, as you
make the spin_loop() time delay smaller you'll see a transition as
libzmq goes from packing one message per TCP packet (message rate is
"too slow") to as many as will fit (message rate is "fast enough").
Plotting CPU% vs spin_loop() delay time should show a step function or
maybe a softer sigmoid.  I expect the slope of the transition depends on
the message size.

Which brings to:

2) rerun with dramatically bigger messages, say 10 kB?  (I don't know
libzmq's Nagle parameter values, but this is bigger than MTU).  If my
guess is right, then throughput and CPU% will become linearly and weakly
sensitive to the spin_loop() delay time.


Right or wrong, I'd be curious to hear the results!

-Brett.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 865 bytes
Desc: not available
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20210401/092399f5/attachment.sig>