[zeromq-dev] Performance results on 100Gbps direct link

Brett Viren bv at bnl.gov
Wed Oct 2 18:43:57 CEST 2019

Hi Francesco,

I confirm your benchmark using two systems with the same 100 Gbps
Mellanox NICs but with an intervening Juniper QFX5200 switch (100 Gbps

To reach ~25 Gbps with the largest message sizes required "jumbo frame"
MTU.  The default mtu=1500 allows only ~20 Gbps.  I also tried two more
doubling of zmsg size in the benchmark and these produce no significant
increase in throughput.  OTOH, pinning the receiver (local_thr) to a CPU
gets it up to 33 Gbps.

I note that iperf3 can achieve almost 40 Gbps (20 Gbps w MTU=1500).
Multiple simultaneous iperf3 tests can, in aggregate, use 90-100 Gbps.

In both the ZMQ and singular iperf3 tests, it seems that CPU is the
bottleneck.  For ZeroMQ the receiver's I/O thread is pegged at 100%.
With iperf3 it's that of the client/sender.  The other ends in both
cases are at about 50%.

The zguide suggests to use one I/O thread per GByte/s (faq says "Gbps")
so I tried the naive thing and hacked the ZMQ remote_thr.cpp and
local_thr.cpp so each use ten I/O threads.  While I see all ten threads
in "top -H", still only one thread uses any CPU and it remains pegged at
100% on the receiver (local_thr) and about 50% on the sender
(remote_thr).  I think now that I misinterpreted this advice and it's
really relevant to the case of handling a very large number of

Any suggestions on how to let ZeroMQ get higher throughput at 100 Gbps?
If so, I'll give them a try.


Francesco <francesco.montorsi at gmail.com> writes:

> Hi all,
> I placed here:
>   http://zeromq.org/results:100gbe-tests-v432 
> the results I collected using 2 Mellanox ConnectX-5 linked by 100Gbps
> fiber cable.
> The results are not too much different from those at 10gpbs
> (http://zeromq.org/results:10gbe-tests-v432 )... the difference in TCP
> throughput is that
>  - even using 100kB-long messages we still cannot saturate the link
>  - latency is very much improved for messages > 10kB long
> Hopefully we will be able to improve performances in the future to
> improve these benchmarks...
> Francesco
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20191002/3cbe9794/attachment.sig>

More information about the zeromq-dev mailing list