[zeromq-dev] Performance results on 100Gbps direct link
Brett Viren
bv at bnl.gov
Wed Oct 2 18:43:57 CEST 2019
Hi Francesco,
I confirm your benchmark using two systems with the same 100 Gbps
Mellanox NICs but with an intervening Juniper QFX5200 switch (100 Gbps
ports).
To reach ~25 Gbps with the largest message sizes required "jumbo frame"
MTU. The default mtu=1500 allows only ~20 Gbps. I also tried two more
doubling of zmsg size in the benchmark and these produce no significant
increase in throughput. OTOH, pinning the receiver (local_thr) to a CPU
gets it up to 33 Gbps.
I note that iperf3 can achieve almost 40 Gbps (20 Gbps w MTU=1500).
Multiple simultaneous iperf3 tests can, in aggregate, use 90-100 Gbps.
In both the ZMQ and singular iperf3 tests, it seems that CPU is the
bottleneck. For ZeroMQ the receiver's I/O thread is pegged at 100%.
With iperf3 it's that of the client/sender. The other ends in both
cases are at about 50%.
The zguide suggests to use one I/O thread per GByte/s (faq says "Gbps")
so I tried the naive thing and hacked the ZMQ remote_thr.cpp and
local_thr.cpp so each use ten I/O threads. While I see all ten threads
in "top -H", still only one thread uses any CPU and it remains pegged at
100% on the receiver (local_thr) and about 50% on the sender
(remote_thr). I think now that I misinterpreted this advice and it's
really relevant to the case of handling a very large number of
connections.
Any suggestions on how to let ZeroMQ get higher throughput at 100 Gbps?
If so, I'll give them a try.
Cheers,
-Brett.
Francesco <francesco.montorsi at gmail.com> writes:
> Hi all,
>
> I placed here:
> http://zeromq.org/results:100gbe-tests-v432
> the results I collected using 2 Mellanox ConnectX-5 linked by 100Gbps
> fiber cable.
>
> The results are not too much different from those at 10gpbs
> (http://zeromq.org/results:10gbe-tests-v432 )... the difference in TCP
> throughput is that
> - even using 100kB-long messages we still cannot saturate the link
> - latency is very much improved for messages > 10kB long
>
> Hopefully we will be able to improve performances in the future to
> improve these benchmarks...
>
> Francesco
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20191002/3cbe9794/attachment.sig>
More information about the zeromq-dev
mailing list