[zeromq-dev] Performance results on 100Gbps direct link
Francesco
francesco.montorsi at gmail.com
Wed Oct 2 19:27:30 CEST 2019
Il mer 2 ott 2019, 19:05 Doron Somech <somdoron at gmail.com> ha scritto:
>
> You don't need to create multiple sockets, just call connect multiple
> times with same address.
>
Wow, really??
I wish I had known that, I already changed quite a bit of code to use
multiple zmq sockets to make better use of background zmq threads!!
I will try connecting multiple times... At this point I suggest modifying
the benchmark utility to just do this trick and update the performance
graphs in the wiki with new results!
Francesco
On Wed, Oct 2, 2019, 19:45 Brett Viren via zeromq-dev <
> zeromq-dev at lists.zeromq.org> wrote:
>
>> Hi Francesco,
>>
>> I confirm your benchmark using two systems with the same 100 Gbps
>> Mellanox NICs but with an intervening Juniper QFX5200 switch (100 Gbps
>> ports).
>>
>> To reach ~25 Gbps with the largest message sizes required "jumbo frame"
>> MTU. The default mtu=1500 allows only ~20 Gbps. I also tried two more
>> doubling of zmsg size in the benchmark and these produce no significant
>> increase in throughput. OTOH, pinning the receiver (local_thr) to a CPU
>> gets it up to 33 Gbps.
>>
>> I note that iperf3 can achieve almost 40 Gbps (20 Gbps w MTU=1500).
>> Multiple simultaneous iperf3 tests can, in aggregate, use 90-100 Gbps.
>>
>> In both the ZMQ and singular iperf3 tests, it seems that CPU is the
>> bottleneck. For ZeroMQ the receiver's I/O thread is pegged at 100%.
>> With iperf3 it's that of the client/sender. The other ends in both
>> cases are at about 50%.
>>
>> The zguide suggests to use one I/O thread per GByte/s (faq says "Gbps")
>> so I tried the naive thing and hacked the ZMQ remote_thr.cpp and
>> local_thr.cpp so each use ten I/O threads. While I see all ten threads
>> in "top -H", still only one thread uses any CPU and it remains pegged at
>> 100% on the receiver (local_thr) and about 50% on the sender
>> (remote_thr). I think now that I misinterpreted this advice and it's
>> really relevant to the case of handling a very large number of
>> connections.
>>
>>
>> Any suggestions on how to let ZeroMQ get higher throughput at 100 Gbps?
>> If so, I'll give them a try.
>>
>>
>> Cheers,
>> -Brett.
>>
>> Francesco <francesco.montorsi at gmail.com> writes:
>>
>> > Hi all,
>> >
>> > I placed here:
>> > http://zeromq.org/results:100gbe-tests-v432
>> > the results I collected using 2 Mellanox ConnectX-5 linked by 100Gbps
>> > fiber cable.
>> >
>> > The results are not too much different from those at 10gpbs
>> > (http://zeromq.org/results:10gbe-tests-v432 )... the difference in TCP
>> > throughput is that
>> > - even using 100kB-long messages we still cannot saturate the link
>> > - latency is very much improved for messages > 10kB long
>> >
>> > Hopefully we will be able to improve performances in the future to
>> > improve these benchmarks...
>> >
>> > Francesco
>> > _______________________________________________
>> > zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20191002/40242700/attachment.htm>
More information about the zeromq-dev
mailing list