[zeromq-dev] Performance results on 100Gbps direct link

[Doron Somech]

You need to create multiple connections to enjoy the multiple io threads.

So in the remote/local_thr connect to the same endpoint 100 times and
create 10 io threads.

You don't need to create multiple sockets, just call connect multiple times
with same address.

On Wed, Oct 2, 2019, 19:45 Brett Viren via zeromq-dev <
zeromq-dev at lists.zeromq.org> wrote:

> Hi Francesco,
>
> I confirm your benchmark using two systems with the same 100 Gbps
> Mellanox NICs but with an intervening Juniper QFX5200 switch (100 Gbps
> ports).
>
> To reach ~25 Gbps with the largest message sizes required "jumbo frame"
> MTU.  The default mtu=1500 allows only ~20 Gbps.  I also tried two more
> doubling of zmsg size in the benchmark and these produce no significant
> increase in throughput.  OTOH, pinning the receiver (local_thr) to a CPU
> gets it up to 33 Gbps.
>
> I note that iperf3 can achieve almost 40 Gbps (20 Gbps w MTU=1500).
> Multiple simultaneous iperf3 tests can, in aggregate, use 90-100 Gbps.
>
> In both the ZMQ and singular iperf3 tests, it seems that CPU is the
> bottleneck.  For ZeroMQ the receiver's I/O thread is pegged at 100%.
> With iperf3 it's that of the client/sender.  The other ends in both
> cases are at about 50%.
>
> The zguide suggests to use one I/O thread per GByte/s (faq says "Gbps")
> so I tried the naive thing and hacked the ZMQ remote_thr.cpp and
> local_thr.cpp so each use ten I/O threads.  While I see all ten threads
> in "top -H", still only one thread uses any CPU and it remains pegged at
> 100% on the receiver (local_thr) and about 50% on the sender
> (remote_thr).  I think now that I misinterpreted this advice and it's
> really relevant to the case of handling a very large number of
> connections.
>
>
> Any suggestions on how to let ZeroMQ get higher throughput at 100 Gbps?
> If so, I'll give them a try.
>
>
> Cheers,
> -Brett.
>
> Francesco <francesco.montorsi at gmail.com> writes:
>
> > Hi all,
> >
> > I placed here:
> >   http://zeromq.org/results:100gbe-tests-v432
> > the results I collected using 2 Mellanox ConnectX-5 linked by 100Gbps
> > fiber cable.
> >
> > The results are not too much different from those at 10gpbs
> > (http://zeromq.org/results:10gbe-tests-v432 )... the difference in TCP
> > throughput is that
> >  - even using 100kB-long messages we still cannot saturate the link
> >  - latency is very much improved for messages > 10kB long
> >
> > Hopefully we will be able to improve performances in the future to
> > improve these benchmarks...
> >
> > Francesco
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20191002/4b0b45ae/attachment.htm>