[zeromq-dev] Performance results on 100Gbps direct link
Luca Boccassi
luca.boccassi at gmail.com
Fri Oct 25 10:21:38 CEST 2019
On Thu, 2019-10-24 at 16:55 -0400, Brett Viren via zeromq-dev wrote:
> Hi again,
>
> Doron Somech <
> somdoron at gmail.com
> > writes:
>
> > You need to create multiple connections to enjoy the multiple io
> > threads.
> >
> > So in the remote/local_thr connect to the same endpoint 100 times
> > and create 10 io
> > threads.
> >
> > You don't need to create multiple sockets, just call connect
> > multiple times with same
> > address.
>
> I keep working on evaluating ZeroMQ against this 100 Gbps network
> when I
> get a spare moment. You can see some initial results in the attached
> PNG. As-is, things look pretty decent but there are two effects I
> see
> which I don't fully understand and I think are important to achieving
> something closer to saturation.
>
> 1) As you can see in the PNG, as the message size increases beyond 10
> kB
> the 10 I/O threads become less and less active. This activity seems
> correlated with throughput. But, why the die-off as we go to higher
> message size and why the resurgence at ~1 MB? Might there be some
> additional tricks to lift the throughput? Are such large messages
> simply not reasonable?
>
> 2) I've instrumented my tester to include a sequence count in each
> message and it uncovers that this multi-thread/multi-connect trick
> may
> lead to messages arriving to the receiver out-of-order. Given PUSH
> is
> round-robin and PULL is fair queued, I naively didn't expect
> this. But
> seeing it, I have two guesses. 1) I don't actually know what
> "fair-queued" really means :) and 2) if a mute state is getting hit
> then
> maybe all bets are off. I do wonder if adding "credit based"
> transfers
> might solve this ordering. Eg, if N credits (or fewer) are used
> given N
> connects, might the round-robin/fair-queue ordering stay in lock
> step?
>
>
> Any ideas are welcome. Thanks!
>
> -Brett.
The zero-copy receiver uses by default an 8KB buffer. If the message
received is larger, a new buffer is allocated per each message. That's
probably why you are seeing a drop just around that size.
A socket option ZMQ_IN_BATCH_SIZE has been recently added (TO BE USED
WITH CARE) to change that default - maybe try experimenting with that
and see if this assumption holds true.
--
Kind regards,
Luca Boccassi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20191025/cabad331/attachment.sig>
More information about the zeromq-dev
mailing list