[zeromq-dev] Performance results on 100Gbps direct link

[Brett Viren]

Hi again,

Doron Somech <somdoron at gmail.com> writes:

> You need to create multiple connections to enjoy the multiple io threads.
>
> So in the remote/local_thr connect to the same endpoint 100 times and create 10 io
> threads.
>
> You don't need to create multiple sockets, just call connect multiple times with same
> address. 

I keep working on evaluating ZeroMQ against this 100 Gbps network when I
get a spare moment.  You can see some initial results in the attached
PNG.  As-is, things look pretty decent but there are two effects I see
which I don't fully understand and I think are important to achieving
something closer to saturation.

1) As you can see in the PNG, as the message size increases beyond 10 kB
the 10 I/O threads become less and less active.  This activity seems
correlated with throughput.  But, why the die-off as we go to higher
message size and why the resurgence at ~1 MB?  Might there be some
additional tricks to lift the throughput?  Are such large messages
simply not reasonable?

2) I've instrumented my tester to include a sequence count in each
message and it uncovers that this multi-thread/multi-connect trick may
lead to messages arriving to the receiver out-of-order.  Given PUSH is
round-robin and PULL is fair queued, I naively didn't expect this.  But
seeing it, I have two guesses.  1) I don't actually know what
"fair-queued" really means :) and 2) if a mute state is getting hit then
maybe all bets are off.  I do wonder if adding "credit based" transfers
might solve this ordering.  Eg, if N credits (or fewer) are used given N
connects, might the round-robin/fair-queue ordering stay in lock step?


Any ideas are welcome.  Thanks!

-Brett.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: z.png
Type: image/png
Size: 149061 bytes
Desc: not available
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20191024/c86f3f0d/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20191024/c86f3f0d/attachment.sig>