[zeromq-dev] Performance results on 100Gbps direct link

Luca Boccassi luca.boccassi at gmail.com
Fri Oct 25 10:21:38 CEST 2019


On Thu, 2019-10-24 at 16:55 -0400, Brett Viren via zeromq-dev wrote:
> Hi again,
> 
> Doron Somech <
> somdoron at gmail.com
> > writes:
> 
> > You need to create multiple connections to enjoy the multiple io
> > threads.
> > 
> > So in the remote/local_thr connect to the same endpoint 100 times
> > and create 10 io
> > threads.
> > 
> > You don't need to create multiple sockets, just call connect
> > multiple times with same
> > address. 
> 
> I keep working on evaluating ZeroMQ against this 100 Gbps network
> when I
> get a spare moment.  You can see some initial results in the attached
> PNG.  As-is, things look pretty decent but there are two effects I
> see
> which I don't fully understand and I think are important to achieving
> something closer to saturation.
> 
> 1) As you can see in the PNG, as the message size increases beyond 10
> kB
> the 10 I/O threads become less and less active.  This activity seems
> correlated with throughput.  But, why the die-off as we go to higher
> message size and why the resurgence at ~1 MB?  Might there be some
> additional tricks to lift the throughput?  Are such large messages
> simply not reasonable?
> 
> 2) I've instrumented my tester to include a sequence count in each
> message and it uncovers that this multi-thread/multi-connect trick
> may
> lead to messages arriving to the receiver out-of-order.  Given PUSH
> is
> round-robin and PULL is fair queued, I naively didn't expect
> this.  But
> seeing it, I have two guesses.  1) I don't actually know what
> "fair-queued" really means :) and 2) if a mute state is getting hit
> then
> maybe all bets are off.  I do wonder if adding "credit based"
> transfers
> might solve this ordering.  Eg, if N credits (or fewer) are used
> given N
> connects, might the round-robin/fair-queue ordering stay in lock
> step?
> 
> 
> Any ideas are welcome.  Thanks!
> 
> -Brett.

The zero-copy receiver uses by default an 8KB buffer. If the message
received is larger, a new buffer is allocated per each message. That's
probably why you are seeing a drop just around that size.
A socket option ZMQ_IN_BATCH_SIZE has been recently added (TO BE USED
WITH CARE) to change that default - maybe try experimenting with that
and see if this assumption holds true.

-- 
Kind regards,
Luca Boccassi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20191025/cabad331/attachment.sig>


More information about the zeromq-dev mailing list