[zeromq-dev] Message batching in zmq

Doron Somech somdoron at gmail.com
Sun Aug 11 15:14:38 CEST 2019


Actually, zeromq is not waiting for buffers to be filled, the engine batch
as much messages as possible (or until the buffer is full).

If msg is larger than the buffer the message will be sent as is, using the
message own buffer (zerocopy can benefit here as well).
We will have to free the message buffer or continue batching only after the
kernel free the buffer.

On Sun, Aug 11, 2019 at 2:01 PM Luca Boccassi <luca.boccassi at gmail.com>
wrote:

> On Fri, 2019-08-09 at 23:28 +0200, Francesco wrote:
> > Hi Luca,
> > [changing subject to make it easier to search this message in future]
> >
> > Il giorno mer 7 ago 2019 alle ore 16:05 Luca Boccassi
> > <luca.boccassi at gmail.com> ha scritto:
> > > On Wed, 2019-08-07 at 14:50 +0200, Francesco wrote:
> > > > > Another improvement that can help is using the new zero-copy
> > > > > kernel
> > > > > TCP read/write APIs - I had started something a couple of years
> > > > > back, but again didn't have time to complete it.
> > > > >
> > > >
> > > > This looks very interesting as well.. do you have any pointer to
> > > > these new zero-copy kernel APIs?
> > > >
> > > > Thanks,
> > > > Francesco
> > >
> > > Kernel docs:
> > >
> > > https://www.kernel.org/doc/html/v5.2/networking/msg_zerocopy.html
> > >
> > > This is the initial experiment, very much incomplete:
> > >
> > >
> https://github.com/bluca/libzmq/commit/d021ea5f2c7526b388cb8f8005298e30b4cadd62
> >
> > Thanks, this looks really interesting, however the kernel docs state
> > very clearly that "MSG_ZEROCOPY is generally only effective at writes
> > over around 10 KB."...
> >  this raises the question: can we better tune the ZMQ message
> > batching
> > algorithm?
> >
> > I see in
> > zmq::tune_tcp_socket()
> > the following comment:
> >     //  Disable Nagle's algorithm. We are doing data batching on 0MQ
> > level,
> >     //  so using Nagle wouldn't improve throughput in anyway, but it
> > would
> >     //  hurt latency.
> >
> > but did not find (didn't look too much though) the place where ZMQ
> > actually does its own data batching. Nor could I find any option in
> > zmq_setsockopt() to tune somehow that batching.
> >
> > E.g. assuming I don't mind at all about message latency, can I
> > improve
> > the throughput by forcing ZMQ somehow to only call the send() system
> > call when I have queued 10kB of data to the zmq socket?
> >
> > Thanks!
> > Francesco
>
> Batching happens in the engine - the default data size is 8KB, tuned
> via src/config.hpp - although it's extremely tricky and it's very very
> easy to shot on oneself's foot by changing those parameters. The
> default wait time for the buffers to fill before sending is also
> defined there.
>
> Tweaking those can be done, but it needs a very very well defined test
> setup with a precise workload, which we don't really have.
>
> Yes adjusting the heuristics for MSG_ZEROCOPY is one of the TODOs to be
> able to use it effectively.
>
> --
> Kind regards,
> Luca Boccassi
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20190811/24e12808/attachment.htm>


More information about the zeromq-dev mailing list