[zeromq-dev] vector io for TCP sockets

Auer, Jens jens.auer at cgi.com
Mon May 30 08:58:08 CEST 2016


Hi Doron,

I am not working on the code yet, I was just thinking about it. I am also not sure about the performance implications especially for small messages, so I wanted to ask if somebody had more experience with vector io systems.

The reason I am interested in this is not a concrete performance bottleneck. I am more reacting to some concerns on our client side which were raised initially about performance and memory bandwidth allocation, so I checked how ZeroMQ handles IO and if it can be improved. If you remember, I eliminated a memory allocation and a copy operations on the receiving side, so I am looking at the other side. I think I can find some time on the weekend and implement a prototype which can be used for benchmarking. Does anybody have a larger benchmark installation? I only have two computers at home on my private network.

Cheers,
  Jens

--
Dr. Jens Auer | CGI | Software Engineer
CGI Deutschland Ltd. & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: +49 6151 36860 154
jens.auer at cgi.com<mailto:jens.auer at cgi.com>
Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>.

CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI Group Inc. and its affiliates may be contained in this message. If you are not a recipient indicated or intended in this message (or responsible for delivery of this message to such person), or you think for any reason that this message may have been addressed to you in error, you may not use or copy or deliver this message to anyone else. In such case, you should destroy this message and are asked to notify the sender by reply e-mail.

From: zeromq-dev [mailto:zeromq-dev-bounces at lists.zeromq.org] On Behalf Of Doron Somech
Sent: 25 May 2016 11:46
To: ZeroMQ development list
Subject: Re: [zeromq-dev] vector io for TCP sockets

Can you share the repository you are developing on?
What are you trying to improve? Do you have more performance needs from zeromq?

I think it is possible, however I'm not sure about the performance improvement.

We can do this maybe only for large messages (bigger than 8KB).

On Tue, May 24, 2016 at 4:22 PM, Auer, Jens <jens.auer at cgi.com<mailto:jens.auer at cgi.com>> wrote:
Hi,

I have been thinking on how to change the stream_engine_t class to use vector io to save copying the message data into the send buffer. The basic idea is to send the data directly from the messages in the queue. To do this, the headers must be created in an independent array and the lock-free queue needs to provide an iterator to traverse the elements in the queue. I think this should be possible with the current implementation because it is a producer/consumer queue where only one side can remove items, and this is the stream_engine. The other side can only add new elements, and that should not invalidate the iterator (which is probably a pointer to the current element).

Every message then uses two io_vec elements, the first pointing to the header data and the second one pointing to the message data.

I've tried to write down a pseudo-code implementation of the algorithm:
- The write buffer in stream_engine_t is removed.
- stream_engine_t gets an array iovecs of IOV_MAX io_vec elements
- stream_engine_t gets an array headerbuf of IOV_MAX/2 * 10 bytes to hold the headers
- stream_engine_t stores an offset counting the already sent bytes of the first message to be sent

When called, out_event will check if the mailbox Q is not empty. If so:
char* h = &headerbuf[0];
io_vec* v = &iovecs[0];
msg = Q.cbegin();
last_msg = Q.cend();

bool firstMessage = true;
while (number of processed messages <= IOV_MAX/2 && msg != last_msg) {
    hlen = create_header_at(h, *msg); // hlen == 2 || hlen == 10
    h += hlen;

    // send header
    v->iov_base = h;
    v->iov_len = hlen;
    v += 1;

    // send data
    v->io_base = msg->data() + offset;
    offset = 0;
    v->iov_len = msg->size();
    v += 1;

    ++msg;
}

// v - iovecs is the number of created io_vec elements
ssize_t sent = writev(socket, iovecs, (v - iovecs) );

// error checking ...

// no pop messages from the queue and check if there is a remainder
while ( (sent - Q.front().size() > 0) {
    msg_t msg = Q.pop();
    sent -= msg.size();
}
if (sent >= 0) {
    offset = sent;
}

It looks reasonable to me, but I am not sure about the implications, especially when considering different message sizes. For very small messages, this will probably be slower than the original stream_engine. IOV_MAX is 1024 on my Linux machine, so for one-byte messages this would send 512 bytes of message data with one syscall, compared to 8192/3=2730 messages with the old implementation. There should be a message size where it will get faster, but I have not much experience with vector io to guess anything. Maybe somebody could provide some feedback?

Best wishes,
  Jens

--
Dr. Jens Auer | CGI | Software Engineer
CGI Deutschland Ltd. & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: +49 6151 36860 154
jens.auer at cgi.com<mailto:jens.auer at cgi.com>
Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>.

CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI Group Inc. and its affiliates may be contained in this message. If you are not a recipient indicated or intended in this message (or responsible for delivery of this message to such person), or you think for any reason that this message may have been addressed to you in error, you may not use or copy or deliver this message to anyone else. In such case, you should destroy this message and are asked to notify the sender by reply e-mail.


_______________________________________________
zeromq-dev mailing list
zeromq-dev at lists.zeromq.org<mailto:zeromq-dev at lists.zeromq.org>
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20160530/63a6c645/attachment.htm>


More information about the zeromq-dev mailing list