[zeromq-dev] New option to set receive buffer for tcp

[Auer, Jens]

Hi Pieter,

How would we come up with a value for FAST? As I see it, the value depends on
- the data rate (msg/s)
- the message size
- the amount of memory the developer wants to spend

We can define a small struct as parameter to the new socket option with data rate, message size and the maximum memory available to zeroMQ, and then decide internally how much space we allocate. But I still it is hard to come up with a general rule.

I also thought about new socket options to set allocation/deallocation functions. This would allow users to use a memory pool for the buffers, but it is a very risky option with a lot of potential for problems. On the other hand, many people working on systems with resource constraints would love it. What is your opinion on this?

Best wishes,
  Jens

--
Dr. Jens Auer | CGI | Software Engineer
CGI Deutschland Ltd. & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: +49 6151 36860 154
jens.auer at cgi.com
Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter de.cgi.com/pflichtangaben.

CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI Group Inc. and its affiliates may be contained in this message. If you are not a recipient indicated or intended in this message (or responsible for delivery of this message to such person), or you think for any reason that this message may have been addressed to you in error, you may not use or copy or deliver this message to anyone else. In such case, you should destroy this message and are asked to notify the sender by reply e-mail.


-----Original Message-----
From: zeromq-dev-bounces at lists.zeromq.org [mailto:zeromq-dev-bounces at lists.zeromq.org] On Behalf Of Pieter Hintjens
Sent: 13 July 2015 23:44
To: ZeroMQ development list
Subject: Re: [zeromq-dev] New option to set receive buffer for tcp

A socket option is a good idea. Is there any way we can abstract this so users don't need to calculate buffer sizes (they will get it wrong often enough)? E.g. _SMALL vs. _FAST?

On Mon, Jul 13, 2015 at 5:29 PM, Auer, Jens <jens.auer at cgi.com> wrote:
> Hi,
>
>
>
> I ran some benchmarks with the current master to see if the zero-copy 
> changes have an effect on the performance in my system. As it turns 
> out, in my setup, the effect is not so large because of the number of 
> malloc calls still being done. In my setup, I send and receive data 
> streams with up to
> 200000 messages per second with a message size of 1125 bytes. The 
> receive buffer is fixed to 8kb, so I need one allocation for every 7 
> messages, and one deallocation. I experimentally increased the size of 
> the receive buffer to 65kb, and this resulted in a good performance 
> gain, basically trading memory (which I have lots of) for speed. What 
> is the opinion on a new socket option to let users set the size of the 
> receive buffer? I think the size depends on the expected data rate and 
> message size, so it seems fair for users to be able to influence it.
>
>
>
> Best wishes,
>
>   Jens
>
>
>
> --
>
> Dr. Jens Auer | CGI | Software Engineer
>
> CGI Deutschland Ltd. & Co. KG
> Rheinstraße 95 | 64295 Darmstadt | Germany
>
> T: +49 6151 36860 154
>
> jens.auer at cgi.com
>
> Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie 
> unter de.cgi.com/pflichtangaben.
>
>
>
> CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging 
> to CGI Group Inc. and its affiliates may be contained in this message. 
> If you are not a recipient indicated or intended in this message (or 
> responsible for delivery of this message to such person), or you think 
> for any reason that this message may have been addressed to you in 
> error, you may not use or copy or deliver this message to anyone else. 
> In such case, you should destroy this message and are asked to notify the sender by reply e-mail.
>
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
_______________________________________________
zeromq-dev mailing list
zeromq-dev at lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev