[zeromq-dev] (almost) zero-copy message receive

[Pieter Hintjens]

Hi Jens,

Thank you very much for this patch, which I've merged to libzmq
master. It is a smart change, one to be proud of. I'd expect people to
test this over the next days and weeks, and raise any issues if they
find them.

In the meantime, bravo!

-Pieter

On Wed, Jun 3, 2015 at 4:19 PM, Thomas Rodgers <rodgert at twrodgers.com> wrote:
> Glad it worked for you, though I think you really do need to make the array
> type char or unsigned char to not run afoul (though for 99% of the hardware
> in the world it makes no difference) of section 3.9.2 of the standard -
>
>> For any object (other than a base-class subobject) of trivially copyable
>> type T, whether or not the object holds a valid value of type T, the
>> underlying bytes (1.7) making up the object can be copied into an array of
>> char or unsigned char. 40 If the content of the array of char or unsigned
>> char is copied back into the object, the object shall subsequently hold its
>> original value
>
>
> In trying to follow up on the strict "legality" of the technique and the
> strict aliasing warning issued by GCC, one of the implementers of GCC's
> libstdc++ remarked -
>
>> >> The double static_cast might as well just be a reinterpret_cast, which
>> >> is equivalent but less typing.
>> >>
>> >>   return *reinterpret_cast<atomic_counter_t*>(m.lmsg.ctr_stg);
>> >
>> >
>> > I agree that's what *should* happen with reinterpret_cast, but with
>> > GCC4.9.2
>> > and -Wall -Wextra -Werror this results in -
>> >
>> > 'error: dereferencing type-punned pointer will break strict-aliasing
>> > rules'
>>
>> Hmm, so it does.
>>
>> Using the pointer returned by the placement-new expression doesn't get
>> a warning, even though it's doing the same thing (saying "I know this
>> looks like chars, but I've put an object here). I wonder if that's
>> because the placement-new just obscures the code (like the double
>> static_cast) or if the compiler gives ::new some magic powers.
>>
>>
>>
>> And if that's an aliasing violation then std::aligned_storage is
>>
>> impossible to use correctly.
>
>
>
>
> On Tue, Jun 2, 2015 at 4:29 PM, Jens Auer <jens.auer at betaversion.net> wrote:
>>
>> Hi Thomas,
>>
>>
>>
>> I changed my patch to use a
>>
>> union {
>>
>> zmq::atomic_counter_t aligner;
>>
>> uint8_t counter[ sizeof(zmq::atomic_counter_t) ];
>>
>> }
>>
>> to have a storage which is hopefully aligned. I think this this is
>> probably ok, because the values before that in the struct are all pointers
>> or integer values which should have a proper alignment, and the
>> atomic_counter_t follows. ‘ve also added the double static_cast in an
>> accessor function and it compiles without C++11and seems to work.
>>
>>
>>
>> Thanks for the suggestion,
>>
>>   Jens
>>
>>
>>
>>
>>
>> Von: zeromq-dev-bounces at lists.zeromq.org
>> [mailto:zeromq-dev-bounces at lists.zeromq.org] Im Auftrag von Thomas Rodgers
>> Gesendet: Dienstag, 2. Juni 2015 19:59
>> An: Jens Auer; ZeroMQ development list
>>
>>
>> Betreff: Re: [zeromq-dev] (almost) zero-copy message receive
>>
>>
>>
>> FWIW, I made a local change to msg_t::content_t to use a type erased
>> zmq::atomic_counter_t and it compiles (without relaxing warnings) and passes
>> all tests.
>>
>>
>>
>> On Tue, Jun 2, 2015 at 12:14 PM, Thomas Rodgers <rodgert at twrodgers.com>
>> wrote:
>>
>> Current vector implementations (AFAIK) don't do small object
>> optimizations, but I have it on good authority (the author sits next to me
>> at work) there is a proposal coming for such a vector, and it would face the
>> exact same problem. std::vector does use a dynamically allocated array, but
>> not of type T, otherwise reserve() would unimplementable. Is the point that
>> a void* that comes via malloc is somehow different than a member array in
>> terms of type punning? This may well be strictly true (see aforementioned
>> malloc "standards weasel wording" comment), but there are many many
>> libraries which do small object optimizations via the double static_cast
>> approach that would break if it didn't work (never mind whether it is
>> legal). The standard already includes types that do this sort of punning
>> (std::function, std::aligned_storage, proposed std::optional, proposed
>> std::variant).
>>
>>
>>
>> In any event, I've dropped an email to a couple of fellow C++ standards
>> committee members (including a Clang developer) to see what their thoughts
>> on this are.
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Jun 2, 2015 at 11:53 AM, Jens Auer <jens.auer at betaversion.net>
>> wrote:
>>
>> I always thought that vector uses a dynamically allocated array and not a
>> char []. That would give us a char* which we can savely cast into another
>> pointer. I don't have the source available, but if it uses small object
>> optimization, it has to use a static array backend. I will take a look at
>> the vector source code.
>>
>>
>>
>> Thanks,
>>
>>  Jens
>>
>>
>>
>>
>> -------- Ursprüngliche Nachricht --------
>> Von: Thomas Rodgers <rodgert at twrodgers.com>
>> Datum: 02.06.2015 18:37 (GMT+01:00)
>> An: ZeroMQ development list <zeromq-dev at lists.zeromq.org>
>>
>> Betreff: Re: [zeromq-dev] (almost) zero-copy message receive
>>
>> If it were completely broken, then how could vector work (it is the
>> current poster child for type erased aligned storage)? std::aligned_storage
>> (and boost::aligned_storage) also do type erasure through a char backing
>> array. It might well be "strictly illegal" relative to standards wording
>> (there was some discussion along these lines at the fall Standards meeting),
>> but it would essentially preclude any sort of placement new except the via
>> the "magic weasel" words that allow malloc.
>>
>>
>>
>> On Tue, Jun 2, 2015 at 11:30 AM, Auer, Jens <jens.auer at cgi.com> wrote:
>>
>> Yes, this doesn't change anything because it only depends on the existence
>> of two pointers of different type pointing to the same address. There is an
>> exception for char*, but here we would have a char array and an
>> atomic_counter_t pointer pointing to it. This is illegal.
>>
>> Cheers,
>>   Jens
>>
>>
>>
>> --
>>
>> Jens Auer | CGI | Software-Engineer
>>
>> CGI (Germany) GmbH & Co. KG
>> Rheinstraße 95 | 64295 Darmstadt | Germany
>>
>> T: +49 6151 36860 154
>> jens.auer at cgi.com
>> Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie
>> unter de.cgi.com/pflichtangaben.
>>
>>
>>
>> CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to
>> CGI Group Inc. and its affiliates may be contained in this message. If you
>> are not a recipient indicated or intended in this message (or responsible
>> for delivery of this message to such person), or you think for any reason
>> that this message may have been addressed to you in error, you may not use
>> or copy or deliver this message to anyone else. In such case, you should
>> destroy this message and are asked to notify the sender by reply e-mail.
>>
>> ________________________________
>>
>> Von: zeromq-dev-bounces at lists.zeromq.org
>> [zeromq-dev-bounces at lists.zeromq.org]" im Auftrag von "Thomas Rodgers
>> [rodgert at twrodgers.com]
>> Gesendet: Dienstag, 2. Juni 2015 18:10
>>
>>
>> An: ZeroMQ development list
>> Betreff: Re: [zeromq-dev] (almost) zero-copy message receive
>>
>>
>>
>> with reinterpret_cast?
>>
>>
>>
>> On Tue, Jun 2, 2015 at 10:55 AM, Auer, Jens <jens.auer at cgi.com> wrote:
>>
>> I already tried this, but then the compiler complaint about a violation of
>> the strict alias rules and the code doesn't compile anymore. The problem is
>> that I access something of type uint8_t[] with a pointer of type
>> atomic_counter_t.
>>
>> Cheers,
>>   Jens
>>
>>
>>
>> --
>>
>> Jens Auer | CGI | Software-Engineer
>>
>> CGI (Germany) GmbH & Co. KG
>> Rheinstraße 95 | 64295 Darmstadt | Germany
>>
>> T: +49 6151 36860 154
>> jens.auer at cgi.com
>> Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie
>> unter de.cgi.com/pflichtangaben.
>>
>>
>>
>> CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to
>> CGI Group Inc. and its affiliates may be contained in this message. If you
>> are not a recipient indicated or intended in this message (or responsible
>> for delivery of this message to such person), or you think for any reason
>> that this message may have been addressed to you in error, you may not use
>> or copy or deliver this message to anyone else. In such case, you should
>> destroy this message and are asked to notify the sender by reply e-mail.
>>
>> ________________________________
>>
>> Von: zeromq-dev-bounces at lists.zeromq.org
>> [zeromq-dev-bounces at lists.zeromq.org]" im Auftrag von "Thomas Rodgers
>> [rodgert at twrodgers.com]
>> Gesendet: Dienstag, 2. Juni 2015 17:34
>>
>>
>> An: ZeroMQ development list
>> Betreff: Re: [zeromq-dev] (almost) zero-copy message receive
>>
>>
>>
>> One other point, you would need to guarantee appropriate alignment of the
>> backing array (sadly, C++11 gave us std::aligned_storage but 03 not so
>> much).
>>
>>
>>
>> On Tue, Jun 2, 2015 at 10:11 AM, Thomas Rodgers <rodgert at twrodgers.com>
>> wrote:
>>
>> I am unaware of any list, it seems (to me at least) it's whatever people
>> have contributed support for.
>>
>>
>>
>> As for a concrete suggestion. I am not sure this is any less ugly but...
>>
>>
>>
>> libzmq goes out of it's way generally to avoid calling new() in many cases
>> (non-throwing failure on OOM presumably), preferring malloc() and placement
>> new instead, content_t was no exception. It did embed an explicit
>> atomic_counter_t and would placement new it, and explicitly call the
>> destructor when the lmsg was destroyed.
>>
>>
>>
>> You could, instead declare the storage for atomic_counter_t as an array,
>> uint8_t ctr_storage[sizeof(atomic_counter_t)] and placement new/explicit
>> dtor as before, and then provide an accessor to return an atomic_counter_t&
>> by returning *reinterpret_cast<atomic_counter_t*>(&ctr_storage[0]).
>>
>>
>>
>> It doesn't change the size of the lmsg type, but it does remove one level
>> of indirection, and you have hidden the ugly bits behind an accessor method.
>>
>>
>>
>> On Tue, Jun 2, 2015 at 9:40 AM, Auer, Jens <jens.auer at cgi.com> wrote:
>>
>> Hi Thomas,
>>
>>
>>
>> I did not want to say that you intended to start a flame war and you
>> certainly did not. I know that things like these easily turn into flame
>> wars, so I wanted to prevent that. All I wanted say was that I know that
>> switching to C++11 may not be applicable so I want to discuss the patch
>> first and search for an alternative to C++11. Do you have a suggestion for
>> the issue with the non-POD atomic_counter_t? My experience is quite limited
>> because I normally don’t write code which has to support older compilers.
>>
>>
>>
>> Just out of curiosity, do you know a definite list of compilers zeroMQ
>> intends to supports? I only found the document
>> http://zeromq.org/docs:builds, but this is quite outdated.
>>
>>
>>
>> Best wishes,
>>
>>   Jens
>>
>>
>>
>> --
>>
>> Dr. Jens Auer | CGI | Software Engineer
>>
>> CGI Deutschland Ltd. & Co. KG
>> Rheinstraße 95 | 64295 Darmstadt | Germany
>>
>> T: +49 6151 36860 154
>>
>> jens.auer at cgi.com
>>
>> Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie
>> unter de.cgi.com/pflichtangaben.
>>
>>
>>
>> CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to
>> CGI Group Inc. and its affiliates may be contained in this message. If you
>> are not a recipient indicated or intended in this message (or responsible
>> for delivery of this message to such person), or you think for any reason
>> that this message may have been addressed to you in error, you may not use
>> or copy or deliver this message to anyone else. In such case, you should
>> destroy this message and are asked to notify the sender by reply e-mail.
>>
>>
>>
>> From: zeromq-dev-bounces at lists.zeromq.org
>> [mailto:zeromq-dev-bounces at lists.zeromq.org] On Behalf Of Thomas Rodgers
>> Sent: 02 June 2015 16:07
>> To: ZeroMQ development list
>> Subject: Re: [zeromq-dev] (almost) zero-copy message receive
>>
>>
>>
>> A better chart of compiler conformance is probably here -
>>
>>
>>
>> http://en.cppreference.com/w/cpp/compiler_support
>>
>>
>>
>> I don't know that there's been a "flamewar" around this topic, but
>> switching to -std=c++11 has been discussed recently in the context of
>> implementing thread safe sockets. IIRC the consensus at the time was to
>> stick with 98/03 conforming code.
>>
>>
>>
>> In my experience, supporting C++11, opens opens up a bit of a minefield of
>> portability "gotchas". For https://github.com/zeromq/azmq I can generally
>> fall back on Boost to paper over these issues, but that's not a realistic
>> option with libzmq. So, while I get that *this* change wouldn't necessarily
>> run afoul of the current state of C++ conformance for MSVC, I am personally
>> a bit leary of the general prospects of switching to C++11 in libzmq because
>> the next change that assumes C++11 conformance might not be supported by the
>> range of compilers used to build libzmq.
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Jun 2, 2015 at 8:44 AM, Auer, Jens <jens.auer at cgi.com> wrote:
>>
>> Hi,
>>
>> I don't want to start a flame war on this because I know that there can be
>> very good reasons to not rely on C++11. That's why I did not just send the
>> patch, but wanted to discuss it first and see if there are any other ways to
>> do this.
>> The main issue is that I want to put an atomic_counter_t into a union in
>> msg_t, which C++98/C++03 does not allow because it has constructors and
>> non-public data members. So an obvious way to solve this would be to make
>> atomic_counter_t a plain C struct with free function to create and modify
>> it. I was hoping for other suggestions.
>>
>> For what it's worth, the change does not need a fully compliant C++, but
>> just standard layout and trivial types which are implemented in MSVC since
>> 2012.
>>
>> Cheers,
>>   Jens
>>
>>
>>
>> --
>>
>> Jens Auer | CGI | Software-Engineer
>>
>> CGI (Germany) GmbH & Co. KG
>> Rheinstraße 95 | 64295 Darmstadt | Germany
>>
>> T: +49 6151 36860 154
>> jens.auer at cgi.com
>> Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie
>> unter de.cgi.com/pflichtangaben.
>>
>>
>>
>> CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to
>> CGI Group Inc. and its affiliates may be contained in this message. If you
>> are not a recipient indicated or intended in this message (or responsible
>> for delivery of this message to such person), or you think for any reason
>> that this message may have been addressed to you in error, you may not use
>> or copy or deliver this message to anyone else. In such case, you should
>> destroy this message and are asked to notify the sender by reply e-mail.
>>
>> ________________________________
>>
>> Von: zeromq-dev-bounces at lists.zeromq.org
>> [zeromq-dev-bounces at lists.zeromq.org]" im Auftrag von "Thomas Rodgers
>> [rodgert at twrodgers.com]
>> Gesendet: Dienstag, 2. Juni 2015 15:37
>>
>>
>> An: ZeroMQ development list
>> Betreff: Re: [zeromq-dev] (almost) zero-copy message receive
>>
>>
>>
>> Personally, I think that 4 years of C++11, this should not be an issues,
>> but there may be platforms with old compilers which you want to support.
>>
>>
>>
>> 4 years of C++11 *should* be enough, but wide-spread use of fully
>> conforming compilers is still an issue, for instance -
>>
>>
>>
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>