[zeromq-dev] On hooking memory allocations

Tue Nov 29 12:55:39 CET 2016

1) yes

2) depends on who you ask :-D

Don't worry too much about it, if it works on your development platform
it's good enough for the initial iteration.

We have automated CI for Linux (Travis), OSX (Travis) and Windows
(appveyor with gcc and jenkins with vsc++). If anything breaks we'll see
it quickly and will ask you to fix it :-)

If you prefer to check it beforehand you can set up Travis and Appveyor
to run from your Github personal fork when you push to your personal
branch, but it's not a hard requirement. I can give you instructions if
you would like that.

On Tue, 2016-11-29 at 13:39 +0200, Petteri Salo wrote:
> Hi,
> 
> Few questions:
> 1. PR == Pull Request ? git and github are pretty much unknown to me so I'm
> just checking my assumptions...
> 2. From https://rfc.zeromq.org/spec:42/C4/ section 2.3.6, I assume the
> "principle target platform" is Linux?
> 
> -Petteri
> 
> On Tue, Nov 29, 2016 at 12:51 PM, Luca Boccassi <luca.boccassi at gmail.com>
> wrote:
> 
> > That works on Linux (and I guess *NIX) yes, but I'm not sure if Windows
> > has a similar functionality. Even if it does, the use case here (a game
> > engine/library) is most likely a statically linked binary, so I think it
> > won't work there :-)
> >
> > All in all this is an interesting discussion, but let's see some PRs and
> > take it from there :-)
> >
> > On Mon, 2016-11-28 at 11:48 -0800, Max Kozlovsky wrote:
> > > Each program can be linked with a separate malloc implementation if the
> > > user so desires. Libraries don't need to be aware which implementation it
> > > is. Different malloc implementation can be even substituted at run time
> > on
> > > platforms with dynamic linking support (LD_PRELOAD etc).
> > >
> > > On Mon, Nov 28, 2016 at 11:39 AM, Jens Auer <jens.auer at betaversion.net>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > >
> > > >
> > > > yes and no. If you overwrite it globally at compute-time every program
> > on
> > > > the system has to use your custom implementation. So if you deliver
> > your
> > > > ZeroMQ library with your program it will work, but what if my program
> > wants
> > > > a different custom allocator?
> > > >
> > > >
> > > >
> > > > Cheers,
> > > >
> > > > Jens
> > > >
> > > >
> > > >
> > > > *Von:* zeromq-dev [mailto:zeromq-dev-bounces at lists.zeromq.org] *Im
> > > > Auftrag von *Max Kozlovsky
> > > > *Gesendet:* Montag, 28. November 2016 20:36
> > > > *An:* ZeroMQ development list
> > > > *Betreff:* Re: [zeromq-dev] On hooking memory allocations
> > > >
> > > >
> > > >
> > > > Hi,
> > > >
> > > >
> > > >
> > > > Would not globally overwriting malloc/free with the custom
> > implementation
> > > > achieve the desired behavior (instead of providing hooks for installing
> > > > malloc overrides in each and every library)?
> > > >
> > > >
> > > >
> > > > Max
> > > >
> > > >
> > > >
> > > > On Mon, Nov 28, 2016 at 5:08 AM, Auer, Jens <jens.auer at cgi.com> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I don't see a big problem with the C API except that C doesn't support
> > > > overloads. So if the function has a new name, e.g.
> > > > zmq_ctx_new_with_allocator, everything stays plain C. The default
> > instance
> > > > would be a
> > > >
> > > > static void* malloc_(size_t n, void*) {return malloc(n);}
> > > > static void free_(void* ptr, size_t n, void*) {free(ptr);}
> > > >
> > > > allocator_t alloc{
> > > > NULL,
> > > > malloc_,
> > > > free_
> > > > };
> > > >
> > > > context_t then stores the member and gets methods to forward memory
> > > > allocations to the function pointers, passing the hint pointer as an
> > > > additional argument.
> > > >
> > > > In my C++ code, I can then use an allocator
> > > > static void* allocate(size_t n, void* obj) {return
> > > > static_cast<std::allocator<char>>(obj)->allocate(n); }
> > > > static void free_(void* ptr, size_t n, void*obj) {
> > > > static_cast<std::allocator<char>>(obj)->deallocate(ptr, n); }
> > > >
> > > > std::allocator<char> a;
> > > > allocator_t zmqAlloc{
> > > >    &a,
> > > >    allocate,
> > > >    free_
> > > > };
> > > >
> > > > void* ctx = zmq_ctx_new_with_allocator(&zmqAlloc);
> > > >
> > > > I think this should work?
> > > >
> > > > Best wishes,
> > > > Jens
> > > >
> > > > --
> > > > Dr. Jens Auer | CGI | Software Engineer
> > > > CGI Deutschland Ltd. & Co. KG
> > > > Rheinstraße 95 | 64295 Darmstadt | Germany
> > > > T: +49 6151 36860 154
> > > > jens.auer at cgi.com
> > > > Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie
> > > > unter de.cgi.com/pflichtangaben.
> > > >
> > > > CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging
> > to
> > > > CGI Group Inc. and its affiliates may be contained in this message. If
> > you
> > > > are not a recipient indicated or intended in this message (or
> > responsible
> > > > for delivery of this message to such person), or you think for any
> > reason
> > > > that this message may have been addressed to you in error, you may not
> > use
> > > > or copy or deliver this message to anyone else. In such case, you
> > should
> > > > destroy this message and are asked to notify the sender by reply
> > e-mail.
> > > >
> > > > > -----Original Message-----
> > > > > From: zeromq-dev [mailto:zeromq-dev-bounces at lists.zeromq.org] On
> > Behalf
> > > > Of
> > > > > Luca Boccassi
> > > > > Sent: 28 November 2016 12:30
> > > > > To: ZeroMQ development list
> > > > > Subject: Re: [zeromq-dev] On hooking memory allocations
> > > > >
> > > > > That would work for an internal API, but given we expose a C API
> > > > unfortunately I
> > > > > don't think that would work as a public API :-( And I think for this
> > use
> > > > case they
> > > > > would require a public API.
> > > > >
> > > > > As an external API, a new zmq_ctx_set that takes a callback would
> > have
> > > > been ideal,
> > > > > but it only takes int. So perhaps a new zmq_ctx_set_allocator that
> > takes
> > > > a callback
> > > > > pointer would be the next best.
> > > > >
> > > > > An alternative would be to have a system similar to what we use for
> > the
> > > > poll
> > > > > implementation (epoll kqueue select etc), but this would be a
> > build-time
> > > > option,
> > > > > and the implementation would have to be checked in, which I don't
> > think
> > > > is an
> > > > > option for this case, right?
> > > > >
> > > > > On Mon, 2016-11-28 at 10:51 +0000, Auer, Jens wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I am just a user, but I would love to see this change. I have
> > thinking
> > > > > > about this and I would like to be able to pass a C++ allocator
> > object
> > > > > > to ZeroMQ, so a simple function hook is not enough. My idea is to
> > > > > > define a struct in the interface
> > > > > >
> > > > > > struct allocator_t
> > > > > > {
> > > > > >     void* hint;
> > > > > >     void* (allocate)(size_t, void*);
> > > > > >     void (deallocate)(void*, size_t, void*); };
> > > > > >
> > > > > > and store this in the context object. Since I don't think that this
> > > > should be
> > > > > changed during runtime, I would create a new zmq_ctx_new overload
> > which
> > > > takes a
> > > > > parameter of type allocator_t. The default value would be to call
> > > > malloc/free.
> > > > > >
> > > > > > Cheers,
> > > > > > Jens
> > > > > >
> > > > > > --
> > > > > > Jens Auer | CGI | Software-Engineer
> > > > > > CGI (Germany) GmbH & Co. KG
> > > > > > Rheinstraße 95 | 64295 Darmstadt | Germany
> > > > > > T: +49 6151 36860 154
> > > > > > jens.auer at cgi.com<mailto:jens.auer at cgi.com>
> > > > > > Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden
> > Sie
> > > > unter
> > > > > de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>.
> > > > > >
> > > > > > CONFIDENTIALITY NOTICE: Proprietary/Confidential information
> > belonging
> > > > to CGI
> > > > > Group Inc. and its affiliates may be contained in this message. If
> > you
> > > > are not a
> > > > > recipient indicated or intended in this message (or responsible for
> > > > delivery of this
> > > > > message to such person), or you think for any reason that this
> > message
> > > > may have
> > > > > been addressed to you in error, you may not use or copy or deliver
> > this
> > > > message to
> > > > > anyone else. In such case, you should destroy this message and are
> > asked
> > > > to notify
> > > > > the sender by reply e-mail.
> > > > > > ________________________________
> > > > > > Von: zeromq-dev [zeromq-dev-bounces at lists.zeromq.org]" im Auftrag
> > von
> > > > > > "Petteri Salo [petteri.salo at gmail.com]
> > > > > > Gesendet: Montag, 28. November 2016 09:40
> > > > > > An: zeromq-dev at lists.zeromq.org
> > > > > > Betreff: [zeromq-dev] On hooking memory allocations
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > Let me first do a little introduction as I'm new to this list. I'm
> > a
> > > > software engineer
> > > > > with 15+ years of experience working on games at a company called
> > Remedy
> > > > > Entertainment Ltd. We've done games for PC, and various games
> > consoles
> > > > over the
> > > > > years. Most recently we did Quantum Break for Xbox One.
> > > > > >
> > > > > > I've now been tasked with evaluating ZeroMQ. One important feature
> > of
> > > > any
> > > > > library we use in games is the ability to hook all memory
> > allocations -
> > > > this is to allow
> > > > > the use of custom memory allocators and/or for tracking when and
> > where
> > > > memory is
> > > > > allocated.
> > > > > >
> > > > > > I've searched the libzmq source code and there is about 150 uses of
> > > > new, malloc,
> > > > > realloc , etc.
> > > > > >
> > > > > > If we were to adopt libzmq we'd like to put in allocation hooks and
> > > > that work
> > > > > would then be something that we'd like to contribute back to the
> > > > project. Having
> > > > > those hooks in the main repository would then make it easier for us
> > to
> > > > adopt future
> > > > > changes to the library.
> > > > > >
> > > > > > So, my question is would this kind of change be something that
> > would be
> > > > > accepted? Of course assuming that coding conventions, proper way of
> > > > submitting
> > > > > the patch etc. are followed. I do realize that one would want to see
> > the
> > > > actual code
> > > > > before accepting. I'm interested in the principle of accepting a
> > change
> > > > such as this,
> > > > > since it would introduce a new "rule" for working ión libzmq source
> > code
> > > > : "All
> > > > > allocations shall go through an allocation hook."
> > > > > >
> > > > > > Best Regards,
> > > > > >
> > > > > > Petteri Salo
> >
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20161129/6eb1249c/attachment.sig>