[zeromq-dev] On hooking memory allocations

[Max Kozlovsky]

Hi,

Would not globally overwriting malloc/free with the custom implementation
achieve the desired behavior (instead of providing hooks for installing
malloc overrides in each and every library)?

Max

On Mon, Nov 28, 2016 at 5:08 AM, Auer, Jens <jens.auer at cgi.com> wrote:

> Hi,
>
> I don't see a big problem with the C API except that C doesn't support
> overloads. So if the function has a new name, e.g.
> zmq_ctx_new_with_allocator, everything stays plain C. The default instance
> would be a
>
> static void* malloc_(size_t n, void*) {return malloc(n);}
> static void free_(void* ptr, size_t n, void*) {free(ptr);}
>
> allocator_t alloc{
> NULL,
> malloc_,
> free_
> };
>
> context_t then stores the member and gets methods to forward memory
> allocations to the function pointers, passing the hint pointer as an
> additional argument.
>
> In my C++ code, I can then use an allocator
> static void* allocate(size_t n, void* obj) {return
> static_cast<std::allocator<char>>(obj)->allocate(n); }
> static void free_(void* ptr, size_t n, void*obj) {
> static_cast<std::allocator<char>>(obj)->deallocate(ptr, n); }
>
> std::allocator<char> a;
> allocator_t zmqAlloc{
>    &a,
>    allocate,
>    free_
> };
>
> void* ctx = zmq_ctx_new_with_allocator(&zmqAlloc);
>
> I think this should work?
>
> Best wishes,
> Jens
>
> --
> Dr. Jens Auer | CGI | Software Engineer
> CGI Deutschland Ltd. & Co. KG
> Rheinstraße 95 | 64295 Darmstadt | Germany
> T: +49 6151 36860 154
> jens.auer at cgi.com
> Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie
> unter de.cgi.com/pflichtangaben.
>
> CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to
> CGI Group Inc. and its affiliates may be contained in this message. If you
> are not a recipient indicated or intended in this message (or responsible
> for delivery of this message to such person), or you think for any reason
> that this message may have been addressed to you in error, you may not use
> or copy or deliver this message to anyone else. In such case, you should
> destroy this message and are asked to notify the sender by reply e-mail.
>
>
> > -----Original Message-----
> > From: zeromq-dev [mailto:zeromq-dev-bounces at lists.zeromq.org] On Behalf
> Of
> > Luca Boccassi
> > Sent: 28 November 2016 12:30
> > To: ZeroMQ development list
> > Subject: Re: [zeromq-dev] On hooking memory allocations
> >
> > That would work for an internal API, but given we expose a C API
> unfortunately I
> > don't think that would work as a public API :-( And I think for this use
> case they
> > would require a public API.
> >
> > As an external API, a new zmq_ctx_set that takes a callback would have
> been ideal,
> > but it only takes int. So perhaps a new zmq_ctx_set_allocator that takes
> a callback
> > pointer would be the next best.
> >
> > An alternative would be to have a system similar to what we use for the
> poll
> > implementation (epoll kqueue select etc), but this would be a build-time
> option,
> > and the implementation would have to be checked in, which I don't think
> is an
> > option for this case, right?
> >
> > On Mon, 2016-11-28 at 10:51 +0000, Auer, Jens wrote:
> > > Hi,
> > >
> > > I am just a user, but I would love to see this change. I have thinking
> > > about this and I would like to be able to pass a C++ allocator object
> > > to ZeroMQ, so a simple function hook is not enough. My idea is to
> > > define a struct in the interface
> > >
> > > struct allocator_t
> > > {
> > >     void* hint;
> > >     void* (allocate)(size_t, void*);
> > >     void (deallocate)(void*, size_t, void*); };
> > >
> > > and store this in the context object. Since I don't think that this
> should be
> > changed during runtime, I would create a new zmq_ctx_new overload which
> takes a
> > parameter of type allocator_t. The default value would be to call
> malloc/free.
> > >
> > > Cheers,
> > > Jens
> > >
> > > --
> > > Jens Auer | CGI | Software-Engineer
> > > CGI (Germany) GmbH & Co. KG
> > > Rheinstraße 95 | 64295 Darmstadt | Germany
> > > T: +49 6151 36860 154
> > > jens.auer at cgi.com<mailto:jens.auer at cgi.com>
> > > Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie
> unter
> > de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>.
> > >
> > > CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging
> to CGI
> > Group Inc. and its affiliates may be contained in this message. If you
> are not a
> > recipient indicated or intended in this message (or responsible for
> delivery of this
> > message to such person), or you think for any reason that this message
> may have
> > been addressed to you in error, you may not use or copy or deliver this
> message to
> > anyone else. In such case, you should destroy this message and are asked
> to notify
> > the sender by reply e-mail.
> > > ________________________________
> > > Von: zeromq-dev [zeromq-dev-bounces at lists.zeromq.org]" im Auftrag von
> > > "Petteri Salo [petteri.salo at gmail.com]
> > > Gesendet: Montag, 28. November 2016 09:40
> > > An: zeromq-dev at lists.zeromq.org
> > > Betreff: [zeromq-dev] On hooking memory allocations
> > >
> > > Hello,
> > >
> > > Let me first do a little introduction as I'm new to this list. I'm a
> software engineer
> > with 15+ years of experience working on games at a company called Remedy
> > Entertainment Ltd. We've done games for PC, and various games consoles
> over the
> > years. Most recently we did Quantum Break for Xbox One.
> > >
> > > I've now been tasked with evaluating ZeroMQ. One important feature of
> any
> > library we use in games is the ability to hook all memory allocations -
> this is to allow
> > the use of custom memory allocators and/or for tracking when and where
> memory is
> > allocated.
> > >
> > > I've searched the libzmq source code and there is about 150 uses of
> new, malloc,
> > realloc , etc.
> > >
> > > If we were to adopt libzmq we'd like to put in allocation hooks and
> that work
> > would then be something that we'd like to contribute back to the
> project. Having
> > those hooks in the main repository would then make it easier for us to
> adopt future
> > changes to the library.
> > >
> > > So, my question is would this kind of change be something that would be
> > accepted? Of course assuming that coding conventions, proper way of
> submitting
> > the patch etc. are followed. I do realize that one would want to see the
> actual code
> > before accepting. I'm interested in the principle of accepting a change
> such as this,
> > since it would introduce a new "rule" for working ión libzmq source code
> : "All
> > allocations shall go through an allocation hook."
> > >
> > > Best Regards,
> > >
> > > Petteri Salo
> > >
> > >
> > > _______________________________________________
> > > zeromq-dev mailing list
> > > zeromq-dev at lists.zeromq.org
> > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20161128/614c017d/attachment.htm>