[zeromq-dev] On hooking memory allocations

Tue Nov 29 12:39:24 CET 2016

Hi,

Few questions:
1. PR == Pull Request ? git and github are pretty much unknown to me so I'm
just checking my assumptions...
2. From https://rfc.zeromq.org/spec:42/C4/ section 2.3.6, I assume the
"principle target platform" is Linux?

-Petteri

On Tue, Nov 29, 2016 at 12:51 PM, Luca Boccassi <luca.boccassi at gmail.com>
wrote:

> That works on Linux (and I guess *NIX) yes, but I'm not sure if Windows
> has a similar functionality. Even if it does, the use case here (a game
> engine/library) is most likely a statically linked binary, so I think it
> won't work there :-)
>
> All in all this is an interesting discussion, but let's see some PRs and
> take it from there :-)
>
> On Mon, 2016-11-28 at 11:48 -0800, Max Kozlovsky wrote:
> > Each program can be linked with a separate malloc implementation if the
> > user so desires. Libraries don't need to be aware which implementation it
> > is. Different malloc implementation can be even substituted at run time
> on
> > platforms with dynamic linking support (LD_PRELOAD etc).
> >
> > On Mon, Nov 28, 2016 at 11:39 AM, Jens Auer <jens.auer at betaversion.net>
> > wrote:
> >
> > > Hi,
> > >
> > >
> > >
> > > yes and no. If you overwrite it globally at compute-time every program
> on
> > > the system has to use your custom implementation. So if you deliver
> your
> > > ZeroMQ library with your program it will work, but what if my program
> wants
> > > a different custom allocator?
> > >
> > >
> > >
> > > Cheers,
> > >
> > > Jens
> > >
> > >
> > >
> > > *Von:* zeromq-dev [mailto:zeromq-dev-bounces at lists.zeromq.org] *Im
> > > Auftrag von *Max Kozlovsky
> > > *Gesendet:* Montag, 28. November 2016 20:36
> > > *An:* ZeroMQ development list
> > > *Betreff:* Re: [zeromq-dev] On hooking memory allocations
> > >
> > >
> > >
> > > Hi,
> > >
> > >
> > >
> > > Would not globally overwriting malloc/free with the custom
> implementation
> > > achieve the desired behavior (instead of providing hooks for installing
> > > malloc overrides in each and every library)?
> > >
> > >
> > >
> > > Max
> > >
> > >
> > >
> > > On Mon, Nov 28, 2016 at 5:08 AM, Auer, Jens <jens.auer at cgi.com> wrote:
> > >
> > > Hi,
> > >
> > > I don't see a big problem with the C API except that C doesn't support
> > > overloads. So if the function has a new name, e.g.
> > > zmq_ctx_new_with_allocator, everything stays plain C. The default
> instance
> > > would be a
> > >
> > > static void* malloc_(size_t n, void*) {return malloc(n);}
> > > static void free_(void* ptr, size_t n, void*) {free(ptr);}
> > >
> > > allocator_t alloc{
> > > NULL,
> > > malloc_,
> > > free_
> > > };
> > >
> > > context_t then stores the member and gets methods to forward memory
> > > allocations to the function pointers, passing the hint pointer as an
> > > additional argument.
> > >
> > > In my C++ code, I can then use an allocator
> > > static void* allocate(size_t n, void* obj) {return
> > > static_cast<std::allocator<char>>(obj)->allocate(n); }
> > > static void free_(void* ptr, size_t n, void*obj) {
> > > static_cast<std::allocator<char>>(obj)->deallocate(ptr, n); }
> > >
> > > std::allocator<char> a;
> > > allocator_t zmqAlloc{
> > >    &a,
> > >    allocate,
> > >    free_
> > > };
> > >
> > > void* ctx = zmq_ctx_new_with_allocator(&zmqAlloc);
> > >
> > > I think this should work?
> > >
> > > Best wishes,
> > > Jens
> > >
> > > --
> > > Dr. Jens Auer | CGI | Software Engineer
> > > CGI Deutschland Ltd. & Co. KG
> > > Rheinstraße 95 | 64295 Darmstadt | Germany
> > > T: +49 6151 36860 154
> > > jens.auer at cgi.com
> > > Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie
> > > unter de.cgi.com/pflichtangaben.
> > >
> > > CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging
> to
> > > CGI Group Inc. and its affiliates may be contained in this message. If
> you
> > > are not a recipient indicated or intended in this message (or
> responsible
> > > for delivery of this message to such person), or you think for any
> reason
> > > that this message may have been addressed to you in error, you may not
> use
> > > or copy or deliver this message to anyone else. In such case, you
> should
> > > destroy this message and are asked to notify the sender by reply
> e-mail.
> > >
> > > > -----Original Message-----
> > > > From: zeromq-dev [mailto:zeromq-dev-bounces at lists.zeromq.org] On
> Behalf
> > > Of
> > > > Luca Boccassi
> > > > Sent: 28 November 2016 12:30
> > > > To: ZeroMQ development list
> > > > Subject: Re: [zeromq-dev] On hooking memory allocations
> > > >
> > > > That would work for an internal API, but given we expose a C API
> > > unfortunately I
> > > > don't think that would work as a public API :-( And I think for this
> use
> > > case they
> > > > would require a public API.
> > > >
> > > > As an external API, a new zmq_ctx_set that takes a callback would
> have
> > > been ideal,
> > > > but it only takes int. So perhaps a new zmq_ctx_set_allocator that
> takes
> > > a callback
> > > > pointer would be the next best.
> > > >
> > > > An alternative would be to have a system similar to what we use for
> the
> > > poll
> > > > implementation (epoll kqueue select etc), but this would be a
> build-time
> > > option,
> > > > and the implementation would have to be checked in, which I don't
> think
> > > is an
> > > > option for this case, right?
> > > >
> > > > On Mon, 2016-11-28 at 10:51 +0000, Auer, Jens wrote:
> > > > > Hi,
> > > > >
> > > > > I am just a user, but I would love to see this change. I have
> thinking
> > > > > about this and I would like to be able to pass a C++ allocator
> object
> > > > > to ZeroMQ, so a simple function hook is not enough. My idea is to
> > > > > define a struct in the interface
> > > > >
> > > > > struct allocator_t
> > > > > {
> > > > >     void* hint;
> > > > >     void* (allocate)(size_t, void*);
> > > > >     void (deallocate)(void*, size_t, void*); };
> > > > >
> > > > > and store this in the context object. Since I don't think that this
> > > should be
> > > > changed during runtime, I would create a new zmq_ctx_new overload
> which
> > > takes a
> > > > parameter of type allocator_t. The default value would be to call
> > > malloc/free.
> > > > >
> > > > > Cheers,
> > > > > Jens
> > > > >
> > > > > --
> > > > > Jens Auer | CGI | Software-Engineer
> > > > > CGI (Germany) GmbH & Co. KG
> > > > > Rheinstraße 95 | 64295 Darmstadt | Germany
> > > > > T: +49 6151 36860 154
> > > > > jens.auer at cgi.com<mailto:jens.auer at cgi.com>
> > > > > Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden
> Sie
> > > unter
> > > > de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>.
> > > > >
> > > > > CONFIDENTIALITY NOTICE: Proprietary/Confidential information
> belonging
> > > to CGI
> > > > Group Inc. and its affiliates may be contained in this message. If
> you
> > > are not a
> > > > recipient indicated or intended in this message (or responsible for
> > > delivery of this
> > > > message to such person), or you think for any reason that this
> message
> > > may have
> > > > been addressed to you in error, you may not use or copy or deliver
> this
> > > message to
> > > > anyone else. In such case, you should destroy this message and are
> asked
> > > to notify
> > > > the sender by reply e-mail.
> > > > > ________________________________
> > > > > Von: zeromq-dev [zeromq-dev-bounces at lists.zeromq.org]" im Auftrag
> von
> > > > > "Petteri Salo [petteri.salo at gmail.com]
> > > > > Gesendet: Montag, 28. November 2016 09:40
> > > > > An: zeromq-dev at lists.zeromq.org
> > > > > Betreff: [zeromq-dev] On hooking memory allocations
> > > > >
> > > > > Hello,
> > > > >
> > > > > Let me first do a little introduction as I'm new to this list. I'm
> a
> > > software engineer
> > > > with 15+ years of experience working on games at a company called
> Remedy
> > > > Entertainment Ltd. We've done games for PC, and various games
> consoles
> > > over the
> > > > years. Most recently we did Quantum Break for Xbox One.
> > > > >
> > > > > I've now been tasked with evaluating ZeroMQ. One important feature
> of
> > > any
> > > > library we use in games is the ability to hook all memory
> allocations -
> > > this is to allow
> > > > the use of custom memory allocators and/or for tracking when and
> where
> > > memory is
> > > > allocated.
> > > > >
> > > > > I've searched the libzmq source code and there is about 150 uses of
> > > new, malloc,
> > > > realloc , etc.
> > > > >
> > > > > If we were to adopt libzmq we'd like to put in allocation hooks and
> > > that work
> > > > would then be something that we'd like to contribute back to the
> > > project. Having
> > > > those hooks in the main repository would then make it easier for us
> to
> > > adopt future
> > > > changes to the library.
> > > > >
> > > > > So, my question is would this kind of change be something that
> would be
> > > > accepted? Of course assuming that coding conventions, proper way of
> > > submitting
> > > > the patch etc. are followed. I do realize that one would want to see
> the
> > > actual code
> > > > before accepting. I'm interested in the principle of accepting a
> change
> > > such as this,
> > > > since it would introduce a new "rule" for working ión libzmq source
> code
> > > : "All
> > > > allocations shall go through an allocation hook."
> > > > >
> > > > > Best Regards,
> > > > >
> > > > > Petteri Salo
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20161129/da31f184/attachment.htm>