[zeromq-dev] Important: backward incompatible changes for 0MQ/3.0!

Martin Sustrik sustrik at 250bpm.com
Thu Mar 24 19:07:20 CET 2011


On 03/24/2011 06:16 PM, Chuck Remes wrote:

> True, I am looking for a global replacement for all malloc/free
> calls. Even if the existing ones were just replaced by a macro that
> was easily modified at compile time that would make me happy.

Yes. If we put aside tcmalloc-style solution, that would be the only 
remaning solution of the problem.

> I don't agree. While the experiments showed that the yqueue memory
> was being deallocated as expected, it may have still been causing
> some kind of fragmentation that prevented the memory from being
> returned back to the OS.
>
> When I was reading up on jemalloc [1] I noticed a passage early in
> the text regarding allocator-induced fragmentation. If the allocator
> let's too much fragmentation occur, this "leak" could make the
> Facebook application heaps grow beyond the bounds of physical memory
> and cause problems. My use-case is very similar in the sense that I
> have distributed applications that run for days or weeks at a time
> wherein a lot of sockets are created and destroyed over that time
> period. There appears to something about the way 0mq is allocating
> and freeing memory that creates this fragmentation and slow heap
> growth over time. If jemalloc can solve it, then I'm very
> interested.
>
> Note that even though some of the 0mq allocations are 8kB or 12kB,
> unless they are page aligned then even these sizes could cause
> fragmentation. If even one byte on a page is being held by the
> process, it cannot be relinquished to the OS hence the heap
> fragmentation.

Ok. I have no strong opinion here. We can proceed by allowing to 
redefine malloc/free at compile time, as discussed above and you can 
check whether alternate mechanism actually solves your problem.

Martin



More information about the zeromq-dev mailing list