[zeromq-dev] Important: backward incompatible changes for 0MQ/3.0!

[Chuck Remes]

On Mar 25, 2011, at 1:14 AM, Martin Sustrik wrote:

> Hi all,
> 
> As already noted there are 2 disctinct problems. One is using a better 
> performing allocator for everything to alleviate performance issues in 
> built-in allocator provided by OS, the other is using a specific 
> user-supplied allocator for message bodies.
> 
> Let's focus on the former now, ignoring the latter for a while so that 
> the discussion doesn't get confused.
> 
> 1. The performance problem (along the CPU/memory tradeoff) happens on 
> process level (process consuming too much memory etc.) not on library 
> level. All the libraries loaded to the process contribute to the 
> problem. Thus, the solution should be preferably at process level.

I ran a *long* test (2 days) using the jemalloc allocator on linux (i.e. replacing the allocator at the process level).

It improved the usage and allocation of heap memory by 0mq by about 50%. The heap still slowly grew due to fragmentation, but with jemalloc the growth was about twice as slow as the standard glibc malloc. This is a partial win.

Unfortunately, jemalloc didn't solve the problem.

Looking deeper into the issue, it looks like the long-term fix is to use a SLAB allocator [1]. Each 0mq structure would essentially get its own Cache + Slab for doing memory allocations. This would *eliminate* memory fragmentation using the exact same techniques that kernels use to avoid it for their own internals structures. If 0mq is still intended for inclusion in the kernel some day, it might be a good idea to start exercising these techniques now.

I have a few other things in front of this (primarily doing a spike using cxxtest to do some low-level unit testing of 0mq). In a few weeks/months I'll see about putting together a patch that illustrates the concept. I'll pick one specific 0mq structure and do all memory allocation/deallocation using a SLAB of some kind.

cr

[1] http://en.wikipedia.org/wiki/Slab_allocation