[zeromq-dev] Memory allocation/Exception

[Martin Sustrik]

Pieter Hintjens wrote:
> On Tue, Mar 24, 2009 at 12:27 PM, Steven McCoy <steven.mccoy at miru.hk> wrote:
> 
>>> Because there's very little you can do when computer is already out of
>>> resources. Even simplest attempts to fix the problem tend to fail in
>>> that case. The goal is never to let such a thing happen. That's why
>>> we've introduced queue limits and (with version 0.6) on-disk offload of
>>> the queues that allows you to store gigabytes of data without affecting
>>> memory footprint etc.
>> FYI:  GLib takes the stance of immediately terminating.
> 
> OpenAMQ does the same: reduce the risk and terminate immediately if it
> happens anyhow.  Very long ago we considered trying to recover from
> out of memory situations but it creates very fragile code.
> 
> The one thing is: when aborting due to lack of memory, print a useful
> error message.  While Linux is generous in allowing processes to use
> up all available memory, other UNIX systems often set quite strict
> per-process limits that a broker can rapidly hit.  The usual action is
> to reconfigure the kernel to raise per-process limits.
> 
> In OpenAMQ the low-level allocator prints a nice message if it can't
> get memory from the heap.  Presumably you can do the same in 0MQ.
> 
> Offloading queues to disk is something you should do only when the
> application explicitly asks for it, IMO, since it'll totally change
> the performance of the system.  In many cases if a queue gets too
> large, that's indicative of a problem that needs fixing (e.g. too slow
> client) and offloading to disk won't help.  It'll delay fixing the
> issue, not encourage it.

Yes, it's optional. By default it's switched off.

The use case behind the functionality is to be able to handle unexpected 
peeks in traffic without loosing messages. Thus, if your system is 
slashdotted, the requests are saved on disk and processed gradually. 
However, if the memory limits are not exceeded, there's no disk access 
involved altogether, keeping the system fast.

Martin