[zeromq-dev] Memory allocation/Exception

[Pieter Hintjens]

On Tue, Mar 24, 2009 at 12:27 PM, Steven McCoy <steven.mccoy at miru.hk> wrote:

>> Because there's very little you can do when computer is already out of
>> resources. Even simplest attempts to fix the problem tend to fail in
>> that case. The goal is never to let such a thing happen. That's why
>> we've introduced queue limits and (with version 0.6) on-disk offload of
>> the queues that allows you to store gigabytes of data without affecting
>> memory footprint etc.
>
> FYI:  GLib takes the stance of immediately terminating.

OpenAMQ does the same: reduce the risk and terminate immediately if it
happens anyhow.  Very long ago we considered trying to recover from
out of memory situations but it creates very fragile code.

The one thing is: when aborting due to lack of memory, print a useful
error message.  While Linux is generous in allowing processes to use
up all available memory, other UNIX systems often set quite strict
per-process limits that a broker can rapidly hit.  The usual action is
to reconfigure the kernel to raise per-process limits.

In OpenAMQ the low-level allocator prints a nice message if it can't
get memory from the heap.  Presumably you can do the same in 0MQ.

Offloading queues to disk is something you should do only when the
application explicitly asks for it, IMO, since it'll totally change
the performance of the system.  In many cases if a queue gets too
large, that's indicative of a problem that needs fixing (e.g. too slow
client) and offloading to disk won't help.  It'll delay fixing the
issue, not encourage it.

-Pieter