[zeromq-dev] ZMQ 2.1.0 and OpenPGM Memory Usage

Bob Beaty bobbeaty at comcast.net
Fri Dec 3 16:56:08 CET 2010

  I've been working on what I *thought* was a memory leak in the latest
ZeroMQ (2.1.0) using OpenPGM URLs - "epgm://". Good news - it's not a
memory leak! Bad news - the memory usage of ZMQ depends radically on the
value of ZMQ_RATE. And not in a good way.
  I've created a little test application: https://gist.github.com/726145
  In the comments I've laid out the tests I've done, but I'll repeat them
here as well...

  - Running on CentOS 5
  - Using latest ZMQ 2.1.0
  - 10GbE NICs
  - 8 cores... 32 GB RAM (not that this matters)

  - the code sends the SAME message over and over again. Every 1000 times
    it logs the fact that it did another 1000.
  - the speed is approximately 64kbps as regulated by the usleep() in
    the loop. I wanted to keep it under the default 100kbps to be safe.

  After the first 1000 messages, the memory utilization should remain
constant. After all... it's the SAME message, and it's creating and
destroying the zmq::message_t so it should be a zero-sum game after it's
done the first 1000 or so. OK... maybe 2000. But it should "level off"
at a "terminal memory footprint" that's not that far off from the value
it has after the first few thousand.
  Secondly, the size of the ZMQ_RATE option should have *no* effect on
the memory size assuming that the actual rate is *below* the stated value.
In my case, sending less than 100kbps means any value of ZMQ_RATE greater
than the default (100) should have no effect whatsoever on the terminal
memory value.

  This is where it gets interesting.
  Below is the table of ZMQ_RATE size versus the "terminal memory
footprint" of the application.

	bps       ZMQ_RATE     Size (after 2000 ... terminal)
	10Mbps      10000         7MB ... 18MB
	50Mbps      50000         7MB ... 73MB
	200Mbps    200000         7MB ... 280MB

  At this point I realized that it's not a leak - it's something like a
buffer growth related to the anticipated need based on the maximum speed.

  At this point I'm stumped. I know it's always an option to go into the
code and start trying to find the cause of this behavior, but I wanted to
throw these test results out to the mailing list to see if anyone really
familiar with the OpenPGM usage of the ZMQ_RATE parameter can shed some
  I'm not adverse to digging into the code, but if there's someone that
can point me to the right file(s), class(es), or method(s), to make it
a little easier for me, that would be fantastic.

        Bob (drbob at TheManFromSPUD.com)
    The Man from S.P.U.D.
    We will write no code before it's designed.

More information about the zeromq-dev mailing list