[zeromq-dev] Mb vs. MB in tests & on the Wiki

[Holger Hoffstätte]

Martin Sustrik wrote:
>> ..sigh! You were right of course. I keep getting 60-70 MB/sec with the
>> local_thr test over localhost:lo and messages from ~1k - 8k. With 32k and
>> above it levels out at ~100MB/sec
> 
> I assume you've meant Mb (this B/b thing is really confusing) which

No, I meant megabyte as in MB. See numbers below.

> equals to some 7500 messages a second. Even with 60MB is would be 60,000
> msgs/sec which is really unacceptable.

I know people who would be happy to get that from their Java bloatware. :D

> On what kind of hardware are you running?

2.2GB RAM, 2.4GHz P4 Northwood with 512k cache (no hyperthreading), Gentoo
Linux "pimped out", gcc 4.3.2, glibc 2.8, plain 2.6.27 kernel with 250HZ
and voluntary preemption. It's my file/postgres/mail/news/web server with
otherwise 0.1% load. Fast (for a ) and stable as a rock. RAM is "only"
DDR266 though, so a bit slow by todays standards.

> I've tried to run on loopback on a single-core P4 box and this is what
> I've got:
> 
> threads: 1
> message size: 1024 [B]
> message count: 1000000
> Your average throughput is 196822 [msg/s]
> Your average throughput is 1612 [Mb/s]

Run "A":
holger>./local_thr localhost lo 1024 1000000 1
threads: 1
message size: 1024 [B]
message count: 1000000
Your average throughput is 99298 [msg/s]
Your average throughput is 813 [Mb/s]

With larger message size:

Run "B":
holger>./local_thr localhost lo 262144 1000 1
threads: 1
message size: 262144 [B]
message count: 1000
Your average throughput is 583 [msg/s]
Your average throughput is 1222 [Mb/s]

> An idea: What message count are you using? Small amount of messages in
> the test (<100000) may cause throughput figure to change unpredictably
> from run to run...

I always run several tests to get an average.

>> and sometimes fails, I assume with OOMs
>> (zmq::raw_message_init: no msg_->content).
> 
> We haven't seen this kind of behaviour. Looks like some nasty bug. Would
> you mind running debug version (export CXXFLAGS="-g -O0") under gdb and
> sending us backtraces? Optionally, you can report the bug at
> jira.fastmq.org.

I just checked again with a top in aother window - it is indeed an OOM.
During run "A", remote_thr grows to about ~1.3GB rsize but finishes.
local_thr stays constant in both runs (as far as I can tell). If I
increase the number of messages or their size, remote_thr will repeatably
run out of memory and fail in asserts, mostly like this:

holger>./remote_thr localhost 131072 100000 1
threads: 1
message size: 131072 [B]
message count: 100000

lt-remote_thr: ../../../zmq/raw_message.hpp:83: void
zmq::raw_message_init(zmq::raw_message_t*, size_t): Assertion
`msg_->content' failed.
zsh: abort      ./remote_thr localhost 131072 100000 1

I'll file a bug in JIRA, though I'm not sure what else I can provide - I
just run the examples, though I do have a picture of top that shows
remote_thr at 1.8GB as proof.

>> FWIW lmbench says this rusty tin pushes ~1.1GB (not bit) between
>> processes
>> with AF_UNIX but of course that is without zmq overhead. Still, shouldn't
>> throughput over loopback be higher than ~60-70 MB even with only one
>> core?
> 
> Yup. See the figures above.

Well I now get ~152MB and your 1612Mb are ~200MB so we're not that far
apart anymore, which is good.

>> Then again, latency is much more interesting to me anyway. I'll run some
>> tests on a dual-core machine with the -rt kernel later and can report
>> back
>> if it yields anthing useful.
> 
> We are doing tests on RT kernel right now, so you joining the effort
> would be helpful.

Sure, see my other mail for preliminary results. They confirm what you
write below so that's good.

> In overall it looks like you won't see better latencies for RT kernel.
> Presumably, they'll be a little bit hgiher, however, the jitter should
> be  much lower. To test this you need a specific test case that measures
> latency for *each* message + a statistical tool to compute metrics from
> the mass of output data. We can share that with you in case you are
> interested.

I'd be happy to run that and report back whatever it tells me, so please
feel free to send. Currently collecting data is a bit of a pain and with
-rt you even get close to the regular clock granularities, with skew and
all, not to mention that I'm running on a regular notebook and not some
specialized realtime HW.

cheers!
Holger