[zeromq-dev] Measuring PUB/SUB performance on resource-constrained devices

[J.S. Lischeid]

Dear ZeroMQ community,

Are there any established throughput benchmarking practices for PUB/SUB on resource-constrained devices that are possibly bottlenecked by CPU/memory consumption instead of network bandwidth?

I'm asking because I'm trying to benchmark an IoT messaging middleware that uses ZMQ PUB/SUB queues under the hood. More specifically, I'm trying to find the maximum theoretical throughput for given hardware configurations (e.g. 256MB/512MB/1GB RAM, different CPU speeds, network interfaces).

These are my thoughts so far:
- Ideally, you'd want to keep the publisher-side ZMQ-internal message queue filled with a low number of messages throughout the benchmarking interval. There's not enough memory on the devices to keep it filled with a high number of larger messages (KB-MB range) but you'd also want to avoid having an empty queue at any time since you're missing out on send operations you could do in the meantime (for small messages, there also might be batching advantages when having > 1 message in the queue).
- ZMQ does not expose the internal queue fill level.
- But just spamming a PUB socket with a low high water mark also distorts measurements because it introduces middleware overhead for messages that will not be sent eventually (probably especially important on uniprocessors).
- My currently favoured approach is performing a (binary) search for the maximum number of messages that can be transferred in a given time frame by evenly spacing out (small batches of) messages and sending the producer thread to sleep in between. 

Do you have any thoughts on this or has someone here encountered a similar problem in the past?

Thanks in advance!

Julius