[zeromq-dev] architecture performance issues

[Pieter Hintjens]

On Fri, May 20, 2011 at 11:18 AM, Andrew Hume <andrew at research.att.com> wrote:

> i think i need to rework some of my architecture because its not scaling
> like i need it to.

Could you explain why the forwarders are getting overloaded? What's
your message volume, message size, and work per message in the
forwarder?

Normally, you could expect devices to become bottlenecks, but that's
balanced against them doing much less work per message than endpoints.
And a device will essentially be CPU bound. So if a device starts to
do too much work per message, it'll degrade. Or, if the overall
volumes combine to more than the device can handle, it'll degrade.

Options are therefore to reduce the number of messages, to reduce the
work done per message, in devices, and eventually to remove devices
altogether.

The pattern which perhaps works best for such a network is a single
device for coordination and registration, but all traffic flowing peer
to peer. 80 sockets per node is not an issue, you can open thousands
if needed. The issue is usually to address these, and you can solve
that with a broker / name server device.

It's not been covered in the Guide yet so whatever you come up with,
please do share, it'll be useful to many people.

-Pieter