[zeromq-dev] Design for a Pub/Sub system

Bill Torpey wallstprog at gmail.com
Tue Nov 3 16:06:58 CET 2020


> In any case, one question of mine has remained unanswered so far:
> How does one realize horizontal scaling? So far everyone seems to assume a single machine/node.
> 
> Regardless of the socket type used, I will run into some limit at some point. Then I need to add nodes to the cluster. How is the load balanced between them?

Maybe not … it sounds like you’re sending a small number of small messages infrequently, but to a VERY large number of recipients.

With a point-to-point protocol like TCP you’re correct — you’ll hit some limit on what a single machine can handle, so will need to scale horizontally, which can be quite tricky.

Which is the beauty of multicast — you’re leveraging the network itself to do the fan-out.  So multicast may enable you to avoid scalability problems, but only if multicast works in your use case.

If you need to go point-to-point, then you will likely need to do some kind of “sharding”, and like anything else simpler solutions are, well, simpler.  For instance, when you exceed what a single machine can handle (quite a lot depending on the machine), then you go to two machines, then four, etc.  At each step you need some kind of maintenance window to re-balance traffic across servers, and some criteria to decide which recipients belong in which “shard”.

You can also make the sharding dynamic, but that is much more work — the advantage in your case is that it sounds like the traffic is not “transactional”, so you don’t care if a message gets delivered more than once, which makes the dynamic approach more feasible (but still tricky).





More information about the zeromq-dev mailing list