[zeromq-dev] PUB-SUB scalability

[Steven Clark]

On Mon, Aug 16, 2010 at 1:16 PM, Pieter Hintjens <ph at imatix.com> wrote:

> On Mon, Aug 16, 2010 at 4:44 PM, Steven Clark <steven.p.clark at gmail.com>
> wrote:
>
> > I'm interested in how well PUB-SUB scales via TCP.
>
> The maths is fairly simple.  The network traffic will be sum P(S x M x Z)
> where:
>
> P is the publisher socket
> S is the number of subscribers
> M is the number of messages per second
> Z is the size of messages on average
>
> Summed up over all publisher sockets.
>
> > Any guesses as to how high N could go before things go haywire? Is the
> > network bandwidth the limiting factor?
>
> Yes, that's all.
>
> > Does the situation improve if I provide a separate PUB socket for each
> > topic?
>
> Nope.  Don't create multiple PUB sockets for this.
>
>
Don't create multiple PUB sockets because it's unnecessarily complex, or
because it wouldn't help the network data rate at all? It seems like doing
so *would* reduce data rate / allow more topics/subscribers, especially if
each subscriber A) was only interested in 1 or 2 topics out of a 1000 at a
time, and B) switched topics very infrequently.

If there were many PUB sockets (1 per topic), then when a new client
connects, the marginal increase in network traffic is only that of the 1 or
2 (low rate) PUB sockets going to one more endpoint. Whereas, with only one
PUB socket, a new client connection = ALL data going to one more endpoint.

Maybe PUB-SUB is the wrong pattern for this type of sparse-topic interest,
if one really wants to maximize scalability? (there's also the security
concern of ALL data going to ALL users, which is a whole 'nother bag of
worms...)

-Steven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20100816/4c683059/attachment.htm>