[zeromq-dev] Understanding ØMQ PubSub Scaling

[dave]

I am looking into products for pub-sub problem that I am working through and
I think thank ZeroMQ might fit the bill. I am curious to understand the
economies of scale that ZeroMQ works upon and how I can efficiently tackle
the problem at hand. My problem is best described with a simple scenario:

10 machines, each acting as both publishers and subscribers.
1 publisher per machine.
10,000 subscribers per machine.
Subscribers are non-blocking and being looped over in a single thread
looking for data.
Messages being passed are relatively small. Let's 1K for the sake of this
scenario.
It is not a pure fan-out scenario. A single subscriber will be interested in
only a subset of the messages published. From the 100,000 subscribers, maybe
1, maybe 20, maybe 80,000 will be interested in the message, depending on
the routing key for the message.
A subscriber/consumer will be interested in several routing keys, but it
will have an explicit list of what those are. There will be no fuzzy
matching on the routing key. A routing key will either match exactly
something in its list or it won't.

How does scenario like this perform, theoretically. Assuming an adequately
beefy machine (whatever that means), if I send a single message to all
100,000 subscribers, how quickly can I expect them to see it as available?
How does memory usage correlate? What happens if I have more than 10,000
subscribers in a single thread? What happens if I have each machine run 4
processes instead of 1, thus having 4 publishers and 40,000 subscribers on a
machine? What happens if I increase the number of machines from 10 to 100,
with 1 million total subscribers?

And what sort of topology best makes this work? Can I setup multicast and do
it all over TCP or should I have relays at each of the servers that
subscribe to each other indiscriminately and then forward the messages with
the routing keys over something like ipc/inproc?

I know that I'm asking a lot and I'm sure that there's more variables I
haven't thought of here. I'm looking for some sage advice or direction that
can help me understand ZeroMQ better and what it can or can't do. Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110111/6414aea2/attachment.htm>