[zeromq-dev] How to scale?
sustrik at fastmq.com
Wed Feb 25 09:38:33 CET 2009
> I have a question about how to scale with 0MQ. Suppose I have a node N1
> which serves m clients and sends messages via 0MQ to another node N2
> which does some business logic and sends messages back to N1 who in turn
> speaks with its clients.
>>From what I understood, N1 and N2 would both declare an exchange and a
> queue for bidirectional communication.
> Now lets say the load on N1 becomes too high and another node N3 is
> added, behaving the same as N1, i.e. sends to N2 and talks to its
> clients. But N2 doesn't know about this added node N3 and I wonder how
> to setup 0MQ networks in a cluster-like manner in order to scale
You should have a look at the trunk. There's a "butterfly" example
(examples/butterfly) that does exactly the thing you want. We're writing
a tutorial for this use case at the moment, but it'll take some time to
get it released. In the meantime, I'm attaching diagram of "butterfy"
do_a and do_b components are doing the actual work. They have no global
queues or exchanges. They connect to send_request, intermediate and
receive_replies applications - those are singletons exposing global
queues and/or exchanges. You can run as many do_a's and do_b's as needed
to handle the load.
> Also I have read the following comment about 0MQ:
> "The real beauty of ZeroMQ is the way it lets developers build fast
> multithreaded applications without ever worrying about how to scale -
> everything works as asynchronous messages, between processes on one
> machine, or processes distributed across wide networks." 
> "ØMQ is fully distributed: no central servers to crash, millions of WAN
> and LAN nodes." 
> This sounds quite a bit like Erlang-style concurrency.
Yes, 0MQ is similar to Erlang in a way, except that it is meant as an
language-agnostic infrastructure rather then a programming language.
> Does that mean it
> would actually make more sense to break an application into smaller
> parts and use 0MQ for this inter-process communication?
Breaking the application into smaller parts can make sense, however,
finding out whether is would make any difference is kind of complex.
We'll cover this problem in "butterfly" tutorial (if you are interested
in reviewing the text, I'll send you the link).
Using 0MQ for inter-process and inter-thread communication is strongly
encouraged. There are three important reasons to adopt this way of
writing the code:
1. It makes your business logic single-threaded. When you are writing it
you can focus on what the thing actually does rather then polluting the
code with unpenetrable and error-prone synchronisation code.
2. Once one process is not able to handle the load. You can simply move
one (or several) threads to a different box. The messages are passed the
same way whether they are passed in-process, between processes or
between different machines.
3. Thread synchronisation is much more efficient with 0MQ than with
standard critical section based approach.
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 14086 bytes
Desc: not available
More information about the zeromq-dev