[zeromq-dev] ooc bindings for ØMQ

Douglas Creager dcreager at dcreager.net
Mon Jun 21 19:20:32 CEST 2010

> Ok. I see. So basically, you want messages to be load-balanced by 0MQ 
> but at the same time ensure that all messages with a particular 
> "subject" (whatever that means) go to the same downstream component.

Yep, that's the behavior we need — though I should say that I'm not
necessarily asking that 0MQ support this directly.  You had asked if the
current UP/DOWNSTREAM sockets handle all pipeline/workflow use cases.
In our case, they don't.  We're handling this kind of partitioning at
the application level with an extra component:

                                     /-->| Sum |
                                     |   +-----+
                      +-----------+  |
   +--------------+   |           |--/   +-----+
   | Read from DB |-->| Partition |----->| Sum |
   +--------------+   |           |--\   +-----+
                      +-----------+  |
                                     |   +-----+
                                     \-->| Sum |

Note that there are three separate 0MQ connections leaving the Partition
component, instead of one connection with three downstream sockets.  All
of the connections are PAIR sockets, since we only ever have one
component on either end.

> The problem I see with this scenario is that it doesn't comply with 
> basic "scalability" requirement that underlies all 0MQ messaging 
> patterns (except PAIR socket... ugh).
> "Scalability" requirement says that when your distributed system is not 
> catching up with load you can add more boxes on the fly to solve it 
> (think of buying more processing time in the cloud).
> With the above scenario adding a new box wouldn't help. The messages 
> have to be deterministically dispatched from the very beginning. Thus no 
> message would be ever dispatched to the newly added box.
> Can we possibly console the requirement for "grouping" with "scalability"?

Yep, that's definitely an issue.  For us, these aren't long running
processes, though, so we don't need hot-swap scalability.  Think batch
processing of a large pile of data, instead of continuous processing of
a never-ending stream.  An analyst generates one of these workflows to
query a handful of traffic analysis data repositories.  If we need to
add a new box, the currently executing queries won't take advantage of
it, but any new queries will.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20100621/871e34f3/attachment.sig>

More information about the zeromq-dev mailing list