[zeromq-dev] ooc bindings for ØMQ

Douglas Creager dcreager at dcreager.net
Mon Jun 21 16:19:24 CEST 2010

> Does that mean you have a cluster of components that processes requests 
> of type X, another cluster that processes requests of type Y and so on?

Not really, I think.  Here's a simplified picture:

  +--------------+     +-----+
  | Read from DB |---->| Sum |+
  +--------------+     +-----+|

So the first component reads some records from a database (or file or
whatever), and we want to calculate the equivalent of an SQL SUM/GROUP
BY.  Each record is an individual 0MQ message.  If the Read component
outputs the following list of records:

  Key   Value
   A      1
   B      3
   A      1
   A      2
   B      4

then we want the final result to be <(A,4), (B,7)>.

We've got multiple copies of the Sum component, so that we can process a
huge number of records in parallel.  If we implement the arrow as an
UP/DOWNSTREAM connection, with multiple sockets on the downstream end,
then the records will be round-robin distributed to the different Sums.
The first Sum will give us an output of <(A,2), (B,4)>, and the second
Sum will give us <(A,2), (B,3)>.

To get the right answer, the part that does the partitioning has to
ensure that all "A" records go to the same Sum instance.  This is
usually done by hashing the key and using that to choose which instance
to send each record to.  (That way the partitioner can be stateless, so
we can scale up to any number of records.)  So if we want 0MQ to handle
this behavior, we'd have to pass in this hash value when we submit a
message using zmq_send.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20100621/c017f69a/attachment.sig>

More information about the zeromq-dev mailing list