[zeromq-dev] ooc bindings for ØMQ

Douglas Creager dcreager at dcreager.net
Fri Jun 18 22:45:46 CEST 2010


>> If you have theoretical background in this area, please do check
>> UPSTREAM/DOWNSTEAM sockets and let me know if you believe something is
>> broken or missing there.
> 
> The first is a very flexible, even autonomous load
> distribution/balancing scheme(s) that take into consideration various
> aspects of distributed pipeline processing like the distribution of
> components (local, lan, wan, etc) as well as the various individual
> machine characteristics like cpu, memory, etc. Currently I believe,
> only a round-robin scheme is available.

I'm developing something similar for the network analytics company I work for.  Unfortunately, you can't always rely on the messaging layer to do the partitioning for you — in a couple of our use cases, we're doing some key-based aggregation, and so we have to ensure that all records with the same key get sent to the same downstream partition.  In theory, you could have an UP/DOWNSTREAM socket handle this for you, but you'd have to pass in a hash of the key for every message.  Maybe at some point that becomes another message option?

In the meantime, I just have all partitioning — both round-robin and key-based — happen at the application layer in separate components.  That means that every 0mq socket is single-source, single-sink, which means (I think, correct me if I'm wrong) that I can get away with using PAIR sockets instead UP/STREAM sockets.

–doug


More information about the zeromq-dev mailing list