[zeromq-dev] ooc bindings for ØMQ

Martin Sustrik sustrik at 250bpm.com
Sat Jun 19 09:11:13 CEST 2010


Peter Alexander wrote:

> The first is a very flexible, even autonomous load
> distribution/balancing scheme(s) that take into consideration various
> aspects of distributed pipeline processing like the distribution of
> components (local, lan, wan, etc) as well as the various individual
> machine characteristics like cpu, memory, etc. Currently I believe,
> only a round-robin scheme is available.

What's currently done is that size of the queue leading to a downstream 
component can be limited (ZMQ_HWM socket option). Thus, if the component 
is not catching up, its queue gets full and the scheduler stops sending 
more requests to the component. Instead it dispatches requests to the 
components that are working OK.

What's your idea with the resource-monitoring approach?

Btw, one thing that would be worth implementing in the scheduler IMO are 
priorities. That way you would be able to say: load balance the work 
among components A, B and C and only if none of those is available use 
component D (which may be off-site or too expensive etc.)

> The second is a transparent standard-socket adapter, which I believe
> is what you have in mind and are currently planning. I think atm only
> zmq_poll() is available for adapting a standard-socket into the zmq
> system.

Yes. That's what I am working on although it's technically impossible in 
POSIX-y environment :)

Martin



More information about the zeromq-dev mailing list