[zeromq-dev] Thinking out loud ...
sustrik at 250bpm.com
Sun Jun 26 22:44:13 CEST 2011
On 06/26/2011 04:26 PM, Pieter Hintjens wrote:
> It has to be handled on a pattern-by-pattern basis. My belief today is
> that if you want synchronization over push-pull, you must switch to a
> router-based pattern, in the same way as we use routers to synchronize
> request-reply workers.
My feeling at the moment is that Andew Hume is right when he contrasts
message routing with job scheduling.
The two are different problems. Specifically, job scheduling system
needs to know about computational resources available, possibly even be
able to estimate resource usage associated with specific task etc.
This seems to pretty much out of scope of 0MQ. So, the question is
whether there's a way for 0MQ to be used as an underlying layer to do
routing for job scheduling systems.
Naive approach would be to allow for explicit acks so that there could
be at most one request sent to a specific peer at any single moment.
That would at least allow load-balancer to distinguish "busy" peers from
"idle" peers and base the load-balancing on that.
However, that doesn't seem to play well with intermediate devices. If
there are multiple workers attached to a particular device, it doesn't
make sense to send at most one request a time to that device -- that way
at most one worker would be used at any given moment, which is clearly
Alternative approach would to bubble up number of idle workers
accessible via a particular peer to the load-balancer.
Even more sophisticated approach would be to report "available
resources" metric instead of idle workers count, ie. a big multi-core
box can report more available resource than an embedded device. Etc.
The whole thing gets complex pretty fast. There's definitely more
thinking to be done here.
More information about the zeromq-dev