[zeromq-dev] Thinking out loud ...

[Martin Sustrik]

On 06/26/2011 04:26 PM, Pieter Hintjens wrote:

> It has to be handled on a pattern-by-pattern basis. My belief today is
> that if you want synchronization over push-pull, you must switch to a
> router-based pattern, in the same way as we use routers to synchronize
> request-reply workers.

My feeling at the moment is that Andew Hume is right when he contrasts 
message routing with job scheduling.

The two are different problems. Specifically, job scheduling system 
needs to know about computational resources available, possibly even be 
able to estimate resource usage associated with specific task etc.

This seems to pretty much out of scope of 0MQ. So, the question is 
whether there's a way for 0MQ to be used as an underlying layer to do 
routing for job scheduling systems.

Naive approach would be to allow for explicit acks so that there could 
be at most one request sent to a specific peer at any single moment. 
That would at least allow load-balancer to distinguish "busy" peers from 
"idle" peers and base the load-balancing on that.

However, that doesn't seem to play well with intermediate devices. If 
there are multiple workers attached to a particular device, it doesn't 
make sense to send at most one request a time to that device -- that way 
at most one worker would be used at any given moment, which is clearly 
suboptimal.

Alternative approach would to bubble up number of idle workers 
accessible via a particular peer to the load-balancer.

Even more sophisticated approach would be to report "available 
resources" metric instead of idle workers count, ie. a big multi-core 
box can report more available resource than an embedded device. Etc.

The whole thing gets complex pretty fast. There's definitely more 
thinking to be done here.

Martin