[zeromq-dev] LRU broker queue in intuitive way on 3.0

[Andrew Hume]

i like creative ways to solve problems as much as the next sapient,
but this problem (optimal job scheduling) is much more complicated
than that, and to run well, requires adaptive scheduling stuff in realtime.
you can't really do it with just load balancing and back pressure.

how accurately can you predict teh runtime of a task?
how accurately can you predict the task capacity of a server or worker?
what is the model relating task capacity, workers and servers?
is there a significant overhead to starting and documenting a task?
do you have a real job scheduler? (or are you trying to wrench this functionality
out of 0mq?)
do you need to consider failures (or servers, workers, ...)?

if you have the answers to these, i can advise you on a path forward.

	andrew

On Apr 4, 2011, at 8:16 PM, Henry Baragar wrote:
> I think that this is the pattern that I am trying to figure out if it would be easy to implement in zeromq.
> Here is my use case...
> I have a "day end" task that can be split up into 200 subtask and I want the task to run as fast as possible. The interesting thing is that the longest subtask could take 10K times as long to run as the shortest, and I have information that allows me to sort them in descending order from longest to shortest run time.
> I want the task manager (the Boss) to hand out subtasks to workers one at a time, so that the first worker to connect gets the longest running task, the second one gets second longest, etc.  I would only start 4 workers, so I want to assign the fifth longest task to the first available worker, which probably would be the fourth worker.  The sixth longest subtask would not be assigned until there was an available worker, etc. until all the subtasks are complete, at which point all the processes would be shut down.
> Oh, if the task is taking too long, I would like to be able to add workers as needed (probably on other servers).
> Would this BOSS/WORKER pattern you envision address my use case?
> Regards,
> Henry
> 
> -- 
> Henry Baragar
> Instantiated Software
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev


------------------
Andrew Hume  (best -> Telework) +1 623-551-2845
andrew at research.att.com  (Work) +1 973-236-2014
AT&T Labs - Research; member of USENIX and LOPSA




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110404/c5e70e94/attachment.htm>