[zeromq-dev] LRU broker queue in intuitive way on 3.0

[Henry Baragar]

On April 4, 2011 11:35:25 pm Andrew Hume wrote:
> i like creative ways to solve problems as much as the next sapient,
> but this problem (optimal job scheduling) is much more complicated
> than that, and to run well, requires adaptive scheduling stuff in realtime.
> you can't really do it with just load balancing and back pressure.
> 

Right now, all the subtasks are being run on a single server in sequential 
order in a single process.  Currently, the overall task is completing within 
its operational window, but the transaction volumes are going at 20% per 
month.  My initial thoughts for a solution were to fork a few subprocesses and 
use polling to find the next available subprocess.


> how accurately can you predict teh runtime of a task?
I know the transaction volumes that need to be processed by each subtask.

> how accurately can you predict the task capacity of a server or worker?
Initially, I would like to be able to take advantage of all 4 CPU's on the 
server, meaning the capacities should be identical.  Until I saturate the 
single server (potentially with the addition of more CPU's) I don't need to 
worry about where the other workers will be run.

> what is the model relating task capacity, workers and servers?
I'm not sure that I understand this question.  Currently, I hope to be able to 
run everything on one server.

> is there a significant overhead to starting and documenting a task?
No.

> do you have a real job scheduler? (or are you trying to wrench this
> functionality out of 0mq?)
No. A job scheduler does not seem to be at the correct level of granularity 
for this project and I suspect would add unnecessary complications.

> do you need to consider failures (or servers, workers, ...)?
> 
That would be nice, but essential.  The application already has a number of 
checks to verify that tasks have completed successfully.

Regards,
Henry

> if you have the answers to these, i can advise you on a path forward.
> 
> 	andrew
> 
> On Apr 4, 2011, at 8:16 PM, Henry Baragar wrote:
> > I think that this is the pattern that I am trying to figure out if it
> > would be easy to implement in zeromq. Here is my use case...
> > I have a "day end" task that can be split up into 200 subtask and I want
> > the task to run as fast as possible. The interesting thing is that the
> > longest subtask could take 10K times as long to run as the shortest, and
> > I have information that allows me to sort them in descending order from
> > longest to shortest run time. I want the task manager (the Boss) to hand
> > out subtasks to workers one at a time, so that the first worker to
> > connect gets the longest running task, the second one gets second
> > longest, etc.  I would only start 4 workers, so I want to assign the
> > fifth longest task to the first available worker, which probably would
> > be the fourth worker.  The sixth longest subtask would not be assigned
> > until there was an available worker, etc. until all the subtasks are
> > complete, at which point all the processes would be shut down. Oh, if
> > the task is taking too long, I would like to be able to add workers as
> > needed (probably on other servers). Would this BOSS/WORKER pattern you
> > envision address my use case? Regards,
> > Henry
> 
> ------------------
> Andrew Hume  (best -> Telework) +1 623-551-2845
> andrew at research.att.com  (Work) +1 973-236-2014
> AT&T Labs - Research; member of USENIX and LOPSA

-- 
Henry Baragar
Instantiated Software
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110405/129f73cc/attachment.htm>