[zeromq-dev] LRU broker queue in intuitive way on 3.0

[Henry Baragar]

On April 6, 2011 09:14:28 am Andrew Hume wrote:
> On Apr 6, 2011, at 9:00 AM, Henry Baragar wrote:
> > On April 5, 2011 10:30:31 am Andrew Hume wrote:
> > > given this degenerate case of a single server and no per-task overhead,
> > > then greedy will be your friend. set up one worker thread per cpu,
> > > connected to the scheduler by a PULL socket with a queue length of 1.
> > > each worker pulls a task description, performs it, and then asks for
> > > more. when it receives a 'all done' message, it exits.
> > 
> > Basically, that is what I was thinking.  However, my understading is that
> > you have to have all workers connected to the boss before you start
> > sending out tasks. Otherwise the longest running task, as well as the
> > second longest running task get assigned the the first worker that
> > connects (which is very suboptimal). If I have to do a lot of
> > bookkeeping to keep track of the workers, then I suspect piping to
> > subprocesses is an easier solution.  I was intrigued by the BOSS/WORKER
> > pattern email because it sounded like it handled all this bookkeeping,
> > resulting in much simpler code in my application. Now that I have
> > elaborated my use case, is the BOSS/WORK pattern would be appropriate
> > for this use case?
> 
> no, you don't have to have all the workers connected.
> and if you set the queue length to 1, early arrivees won't snag all teh
> jobs.
> 

I understand that, but I am worried that the earliest arrivee will get the 
second longest running job while it is working on the longest running job 
(which could last 10's of minutes), since it takes the first job of the queue 
freeing it up for the second job.

> i promise you that using pipes will be more work,
> and less flexible to boot.
> 

The flexibility is the reason I am looking at zeromq and I would agree that 
pipes will be more work for one who has drunk the kool-aid, but I understand 
pipes and don't know how much work it will take to learn zeromq.

> surrender to the flow, drink the kool-aid, and enjoy the bliss.
> 

Thanks for your help.
Henry

> ------------------
> Andrew Hume  (best -> Telework) +1 623-551-2845
> andrew at research.att.com  (Work) +1 973-236-2014
> AT&T Labs - Research; member of USENIX and LOPSA

-- 
Henry Baragar
Instantiated Software
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110406/1755b466/attachment.htm>