[zeromq-dev] LRU broker queue in intuitive way on 3.0

Andrew Hume andrew at research.att.com
Thu Apr 7 11:19:12 CEST 2011


point taken.

i remember being your position and it really took just a few days
to progress from some simple programs working to devising
more robust and complex schemes.

actually, the last time i had your exact problem, i solved it in Ruby
using a distributed queue, but that would count as a cheat, probably.

On Apr 6, 2011, at 10:39 AM, Henry Baragar wrote:

> On April 6, 2011 09:14:28 am Andrew Hume wrote:
> > On Apr 6, 2011, at 9:00 AM, Henry Baragar wrote:
> > > On April 5, 2011 10:30:31 am Andrew Hume wrote:
> > > > given this degenerate case of a single server and no per-task overhead,
> > > > then greedy will be your friend. set up one worker thread per cpu,
> > > > connected to the scheduler by a PULL socket with a queue length of 1.
> > > > each worker pulls a task description, performs it, and then asks for
> > > > more. when it receives a 'all done' message, it exits.
> > > 
> > > Basically, that is what I was thinking.  However, my understading is that
> > > you have to have all workers connected to the boss before you start
> > > sending out tasks. Otherwise the longest running task, as well as the
> > > second longest running task get assigned the the first worker that
> > > connects (which is very suboptimal). If I have to do a lot of
> > > bookkeeping to keep track of the workers, then I suspect piping to
> > > subprocesses is an easier solution.  I was intrigued by the BOSS/WORKER
> > > pattern email because it sounded like it handled all this bookkeeping,
> > > resulting in much simpler code in my application. Now that I have
> > > elaborated my use case, is the BOSS/WORK pattern would be appropriate
> > > for this use case?
> > 
> > no, you don't have to have all the workers connected.
> > and if you set the queue length to 1, early arrivees won't snag all teh
> > jobs.
> > 
> I understand that, but I am worried that the earliest arrivee will get the second longest running job while it is working on the longest running job (which could last 10's of minutes), since it takes the first job of the queue freeing it up for the second job.
> > i promise you that using pipes will be more work,
> > and less flexible to boot.
> > 
> The flexibility is the reason I am looking at zeromq and I would agree that pipes will be more work for one who has drunk the kool-aid, but I understand pipes and don't know how much work it will take to learn zeromq.
> > surrender to the flow, drink the kool-aid, and enjoy the bliss.
> > 
> Thanks for your help.
> Henry
> > ------------------
> > Andrew Hume  (best -> Telework) +1 623-551-2845
> > andrew at research.att.com  (Work) +1 973-236-2014
> > AT&T Labs - Research; member of USENIX and LOPSA
> 
> -- 
> Henry Baragar
> Instantiated Software
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev


------------------
Andrew Hume  (best -> Telework) +1 623-551-2845
andrew at research.att.com  (Work) +1 973-236-2014
AT&T Labs - Research; member of USENIX and LOPSA




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110407/143b9ee3/attachment.htm>


More information about the zeromq-dev mailing list