[zeromq-dev] LRU broker queue in intuitive way on 3.0

[Henry Baragar]

On April 4, 2011 06:30:06 pm Pieter Hintjens wrote:
> On Tue, Apr 5, 2011 at 12:16 AM, Paul Colomiets <tailhook at yandex.ru> wrote:
> > There are two patterns in the guide for request/reply broker. One
> > with using XREQ on router and REP on worker. And the other is
> > based on XREP on router REQ on worker. The problem is that
> > first is being obvious but basically does not work. The latter
> > is counter-intuitive and code is much more comples but works much
> > better.
> 
> This is a good point. Having to use a REQ socket for workers is weird.
> It works, but it's surprising, which is a bad sign.
> 
> In fact the LRU mechanism could be done by the socket layer, because
> it's completely generic. It would IMO be a new socket pattern:
> 
> * BOSS socket, which organizes a group of workers, and
> * WORKER socket, which takes orders from a BOSS socket
> 
> The BOSS socket keeps a list of ready workers, which signal presence
> to the BOSS socket, when they connect, and when they reply to a
> request. They work synchronously, and look just like REP sockets to
> applications. They could fan-in from multiple BOSS sockets. The BOSS
> socket will queue messages until it can send them, and will return
> replies that can be sent back up to clients.
> 
> Then a LRU queue device has a ROUTER socket to talk to clients, and a
> BOSS socket to talk to workers, and becomes a simple read/write device
> between the two.
> 

I think that this is the pattern that I am trying to figure out if it would be 
easy to implement in zeromq.

Here is my use case...

I have a "day end" task that can be split up into 200 subtask and I want the 
task to run as fast as possible. The interesting thing is that the longest 
subtask could take 10K times as long to run as the shortest, and I have 
information that allows me to sort them in descending order from longest to 
shortest run time.

I want the task manager (the Boss) to hand out subtasks to workers one at a 
time, so that the first worker to connect gets the longest running task, the 
second one gets second longest, etc.  I would only start 4 workers, so I want 
to assign the fifth longest task to the first available worker, which probably 
would be the fourth worker.  The sixth longest subtask would not be assigned 
until there was an available worker, etc. until all the subtasks are complete, 
at which point all the processes would be shut down.

Oh, if the task is taking too long, I would like to be able to add workers as 
needed (probably on other servers).

Would this BOSS/WORKER pattern you envision address my use case?

Regards,
Henry

> -Pieter
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

-- 
Henry Baragar
Instantiated Software
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110404/21ce56a9/attachment.htm>