[zeromq-dev] Distributed Q with **lots** of consumers

[Sean Donovan]

Many thanks Pieter,

I will compile it up.  I had tried with the C++ examples version of majordomo on Windows, and immediately ran into a thread deadlock.  I'll try again with the C version, and see if that works (am guessing the C version is better supported).

Sean


________________________________
 From: Pieter Hintjens <ph at imatix.com>
To: Sean Donovan <sdonovan_uk at yahoo.com>; ZeroMQ development list <zeromq-dev at lists.zeromq.org> 
Sent: Monday, November 12, 2012 9:43 PM
Subject: Re: [zeromq-dev] Distributed Q with **lots** of consumers
 
Please read the Guide and specifically the Load-balancing and
Majordomo patterns which do precisely this:

* http://zguide.zeromq.org/page:all#The-Load-balancing-Pattern
* http://zguide.zeromq.org/page:all#Service-Oriented-Reliable-Queuing-Majordomo-Pattern

Majordomo takes load-balancing a step further by doing service-based
lookup of the worker to route to.

You can find an implementation at https://github.com/zeromq/majordomo
although the Guide examples are also a good start if you want to
create your own broker.

-Pieter

On Tue, Nov 13, 2012 at 7:58 AM, Sean Donovan <sdonovan_uk at yahoo.com> wrote:
> Any suggestions for implementing the following in ZMQ?
>
> Imagine a single Q containing millions of entries, which is constantly being
> added to.  This Q would be fully persistent, probably not managed by ZMQ,
> and run in it's own process.
>
> We would like N workers.  Those workers need to start/stop ad-hoc, and
> reconnect to the Q host process.  Each worker would take a single item from
> the Q, process, acknowledge completion, then repeat (to request another
> item).  Processing time for each task is 3ms+ (occasionally minutes).
>
> Because of the variance in compute time it is important that the workers
> don't pre-fetch/cache tasks.  As an optimization, we'll add a heuristic so
> we can batch short-running tasks together (but, we'd like the control -- a
> load-balancing algorithm wouldn't know how to route efficiently, unless it
> could take hints).
>
> Need a pattern that would allow us to scale to 100s of workers.
>
> MANY THANKS!
>
> Sean Donovan
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20121113/b4e3c9c1/attachment.htm>