[zeromq-dev] Distributed Q with **lots** of consumers

Pieter Hintjens ph at imatix.com
Tue Nov 13 06:43:32 CET 2012


Please read the Guide and specifically the Load-balancing and
Majordomo patterns which do precisely this:

* http://zguide.zeromq.org/page:all#The-Load-balancing-Pattern
* http://zguide.zeromq.org/page:all#Service-Oriented-Reliable-Queuing-Majordomo-Pattern

Majordomo takes load-balancing a step further by doing service-based
lookup of the worker to route to.

You can find an implementation at https://github.com/zeromq/majordomo
although the Guide examples are also a good start if you want to
create your own broker.

-Pieter

On Tue, Nov 13, 2012 at 7:58 AM, Sean Donovan <sdonovan_uk at yahoo.com> wrote:
> Any suggestions for implementing the following in ZMQ?
>
> Imagine a single Q containing millions of entries, which is constantly being
> added to.  This Q would be fully persistent, probably not managed by ZMQ,
> and run in it's own process.
>
> We would like N workers.  Those workers need to start/stop ad-hoc, and
> reconnect to the Q host process.  Each worker would take a single item from
> the Q, process, acknowledge completion, then repeat (to request another
> item).  Processing time for each task is 3ms+ (occasionally minutes).
>
> Because of the variance in compute time it is important that the workers
> don't pre-fetch/cache tasks.  As an optimization, we'll add a heuristic so
> we can batch short-running tasks together (but, we'd like the control -- a
> load-balancing algorithm wouldn't know how to route efficiently, unless it
> could take hints).
>
> Need a pattern that would allow us to scale to 100s of workers.
>
> MANY THANKS!
>
> Sean Donovan
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>



More information about the zeromq-dev mailing list