[zeromq-dev] Distributed Q with **lots** of consumers

Sean Donovan sdonovan_uk at yahoo.com
Mon Nov 12 23:58:58 CET 2012


Any suggestions for implementing the following in ZMQ?

Imagine a single Q containing millions of entries, which is constantly being added to.  This Q would be fully persistent, probably not managed by ZMQ, and run in it's own process.

We would like N workers.  Those workers need to start/stop ad-hoc, and reconnect to the Q host process.  Each worker would take a single item from the Q, process, acknowledge completion, then repeat (to request another item).  Processing time for each task is 3ms+ (occasionally minutes).

Because of the variance in compute time it is important that the workers don't pre-fetch/cache tasks.  As an optimization, we'll add a heuristic so we can batch short-running tasks together (but, we'd like the control -- a load-balancing algorithm wouldn't know how to route efficiently, unless it could take hints).

Need a pattern that would allow us to scale to 100s of workers.  

MANY THANKS!

Sean Donovan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20121112/db341508/attachment.htm>


More information about the zeromq-dev mailing list