[zeromq-dev] Distributed Q with **lots** of consumers
Sean Donovan
sdonovan_uk at yahoo.com
Mon Nov 12 23:58:58 CET 2012
Any suggestions for implementing the following in ZMQ?
Imagine a single Q containing millions of entries, which is constantly being added to. This Q would be fully persistent, probably not managed by ZMQ, and run in it's own process.
We would like N workers. Those workers need to start/stop ad-hoc, and reconnect to the Q host process. Each worker would take a single item from the Q, process, acknowledge completion, then repeat (to request another item). Processing time for each task is 3ms+ (occasionally minutes).
Because of the variance in compute time it is important that the workers don't pre-fetch/cache tasks. As an optimization, we'll add a heuristic so we can batch short-running tasks together (but, we'd like the control -- a load-balancing algorithm wouldn't know how to route efficiently, unless it could take hints).
Need a pattern that would allow us to scale to 100s of workers.
MANY THANKS!
Sean Donovan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20121112/db341508/attachment.htm>
More information about the zeromq-dev
mailing list