[zeromq-dev] Distributed Q with **lots** of consumers

Bennie Kloosteman bklooste at gmail.com
Tue Nov 13 08:48:50 CET 2012


This is a fairly standard bus pattern , though personally i
HATE persistent queues with  a passion  , they kill all performance , you
have to manage the infrastructure and the possibility of poison messages -
for most business cases there are better ways to guarantee delivery eg send
update events to the client which can resend or terminate  as desired .

If you run into a limit you can chain them , eg  20 servers with 20 workes
each serve 400 clients , those 20 servers put everything into 1 queue.

Why bother using a fast system like ZeroMQ with a persistent queue anyway ,
the queue will just bottle neck everything writing the message to disk and
create contention on reading it back. Surely MSMQ or the other Service Bus
queues can do this better  - since they have optimized persistence .


On Tue, Nov 13, 2012 at 6:58 AM, Sean Donovan <sdonovan_uk at yahoo.com> wrote:

> Any suggestions for implementing the following in ZMQ?
>
> Imagine a single Q containing millions of entries, which is constantly
> being added to.  This Q would be fully persistent, probably not managed by
> ZMQ, and run in it's own process.
>
> We would like N workers.  Those workers need to start/stop ad-hoc, and
> reconnect to the Q host process.  Each worker would take a single item
> from the Q, process, acknowledge completion, then repeat (to request
> another item).  Processing time for each task is 3ms+
> (occasionally minutes).
>
> Because of the variance in compute time it is important that the workers
> don't pre-fetch/cache tasks.  As an optimization, we'll add a heuristic so
> we can batch short-running tasks together (but, we'd like the control -- a
> load-balancing algorithm wouldn't know how to route efficiently, unless it
> could take hints).
>
> Need a pattern that would allow us to scale to 100s of workers.
>
> MANY THANKS!
>
> Sean Donovan
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20121113/87ef3370/attachment.htm>


More information about the zeromq-dev mailing list