[zeromq-dev] Extending pipeline model, distributing jobs based on the type of job to specific workers

artemv zmq artemv.zmq at gmail.com
Mon Feb 17 21:54:41 CET 2014


You have to learn concept of 0mq' "socket identity". Read that part as many
times as needed. Your problem is just a piece of cake.
I don't want to rob you, guiding you step by step to the end solution (if
you know what I mean) ;) .

2014-02-17 16:57 GMT+02:00 Randall Nortman <rnzmq at wonderclown.net>:

> > On Mon, Feb 17, 2014 at 8:08 AM, aman mangal <mangalaman93 at gmail.com>
> wrote:
> >
> > > I am trying to solve the following problem and I can't think of any
> model
> > > which I can use: I have n workers and 1 distributor. Distributor has
> jobs
> > > for n workers and every worker can solve only a specific type of job.
> It has
> > > to distribute jobs to the workers such that no job is dropped.
> It seems to me that if each worker handles a different type of job,
> then you do not have one job queue, you have N job queues.  The
> question then becomes, who decides which worker handles which type of
> job?  If only the worker knows, then you need one of the "reliable
> pub/sub" patterns from the guide.  None of these are as simple as just
> a single PUB connected to multiple SUBs, which is inherently
> unreliable.  (Because fan-out can't be implemented in a way that's
> reliable, fast/scalable, and suitable for a wide range of fan-out
> problems, so the default pub/sub is fast/scalable and adaptable, but
> not reliable.)
> If the distributor knows how to map job types to workers (probably a
> better idea), then have each worker create its own queue, using
> push/pull if you can accept occasional undetected failures in the
> event of disconnects, or one of the reliable req/rep patterns if
> required, or if results need to be returned from workers.  The
> distributor then connects to those queues (each queue using a
> *different socket* on the distributor side, not a shared PUSH, which
> has the round-robin problem you discovered.)  Alternately, the
> distributor can bind a separate socket for each type of queue and
> workers can connect to that.  (The latter allows you to easily grow to
> support multiple workers per queue without modification to the
> distributor.)
> If a static configuration of what workers are where, and what types of
> jobs are handled, is not acceptable, you can have a separate socket
> (out of band with respect to the main job queues) on the distributor
> that workers connect to in order to advertise their services and setup
> the queues dynamically.
> Randall
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140217/887d4ef6/attachment.htm>

More information about the zeromq-dev mailing list