[zeromq-dev] Extending pipeline model, distributing jobs based on the type of job to specific workers

Randall Nortman rnzmq at wonderclown.net
Mon Feb 17 15:57:48 CET 2014


> On Mon, Feb 17, 2014 at 8:08 AM, aman mangal <mangalaman93 at gmail.com> wrote:
> 
> > I am trying to solve the following problem and I can't think of any model
> > which I can use: I have n workers and 1 distributor. Distributor has jobs
> > for n workers and every worker can solve only a specific type of job. It has
> > to distribute jobs to the workers such that no job is dropped.

It seems to me that if each worker handles a different type of job,
then you do not have one job queue, you have N job queues.  The
question then becomes, who decides which worker handles which type of
job?  If only the worker knows, then you need one of the "reliable
pub/sub" patterns from the guide.  None of these are as simple as just
a single PUB connected to multiple SUBs, which is inherently
unreliable.  (Because fan-out can't be implemented in a way that's
reliable, fast/scalable, and suitable for a wide range of fan-out
problems, so the default pub/sub is fast/scalable and adaptable, but
not reliable.)

If the distributor knows how to map job types to workers (probably a
better idea), then have each worker create its own queue, using
push/pull if you can accept occasional undetected failures in the
event of disconnects, or one of the reliable req/rep patterns if
required, or if results need to be returned from workers.  The
distributor then connects to those queues (each queue using a
*different socket* on the distributor side, not a shared PUSH, which
has the round-robin problem you discovered.)  Alternately, the
distributor can bind a separate socket for each type of queue and
workers can connect to that.  (The latter allows you to easily grow to
support multiple workers per queue without modification to the
distributor.)

If a static configuration of what workers are where, and what types of
jobs are handled, is not acceptable, you can have a separate socket
(out of band with respect to the main job queues) on the distributor
that workers connect to in order to advertise their services and setup
the queues dynamically.

Randall



More information about the zeromq-dev mailing list