[zeromq-dev] Extending pipeline model, distributing jobs based on the type of job to specific workers

aman mangal mangalaman93 at gmail.com
Tue Feb 18 17:54:48 CET 2014

Thank you so much for everyone's reply. I tried using push/pull on
different sockets (because it was the simplest thing I could try), and I
believe it worked (I tried it even before sending this email but it wasn't
working for reasons I could not understand). I will try Majordomo and Freelance
when I have more time, they were too complicated for me to understand as I
am a beginner. I tried to understand the concept of socket identity which,
as I understand, is handled by zeromq for me in the simple models.

Though I still do not understand why in case of pub/sub HWM option is hard
coded. Giving a configuration option would have made much more sense to me.
I can understand that it is not a good idea to have publisher blocked
because of a slow subscriber but let the programmer decide.

Thank you so much once again, It was my first experience with zeromq and I
loved it. I will definitely use it in future. Zeromq is amazing and solves
one of the biggest challenge of Distributed System quite efficiently.

Aman Mangal
4th year Undergraduate Student
Department of Computer Science & Engineering
IIT Bombay

On Tue, Feb 18, 2014 at 2:24 AM, artemv zmq <artemv.zmq at gmail.com> wrote:

> hi,
> You have to learn concept of 0mq' "socket identity". Read that part as
> many times as needed. Your problem is just a piece of cake.
> I don't want to rob you, guiding you step by step to the end solution (if
> you know what I mean) ;) .
> 2014-02-17 16:57 GMT+02:00 Randall Nortman <rnzmq at wonderclown.net>:
> > On Mon, Feb 17, 2014 at 8:08 AM, aman mangal <mangalaman93 at gmail.com>
>> wrote:
>> >
>> > > I am trying to solve the following problem and I can't think of any
>> model
>> > > which I can use: I have n workers and 1 distributor. Distributor has
>> jobs
>> > > for n workers and every worker can solve only a specific type of job.
>> It has
>> > > to distribute jobs to the workers such that no job is dropped.
>> It seems to me that if each worker handles a different type of job,
>> then you do not have one job queue, you have N job queues.  The
>> question then becomes, who decides which worker handles which type of
>> job?  If only the worker knows, then you need one of the "reliable
>> pub/sub" patterns from the guide.  None of these are as simple as just
>> a single PUB connected to multiple SUBs, which is inherently
>> unreliable.  (Because fan-out can't be implemented in a way that's
>> reliable, fast/scalable, and suitable for a wide range of fan-out
>> problems, so the default pub/sub is fast/scalable and adaptable, but
>> not reliable.)
>> If the distributor knows how to map job types to workers (probably a
>> better idea), then have each worker create its own queue, using
>> push/pull if you can accept occasional undetected failures in the
>> event of disconnects, or one of the reliable req/rep patterns if
>> required, or if results need to be returned from workers.  The
>> distributor then connects to those queues (each queue using a
>> *different socket* on the distributor side, not a shared PUSH, which
>> has the round-robin problem you discovered.)  Alternately, the
>> distributor can bind a separate socket for each type of queue and
>> workers can connect to that.  (The latter allows you to easily grow to
>> support multiple workers per queue without modification to the
>> distributor.)
>> If a static configuration of what workers are where, and what types of
>> jobs are handled, is not acceptable, you can have a separate socket
>> (out of band with respect to the main job queues) on the distributor
>> that workers connect to in order to advertise their services and setup
>> the queues dynamically.
>> Randall
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140218/ccf2cce3/attachment.htm>

More information about the zeromq-dev mailing list