[zeromq-dev] Publish / Subscribe vs Multicast

Martin Sustrik sustrik at 250bpm.com
Mon Feb 15 10:53:06 CET 2010

Hi Gonzalo,

>> This is a classic example of multi-hop request/reply scenario.
>> Supporting it is on the roadmap, part of the functionality is already
>> implemented, resources for implementing the rest are still missing :(
> As I said, I intend this to be more of a pipeline scenario, not so much
> request/reply. Please see comments below.
>>>    1. What would be the practical differences between using a PubSub
>>>       approach and using Multicast to pass the requests from
> distributor
>>>       to workers?
>> In this scenario each message is passed to a single worker so using
>> multicast would be an overkill.
> As I see it, there are two alternatives:
> 1. The distributor has the logic to decide which worker should get a
> specific message, and sends it directly to it. This means: the
> distributor knows how many workers there are (in order to evenly
> distribute messages); it knows what workers are not responding; it is
> made aware of new workers. I agree in this scenario Multicast it not
> needed, nor is PubSub.
> 2. The distributor sends every message to all workers, and they decide
> whether to ignore it or if they are the ones to process it. This means:
> the distributor doesn't need any special knowledge about the workers; it
> should use PubSub or Multicast; the workers need logic to determine
> whether to process or ignore the message, and this logic would most
> likely require knowing about all the other workers and their state.
> What do you think?

The former is better IMO. In the latter case you would have to handle 
communication between individual workers which turns out to be pretty 
complex, especially when they become mutually inaccessible.

>>>    2. By going with PubSub or Multicast, all the workers will
> receive
>>>       all task requests and will have to decide whether they are the
>>>       worker which should process it. What are practical ways of
> making
>>>       this decision? It looks like this approach requires the
> workers to
>>>       know in advance the total number of workers in the pool,
> right?
>> As noted above, there's little point in distributing the request to
> all
>> the workers (unless you are aiming for hot-hot failover) thus TCP
>> transport should be used.
> Maybe I should have described first my ideal scenario. I would love to
> be able to add workers at will, or even kill them at will, without
> having to restart the distributor or any current workers. This is not
> strictly hot-hot failover, but you might call it "dynamic load
> distribution". That is why I think sending the messages to all workers
> might make sense.

I would definitely use ZMQ_UPSTREAM and ZMQ_DOWNSTREM sockets. Hava a 
look at the following article:


(The code samples use old API so just ignore them.)

Is that what you are aiming for?

>>>    3. How to handle crashed workers? How about workers that are not
>>>       responding? What if I want to add workers?
>> The only 100% reliable algorithm is end-to-end reliability, meaning
> that
>> sending application tags request with an unique tag and waits for a
>> reply with the same tag. In the meanwhile it drops all non-matching
>> replies. If the reply is not delivered within specified time, the
>> request is resent.
> In my pipeline, I could also do this:
> 1. The distributor receives a request for work.
> 2. It creates a unique tag for it and sends it, together with current
> time, to the final program in the pipeline (let's call it the checker).
> 3. It then passes the request to the first stage in the pipeline (one of
> N workers for this stage).
> 4. It moves from stage to stage, where each stage has Ni workers.
> 5. The final stage passes a notification to the checker including the
> request id and current time.
> Strict pipeline, no request-reply (so I can fire and forget), still can
> know when requests end, how long they take and even take measures for
> requests taking too long.

Yes. Exactly. In pipelined scenario the tag should be a sequence number. 
That way final receiver can identify holes in the sequence (caused by 
crashed workers or crashed distributors) and notify the original sender 
about the failure.

>>>    4. Maybe I should have the distributor handle the load
> distribution,
>>>       not using PubSub or Multicast, but choosing a specific worker
> and
>>>       sending the task request directly to it. Same questions apply,
>> right?
>> This scenario can be implemented even now. However, requester would
> have
>> to have addresses of all the workers so that it is able to connect to
>> them. Probably not what you want.
> Exactly.
>> In case you would like to give a hand with the implementation, let us
>> know.
> Would love too. I still am at the beginning stages of wrapping my head
> around 0mq, so please give me some time.

Sure. Take your time!


More information about the zeromq-dev mailing list