[zeromq-dev] Publish / Subscribe vs Multicast

[gonzalo diethelm]

Hi Martin,

> > 1. The distributor has the logic to decide which worker should get a
> > specific message, and sends it directly to it. This means: the
> > distributor knows how many workers there are (in order to evenly
> > distribute messages); it knows what workers are not responding; it
is
> > made aware of new workers. I agree in this scenario Multicast it not
> > needed, nor is PubSub.
> >
> > 2. The distributor sends every message to all workers, and they
decide
> > whether to ignore it or if they are the ones to process it. This
means:
> > the distributor doesn't need any special knowledge about the
workers; it
> > should use PubSub or Multicast; the workers need logic to determine
> > whether to process or ignore the message, and this logic would most
> > likely require knowing about all the other workers and their state.
> >
> > What do you think?
> 
> The former is better IMO. In the latter case you would have to handle
> communication between individual workers which turns out to be pretty
> complex, especially when they become mutually inaccessible.

I agree, as long as the first case allows for building in the logic to
handle a (possibly) dynamic number of workers, stalled workers, etc.

> > Maybe I should have described first my ideal scenario. I would love
to
> > be able to add workers at will, or even kill them at will, without
> > having to restart the distributor or any current workers. This is
not
> > strictly hot-hot failover, but you might call it "dynamic load
> > distribution". That is why I think sending the messages to all
workers
> > might make sense.
> 
> I would definitely use ZMQ_UPSTREAM and ZMQ_DOWNSTREM sockets. Hava a
> look at the following article:
> 
> http://www.zeromq.org/tutorials:butterfly
> 
> (The code samples use old API so just ignore them.)
> 
> Is that what you are aiming for?

Sweet! Yes, that is the scenario I am researching. However, I could not
find that tutorial anywhere in the downloaded source code (for Windows).
Do I need to get it from the repo? Or perhaps it is not included anymore
(since you mention an old API)?

> > In my pipeline, I could also do this:
> >
> > 1. The distributor receives a request for work.
> > 2. It creates a unique tag for it and sends it, together with
current
> > time, to the final program in the pipeline (let's call it the
checker).
> > 3. It then passes the request to the first stage in the pipeline
(one of
> > N workers for this stage).
> > 4. It moves from stage to stage, where each stage has Ni workers.
> > 5. The final stage passes a notification to the checker including
the
> > request id and current time.
> >
> > Strict pipeline, no request-reply (so I can fire and forget), still
can
> > know when requests end, how long they take and even take measures
for
> > requests taking too long.
> 
> Yes. Exactly. In pipelined scenario the tag should be a sequence
number.
> That way final receiver can identify holes in the sequence (caused by
> crashed workers or crashed distributors) and notify the original
sender
> about the failure.

Yes. Sort of like TCP/IP does it.

> >> In case you would like to give a hand with the implementation, let
us
> >> know.
> >
> > Would love too. I still am at the beginning stages of wrapping my
head
> > around 0mq, so please give me some time.
> 
> Sure. Take your time!

I'm slowly gearing up to it. Thanks for all the help.

-- 
Gonzalo Diethelm



----------------------------------------- 
Declaración de confidencialidad: Este Mensaje esta destinado para
el uso de la o las personas o entidades a quien ha sido dirigido y
puede contener información reservada y confidencial que no puede
ser divulgada, difundida, ni aprovechada en forma alguna. El uso no
autorizado de la información contenida en este correo podrá ser
sancionado de conformidad con la ley chilena. 
Si usted ha recibido este correo electrónico por error, le pedimos
eliminarlo junto con los archivos adjuntos y avisar inmediatamente
al remitente, respondiendo este mensaje. 

"Before printing this e-mail think if is really necesary".
Disclosure: This Message is to be used by the individual,
individuals or entities that it is addressed to and may include
private and confidential information that may not be disclosed,
made public nor used in any way at all. Unauthorized use of the
information in this electronic mail message may be subject to the
penalties set forth by Chilean law. 
If you have received this electronic mail message in error, we ask
you to destroy the message and its attached file(s) and to
immediately notify the sender by answering this message.