[zeromq-dev] Single-Point-of-Failure removal

[Pieter Hintjens]

There are different ways to do this. Try sending redundant messages
and filtering out duplicates on arrival. "Exactly one" is
hard/impossible. "At least one" is rather easier.

On Sun, Apr 8, 2012 at 7:09 AM, Gregory Taylor <gtaylor at gc-taylor.com> wrote:
> I'm working on a set of daemons that will accept raw JSON messages from a
> potentially large number of external clients. The application is built to
> allow every component in the "network" of daemons to be made redundant.
> ZeroMQ makes this very easy. Here's what I've got so far:
>
> https://docs.google.com/drawings/d/1KJInuBtNu9QMiYAPrC5FDq6QDGHgQYutv8M2Bwcw3Nw/edit
>
> The only thing left to figure out is how to re-factor this to make it
> possible to have more than one announcer without message duplication to the
> Clients. There can be no single point of failure, or the application is
> unsuitable for our purposes. I am stumped on the best way to do this.
>
> Notes/needs:
>
> The Clients may connect to multiple Announcers if need be.
> Workers may connect to multiple announcers, if need be.
> Clients should not get the same message twice, if connected to more than one
> announcer.
> For each message uploaded to the Gateway, each Client should get a single
> copy of the message on the bottom from the Announcer (after some parsing at
> the Worker level).
>
> Current idea:
>
> Change the link between Worker and Announcer to be Worker(Push),
> Announcer(Pull), and have each Worker connect to two Announcers. Messages
> would be fair-queued across Announcers, and Clients(sub) would connect to
> the two announcers(pub), getting only one copy of each message. If one of
> the announcers went down, the workers would send everything through the
> announcer that is still up (I think?).
>
> Am I on the right track with this? I'd love to hear more ideas from those
> with better ZeroMQ-fu than I.
>
> Greg
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>