[zeromq-dev] Single-Point-of-Failure removal

Gregory Taylor gtaylor at gc-taylor.com
Mon Apr 9 03:21:29 CEST 2012


If I understand correctly, the possibility I outlined (changing the
Worker->Announcer link to Push/Pull) would eliminate the duplicate
messages, since Push/Pull is fair-queued when multiple connections are made
(each Worker to multiple Announcers), and the Clients would be Sub'd to the
two announcers (Pub).

Am I on the right track with this?

Greg

On Sun, Apr 8, 2012 at 5:22 AM, Pieter Hintjens <ph at imatix.com> wrote:

> There are different ways to do this. Try sending redundant messages
> and filtering out duplicates on arrival. "Exactly one" is
> hard/impossible. "At least one" is rather easier.
>
> On Sun, Apr 8, 2012 at 7:09 AM, Gregory Taylor <gtaylor at gc-taylor.com>
> wrote:
> > I'm working on a set of daemons that will accept raw JSON messages from a
> > potentially large number of external clients. The application is built to
> > allow every component in the "network" of daemons to be made redundant.
> > ZeroMQ makes this very easy. Here's what I've got so far:
> >
> >
> https://docs.google.com/drawings/d/1KJInuBtNu9QMiYAPrC5FDq6QDGHgQYutv8M2Bwcw3Nw/edit
> >
> > The only thing left to figure out is how to re-factor this to make it
> > possible to have more than one announcer without message duplication to
> the
> > Clients. There can be no single point of failure, or the application is
> > unsuitable for our purposes. I am stumped on the best way to do this.
> >
> > Notes/needs:
> >
> > The Clients may connect to multiple Announcers if need be.
> > Workers may connect to multiple announcers, if need be.
> > Clients should not get the same message twice, if connected to more than
> one
> > announcer.
> > For each message uploaded to the Gateway, each Client should get a single
> > copy of the message on the bottom from the Announcer (after some parsing
> at
> > the Worker level).
> >
> > Current idea:
> >
> > Change the link between Worker and Announcer to be Worker(Push),
> > Announcer(Pull), and have each Worker connect to two Announcers. Messages
> > would be fair-queued across Announcers, and Clients(sub) would connect to
> > the two announcers(pub), getting only one copy of each message. If one of
> > the announcers went down, the workers would send everything through the
> > announcer that is still up (I think?).
> >
> > Am I on the right track with this? I'd love to hear more ideas from those
> > with better ZeroMQ-fu than I.
> >
> > Greg
> >
> >
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20120408/6c18bce7/attachment.htm>


More information about the zeromq-dev mailing list