[zeromq-dev] Single-Point-of-Failure removal

Michel Pelletier pelletier.michel at gmail.com
Mon Apr 9 04:24:55 CEST 2012


Turning that PUB/SUB into a PUSH/PULL sounds like a reasonable
approach to me to introduce multiple announcers into the mix.

-Michel

On Sun, Apr 8, 2012 at 6:21 PM, Gregory Taylor <gtaylor at gc-taylor.com> wrote:
> If I understand correctly, the possibility I outlined (changing the
> Worker->Announcer link to Push/Pull) would eliminate the duplicate messages,
> since Push/Pull is fair-queued when multiple connections are made (each
> Worker to multiple Announcers), and the Clients would be Sub'd to the two
> announcers (Pub).
>
> Am I on the right track with this?
>
> Greg
>
>
> On Sun, Apr 8, 2012 at 5:22 AM, Pieter Hintjens <ph at imatix.com> wrote:
>>
>> There are different ways to do this. Try sending redundant messages
>> and filtering out duplicates on arrival. "Exactly one" is
>> hard/impossible. "At least one" is rather easier.
>>
>> On Sun, Apr 8, 2012 at 7:09 AM, Gregory Taylor <gtaylor at gc-taylor.com>
>> wrote:
>> > I'm working on a set of daemons that will accept raw JSON messages from
>> > a
>> > potentially large number of external clients. The application is built
>> > to
>> > allow every component in the "network" of daemons to be made redundant.
>> > ZeroMQ makes this very easy. Here's what I've got so far:
>> >
>> >
>> > https://docs.google.com/drawings/d/1KJInuBtNu9QMiYAPrC5FDq6QDGHgQYutv8M2Bwcw3Nw/edit
>> >
>> > The only thing left to figure out is how to re-factor this to make it
>> > possible to have more than one announcer without message duplication to
>> > the
>> > Clients. There can be no single point of failure, or the application is
>> > unsuitable for our purposes. I am stumped on the best way to do this.
>> >
>> > Notes/needs:
>> >
>> > The Clients may connect to multiple Announcers if need be.
>> > Workers may connect to multiple announcers, if need be.
>> > Clients should not get the same message twice, if connected to more than
>> > one
>> > announcer.
>> > For each message uploaded to the Gateway, each Client should get a
>> > single
>> > copy of the message on the bottom from the Announcer (after some parsing
>> > at
>> > the Worker level).
>> >
>> > Current idea:
>> >
>> > Change the link between Worker and Announcer to be Worker(Push),
>> > Announcer(Pull), and have each Worker connect to two Announcers.
>> > Messages
>> > would be fair-queued across Announcers, and Clients(sub) would connect
>> > to
>> > the two announcers(pub), getting only one copy of each message. If one
>> > of
>> > the announcers went down, the workers would send everything through the
>> > announcer that is still up (I think?).
>> >
>> > Am I on the right track with this? I'd love to hear more ideas from
>> > those
>> > with better ZeroMQ-fu than I.
>> >
>> > Greg
>> >
>> >
>> > _______________________________________________
>> > zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>



More information about the zeromq-dev mailing list