[zeromq-dev] Single-Point-of-Failure removal

[Gregory Taylor]

I'm working on a set of daemons that will accept raw JSON messages from a
potentially large number of external clients. The application is built to
allow every component in the "network" of daemons to be made redundant.
ZeroMQ makes this very easy. Here's what I've got so far:

https://docs.google.com/drawings/d/1KJInuBtNu9QMiYAPrC5FDq6QDGHgQYutv8M2Bwcw3Nw/edit

The only thing left to figure out is how to re-factor this to make it
possible to have more than one announcer without message duplication to the
Clients. There can be no single point of failure, or the application is
unsuitable for our purposes. I am stumped on the best way to do this.

Notes/needs:


   - The Clients may connect to multiple Announcers if need be.
   - Workers may connect to multiple announcers, if need be.
   - Clients should not get the same message twice, if connected to more
   than one announcer.
   - For each message uploaded to the Gateway, each Client should get a
   single copy of the message on the bottom from the Announcer (after some
   parsing at the Worker level).

Current idea:

   - Change the link between Worker and Announcer to be Worker(Push),
   Announcer(Pull), and have each Worker connect to two Announcers. Messages
   would be fair-queued across Announcers, and Clients(sub) would connect to
   the two announcers(pub), getting only one copy of each message. If one of
   the announcers went down, the workers would send everything through the
   announcer that is still up (I think?).

Am I on the right track with this? I'd love to hear more ideas from those
with better ZeroMQ-fu than I.

Greg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20120408/91543e6d/attachment.htm>