[zeromq-dev] high availability and persistance

[Pieter Hintjens]

On Sun, Jun 5, 2011 at 8:32 PM, Sean Ochoa <sean.m.ochoa at gmail.com> wrote:

> 1.)  One or more middle-ware messaging systems which utilizes zeromq as its
> core component has recently (within the last year) experienced a high
> availability failure.  How does one guarantee a high level of availability?

In general, it depends on the kind of messaging you're doing (the
pattern). I.e. reliable pubsub is different from reliable
request-reply, and reliable pipeline. There are a number of techniques
you can use, including heartbeating, redundancy, persistence,
acknowledgements, etc. I've covered a lot of these in Ch4 and Ch5 of
the Guide, and my advice is to start with a brute force approach, e.g.
full acknowledgements on every message, and then improve step by step
for performance.

If you need architectural help, drop me a line.

> 2.)  Could a system which utilizes zeromq as its core messaging component
> also utilize a distributed non-relational data store like Apache Cassandra
> to persist the information coming across the wire and so make the system
> more durable?

Certainly, but it has to fit into an overall design that makes sense.
You can't simply plug persistence into a 0MQ network and hope for
reliability.

> And, if we did, would this totally bork the performance
> benefits that switching to zeromq and a brokerless architecture buys us?

If you do it right (and that can take a few iterations) you can
generally get performance where it matters, and reliability where it
matters. Both at once can be tricky.

General advice is to invest the necessary time in learning the 0MQ
patterns, including all the reliability patterns explained in the
Guide, and then use that knowledge carefully to scale out your system,
solving problems one at a time in the simplest manner possible each
time.

-Pieter