[zeromq-dev] Efficient and reliable pub/sub mechanism

[Pieter Hintjens]

On Mon, Jan 24, 2011 at 1:06 AM, Yusuf Simonson <simonson at gmail.com> wrote:

> I'm trying to build a pub/sub engine on top of jzmq for performing
> distributed computation in cloud environments. Systems have a relatively
> high chance of failure, so I want to ensure reliability.

First of all, do read the Guide and work through the different
examples. You can do reliable workload distribution but not using a
pubsub pattern, rather you need custom routing (xreq). Chapter 3
explores this in detail.

The simplest reliability model for workload distribution is probably as follows:

- you have a pool of workers
- you distribute work to workers that signal 'ready'
- when a worker fails to respond within X seconds you flag it as
'dead' and redistribute its last task to another worker

You can build this as a device that connects clients to workers, or
you can build it into the clients directly.

-
Pieter Hintjens
iMatix