[zeromq-dev] Thoughts on a pattern: "The Noisy Gymnasium"

[Randall Nortman]

I am developing a new application using zmq in an unusual (from what
I've seen so far) way.  I'd like to solicit input, particularly about
reasons this may be a terrible idea.  It is essentially a variation on
an event-driven architecture.

The application essentially collects data from a variety of sources,
does various sorts of processing on the data, and reacts to data
(including both raw data and at various stages of processing) in
various ways, including taking actions, reporting on action results
(i.e., injecting new data), or emitting events.

This application, unlike many that fit the above description, is NOT
concerned with distributing the workload among multiple nodes for
scaling or reliability.  The fact that I'm using zmq means I could
distribute it, but in this case there's really no reason (so far).
The volume of data and events is quite small.  I am not looking for
scaling as much as I'm looking for making it easy to develop and build
an application with is flexible and robust in the face of complex and
changing requirements.

My tentative solution is to locally run a "broker" which does nothing
other than re-broadcast all messages it receives on a SUB socket to a
PUB socket.  It binds both the SUB and the PUB (to different ports),
as opposed to the typical pattern of binding PUB and connecting the
SUB to it.  I then run a bunch of small, single-purpose daemons (call
them "agents" if you prefer) which connect their own SUB to the
broker's PUB to hear messages, and their own PUB to the broker's SUB
in order to broadcast messages.

Basically, this creates an environment where all these daemons are
shouting into the same room, and everybody hears everybody else,
including getting their own messages fed back to themselves.  Each
daemon can filter this by putting a filter on their SUB socket if they
like.  Unlike in some event-driven architectures, there is no concept
of "handling" an event or request in a way that halts further
processing by other "handlers".  The event gets shouted into the room,
and anybody that cares to do something with it can.  Like a noisy
gymnasium, you mostly have small groups of people playing at different
games with different balls, but you can also sit in the bleachers and
watch everything all at once.

The advantages I see are:

Decoupling: Unit testing is trivial, and you are encouraged by the
architecture to divide the work into very small, very simple,
independently reusable routines that each do one and only one thing.

Transparency: It is easy to trace and log a system like this.  Unlike
a system where handlers can "consume" events that then nobody else
gets to act on, in this system multiple daemons can act on every
message.  This makes it easy to implement rules like "No matter what
else happens, if X happens, do this."  In fact, every rule/processing
step you implement in a system like this will be of that form; every
matching rule always fires.  This, I believe, makes the system more
flexible, because today you may think you know how to "handle" an
event, but tomorrow you will find that you need to do all of that,
plus another 10 things that you didn't know about yesterday.  If you
have a system where events are explicitly handled, you need to go back
to that single event handler and add those 10 things to it.

Configurability: Continuing that thought, if you need to run different
configurations of the application which have different sets of rules,
it is easy to do without ending up with huge event handlers that have
a bunch of conditional statements in them based on settings from a
configuration file or database.  Instead, in each environment, you
either run the daemon that implements a given rule/process, or you do
not run that daemon.  The source code for the daemons themselves can
then be much simpler and uncluttered, with little if any configuration
involved.

Language independence: Code each rule/process in the language that
makes the most sense for that individual piece.  This is actually the
most trivial benefit for me, but possibly quite useful for others.

What are the downsides?  Scalability, obviously, because
rebroadcasting all events to everybody is not going to be very
performant with either very high volume of messages or very large
numbers of cooperating daemons.  Reliability may also be an issue;
obviously PUB/SUB can drop messages, and if a given daemon is not up
and running when a message comes across the wire, then whatever that
daemon was supposed to do with it is not going to happen.  But the
idea is that this is all happening on localhost, or at least a
reliable LAN, and that there will be some supervisor process that
first starts the broker and then everything else, and restarts dead
daemons as necessary.  Still, this is not a solution with 100%
reliability.

Anybody else have other thoughts?  Is this is bad idea?  A useless
idea?  Better accomplished another way?  Any "Yeah, I did something
like that once -- never again" stories?