[zeromq-dev] Upstream/Downstream simple example

Peter Alexander vel.accel at gmail.com
Tue Jun 22 05:52:50 CEST 2010

Hi Martin,

On Wed, Jun 9, 2010 at 12:09 PM, Martin Sustrik <sustrik at 250bpm.com> wrote

> I believe the "send_requests" is the initial node of the pipeline. It
> produces request. Down the stream there's arbitrary number of
> "component"s which do the actual processing and generate replies.
> Further down there's "receive_replies" component that gathers all the
> replies.

A pipeline shouldn't be considered in the request/reply paradigm, but
rather in the context of "pushing data". In other words, the pattern
is data centric.

The entire string of components should be thought of as a virtual
single process that has an input and an output where the first "root"
node has the responsibility of obtaining the data (input) which is to
be passed along to the following nodes. The string/pipeline is a
directed, normally acyclic, graph. The last "leaf" node(s) is then
responsible as an output of the data in it's final form to, for
instance, a file, database, stdout, or a display (like a graph/plot).

Each inner component/node, within the string/pipeline, has a
responsibility to massage or derive-from the dataset buffer which has
been passed to it.

Each component/node knows apriori, how to handle this "chunk" of data
that it has been handed. If its a raw chunk of data, for instance,
then it is familiar with a dataset descriptor that allows it to
interpret the chunk into some sort of workable structure.

Each component/node is a _single_ "black box" process that has an
input(s) and an output(s) just like a typical function call.

Request/Reply is not needed because that is _implied_ by the existence
of the component/node itself.. within the string/pipeline.

Finally, a pipeline can be an assembly of other (pre-composed) "sub"
pipelines. Think "cut and paste".. ;)

This is a very.. very flexible and powerful programming paradigm.


More information about the zeromq-dev mailing list