[zeromq-dev] Start sequence problem for cycle pup-sub processes.

Martin Sustrik sustrik at 250bpm.com
Mon Dec 6 17:06:44 CET 2010


On 12/06/2010 05:00 PM, Jeff Vienneau wrote:
> I'm not familiar enough with PUSH/PULL. How does PUSH/PULL solve flow
> control? Would I still be using a "flow" socket from the SINK to the
> SOURCE? Looking at the zmq_socket(3) documentation, I am getting that
> you are suggesting setting a high watermark at each point in the
> pipeline and generating at the source until the pipeline blocks.
>
> With the scheme I am using I am sending one flow control message per
> 1000 actual data messages, I am trying to avoid a situation where a flow
> message is needed for every data message at every stage as would be seen
> with REQ/REP pattern. It seems this lock-step linkage would create a
> large amount of latency.

PUSH/PULL is using TCP flow control to achieve it. No extra messages.

> Is there an inherent problem with a cycle PUB-SUB? What is happening
> that I cannot start the processes and have them run, but then a restart
> on one will kick start the data flow?

PUB/SUB is like a radio broadcast. PUB is boradcaster, SUBs are radio 
receivers. If the receiver is not turned on, the broadcast can be 
missed. In short, PUB/SUB is not really reliable.

Martin

> On Mon, Dec 6, 2010 at 10:59 AM, Martin Sustrik <sustrik at 250bpm.com
> <mailto:sustrik at 250bpm.com>> wrote:
>
>     Jeff,
>
>
>         The straight through stream is a simplified model for now.
>
>         We need the ability to distribute data at some points in the
>         stream, to
>         effectively monitor data by tapping in a certain points with a
>         second
>         subscriber. The flexibility of using all ZMQ PUB-SUB will allow
>         us to
>         build out pipeline components in an ad hock manner. For instance
>         we have
>         a splitter component and an combiner component that can
>         decompose and
>         build larger messages. We also have a decimator that can reduce
>         a data
>         rate of messages by dropping N and sending M messages. All there
>         components are command line apps the are passed URIs to
>         configure their
>         inputs and outputs.
>
>         While it is true that mostly we are point to point, there are places
>         where we need to publish one to many. It would be difficult to
>         design
>         the app such components are capable of subscribing to PUB-SUB in one
>         point in the pipeline and used in a PUSH-PULL mode at another point.
>
>         It is all working pretty well so far. I would rather solve the
>         startup
>         issue and keep with our current tack.
>
>
>     The problems you are having are stemming from the fact that you are
>     using wrong messaging pattern. Pipeline pattern (PUSH/PULL) solves
>     the flow control problem and many more.
>
>     What you can do is implement the backbone pipeline using PUSH/PULL
>     while redistributing data at certain nodes via PUB/SUB to allow
>     wiretapping.
>
>     Martin
>
>




More information about the zeromq-dev mailing list