[zeromq-dev] help with feng shui

[Pieter Hintjens]

Andrew,

It looks like a good brute-force design, and quite probably when you
have made this work you will understand how you could have solved it
more elegantly.

-Pieter

On Fri, Aug 27, 2010 at 4:05 AM, Andrew Hume <andrew at research.att.com> wrote:
> i need some advice. i do not yet grok the feng shui of zeromq,
> and thus seek advice from those who do.
> i have a fairly normal setup similiar to the parallel pipeline example in
> teh guide.
> except that i have a handful of ventilators, and a handful of sinks.
> so far, so good. we just use the PUSH/PULL pattern.
> here is where it gets harder. i need to be able to essentially pause
> the ventilators, adjust the number of workers and sinks, and then
> unpause the ventilators WITHOUT losing any packets.
> the best (!?) solution i have so far is
> a) add a PUSH/PULL feedback socket (with all sinks and workers PUSH,
> and the master is a PULL)
> b) add a PUB/SUB command socket (with all ventilators, sinks and workers
> SUB,
> and the master PUB)
> c) we send an "IDLE" command to the ventilators; they pause their normal
> work
> and start sending NO-OP work items
> d) as each worker starts getting NO-OPs, they push a "LAZY" message to the
> master.
> they orward the NO-OP to the sinks.
> e) when the master sees k LAZY messages (where k is the existing number of
> workers),
> it rearranges teh workers (killing some or starting new ones). new workers
> send NO-OPs.
> f) when each sink starts getting NO-OPs, it sends a "LAZY" message to the
> master.
> g) when the master has done e), and seen NO-OPs from each of the j sinks, it
> rearranges the sinks. when each new sink starts getting NO-OPs, it send s a
> LAZY to teh master.
> h) when the master receives m "LAZY"s (where m is the number of new sinks),
> it send an "GO"
> command to teh ventilators, who then stop sending NO-OPs and start sending
> real work.
> -------------------------------------
> pros: i believe this scheme will work. and the additional cost of two
> sockets is modest.
> cons: it is tedious to send NO-OPs, but i don't know how else to flush teh
> buffers
> and synchronise everyone. it does involve knowing how many things there are,
> but that is part of an external configuration in any case.
> is this the (or a) right way to do this? is there a better way?
> andrew
> ------------------
> Andrew Hume  (best -> Telework) +1 732-886-1886
> andrew at research.att.com  (Work) +1 973-360-8651
> AT&T Labs - Research; member of USENIX and LOPSA
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>



-- 
-
Pieter Hintjens
iMatix - www.imatix.com