[zeromq-dev] dealer responsibility with the DEALER->REP pattern

[Michel Pelletier]

Mikhail,

Setting a HWM to 1 is, to me, an anti-pattern.  It specifically makes
you have to think about these very issues you bring up, and it negates
the ability for the message paths to naturally surge and and ebb as
load varies.  I suppose it may have actual use cases in the real
world; I've never encountered one, but again that's just me and I've
not had to solve your problem.

Perhaps another approach would be to use the LRU pattern so that at
any time you know how many of your workers are busy and how many are
free.  Then you can immediately turn away requests when your load is
high.  It also lets you buffer up more work into queues than just 1 so
that workers can be constantly busy.

http://zguide.zeromq.org/page:all#Least-Recently-Used-Routing-LRU-Pattern

-Michel

On Thu, Apr 26, 2012 at 10:38 PM,  <mikhail at sirena2000.ru> wrote:
> Hello,
>
> I'm building a distributed system on top of 0mq and I am anxious if
> I am going to encounter kinds of bottleneck problems.
>
> So, the prehistory.
>
> The system consists of a gate (Inet->Hive) and a bunch of workers,
> which are simple blocking responders; the nature of task often make
> a worker take a while to respond with an answer.
>
> So far, the DEALER->REP pattern seems to me fitting pretty well.
>
> Next, I want to have every task that is not being sent or processed
> right now under my control to have an option to throw it back to the
> customer with a note «hey, we are a bit overloaded atm, but you can
> try again in just a few seconds».
>
> With this view HWM on the dealer and the reps is set to 1.
>
> Thus, it leads me to a pair of questions:
>
> 1. imagine an immense load on the gate while all the workers are
>   processing their tasks. Doesn't that means that the dealer will
>   be blocked from the outer side (due to hwm) and rep will come to
>   the «exceptional state due to having reached the high water mark
>   for a client, then any replies sent to the client in question
>   shall be dropped until the exceptional state ends»
>   (according to http://api.zeromq.org/2-2:zmq-socket)?
>   In which case «any replies sent ... shall be dropped» really
>   scares me.
>
> 2. imagine now, that all those workers are willing to send their
>   answers (which can be really large) just at one moment of time.
>   What about the opposite situation, with the dealer blocked from
>   the inner side and not possible to take a window to send
>   awaiting task to a fresh-freed worker?
>
> Or, maybe I have just misunderstood the way HWM works and all my
> worries are in vain?
>
> Any help is highly appreciated in understanding the matter.
> Thanks in advance.
>
> -- Mikhail
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev