[zeromq-dev] Getting rid of low watermark

[Brian Granger]

Martin,

+1

When I have used the HWM, I had this thought exactly "how on earth do
I pick the LWM?"  I like your idea - even if the exact algorithm you
use in picking LWM isn't quite optimal, it can be changed in the
future without changing the API.  Great idea.

Cheers,

Brian

On Tue, May 25, 2010 at 1:43 AM, Martin Sustrik <sustrik at 250bpm.com> wrote:
> Hi all,
>
> Currently, when setting limit for number of messages that can be held by  a
> socket you have to specify high & low watermarks.
>
> Here's what documentation says:
>
> "The ZMQ_HWM option shall set the high water mark for the message queue
> associated with the socket. The high water mark is a hard limit on the
> number of outstanding messages in the queue; if this
>       limit has been reached the socket shall enter an "emergency" state and
> depending on the socket type, 0MQ shall take appropriate action such as
> blocking or dropping new messages entering the
>       queue."
>
> "The ZMQ_LWM option shall set the low water mark for the message queue
> associated with the socket. This option only makes sense when used in
> conjunction with the ZMQ_HWM option. A socket which
>       has reached it’s high water mark remains in the "emergency" state
> until the number of outstanding messages in it’s associated message queue
> falls below the low water mark, at which point normal
>       message processing is resumed."
>
> As far as I understand -- correct me if I am wrong -- it's not obvious what
> value LWM should be set to and it's not even clear why the user should
> bother with setting LWM at all.
>
> Given the above, I would like to propose an algorithm for automatic setting
> of LWM.
>
> Following three points are to be considered:
>
> 1. LWM has to be less than HWM.
>
> 2. LWM cannot be set to very low value (such as zero) as after filling the
> queue it would start to refill only after all the messages are read from it
> and thus unnecessarily hold the progress back.
>
> 3. LWM cannot be set to very high value (such as HWM-1) as it would result
> in lock-step filling of the queue -- if a single message is read from a full
> queue, writer thread is resumed to write exactly one message to the queue
> and go back to sleep immediately. This would result in low performance.
>
> Given the 3) it would be good to keep HWM and LWM as far apart as to reduce
> the thread switching overhead to almost zero, say HWM-LWM should be 500.
>
> Still we have to account for the cases where HWM<500 thus driving LWM to
> negative numbers. My suggestion is to make LWM one half of HWM in such
> cases.
>
> The graph showing resulting HWM/LWMs is attached.
>
> Thoughts?
> Martin
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>



-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com