[zeromq-dev] slow consumer

[Martin Sustrik]

John Skyfox wrote:
> 
> 
>  > Date: Fri, 28 May 2010 12:41:01 +0200
>  > From: sustrik at 250bpm.com
>  > To: zeromq-dev at lists.zeromq.org
>  > Subject: Re: [zeromq-dev] slow consumer
>  >
>  > John Skyfox wrote:
>  >
>  > > If HWM is set to unlimited the publisher will eventually crash when it
>  > > will run
>  > > out of resources. What I was thinking (and this is the change that I
>  > > actually did) is to
>  > > allow the publisher to skip only those consumers that are too slow:
>  > >
>  > > Before sending a message, the publisher use the function 
> check_write().
>  > > if the queue is
>  > > full (high watermark hit) the publisher will stop writing new messages
>  > > until the pipe that
>  > > was stalled become available again. On a slow network this can take 
> some
>  > > seconds.
>  > >
>  > > What I did is to change the pipe_full() function and the write()
>  > > function so that when a pipe
>  > > hit the HWM it will stop writing but just release the message counter.
>  > > Those consumers that retrieve normally
>  > > will not be penalized because the publisher will still send them new
>  > > messages as long as they do not hit HWM.
>  > > This is done without any further processing because the pipe_full() is
>  > > done anyway by each pipe. Since it is done
>  > > during writing, the program run as fast as before with the only
>  > > advantage that only the slow consumers are missing
>  > > new messages.
>  >
>  > Hm. How does that differ from the standard behaviour offered by mainline
>  > 0MQ at the moment? I believe PUB/SUB sockets handle slow consumers
>  > exactly this way.
> 
> Not really. If you look at the code for the pub.cc (I am using the 2.0.6 
> that I downloaded from the ømq web pages) you will find the the 
> check_write() is done before writing any
> data to the pipes. And the check_write() will fail if one pipe is 
> stalled (which is what happens when one of them is too slow and you have 
> a HWM, it will get eventually full
>  and pipe_full() will be true). What I did is to postpone this test 
> until the writting. if one pipe is full it is skipped and the system 
> will move to the next pipe that is not full.
> I haven't checked the current devel. version so I do not know if it is 
> done this way.

Yes. You are right.

This have already been fixed in the development version.

Martin