[zeromq-dev] slow consumer

John Skyfox johnskyfox at hotmail.com
Fri May 28 11:41:48 CEST 2010


> Date: Thu, 27 May 2010 12:39:06 +0200
> From: sustrik at 250bpm.com
> To: zeromq-dev at lists.zeromq.org
> Subject: Re: [zeromq-dev] slow consumer
> 
> Hi John.
> 
> > Is there a plan to arm ømq with some kind strategy in order to handle 
> > slow consumer ? In term of reliability it should not be possible to 
> > crash a publisher or to punish all the consumers because of 1 slow
> > consumer. I can contribute to some degree (I have started looking in ømq 
> > code in order to understand its logic. Is there a good doc that I can 
> > read for that ?)
> 
> I think you've hit the right solution. When using PUB/SUB pattern, you 
> should limit message buffers (ZMQ_HWM option) -- if done, when a slow 
> consumer's buffer gets full, new messages will be dropped.
> 
> As far as I understand what you are asking for is that publisher doesn't 
> get killed by OOM killer even if HWM is set to 'unlimited', right?
If HWM is set to unlimited the publisher will eventually crash when it will runout of resources. What I was thinking (and this is the change that I actually did) is toallow the publisher to skip only those consumers that are too slow:
Before sending a message, the publisher use the function check_write(). if the queue is full (high watermark hit) the publisher will stop writing new messages until the pipe thatwas stalled become available again. On a slow network this can take some seconds.
What I did is to change the pipe_full() function and the write() function so that when a pipe hit the HWM it will stop writing but just release the message counter. Those consumers that retrieve normallywill not be penalized because the publisher will still send them new messages as long as they do not hit HWM.This is done without any further processing because the pipe_full() is done anyway by each pipe. Since it is doneduring writing, the program run as fast as before with the only advantage that only the slow consumers are missingnew messages.
It is fully possible to implement other type of behavior but this one seem more fair and it is quite straight forward to implement.
John 		 	   		  
_________________________________________________________________
Your E-mail and More On-the-Go. Get Windows Live Hotmail Free.
https://signup.live.com/signup.aspx?id=60969
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20100528/3a21a36b/attachment.htm>


More information about the zeromq-dev mailing list