[zeromq-dev] slow consumer

John Skyfox johnskyfox at hotmail.com
Fri May 28 13:20:09 CEST 2010




> Date: Fri, 28 May 2010 12:41:01 +0200
> From: sustrik at 250bpm.com
> To: zeromq-dev at lists.zeromq.org
> Subject: Re: [zeromq-dev] slow consumer
> 
> John Skyfox wrote:
> 
> > If HWM is set to unlimited the publisher will eventually crash when it 
> > will run
> > out of resources. What I was thinking (and this is the change that I 
> > actually did) is to
> > allow the publisher to skip only those consumers that are too slow:
> > 
> > Before sending a message, the publisher use the function check_write(). 
> > if the queue is 
> > full (high watermark hit) the publisher will stop writing new messages 
> > until the pipe that
> > was stalled become available again. On a slow network this can take some 
> > seconds.
> > 
> > What I did is to change the pipe_full() function and the write() 
> > function so that when a pipe 
> > hit the HWM it will stop writing but just release the message counter. 
> > Those consumers that retrieve normally
> > will not be penalized because the publisher will still send them new 
> > messages as long as they do not hit HWM.
> > This is done without any further processing because the pipe_full() is 
> > done anyway by each pipe. Since it is done
> > during writing, the program run as fast as before with the only 
> > advantage that only the slow consumers are missing
> > new messages.
> 
> Hm. How does that differ from the standard behaviour offered by mainline 
> 0MQ at the moment? I believe PUB/SUB sockets handle slow consumers 
> exactly this way.

Not really. If you look at the code for the pub.cc (I am using the 2.0.6 that I downloaded from the ømq web pages) you will find the the check_write() is done before writing anydata to the pipes. And the check_write() will fail if one pipe is stalled (which is what happens when one of them is too slow and you have a HWM, it will get eventually full and pipe_full() will be true). What I did is to postpone this test until the writting. if one pipe is full it is skipped and the system will move to the next pipe that is not full.I haven't checked the current devel. version so I do not know if it is done this way.
> Martin
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 		 	   		  
_________________________________________________________________
Hotmail: Trusted email with powerful SPAM protection.
https://signup.live.com/signup.aspx?id=60969
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20100528/616010ec/attachment.htm>


More information about the zeromq-dev mailing list