[zeromq-dev] memory pb

John Skyfox johnskyfox at hotmail.com
Wed May 26 13:38:54 CEST 2010


Hi all,
I would like to thank you all for your reply. I end up with some kind of home made solution because I had 2 requirements:
1) A client should not be able to crash the server (or the publisher) because it is too slow in retrieving data.2) Messages are valid within a certain time limit because they are correlated in time with other events happening somewhere else. So a message    produce 5 seconds ago should not be correlated with an event that that was produced some milliseconds ago.
What I did, was to add an option that allows me to control whether I want to drop messages based on a the difference between msg_written and msg_read.I agree with Martin that I cannot know exactly the last message but actually I do not that but I can remove the last n oldest messages on the queue (or pipe) that is on the server.I fact, if the buffer is big anough, clients do not have any problem reading the messages but if by accident on client is on a slow network, it should be allow to loose message not crashing the server or puting it in a mode where none the of the client is receiving new messages (this was the behavior I was getting if I try to use high watermark in order tolimit the memory consumption of the server). The test I had mad so far seems to work pretty well (I have try a publisher with n clients subscribing to it, and some of them wason a slow network and the behavior was as expected. None of the client that was retrieving data at a "normal" speed loose data but eventually the slow client loose some data and the memory consumption of the server stay within its limit (depending on the size of the buffer which is a function of the difference between the msg_written and msg_read).
--John

> Date: Mon, 24 May 2010 23:50:43 +0200
> From: sustrik at 250bpm.com
> To: zeromq-dev at lists.zeromq.org
> Subject: Re: [zeromq-dev] memory pb
> 
> Hi Burak,
> > It sounds like you need a LIFO queue. It may sound silly, but what
> > you're asking for is done, to some extent, in memcached.
> > 
> > It'd certainly be interesting to generalize zeromq by having
> > customizable queuing schemes, (a la priority_queue) but I don't know
> > whether core devs are working on it, or even willing to accept a patch
> > that introduces such a change at the very core of things.
> 
> Check my previous post in the thread. My argument was that even if you 
> implement a LIFO to store the messages, the rest of the network behaves 
> in FIFO way (that's the nature of networking). Thus what you'll get 
> would be some kind of LIFO/FIFO mixture with no clear semantics.
> 
> On the other hand, priority queues maintain the FIFO semantics so there 
> should be no crucial problem with implementing it.
> 
> Martin
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 		 	   		  
_________________________________________________________________
Your E-mail and More On-the-Go. Get Windows Live Hotmail Free.
https://signup.live.com/signup.aspx?id=60969
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20100526/314203c0/attachment.htm>


More information about the zeromq-dev mailing list