[zeromq-dev] High water mark notification for publisher

Justin Karneges justin at affinix.com
Sun Sep 23 20:34:23 CEST 2012


I am also dealing with a file-sending case and want to avoid sending at read 
speed for the same reasons. I've decided to use the credits-based flow control 
approach Pieter suggested. For pub/sub, you really only need one subscriber 
sending credits, but the nice thing about the approach is that it can be used 
for either fastest subscriber or slowest subscriber flows depending on how you 
organize sending the credits.

On Saturday, September 22, 2012 10:23:16 PM Edwin Amsler wrote:
> Heh. Not quite.
> 
> In my particular case, I'm creating data at about 500MB/s (a file read
> in this case that's limited by in-memory kernel file cache), and I'm
> capping the outbound bandwidth at around 40Mb (~4MB/s) for OpenPGM. Not
> setting a high water mark, I quickly eat all available memory and my
> system swaps heavily (and indefinitely depending on the file I'm
> sending). Setting it to 128 messages (~1MB/message which is big, I'll
> admit), means that the first 128 megabytes go out just fine, then chunks
> here and there make it out when the queues begin to empty.
> 
> I also have some control data here and there in band, but I need to hope
> I have room in the send queue so they don't just get silently thrown out.
> 
> I have a mechanism out of band over TCP that re-requests pieces once the
> transfer is done, but I'm never actually sure when it's done sending so
> I just wait 1 minute before re-requesting.
> 
> If I had some indicator of whether or not the message goes missing, I
> could re-transmit or throttle back the 500MB/s to what the network is
> actually able to provide.
> 
> On 22/09/2012 10:04 PM, Bennie Kloosteman wrote:
> > ADSL upload ???
> > 
> > On Sun, Sep 23, 2012 at 10:48 AM, Edwin Amsler
> > <edwinamsler at thinkboxsoftware.com
> > 
> > <mailto:edwinamsler at thinkboxsoftware.com>> wrote:
> >     It's unlikely that an application can produce more data per second
> >     than
> >     the network hardware is able to handle?
> >     
> >     On 22/09/2012 12:57 AM, Pieter Hintjens wrote:
> >     > On Sat, Sep 22, 2012 at 12:57 AM, Edwin Amsler
> >     > <edwinamsler at thinkboxsoftware.com
> >     
> >     <mailto:edwinamsler at thinkboxsoftware.com>> wrote:
> >     >> It was mentioned that under the hood, the PUB-SUB system had
> >     
> >     individual
> >     
> >     >> outgoing queues, each with their own water mark counters. What
> >     
> >     happens
> >     
> >     >> to a message when all queues are full?
> >     > 
> >     > This is such an unlikely case... almost contrived. The real
> >     
> >     issue with
> >     
> >     > high-speed pub/sub is a small number of clients disconnecting or
> >     > getting swamped by other work, and their queues building up, and
> >     > causing memory exhaustion.
> >     > 
> >     > The best strategy to keep data flowing but also ensure
> >     
> >     reliability is
> >     
> >     > then some kind of out-of-band recovery for clients that need it.
> >     > There's some ideas in the Clone pattern in the Guide (request
> >     
> >     snapshot
> >     
> >     > at startup, then apply changes as they arrive to the snapshot).
> >     > 
> >     > -Pieter
> >     > _______________________________________________
> >     > zeromq-dev mailing list
> >     > zeromq-dev at lists.zeromq.org <mailto:zeromq-dev at lists.zeromq.org>
> >     > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >     
> >     _______________________________________________
> >     zeromq-dev mailing list
> >     zeromq-dev at lists.zeromq.org <mailto:zeromq-dev at lists.zeromq.org>
> >     http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> > 
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev



More information about the zeromq-dev mailing list