[zeromq-dev] High water mark notification for publisher

[Edwin Amsler]

Heh. Not quite.

In my particular case, I'm creating data at about 500MB/s (a file read 
in this case that's limited by in-memory kernel file cache), and I'm 
capping the outbound bandwidth at around 40Mb (~4MB/s) for OpenPGM. Not 
setting a high water mark, I quickly eat all available memory and my 
system swaps heavily (and indefinitely depending on the file I'm 
sending). Setting it to 128 messages (~1MB/message which is big, I'll 
admit), means that the first 128 megabytes go out just fine, then chunks 
here and there make it out when the queues begin to empty.

I also have some control data here and there in band, but I need to hope 
I have room in the send queue so they don't just get silently thrown out.

I have a mechanism out of band over TCP that re-requests pieces once the 
transfer is done, but I'm never actually sure when it's done sending so 
I just wait 1 minute before re-requesting.

If I had some indicator of whether or not the message goes missing, I 
could re-transmit or throttle back the 500MB/s to what the network is 
actually able to provide.

On 22/09/2012 10:04 PM, Bennie Kloosteman wrote:
> ADSL upload ???
>
> On Sun, Sep 23, 2012 at 10:48 AM, Edwin Amsler 
> <edwinamsler at thinkboxsoftware.com 
> <mailto:edwinamsler at thinkboxsoftware.com>> wrote:
>
>     It's unlikely that an application can produce more data per second
>     than
>     the network hardware is able to handle?
>
>     On 22/09/2012 12:57 AM, Pieter Hintjens wrote:
>     > On Sat, Sep 22, 2012 at 12:57 AM, Edwin Amsler
>     > <edwinamsler at thinkboxsoftware.com
>     <mailto:edwinamsler at thinkboxsoftware.com>> wrote:
>     >
>     >> It was mentioned that under the hood, the PUB-SUB system had
>     individual
>     >> outgoing queues, each with their own water mark counters. What
>     happens
>     >> to a message when all queues are full?
>     > This is such an unlikely case... almost contrived. The real
>     issue with
>     > high-speed pub/sub is a small number of clients disconnecting or
>     > getting swamped by other work, and their queues building up, and
>     > causing memory exhaustion.
>     >
>     > The best strategy to keep data flowing but also ensure
>     reliability is
>     > then some kind of out-of-band recovery for clients that need it.
>     > There's some ideas in the Clone pattern in the Guide (request
>     snapshot
>     > at startup, then apply changes as they arrive to the snapshot).
>     >
>     > -Pieter
>     > _______________________________________________
>     > zeromq-dev mailing list
>     > zeromq-dev at lists.zeromq.org <mailto:zeromq-dev at lists.zeromq.org>
>     > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>     _______________________________________________
>     zeromq-dev mailing list
>     zeromq-dev at lists.zeromq.org <mailto:zeromq-dev at lists.zeromq.org>
>     http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20120922/9fef171c/attachment.htm>