[zeromq-dev] Notify send()er that they've hit the high water mark

[Edwin Amsler]

On 10/07/2012 1:59 AM, Paul Colomiets wrote:
> Hi Edwin,
>
> On Mon, Jul 9, 2012 at 11:24 PM, Edwin Amsler
> <edwinamsler at thinkboxsoftware.com>  wrote:
>> So here I am, publishing messages through ZeroMQ's send() function at about
>> 300MB/s, and my network's set to only send at 10MB/s.
>>
>> This is kind of a big problem because, while I don't care if the clients
>> loose data on their end, the server is either using its memory until it
>> crashes, or I'm setting its high water mark and loosing about 29 of every
>> thirty messages I produce because I don't know that ZeroMQ can't keep up.
>>
>> Ideally, when a HWM condition happened, send() would return false, then I'd
>> test EAGAIN so I could decide for myself whether I should drop the message,
>> or retry later. With that kind of functionality, I could throttle back my
>> producer algorithm so that I exactly meet the demand of ZeroMQ instead of
>> overwhelming/starving it out.
>>
>> I'm willing to do the work if this sort of addition makes sense to the rest
>> of the project. I'd rather contribute here instead of forking it off in some
>> forgotten repository.
>>
>> Can/should this be done? Is there someone out there willing to mentor me?
>>
> The behavior is intentional for pub/sub sockets. If you'd have only
> one subscriber you could use push/pull. Push sockets block when reach
> high water mark, so are Req sockets.
>
> The pub/sub sockets can't reliably block in general case because there
> could be multiple subscribers, only one of which reaches high water
> mark. Partially this comes from implementation: when you do zmq_send()
I never want them to block. That'd kind of break my program's state 
machine. What I'd like, is to know when the high water mark was reached 
with a send() error, and and EAGAIN errno so that I can /choose/ whether 
or not I'd like to throw away the message. Right now, the message is 
just silently cast away further down the stack as far as I can tell.
> zeromq starts to push message to the pipe for each connected
> subscribers, if one of the pipes full, there is no way to rollback
> messages already put into pipes, if it would it's not clear whether
> single slow subscriber should stop the publisher.  Also pgm sockets
> can't have backpressure AFAIU.
Is there a way to detect if a pipe is full, and return a send() error 
then? As I said, I feel that pub sockets should never block (that's why 
I changed from using OpenPGM alone to ZeroMQ actually, though I've found 
yet more reasons to stay).
> You can try to implement the behavior as a socket option (it can't be
> default behavior), but you have to be aware of the problems above (And
> I don't know if core developers are willing to accept the patch).
That's more what I'm worried about. I'm quite confident there's no way 
to handle this yet, and I feel that I'm not the only one who'd like to 
have some sense of how fast a pub socket is truthfully sending data. The 
implementation on the API side should be solid since the send() error, 
EAGAIN method is how all ZMQ_NOBLOCK sockets work already. It's just 
that send() doesn't error when a PUB socket starts throwing away data (I 
know that's the case for sure on Windows at least).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20120710/f8595faf/attachment.htm>