[zeromq-dev] Notify send()er that they've hit the high water mark

[Chuck Remes]

On Jul 10, 2012, at 10:31 AM, Edwin Amsler wrote:

> On 10/07/2012 1:59 AM, Paul Colomiets wrote:
>> 
>> Hi Edwin,
>> 
>> The behavior is intentional for pub/sub sockets. If you'd have only
>> one subscriber you could use push/pull. Push sockets block when reach
>> high water mark, so are Req sockets.
>> 
>> The pub/sub sockets can't reliably block in general case because there
>> could be multiple subscribers, only one of which reaches high water
>> mark. Partially this comes from implementation: when you do zmq_send()
> I never want them to block. That'd kind of break my program's state machine. What I'd like, is to know when the high water mark was reached with a send() error, and and EAGAIN errno so that I can choose whether or not I'd like to throw away the message. Right now, the message is just silently cast away further down the stack as far as     I can tell.

I need to stress a particular detail here. The socket has a separate HWM for *each* connected subscriber because each one has its own outgoing queue. If you were to receive EAGAIN (or an error return code) from zmq_send() when a subset of subscribers hit their HWM, your published message would still be received by subscribers that have not fallen behind. It does *not* drop the message for all subscribers, just those that have entered the HWM exceptional state.

I don't think you can (even with a patch specific to your use case) rely on this behavior to achieve your goal.

>> zeromq starts to push message to the pipe for each connected
>> subscribers, if one of the pipes full, there is no way to rollback
>> messages already put into pipes, if it would it's not clear whether
>> single slow subscriber should stop the publisher.  Also pgm sockets
>> can't have backpressure AFAIU.
> Is there a way to detect if a pipe is full, and return a send() error then? As I said, I feel that pub sockets should never block (that's why I changed from using OpenPGM alone to ZeroMQ actually, though I've found yet more reasons to stay). 

Not at this time. However, there has been some work done by others to let developers receive notification for specific internal library events. I'm certain it would be possible to add this type of event as a notification.

Also, PUB sockets never block.

>> You can try to implement the behavior as a socket option (it can't be
>> default behavior), but you have to be aware of the problems above (And
>> I don't know if core developers are willing to accept the patch).
> That's more what I'm worried about. I'm quite confident there's no way to handle this yet, and I feel that I'm not the only one who'd like to have some sense of how fast a pub socket is truthfully sending data. The implementation on the API side should be solid since the send() error, EAGAIN method is how all ZMQ_NOBLOCK sockets work already. It's just that send() doesn't error when a PUB socket starts throwing away data (I know that's the case for sure on Windows at least).

What we really need is a better mechanism for handling this type of situation than HWM. One particular thought is described here:

http://unprotocols.org/blog:15

This can be implemented today using DEALER/ROUTER sockets.

You may also be interested in the Suicidal Snail pattern described in the guide.

http://zguide.zeromq.org/page:all#Slow-Subscriber-Detection-Suicidal-Snail-Pattern

If you haven't read the guide yet, you should do so before getting too much deeper into your problem space. It will save you a lot of time.

cr


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20120710/6baf8de1/attachment.htm>