[zeromq-dev] HWM behaviour & blocking

[Chuck Remes]

On May 9, 2012, at 11:29 AM, Steffen Mueller wrote:

> Hi Chuck,
> 
> On 05/09/2012 03:57 PM, Chuck Remes wrote:
>> On May 9, 2012, at 12:45 AM, Steffen Mueller wrote:
>>> I have an application of 0MQ where I have one or multiple server
>>> instances that use a PUSH socket to send messages that must be
>>> processed by any one of potentially many workers. The servers are
>>> single-threaded apart from 0MQ's IO thread and this is hard to
>>> change.
>>> 
>>> The documentation for the PUSH socket type (and others like XREQ)
>>> explain that sending messages on such a socket will be a blocking
>>> operation IF
>>> 
>>> - the HWM is hit - OR there are no peers
>>> 
>>> In my scenarios, I want to be resilient against intermittent
>>> client failure (due to whatever -- coordinated restart, failure,
>>> ...). But for this time, the server processes will block on the
>>> write and that is not acceptable. Is it reasonable to hope for a
>>> way to achieve the following behaviour?
>>> 
>>> Sending a messages down a socket of this time will block if:
>>> 
>>> - the HWM is hit for all peers (as per docs, presumably, this
>>> means all and each separately since the buffers are probably
>>> per-client) - OR a single global HWM is hit if there are no peers
>>> 
>>> IOW, I'd like to be able to shove up to $HWM messages down a pipe
>>> no matter what, and have the PUSH/whatever socket use a single
>>> $HWM-depth buffer if there is no peer. As soon as a peer connects,
>>> it could do one of two things:
>>> 
>>> - simply swap that buffer in to become the first connecting peer's
>>> send queue (which might be undesirable in some cases since it
>>> doesn't load balance but it's likely much easier to implement and
>>> more efficient) - use that buffer as another queue stage to load
>>> balance from
>>> 
>>> Any chance I could have such a functionality? Of course, being able
>>> to determine whether there are any peers connected would be great,
>>> too.
>> 
>> I recommend that you do non-blocking writes and check the return
>> code. If zmq_errno is equal to EAGAIN, then you know that you have
>> either hit HWM or that there are no peers.
> 
> thanks for your advice.
> 
> Alas, I can't see how that would help me with the particular issue I have. In a nutshell, I want to have the HWM apply also while there's no listeners. Right now, I'd have to do non-blocking writes and implement my own buffering. In particular when you include multi-frame messages into the picture, that's just a lot of silly effort. Thus, I am asking whether having a mode of operation or socket type in 0MQ where 0MQ does buffering and applies HWM even without currently connected peers.

This isn't supported in 0mq. It's such an odd use-case that I doubt it ever would be.

> Furthermore, "there are no peers or HWM hit" doesn't help me AT ALL in this scenario since I am attempting to *distinguish* between the two scenarios. So even if I was to implement my own message queuing in the application (yikes, really?), I would essentially do double buffering if the HWM is hit since it's not distinguishable from having no peers. IOW, what I want is currently NOT POSSIBLE AT ALL with 0MQ unless I'm missing something.

It doesn't help AT ALL? I think it does.

Let's take another stab at this. According to the FAQ (you have read it, right?), when a socket calls zmq_connect() it immediately establishes a local queue. When calling zmq_bind() it does *not* do this. So, there is some minor differences in behavior when connecting versus binding. 

So, if you call zmq_connect() on your socket and you have already set a HWM beforehand, then you can write to it and the messages will queue even if there are no peers. This sounds like what you want. You can still use the non-blocking write technique that I mentioned in my first response to determine when you have hit the HWM if you so desire. There is no way to tell if you have any peers in this (or any) scenario unless you devise a protocol for your clients and servers to use to announce themselves and whatnot. This is not built in to the library.

If this is still insufficient for your purposes, then I think what you want to accomplish is simply impossible. It's probably impossible to do in 0mq or in *any* other networking library.

Patches welcome.

cr