[zeromq-dev] identity bug

[Serge Aleynikov]

On 5/24/2010 9:07 AM, Martin Sustrik wrote:
>> I don't think it's currently possible, but I'll ask anyway - in the
>> absence of callbacks, is there any way for 0MQ socket to receive
>> out-of-band error events?
>
> No. There's no way. The point is that when you start annoying user with
> single-connection-specific events the whole abstraction model of 0MQ
> breaks down. Eventually, you would end with what would equal to raw BSD
> sockets.

This is true for server-side applications when it's handling many 
connections, but for client-side ones there maybe only one or a few 
connections, and the client may very well be concerned about knowing 
that it's indeed connected to a server.  0MQ provides a good level of 
abstraction, however, error notifications are an important factor in 
building distributed systems.  Say if 0MQ is used for replication of 
database content between two master/standby nodes.  It wouldn't be 
appropriate for the standby node to just log a master connectivity error 
to file instead of performing immediate failover and assuming the 
master's role.  Is there another way to handle such use case with 0MQ?

>> Since 0MQ is a library, creating side effects such as writing something
>> to a log is not a very good idea - it's much better to provide some form
>> of control to the upper layer in terms of error handling.
>
> Same with me. However, I haven't thought of any sane alternative so far.
>
> Maybe a context-provided inproc PUB endpoint you can be subscribe to and
> receive error messages that way?

That would be in line with how TibcoRV communicates errors - the 
subscription subject classifies the nature of the error.

>> P.S. BTW, you mentioned that you wrote something on callbacks to the
>> mailing list, could you point me to that email?  I couldn't find
>> anything meaningful when searched for
>> http://search.gmane.org/?query=callback&group=gmane.network.zeromq.devel
>
> I cannot find it myself...
>
> Anyway:
>
> 1. The API should be agnostic about how the underlying implementation
> looks like. What if we choose to move it to kernel space? No callbacks
> from kernel threads to user space I am afraid.

Then possibly read/write calls would end up getting an error and some 
user-level library code could be available to read the error detail and 
convert it to callback...

> 2. There's no thread to execute the callbacks. There are I/O threads
> that do their work and shouldn't be blocked by calling back
> user-supplied functions. Then there are application threads which are
> under users control and thus unavailable for 0MQ to invoke callbacks.

Generally you'd need a callback not for regular use but for overcoming 
some integration problem with bridging a library with another sub-system 
with a goal of not imposing too much overhead.  I don't think it is such 
a huge design problem to permit this level of access to the lower layer 
of the stack for those rare cases that need that level of control with a 
warning that the callback is not thread-safe.  BTW, many middleware 
messaging libraries (including TibcoRV, LBM) provide such callbacks.

> 3. The ultimate goal is to integrate 0MQ API with POSIX socket API.
> There are no callbacks nin POSIX socket API.

Though POSIX AIO permits specifying callbacks (the callback parts are 
defined and implemented in user space).

> 4. Invoking callbacks from worker threads re-introduces the very thing
> 0MQ tries to avoid -- inter-thread synchronisation. Have a look at
> "multithreading magic" blog entry on zeromq.org.

Agree. Worker threads are not the right place for them.

> 5. Real men prefer to be in control rather than being controlled.
> Therefore they don't like callbacks. For quiche-eaters it's easy to
> create a wrapper that would translate received messages into callbacks.

Nice talking to a real man!  ;-)

Serge