[zeromq-dev] identity bug

[Martin Sustrik]

Serge,

>>> I don't think it's currently possible, but I'll ask anyway - in the
>>> absence of callbacks, is there any way for 0MQ socket to receive
>>> out-of-band error events?
>> No. There's no way. The point is that when you start annoying user with
>> single-connection-specific events the whole abstraction model of 0MQ
>> breaks down. Eventually, you would end with what would equal to raw BSD
>> sockets.
> 
> This is true for server-side applications when it's handling many 
> connections, but for client-side ones there maybe only one or a few 
> connections, and the client may very well be concerned about knowing 
> that it's indeed connected to a server. 

I think this have been discussed in the past... My understanding so far 
is the clients who care about error should use a different library, one 
that speaks 0MQ wire protocol but has no message queues, isn't able to 
talk to multiple endpoints etc. In short, a glorified TCP.

> 0MQ provides a good level of 
> abstraction, however, error notifications are an important factor in 
> building distributed systems.  Say if 0MQ is used for replication of 
> database content between two master/standby nodes.  It wouldn't be 
> appropriate for the standby node to just log a master connectivity error 
> to file instead of performing immediate failover and assuming the 
> master's role.  Is there another way to handle such use case with 0MQ?

For example, you can connect your client to 2 servers. If both are 
available, client simply load balances the requests among them. If one 
goes offline, it's queue gets filled (you have to specify the high 
watermark) and subsequent requests go only to the second server. It's 
not yet perfect but that's the idea.

>> 1. The API should be agnostic about how the underlying implementation
>> looks like. What if we choose to move it to kernel space? No callbacks
>> from kernel threads to user space I am afraid.
> 
> Then possibly read/write calls would end up getting an error and some 
> user-level library code could be available to read the error detail and 
> convert it to callback...

Still, I would prefer dumb sync client sockets as described above.

>> 2. There's no thread to execute the callbacks. There are I/O threads
>> that do their work and shouldn't be blocked by calling back
>> user-supplied functions. Then there are application threads which are
>> under users control and thus unavailable for 0MQ to invoke callbacks.
> 
> Generally you'd need a callback not for regular use but for overcoming 
> some integration problem with bridging a library with another sub-system 
> with a goal of not imposing too much overhead.  I don't think it is such 
> a huge design problem to permit this level of access to the lower layer 
> of the stack for those rare cases that need that level of control with a 
> warning that the callback is not thread-safe.  BTW, many middleware 
> messaging libraries (including TibcoRV, LBM) provide such callbacks.

My point was that the separate thread for callbacks is not mandatory (if 
you choose not to use callbacks), thus is should be layered on top of 
existing library, so that it is not forced on those that don't need it.

>> 3. The ultimate goal is to integrate 0MQ API with POSIX socket API.
>> There are no callbacks nin POSIX socket API.
> 
> Though POSIX AIO permits specifying callbacks (the callback parts are 
> defined and implemented in user space).

Good point!

>> 4. Invoking callbacks from worker threads re-introduces the very thing
>> 0MQ tries to avoid -- inter-thread synchronisation. Have a look at
>> "multithreading magic" blog entry on zeromq.org.
> 
> Agree. Worker threads are not the right place for them.
> 
>> 5. Real men prefer to be in control rather than being controlled.
>> Therefore they don't like callbacks. For quiche-eaters it's easy to
>> create a wrapper that would translate received messages into callbacks.
> 
> Nice talking to a real man!  ;-)

Martin