[zeromq-dev] Behavior of Labels, Identities, and Socket Types in 3.0

[Gregory Szorc]

I went to upgrade an application from 2.1.9 to 3.0.3 and was thoroughly 
confused regarding the messaging patterns of XREQ, XREP, ROUTER, and 
DEALER sockets.

First, XREQ used to be an alias to DEALER and XREP was an alias for 
ROUTER. Now, in 3.0 they have different semantics. Wuh!? This is 
extremely confusing. I understand wanting to rename XREQ and XREP to the 
more user friendly DEALER and ROUTER names in 2.1. But then to revive 
the old-style names in 3.0 while introducing different behavior? Is it 
too late to request that XREQ and XREP (or ROUTER and DEALER) disappear 
from 3.0 and be named to something new which accurately differentiates 
them from anything before?

Now, the most confusing aspect of 3.0 has been the behavior of 
identities and labels in multi-part messages. I always thought the 
identity-messages-followed-by-empty-message-followed-by-payload-messages 
was a little awkward. So, I welcome 3.0's idea of adding a label bit to 
the wire format and denoting these "routing" messages specially (i.e. 
determinable from API call) rather than by convention. However, I think 
the current execution/implementation is extremely confusing.

Either the zmq_socket man page sections for DEALER [1] and ROUTER [2] or 
the implementation in 3.0 is completely wrong. The docs and a mailing 
list post [3] indicate that the behavior of ROUTER and DEALER are 
unchanged from 2.1 to 3.0. So, I thought as long as I made the stated 
upgrades from 2.1 to 3.0 [4], I wouldn't need to change anything else. 
In reality, I'm seeing a mix of old and new behavior in my sockets!

My topology is as follows:

[client]         [broker]         [worker]
REQ      -> ROUTER <-> DEALER <-> ROUTER

When shuffling messages from the router to dealer in the broker, I 
noticed that each message sent by the client translated into:

16 byte regular message - 4 byte label message - initial client payload

By the time the message gets to the worker (I had to rewrite my "proxy" 
function to preserve the label flag), things look like:

16 byte - 16 byte - 4 byte + label - payload message

The only difference from 2.1 is the 3rd message is a 4 byte label 
message rather than an empty message.

I thought the behavior of ROUTER and DEALER were unchanged from 2.1? Is 
the fact I'm using a REQ changing things? If REQ and REP are built on 
XREQ and XREP (as the docs say), that must mean they are using labels, 
and not the empty message convention. Is DEALER supposed to convert this 
label message to the empty message convention? Should this happen inside 
libzmq, or must an application be responsible for this conversion logic? 
If an application is responsible, could you please post the rules for 
converting message patterns between the various socket types in 3.0?

Speaking of labels, how important is the 4 byte label message I'm 
seeing. If I want to perform custom routing inside my application (so 
messages can take an independent socket path back to the originating 
peer), what parts of the label do I need to preserve? Is the 4 byte 
label message a delimiter/throwaway message like the 0 byte message in 
2.1? Is it always 4 bytes?

What are the rules on mixing ROUTER/DEALER and XREQ/XREP? zmq_socket's 
man page says they aren't compatible with each other. Yet, REQ and REP 
are compatible with ROUTER and DEALER. If REQ and REP are built upon 
XREQ and XREP, that implies XREQ and XREP are compatible with ROUTER and 
DEALER: you just need to send some leading message parts (like in 2.1). 
What's the true story?

There's a lot of questions in the previous paragraphs. I guess what I'm 
trying to say is I have no clue how things work in 3.0. Can the 3.0 
behavior please be clarified?

[1] https://github.com/zeromq/zeromq3-0/blob/master/doc/zmq_socket.txt#L132
[2] https://github.com/zeromq/zeromq3-0/blob/master/doc/zmq_socket.txt#L162
[3] http://lists.zeromq.org/pipermail/zeromq-dev/2011-September/013313.html
[4] http://www.zeromq.org/docs:3-0-upgrade

Gregory Szorc
gregory.szorc at gmail.com