[zeromq-dev] 0MQ protocol stack

[Martin Sustrik]

On 04/05/2011 06:35 PM, Pieter Hintjens wrote:
> Martin,
>
> Imagine there's a project that wants to use the 0MQ TCP protocol for
> something, but implement its own stacks. We know that there aren't
> proper specifications for the 0MQ stack, but my questions are rather:
> what should we call this protocol, and how should the stack be broken
> up. E.g. do we want a spec for the framing, and another for
> handshaking over TCP, and another for PGM, and another for inproc?
>
> If we can decide on the shape of the stack, and names for the pieces,
> it becomes a lot easier to fill in the blanks.

There's are horizontal pieces of the stack: transport-specific layer (if 
at all), basic framing (that's already documented), hop-by-hop layer and 
end-to-end layer.

Then there are vertical pieces of the stack, i.e. messaging patterns.

> Given that the protocol is going to change for 3.0 I think it's an
> opportunity to spec it out *before* implementing it, rather than
> making the same mistake as before. This would give people a chance to
> review and improve, rather than argue over a defacto implementation.

The only thing changed atm is that the TCP connection initialisation. 
Instead of a simple message, a multipart message is sent where each part 
constitutes one property.

However, I would not document that yet, as I am not at all confident in 
the model. In the first place we should understand why are the 
additional initialisation properties needed on top of TCP stack all. 
Here are the three I am aware of now:

1. Identity. I believe identity is an hack, but one we cannot get rid of 
because of backward compatibility.

2. Socket type. This would help to assume that different messaging 
patterns (protocols) don't cross-connect randomly. However, TCP already 
has a mechanism for dealing with this: the ports. So the question is why 
we should layer duplicit functionality on top (port scarceness? working 
around firewalls? etc.)

3. Subports. Once again, this is a hack to work around firewalls.

In short, the whole initiation thing looks like a collection of hacks. 
It would be good to have some essential use case before formalising it.

Some additional concerns:

1. Using 0MQ message framing for initiation properties is mislayering. 
We can obviously state that the format of initiation data coincides with 
message format only accidentally, but the implementors will be strongly 
tempted to embed this mislayering (i.e. initiation is not part of 
transport, but rather a generic feature common to all transports) into 
the implementations.

2. If the TCP connection initiation should be there, it should lend 
itself to deep packet inspection, so that network can detect it and act 
accordingly. However, that directly contradicts the goal of working 
around firewalls, see above. It looks like in the future deep paket 
inspection will be used for same purpose as firewall.

3. In the future I would like to steer usage of multi-part messages 
towards labeling (to be used for message routing/filtering) as opposed 
to random lists of user data. Using multi-part message to pass 
initiation properties contradicts that goal.

4. I think hardware people and transport layer (L4) people should really 
have a say in this matter, but we lack that kind of expertise here.

> I'm happy to work on writing specs, but only within a framework (i.e.
> that stack and the names for pieces) that's been agreed upon.

Well, we have the framing format written up. I am not sure there is any 
other piece of the stack that is uncontroversial enough to be formally 
specified.

Martin