[zeromq-dev] What differentiates connected peers on a ZMTP point of view ?

[Goswin von Brederlow]

On Mon, Feb 10, 2014 at 05:31:34PM +0100, Laurent Alebarde wrote:
> Le 10/02/2014 10:43, Goswin von Brederlow a écrit :
> >On Thu, Feb 06, 2014 at 04:34:03PM +0100, Laurent Alebarde wrote:
> >>Hi Devs,
> >>
> >>I wonder please what differentiates connected peers on a ZMTP point
> >>of view ?
> >>
> >>To be more clear, let's consider this network:
> >>
> >>Client 1 205.23.12.47
> >>---------------------------------------------------- Server
> >>Client 2 102.12.88.254 --------------------------------------------------/
> >>
> >>In ZMTP internals, I assume it creates a pipe associated with each
> >>origin address, and on an API side, it provides one identity
> >>associated to each one. So it creates one mechanism for each pipe
> >>and knows which instance to use every time in the stateful handcheck
> >>process. Is it correct ? there is a bijective association between
> >>the origin address and the identity ?
> >>
> >>Let's say that now we have a proxy in between:
> >>
> >>
> >>Client 1 205.23.12.47 ----------------------- Proxy
> >>----------------------- Server
> >>Client 2 102.12.88.254 ------------------/92.123.321.22
> >>
> >>ZMTP on server side has no way to differentiate between Client1 and
> >>2. Every message arrives in the same pipe which is the one
> >>corresponding to its closest peer: the proxy and its address. The
> >>only way for the server to differentiate the clients is at
> >>application level, when identities are stacked in the former frames
> >>of each message ?
> >>
> >>I am a bit confused. Can someone clarify for me please and point in
> >>the libzmq code what is taken into account to identify the pipes ?
> >http://zguide.zeromq.org/page:all#The-Extended-Reply-Envelope
> >
> >A zmq message does not have just one identity. It has any number of
> >identity frames followed by an empty delimiter frame followed by data
> >frames. Normaly on each hop an identity frame is added. Depending on
> >the socket type when sending the first identity is used to decied
> >where to send the message and on recieve the identity of the sender is
> >added to show where it came from:
> >
> >Client 1 sends:    '' 'request'                      # REQ adds ''
> >Proxy recieves:    <client 1> '' 'request'           # ROUTER adds <id>
> >Proxy sends:       <client 1> '' 'request'           # DEALER
> >Server recieves:   <proxy> <client 1> '' 'request'   # REP handles <id>s
> >Server replies     <proxy> <client 1> '' 'reply'     # REP handles <id>s
> >Proxy recieves:    <client1> '' 'reply'              # DEALER
> >Proxy sends:       <client1> '' 'reply'              # ROUTER strips <id>
> >Client 1 recieves: '' 'reply'                        # REQ strips ''
> >
> >The REP/REQ sockets handle the identity frames for you so the
> >appliction never sees them. ROUTER adds/strips the first identity
> >automatically while DEALER lets you see them all.
> Thanks Goswin,
> 
> My question here is on the internals of libzmq: how it manages
> identities ws pipes.
> Continued below.
> >>So, if my above hypothesis are right, as I want to manage to proxy a
> >>ZMTP mechanism like in the following:
> >>
> >>_______________client A___________________
> >>____________________server_________________
> >>proxy                  __worker B __
> >>                     _________tunel A______ _____midpoint_____
> >>Client  --inproc--- frontend   /   backend ----tcp---- frontend /
> >>backend --inproc-- Worker
> >>DEALER              ZMQ_STREAM      DEALER ROUTER  ZMQ_STREAM
> >>DEALER
> >>CURVE |              CURVE
> >>|
> >>
> >>I have to create a pool of sockets in the proxy backend, say 1,000
> >>if I want to authorize 1,000 simultaneous connexions, and dispatch
> >>the clients to a sticky socket, and one and only one client per
> >>socket.
> >>\|/
> >>Client 1 ----------------------------------------------> socket 1:
> >>port 10001
> >>Client 2 ----------------------------------------------> socket 2:
> >>port 10002
> >>Client 1000 ----------------------------------------------> socket
> >>1000: port 11000
> >>
> >>So, the worker is binded to all these ports. Still, does it work
> >>here ?  Does ZMTP creates a pipe per address on the worker side,
> >>even if they are binded to the same socket (the answer looks obvious
> >>but I prefer a confirmation) ?
> >>
> >>
> >>Cheers,
> >>
> >>
> >>Laurent
> >No. You use one socket for everything and add the identity of the
> >client to the message as you proxy it. Then on the reply you get the
> >identity of the client back so you know where to send it. Since ROUTER
> >adds the identity implicitly all a proxy has to do is real all message
> >frames from the ROUTER and write them to the DEALER and vice versa.
> >zmq does the rest as if it where magic.
> The worker being set with CURVE, the "classic" backtrack you
> describe cannot work. I cannot attach a backtrack when I send a
> message from a proxy to a CURVE worker.
> 
> >
> >MfG
> >	Goswin

Sure you can. The CURVE encryption is irelevant and transparent to the
design. What you have is a simple case of tunneling:

Client ---- Tunnel A -- T1 -- T2 -- T3 -- Tunnel B ---- Server

You have 2 routing chains here. One is the tunneling itself:

Tunnel A -- T1 -- T2 -- T3 -- Tunnel B

Messages send from Tunnel A should arrive Tunnel B with the IDs of
Tunnel A, T1, T2 and T3 in the message so the reply goes back to
Tunnel A. Although if the tunnel is strickly a 1-1 connection you can
skip the IDs since there is always only one way a message can go. 

The other routing is between Client and Server. If you use STREAM
sockets then for them the tunnel is transparent. 

Client ----<Tunnel>---- Server

So messages send by Client arrive at Server with the ID of Client in
the message, which will be associated with the Tunnel B endpoint.
Replies send to the client will go to the Tunnel B socket and get
forwarded to Client with an ID of Server, associated with the Tunnel A
endpoint.

That is if you use STREAM sockets to tunnel the raw traffic. The
identity the client and server sockets publish will get tunneled and
associated with the tunnel endpoints. Or you get random IDs for them.
For the application it will be the IDs of Client and Server directly.

MfG
	Goswin