[zeromq-dev] PGM and sending interface

[Jim Hague]

On Tuesday 05 Jan 2016 10:20:49 Steven McCoy wrote:
> On 5 January 2016 at 09:41, Jim Hague <jim.hague at acm.org> wrote:
> > ZMQ pgm_socket.cpp calls pgm_getaddrinfo() to parse the transport, and
> > then uses the interface number when setting up the call to pgm_bind3().
> > Inside pgm_bind3(), pgm_if_indextoaddr() is used to get the address used
> > in the call to bind(). The socket is therefore always bound to the first IP
> > address associated with the interface, and packets sent out have that
> > address as their origin. And I need the packet origin address to be the
> > second IP address. Am I missing something?
> 
> I think I remember this is due to IPv6 addressing requiring a bind on the
> local link instead of the global link for addressing.  I'm thus not sure
> how one would distinguish between IPv6 and AIX interface logic.

I think AIX is a bit of a red herring here. I can produce exactly the same 
scenario on Linux using iproute2 (

# ip address
[...]
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP 
group default qlen 1000
    link/ether ec:f4:bb:67:fd:09 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.153/23 brd 10.0.1.255 scope global dynamic eth0
       valid_lft 3445sec preferred_lft 3445sec
    inet6 fe80::eef4:bbff:fe67:fd09/64 scope link 
       valid_lft forever preferred_lft forever
# ip address add 192.168.5.1/24 dev eth0
# ip address
[...]
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP 
group default qlen 1000
    link/ether ec:f4:bb:67:fd:09 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.153/23 brd 10.0.1.255 scope global dynamic eth0                
       valid_lft 3349sec preferred_lft 3349sec                                 
    inet 192.168.5.1/24 scope global eth0                                      
       valid_lft forever preferred_lft forever                                 
    inet6 fe80::eef4:bbff:fe67:fd09/64 scope link                              
       valid_lft forever preferred_lft forever              

Listing the interfaces from one of the OpenPGM examples:
$ ./daytime -i
Info: #1 name lo (lo)         ----                                                
scope 0 status UP   loop YES b/c NO  m/c NO 
Info: #2 name eth0 (eth0)     ----                                                
scope 0 status UP   loop NO  b/c YES m/c YES
Info: #1 name lo (lo)         IPv4 127.0.0.1                                      
scope 0 status UP   loop YES b/c NO  m/c NO 
Info: #2 name eth0 (eth0)     IPv4 10.0.0.153                                     
scope 0 status UP   loop NO  b/c YES m/c YES
Info: #2 name eth0 (eth0)     IPv4 192.168.5.1                                    
scope 0 status UP   loop NO  b/c YES m/c YES
Info: #1 name lo (lo)         IPv6 ::1                                            
scope 0 status UP   loop YES b/c NO  m/c NO 
Info: #2 name eth0 (eth0)     IPv6 fe80::eef4:bbff:fe67:fd09%eth0                 
scope 2 status UP   loop NO  b/c YES m/c YES
Info: Default network: "127.0.1.1;239.192.0.1"

The current OpenPGM API seems to have an assumption that an interface can only 
have one address assigned. With modern Linux IP aliasing (and AIX IP aliasing) 
it looks like that's no longer true :-( Right now I'm playing with adding an 
alias number alongside the interface number.

> What happens with IPv6 on AIX?  With AIX Linux compatibility how do the
> APIs getifaddrs() and  if_indextoaddr()  work?

I haven't tried IPv6 on AIX. And AIX Linux compatibility really a marketing 
fancy. My AIX changes include a commit with the comment:

***
AIX doesn't have getifaddrs(), chiz chiz. Instead we have to use the much
dreaded SIOCGIFCONF.

AIX also has sa_len in struct sockaddr, and the existing SIOCGIFCONF code
doesn't step through the interfaces correctly. Add a configure check
for sa_len and modify the stepping through the data returned from the ioctl.
***

> See the code comment:
> 
> /* interfaces indexes refer to the link layer, we want to find the internet
> layer address.
>  * the big problem is that multiple IPv6 addresses can be bound to one link
> - called scopes.
>  * we can just pick the first scope and let IP routing handle the rest.
>  */

I'm not sure that IPv6 scopes help. A scope seems to me to be an indication of 
the link type. I've just tried, and on Linux at least you can cheerfully add 
IP aliases with the same scope - here's an example with two global scope 
addresses, and you can have multiple link scope addresses too.

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP 
group default qlen 1000
    link/ether ec:f4:bb:67:fd:09 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.153/23 brd 10.0.1.255 scope global dynamic eth0
       valid_lft 2541sec preferred_lft 2541sec
    inet6 2001:41c8:52:189:feff:ff:fe00:b1c/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 2001:41c8:51:189:feff:ff:fe00:b1c/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::eef4:bbff:fe67:fd09/64 scope link 
       valid_lft forever preferred_lft forever
-- 
Jim Hague - jim.hague at acm.org          Never trust a computer you can't lift.