![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
An interesting routing problem
Jul. 5th, 2012 07:43 pmA little while ago, we discovered an interesting GPFS glitch that caused panicky unmounts on some of the systems. When we investigated in detail, we traced the problem to revoke messages from a specific NSD server in the storage cluster and when we examined the network connectivity, we realised that the topology was such that it forbade direct connections to the problem server from the compute environment. Consequently, during token operations involving more than one compute cluster — operations that required arbitration from the NSD server in order to resolve the lack of direct connectivity between the compute clusters — the NSD could not be contacted, triggering a revoke of one of the requesting.
Following this discovery, we've been working on interesting ways to try to work around the communications problem. The US suggested configuring the NSD with an IP alias in a different subnet and using one of the nodes with universal connectivity to act as a router But during a chance conversation with the colleague who was working on implementing the solution, I pointed out that we might be able to solve the problem by altering the subnet mask on the NSD server to partition the topology into directly connected systems and systems that needed to be accessed via that gateway but we would still need come up with a way to route the traffic back from the indirectly connected systems without changing their netmasks, a problem I thought was intractable. So imagine my surprise when my colleague suggested this as a solution!
According to my understanding of IP routing, the determination of whether a system is local or not is made by comparing the network components of an address with the network components of the machine's own interfaces. If the components match, the address is on the local network and the routing table is not examined. If the address is remote, attempts are made to match the target address to entries in the routing table starting with host routes and widening out until the default route is reached as a destination of last resource.
However it turns out that AIX, post 5.1 at least, does not behave like this. Rather it seems to examine the routing table regardless of the network component of the target address, allowing host specific routes to be matched ahead of the local interfaces. Thus my colleague was able to increase the size of the mask on the NSD, add host specific routes on the other side and have the traffic route through the gateway, contrary (to my, at least) theoretic expectations, giving us a nice clean solution to our underlying problem.
Following this discovery, we've been working on interesting ways to try to work around the communications problem. The US suggested configuring the NSD with an IP alias in a different subnet and using one of the nodes with universal connectivity to act as a router But during a chance conversation with the colleague who was working on implementing the solution, I pointed out that we might be able to solve the problem by altering the subnet mask on the NSD server to partition the topology into directly connected systems and systems that needed to be accessed via that gateway but we would still need come up with a way to route the traffic back from the indirectly connected systems without changing their netmasks, a problem I thought was intractable. So imagine my surprise when my colleague suggested this as a solution!
According to my understanding of IP routing, the determination of whether a system is local or not is made by comparing the network components of an address with the network components of the machine's own interfaces. If the components match, the address is on the local network and the routing table is not examined. If the address is remote, attempts are made to match the target address to entries in the routing table starting with host routes and widening out until the default route is reached as a destination of last resource.
However it turns out that AIX, post 5.1 at least, does not behave like this. Rather it seems to examine the routing table regardless of the network component of the target address, allowing host specific routes to be matched ahead of the local interfaces. Thus my colleague was able to increase the size of the mask on the NSD, add host specific routes on the other side and have the traffic route through the gateway, contrary (to my, at least) theoretic expectations, giving us a nice clean solution to our underlying problem.
