Tales of a Fourth Grade Nothing

I've discovered a couple of elegant spin-offs from my current work on GPFS. Playing around with the pool structures and calls in the API, I've found it to be much faster than mmdf, largely because the latter goes out to each NSD and queries it for its usage information — something that's not of much interest if you just want to know whether the pools are full or not.

I've also realised that if I combine a couple of other calls, I can both identify the fileset that owns a particular directory and pull out the three levels of quota — user, group, fileset — for it. This makes it trivially easy to summarise all the limits imposed on a particular user in a way that indicates both their current level of consumption, how much they have left, and the level at which the resource is being constrained.

And best of all, the code to do all this can be wrapped up in a handful of Cython calls. OK so the fileset identification code seems to need an inode scan, which seems to mean it can't be run as a normal user, and I haven't yet found a way to dump out a complete list of filesets defined within the file system — although this information must be available be mmlsfileset and mmrepquota can do it — but it's an extremely handy thing to get out of something originally intended for a completely different purpose...

Today's increadably neat discovery? GPFS callbacks. I know these have been around for a while but until a suggestion from one of my colleagues prodded me into reading the documetation, I hadn't realised just how many things you can do with them.

My immediate ideas include: starting and stopping the batch software when its spool file system mounts or unmounts; automatically exporting file systems via NFS to other systems; requesting hardware replacements when the declustered arrays reach their maximum number of failures; raising alerts on reconstruct failures; and those are just my first ideas. The potential opportunities for laziness, impatience and hubris are vast!

A little while ago, we discovered an interesting GPFS glitch that caused panicky unmounts on some of the systems. When we investigated in detail, we traced the problem to revoke messages from a specific NSD server in the storage cluster and when we examined the network connectivity, we realised that the topology was such that it forbade direct connections to the problem server from the compute environment. Consequently, during token operations involving more than one compute cluster — operations that required arbitration from the NSD server in order to resolve the lack of direct connectivity between the compute clusters — the NSD could not be contacted, triggering a revoke of one of the requesting.

Following this discovery, we've been working on interesting ways to try to work around the communications problem. The US suggested configuring the NSD with an IP alias in a different subnet and using one of the nodes with universal connectivity to act as a router But during a chance conversation with the colleague who was working on implementing the solution, I pointed out that we might be able to solve the problem by altering the subnet mask on the NSD server to partition the topology into directly connected systems and systems that needed to be accessed via that gateway but we would still need come up with a way to route the traffic back from the indirectly connected systems without changing their netmasks, a problem I thought was intractable. So imagine my surprise when my colleague suggested this as a solution!

According to my understanding of IP routing, the determination of whether a system is local or not is made by comparing the network components of an address with the network components of the machine's own interfaces. If the components match, the address is on the local network and the routing table is not examined. If the address is remote, attempts are made to match the target address to entries in the routing table starting with host routes and widening out until the default route is reached as a destination of last resource.

However it turns out that AIX, post 5.1 at least, does not behave like this. Rather it seems to examine the routing table regardless of the network component of the target address, allowing host specific routes to be matched ahead of the local interfaces. Thus my colleague was able to increase the size of the mask on the NSD, add host specific routes on the other side and have the traffic route through the gateway, contrary (to my, at least) theoretic expectations, giving us a nice clean solution to our underlying problem.