Tales of a Fourth Grade Nothing

Today has been even tougher than I predicted yesterday, thanks to a combination of manic firefighting at work and frantic philosophy writing.

Work, in particular, was a real slog, with the day capped by a particularly nasty HSM problem on the old machine which started when we brought down Unitree in an attempt to clear a problem with a non-existant cap. The software shut itself down successfully — always a relief — but, when we attempted to restart it, the tape movers complained that their network port was busy and refused to come back up. Reconciled to a reboot, I started quiescing the batch subsystem and cleared out a whole load of stale ftpd processes. Just as I was about to start bringing the OS down, operations called me to say that the movers had just restarted themselves and that the reboot was no longer necessary.

I'm still not entirely sure what happened but my best guess is that one of the ports normally used by Unitree had been grabbed by an ftpd process which had hung because the HSM NFS archive directory hadn't been unmounted correctly when the software was first taken down. This ftpd process then blocked all the subsequent attempts to bring the HSM back up until I when through and kill them all off, freeing up the troublesome port in the process. Once I'd done this, taped, the tape service minder daemon, was able to bring the movers up, to kick start the tape server and to resume normal production.

My philosophy work too, was a bit of a death-march, but I managed to finish my current chapter ready for today's deadline. It was rather weak, with too much exposition and not enough argument, but hopefully it's enough to give an idea of the sort of thing I'm aiming for.

I'm feeling appropriately smug today: after a chance conversation with one of my cronies about the problems of spotting migrated files, I've come up with a patch for GNU ls that tags purged files with a suitably lurid colour.

The code uses a neat heuristic to spot purged files, which relies on the fact that when DiskXtender clears a file, it changes the block count. Thus, if the actual block count of a file is much lower than the expected block count, based on the size of the file and if the block count also matches the DX stub size (the portion of the file that gets left on disc to deal with commands like file), then it is a pretty safe assumption that the file has been purged. Quick and dirty it may be but, unlike other more correct methods, it has the advantage that it works over NFS.

Perhaps it was my fault. Perhaps I was unclear.

When the two HSM guys turned up to consult with my colleague, they tried to persuade him to find some spare disc space to allow them to mess around with stuff. When my colleague asked me if I knew whether a particular file system could be recycled, I said, "It's Friday. There's a bank holiday coming up. I really wouldn't make any changes to the system. If I were you, I'd leave it completely alone and sort it out next week." He looked suitably comprehending and went off. Then a few hours later, under the watchful eyes of my colleagues, the visitors were allowed to unmount one of the HSM managed file systems, the one that had spent the last 18 hours kicking up a stream of errors, and surprise surprise, the system panicked in a heap. Unwilling to sacrifice my precisely timetabled plans for this afternoon, I left others to dump the system and went home.

To my mind, today's problem simply emphasises two important rules of system administration: never, ever make changes on a Friday; and hierarchical storage managers are a total waste of time, far better to buy a bunch of cheapo discs than arse around with tapes and virtual storage.

Course wasn't as bad as expected, thanks to the combination of a good instructor and bulk quantities of cheap but effective coffee/battery acid. As time went on and we learnt extra details about DiskXtender, I started to notice more and more similarities between DX and FileTek's StorHouse right down to Centera integration...