Tales of a Fourth Grade Nothing

Tape processing is a wretched thing. But it is particularly so on a system runing an obsolete version of a jumped up desktop OS developed by a group of users with little interest in tapes and limited access to large robotic libraries.

O for the days of Unicos and tpdaemon. Sure, it wasn't perfect — I can still remember as series of regular crashes caused by tape daemon crashing with a signal 11 — but it was a damn sight better than nothing. It provided so many nice features. It arbitrated access to a pool of drives, making it easy to share them between applications; it managed both the data path to the device and the control path to the robotics; it allowed the device and channel to be varied; and provided a easy way to monitor the status of each device.

If only Linux had something similar...

Perhaps it was my fault. Perhaps I was unclear.

When the two HSM guys turned up to consult with my colleague, they tried to persuade him to find some spare disc space to allow them to mess around with stuff. When my colleague asked me if I knew whether a particular file system could be recycled, I said, "It's Friday. There's a bank holiday coming up. I really wouldn't make any changes to the system. If I were you, I'd leave it completely alone and sort it out next week." He looked suitably comprehending and went off. Then a few hours later, under the watchful eyes of my colleagues, the visitors were allowed to unmount one of the HSM managed file systems, the one that had spent the last 18 hours kicking up a stream of errors, and surprise surprise, the system panicked in a heap. Unwilling to sacrifice my precisely timetabled plans for this afternoon, I left others to dump the system and went home.

To my mind, today's problem simply emphasises two important rules of system administration: never, ever make changes on a Friday; and hierarchical storage managers are a total waste of time, far better to buy a bunch of cheapo discs than arse around with tapes and virtual storage.

For the last God knows how long, I've had an item on my todo list telling me to create a spreadsheet with details of the disc layout on the supercomputer. Trouble is, it's actually turned out to be really tough task, which probably explains why I've spent such a long time studiously ignoring it in favour of easier, lower hanging, fruit. Not only is the disc data all over the place, but the configuration is way too complicated for a simple spreadsheet and the details need to be routinely updated, so that rules out using Excel to store my results.

My latest idea involves writing a bunch of python modules to munge all the data and spit it out in a Freemind format mindmap. That should let my tie all the details together in a useful way and, if I can get the Java applet working, display it on the intranet. Of course, I expect my colleagues to put up strenuous objections to this on the grounds that (a) they disagree with the use of python and (b) probably disagree with the use of freemind... I suppose if they really don't like it, there's nothing to stop them rewriting it for themselves in Perl and Excel macros.

Thanks to the catharsis obtained through the release of all my bilious choler into my last update and thanks in no small measure to a double espresso, I managed to squeeze in enough work before hometime to prevent the day from being a total waste.

The gist of my cleverness is that after years of complaining about the impossibility of preventing df from hanging at the first sign of an NFS glitch, I bothered to the think about the problem for all of thirty seconds and lo, an answer presented itself. I came to the happy conclusion that I could open the dot directory on the target file system and then use the resulting descriptor with fstatfs to obtain usage figures and effectly dodge the path–to–device lookup badness that hobbles statfs.

Now all I have to do is sit back and wait for an NFS server crash so that I can test it. Going on past experience, I shouldn't have all that long to wait.

This afternoon, I was asked to look into some of the NFS performance problems and to come up with a plan to remove the worst of the bottlenecks. As soon as the subject came up I felt my heart sink and then later, after the discussion had finished and the full horror of the thing had sunk in, I found that I wanted to do nothing more than to lock myself in the lavatory and spend the rest of the day crying quietly...

Is there any reason why, in this day and age, right thinking, God fearing, honest folk should be forced to use the deeply obnoxious Nightmare Failure System for anything?

After spending two hours fixing up the logical volume lossage on the Inorganic Idiot, the High Commander asked me - proxying for whoever he was speaking to on the phone - why it had taken me so long complete the work. I rather haughtily replied, "It was fantastically complicated and intricate and I didn't fancy trashing 15 terabytes of data just because some people have a misplaced sense of urgency." He accepted this response with admirable aplomb.

As Quell says, Thought and Action should be held separate and should not be confused. Thus, if anybody asks, I'm a Thought Phase. God knows, I'm certainly not in an Action Phase.

Bad Day

Had a particularly unproductive morning contemplating the dirtier aspects NFS caching and attempting to determine whether the legendary terabyte FS limit exists in 2.4.18 on IA64. From this I concluded that NFS caching wasn't going to happen this side of NFSv4 and that the terabyte limit is probably mythical given the default variable sizes on IA64.

I then spent an even less productive period badmouthing the appalling support tools that our beloved vendor requires. It's like, "Yeah, we have this crappy database full of problems. Searching? What you mean for string? Who would in their right mind would want to do that? Oh, you would? Really? Oh, well, I'm afraid you can't." It's like worse than infoman, which is really saying something.

Good Day

I recommend Kate Hedstrom's skits on GNU make to anyone and everyone, whether they think they're interested in compilation systems or not. The link to the paper on why recursive make is a bad idea was particular informative - I hadn't realised just how much efficiency could be improved with ":=" until I read it.

Now that I've got all that off my chest, I'm off to write furious political poetry and to anticipate the forthcoming Action phase. Or Monday, as it's more commonly known.