Tales of a Fourth Grade Nothing

A busy few days trying to trace the source of a substantial degradation in the performance of the batch subsystem. Examining the dispatch cycles — normally 10-20 seconds but now inflated to 130-180 seconds — I immediately suspected a bad job but was unable to trace the cause until I enabled full and performance debugging. With this switched on I noticed that each dispatch cycle contained over 330,000 lines referring to a specific job which, when examined, proved to have requested a set of resources that could never be satisfied.

Knowing the likely cause of the problem, I slapped an admin hold on the job and the cycle times promptly dropped back to something like normal. My sense of timing was, as ever, impeccable: I fixed the problem in time for a couple of the others to push off for their Christmas lunch; while I hung around for as long as I could to make sure that everything had settled before pushing off to keep my appointment in town. Excelsior!

And so, after finishing the Work That Must Be Finished and doing a global stop and start LoadLeveler has settled itself back down and started scheduling again as normal. I'm more than slightly relieved — not only has my vision of days of instability followed by a cold start vanished, but my original hypothesis about the cause of the problem seems to have been justified.

Despite being told that the latest LoadLeveler documentation had been released, I was still thrown by the title of the installation guide: Tivoli Workload Scheduler LoadLeveler for AIX V5.1 Installation Guide, as in the AIX version of LoadLeveler 5.1 and not, as I original interpreted it, the AIX 5.1 specific version of LoadLeveler...

I've recently spent some time churning through LoadLeveler's job history file, trying to tease out some trends from the mass of data. What I really want is a way to quantify the relationship between project priority and step queue time, in the hope that this will show that higher priority projects spend less time queuing. Unfortunately the picture is greatly complicated by the fact that step sizes and durations differ between projects, and that the backfill scheduling algorithm is more likely to select smaller, shorter jobs regardless of their priority.

If only I had a better grasp of statistical methods...

I've had a surprisingly productive day testing out some of IBM's suggestions for how we might improve our LoadLeveler configuration. Of these, the most promising seems to be the recommendation that we enable a limited form of preemption by:

1. Setting PREEMPTION_SUPPORT = full;
2. Setting DEFAULT_PREEMPT_METHOD = vc;
3. Setting RESERVATION_PRIORITY = high;

This gives the reservations a higher priority than the running work and, if there are insufficient resources for the reservation when it goes into the SETUP state, the jobs running on the nodes assigned to reservation will be cleared out using the default preempt method. In this case I opted for vacate, which kills the job and requeues it, rather than hold or suspend, because we don't have enough paging space to allow two full memory sized jobs to coexist on a node.

Although I think that preemption may well fix most of our problems, I don't think it is a magic bullet. For one thing it doesn't, by itself, grant us any control over which jobs are preempted — something our current script does for us. But it might be possible to deal with this using another suggestions: that we split the system into two separate pools and use dummy jobs to limit the reservations to a single pool, while allowing the rest of the work to run in either pool.

I think we could probably better this by allowing all lower priority work to run in either pool, while restricting high priority work to the non-reservation pool. Something we could probably do by checking the priority of the job in the filter as it is submitted and by adding a # @ pool = parameter as it passes through.

I've discovered a nasty little problem with LoadLeveler reservations. A problem which means that somes, just sometimes, the reservation actually prevents the job from running, but which is kind of hard to describe.

Imagine a pair of reservations. Both reservations have at least one node in common between them, but because the first reservation is due to finish before the second is due to start, this is not a problem.

Now, consider what happens when RESERVATION_CAN_BE_EXCEEDED is set to true in the LoadL_config file and a job is submitted into the first reservation. If the job has a wallclock time that is longer that the reservation, but which means that it will finish because the second reservation becomes active, there is no problem. But if the wallclock time of the job means that it will end after the second reservation is due to start — i.e. if the current time plus the wallclock time of the job is larger than the start time of the second reservation — the two resource requests will clash and consequently, the job will not run.

Thus, under certain circumstances, running a job in a reservation may actually hamper its execution, rather than assist it. And, worst still, because the placement of reservations is dependent on the state of the system, the problem does not necessarily occur repeatably...

I've spent the last week or two working on a script to create instant LoadLeveler reservations, even in situations where resource shortages would normally prevent them from being created. In the process, I've uncovered a couple of LoadLeveler gotchas.

I've found that a job that starts and finds that it cannot write to its stdout or stderr files will go into user hold. Unless the initial directory is set with "# @ initialdir", then the initial directory is the current working directory of the llsubmit used to submit the job. If the directory is not writable by the job owner and the error and output paths are not explicitly routed to another directory, the job will not start to run.

This unpleasant fact caused us much worry when we discovered that jobs submitted using a script — a script that resided in a directory that was owned by a user other than the one submitting the job — went into user hold when submitted interactively; while near-identical jobs submitted as part of a job chain from a running batch job, whose working directory was set to $HOME ran flawlessly.

Also, if an existing job is cancelled to make room for a new reservation, it takes a few seconds to release its resources. Once the resources are released, the next idle job will jump into the gap unless the scheduling temporarily paused causing the attempt to create a new reservation to fail. If the scheduling is paused by draining the schedulers, no new work will be started and, and here's the real gotcha, now new jobs will be accepted for submission into the batch system, until the schedulers are resumed.

There may be a solution to this dilemma. Perhaps, if all jobs are allocated floating resources at submit time, the resource limit could be reduced slightly prior to the attempt to create the reservation in order to stop any new jobs from starting. Then, once the reservation has grabbed the newly released jobs, the limit could be brought back up to allow normal work to continue to schedule.

Still, these minor problems aside, it looks as though I might be on course to have something flawed but functional working by the end of the week — which precisely matches my initial estimate that it would take two weeks to get sorted out.