Temporary files on GPFS
Nov. 17th, 2011 08:40 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
sawylAfter much consideration and many, many experiments I think I've finally cracked the annoying performance problem I've been investigating for most of past couple of weeks. Essentially, the problem was that the run times for a series of standard builds which used to complete in around 900 seconds had suddenly started taking much longer than expected — up to three times longer in some cases.
With the times varying from routine to routine across runs, indicating that the problem was independent of the amount of compute work being performed, I immediately suspected the root cause might lie somewhere in the IO subsystem. I ran a series of experiments: parallel builds with a range of different source, destination and compiler configurations, all bundled up in resubmitting job chain to control for the usual diurnal fluctuations in load. Initial results were disappointing until I added a step to control for the location of the temporary directory. This generated a clear signal: changing
A further set of experiments, this time using a trivial bit of code to time the performance of the
All of which suggests that it ought to be possible to significantly improve compilation performance by changing the location of
ETA: Finally traced the root cause of the problem back to an application configuration file that was changed a couple of weeks ago, overriding the default value of
With the times varying from routine to routine across runs, indicating that the problem was independent of the amount of compute work being performed, I immediately suspected the root cause might lie somewhere in the IO subsystem. I ran a series of experiments: parallel builds with a range of different source, destination and compiler configurations, all bundled up in resubmitting job chain to control for the usual diurnal fluctuations in load. Initial results were disappointing until I added a step to control for the location of the temporary directory. This generated a clear signal: changing
$TMPDIR to point to something other than the default location — the root directory of a busy GPFS file system optimised for large block IO — gave noticeably superior performance.A further set of experiments, this time using a trivial bit of code to time the performance of the
mkstemp() call under a range of different conditions confirmed the problem: that the large block GPFS file systems offered suboptimal performance when working with small files and that the performance of the call fell off noticeably as the target directory filled up with files. This last test offered some interesting insights in the behaviour of GPFS metadata caching, with the performance falling off when all the inodes were created in quick succession but not if a few minutes were allowed to pass between file creation runs (presumably because this gives sufficient time for the metadata to synchronise).All of which suggests that it ought to be possible to significantly improve compilation performance by changing the location of
$TMPDIR to point to a user-specific empty directory on a GPFS file system optmised for small block IO.ETA: Finally traced the root cause of the problem back to an application configuration file that was changed a couple of weeks ago, overriding the default value of
$TMPDIR and changing it to point to the GPFS file system. Mystery solved!
