GPFS inode and generation numbers...
Apr. 11th, 2014 01:25 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
sawylDuring testing of my current project — a file tree analyser that uses of the GPFS API to speed up access to the metadata — we discovered an interesting bug that resulted in some of the file ownerships being incorrectly attributed.
Investigating, I quickly realised that the fault was caused by a race condition in the code: because the dump of file namespace took a non-trivial amount of time to complete, it was possible for a file to be removed after its name had been retried but before its inode data had been read and for the inode number to be reused by a new file. This meant that it when the file names were joined with the file attributes using the inode number as a unique key, the original file path was assigned the attributes — including the ownership — of the new file. Once I understood this, I changed the join to use inode number and generation ID in place of simple inode number only to find that this completely broke the matching process
Digging deeper into the code and printing out some of the generation numbers, I discovered that the values returned by
Looking more closely, I eventually realised that the lowest 16 bits of the two generation IDs were the same while the highest 16 bits were only set in
Investigating, I quickly realised that the fault was caused by a race condition in the code: because the dump of file namespace took a non-trivial amount of time to complete, it was possible for a file to be removed after its name had been retried but before its inode data had been read and for the inode number to be reused by a new file. This meant that it when the file names were joined with the file attributes using the inode number as a unique key, the original file path was assigned the attributes — including the ownership — of the new file. Once I understood this, I changed the join to use inode number and generation ID in place of simple inode number only to find that this completely broke the matching process
Digging deeper into the code and printing out some of the generation numbers, I discovered that the values returned by
gpfs_ireaddir() in the form of a gpfs_direntx_t failed to match those returned by gpfs_next_inode() in a gpfs_iattr_t structure. From the sorts of values being returned, I wondered whether the problem might be caused by a mismatch between the variable types and replaced the 32 bit routines with their 64 bit equivalents only to experience exactly the same problem.Looking more closely, I eventually realised that the lowest 16 bits of the two generation IDs were the same while the highest 16 bits were only set in
gpfs_iattr_t.ia_gen. Masking the field appropriately, I was able to combine the generation ID with the inode number to create compound key that I was able to use to join both structures in a coherent way, trading one form of inaccuracy — incorrectly assigned ownerships — for a more acceptable one — ignoring files deleted and recreated during the scan.
