Performance problem - reads slower than writes

Fri Feb 3 12:47:23 CST 2012

On Wed, Feb 01, 2012 at 07:25:26AM +1100, Dave Chinner wrote:
> The only thing changing the inode size will have affected is the
> directory structure - maybe your directories are small enough to fit
> in line, or the inode is large enough to keep it in extent format
> rather than a full btree. In either case, though, the directory
> lookup will require less IO.

I've done a whole bunch of testing, which I won't describe in detail unless
you're interested, but I've finally found out what's causing the sudden
change in performance.

With defaults, the files in one directory are spread all over the
filesystem.  But with -i size=1024, the files in a directory are stored
adjacent to each other. Hence reading all the files in one directory
requires far less seeking across the disk, and runs about 3 times faster.

Here is the filesystem on a disk formatted with defaults:

root at storage1:~# find /data/sdc | head -20 | xargs xfs_bmap 
/data/sdc: no extents
/data/sdc/Bonnie.26384:
	0: [0..31]: 567088..567119
/data/sdc/Bonnie.26384/00000:
	0: [0..7]: 567120..567127
/data/sdc/Bonnie.26384/00000/0icoeTRPHKX0000000000:
	0: [0..1015]: 4411196808..4411197823
/data/sdc/Bonnie.26384/00000/Q0000000001:
	0: [0..1543]: 1466262056..1466263599
/data/sdc/Bonnie.26384/00000/JFXQyeq6diG0000000002:
	0: [0..1295]: 2936342144..2936343439
/data/sdc/Bonnie.26384/00000/TK7ciXkkj0000000003:
	0: [0..1519]: 4411197824..4411199343
/data/sdc/Bonnie.26384/00000/0000000004:
	0: [0..1207]: 1466263600..1466264807
/data/sdc/Bonnie.26384/00000/acJKZWAwEnu0000000005:
	0: [0..1223]: 2936343440..2936344663
/data/sdc/Bonnie.26384/00000/9wIgxPKeI4B0000000006:
	0: [0..1319]: 4411199344..4411200663
/data/sdc/Bonnie.26384/00000/C6QLFdND0000000007:
	0: [0..1111]: 1466264808..1466265919
/data/sdc/Bonnie.26384/00000/6xc1Wydh0000000008:
	0: [0..1223]: 2936344664..2936345887
/data/sdc/Bonnie.26384/00000/0000000009:
	0: [0..1167]: 4411200664..4411201831
/data/sdc/Bonnie.26384/00000/HdlN0000000000a:
	0: [0..1535]: 1466265920..1466267455
/data/sdc/Bonnie.26384/00000/52IabyC5pvis000000000b:
	0: [0..1287]: 2936345888..2936347175
/data/sdc/Bonnie.26384/00000/LvDhxcdLf000000000c:
	0: [0..1583]: 4411201832..4411203415
/data/sdc/Bonnie.26384/00000/08P3JAR000000000d:
	0: [0..1255]: 1466267456..1466268711
/data/sdc/Bonnie.26384/00000/000000000e:
	0: [0..1095]: 2936347176..2936348271
/data/sdc/Bonnie.26384/00000/s0gtPGPecXu000000000f:
	0: [0..1319]: 4411203416..4411204735
/data/sdc/Bonnie.26384/00000/HFLOcN0000000010:
	0: [0..1503]: 1466268712..1466270215

And here is the filesystem created with -i size=1024:

root at storage1:~# find /data/sdb | head -20 | xargs xfs_bmap 
/data/sdb: no extents
/data/sdb/Bonnie.26384:
	0: [0..7]: 243752..243759
	1: [8..15]: 5526920..5526927
	2: [16..23]: 7053272..7053279
	3: [24..31]: 24223832..24223839
/data/sdb/Bonnie.26384/00000:
	0: [0..7]: 1465133488..1465133495
/data/sdb/Bonnie.26384/00000/0icoeTRPHKX0000000000:
	0: [0..1015]: 1465134032..1465135047
/data/sdb/Bonnie.26384/00000/Q0000000001:
	0: [0..1543]: 1465135048..1465136591
/data/sdb/Bonnie.26384/00000/JFXQyeq6diG0000000002:
	0: [0..1295]: 1465136592..1465137887
/data/sdb/Bonnie.26384/00000/TK7ciXkkj0000000003:
	0: [0..1519]: 1465137888..1465139407
/data/sdb/Bonnie.26384/00000/0000000004:
	0: [0..1207]: 1465139408..1465140615
/data/sdb/Bonnie.26384/00000/acJKZWAwEnu0000000005:
	0: [0..1223]: 1465140616..1465141839
/data/sdb/Bonnie.26384/00000/9wIgxPKeI4B0000000006:
	0: [0..1319]: 1465141840..1465143159
/data/sdb/Bonnie.26384/00000/C6QLFdND0000000007:
	0: [0..1111]: 1465143160..1465144271
/data/sdb/Bonnie.26384/00000/6xc1Wydh0000000008:
	0: [0..1223]: 1465144272..1465145495
/data/sdb/Bonnie.26384/00000/0000000009:
	0: [0..1167]: 1465145496..1465146663
/data/sdb/Bonnie.26384/00000/HdlN0000000000a:
	0: [0..1535]: 1465146664..1465148199
/data/sdb/Bonnie.26384/00000/52IabyC5pvis000000000b:
	0: [0..1287]: 1465148200..1465149487
/data/sdb/Bonnie.26384/00000/LvDhxcdLf000000000c:
	0: [0..1583]: 1465149488..1465151071
/data/sdb/Bonnie.26384/00000/08P3JAR000000000d:
	0: [0..1255]: 1465151072..1465152327
/data/sdb/Bonnie.26384/00000/000000000e:
	0: [0..1095]: 1465152464..1465153559
/data/sdb/Bonnie.26384/00000/s0gtPGPecXu000000000f:
	0: [0..1319]: 1465153560..1465154879
/data/sdb/Bonnie.26384/00000/HFLOcN0000000010:
	0: [0..1503]: 1465154880..1465156383

All the files in one directory are close to that directory; when you get to
another directory the block offset jumps.

This is a highly desirable property when you want to copy all the files: for
example, using this filesystem I can tar it up and untar it onto another
filesystem at 73MB/s, as compared to about 25MB/sec on a default filesystem.

So now my questions now are:

(1) Is this a fluke? What is it about -i size=1024 which causes this to
happen?

(2) What is the intended behaviour for XFS: that files should be close to
their parent directory or spread across allocation groups?

I did some additional tests:

* -i size=512
Files spread around

* -n size=16384
Files spread around

* -i size=1024 -n size=16384
Files local to directory

* -i size=2048
Files local to directory

Any clues gratefully received. This usage pattern (dumping in a large
library of files, and then processing all those files sequentially) is an
important one for the system I'm working on.

Regards,

Brian.