XFS: Abysmal write performance because of excessive seeking (allocation groups to blame?)

[Emmanuel Florac]

Le Mon, 09 Apr 2012 00:27:29 -0500 vous écriviez:

> In your RAID10 random write testing, was this with a filesystem or
> doing direct block IO? 

Doing random IO in a file lying on an XFS filesystem.

> If the latter, I wonder if its write pattern
> is anything like the access pattern we'd see hitting dozens of AGs
> while creating 10s of thousands of files.

I suppose the file creation process to hit more some defined hot spots
than pure random access.

I just have a machine for testing purposes with 15 4TB drives in
RAID-6, not exactly a IOPS demon :)

So I've build a tar file to make it somewhat similar to OP's
problem :

root at 3[raid]# ls -lh test.tar 
-rw-r--r-- 1 root root 2,6G  9 avril 13:52 test.tar
root at 3[raid]# tar tf test.tar | wc -l
234318

# echo 3 > /proc/sys/vm/drop_caches
# time tar xf test.tar

real    1m2.584s
user    0m1.376s
sys     0m13.643s

Let's rerun it with files cached (the machine has 16 GB RAM, so
every single file must be cached):

# time tar xf test.tar

real    0m50.842s
user    0m0.809s
sys     0m13.767s

Typical IOs during unarchiving: no read, write IO bound.

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0,00  1573,50    0,00  480,50     0,00    36,96   157,52    60,65  124,45   2,08 100,10
dm-0              0,00     0,00    0,00 2067,00     0,00    39,56    39,20   322,55  151,62   0,48 100,10

The OP setup being 6 15k drives, should provide roughly the same
number of true IOPS (1200) as my slow as hell bunch of 7200RPM 
4TB drives (1500). I suppose write cache makes for most of
the difference; or else 15K drives are overrated :)

Alas, I can't run the test on this machine with ext4: I can't 
get mkfs.ext4 to swallow my big device. 

mkfs -t ext4 -v -b 4096 -n /dev/dm-0 2147483647

should work (though drastically limiting the filesystem size),
but dies miserably when removing the -n flag. Mmmph, I suppose it's 
production ready if you don't have much data to store.

JFS doesn't work either. And I was wondering why I'm using XFS?  :)

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac at intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------