XFS performance issues: O_DIRECT and Linux 2.6.6+

[James Foris <james.foris@xxxxxxxxxx>]

I have been using XFS in both IRIX and Linux for many years now, and 
overall I am
quite happy with it - recommend it to everyone, run it on my home systems, etc.

But, recently I ran into something that I need some help in 
understanding/explaining.
The title says it all, really, but the details follow below.

----------------------------------------------------------------------------------------------------------------------

We have a project that requires a sustained 400 - 450 MBytes/sec data rate 
to/from
a U320 SCSI disk array.

Early performance tests demonstrated that we had to go to the Linux 2.6 kernel
to get this kind of sustained performance on commodity hardware (actually, a
standard Intel server motherboard) and this issue went on the backlburner as we
found that we had other kernel-related problems to handle, first.

Those issues are now resolved (or resolved enough - Intel will be feeding patches
into the kernel mainline and Fedora Core group at Red Hat to fix things 
permanently).

On retesting, we discovered that between kernel 2.6.5 and 2.6.6, something happened
that changed the way that O_DIRECT works with XFS; instead of increasing 
performance,
it now degrades it by ~10% (or more).

        As a side note; ext2, ext3, and jfs show the expected increase of 
performance.

These tests were done on a variety of systems, a variety of kernels (both FC and
pure "kernel.org"), and a variety of SCSI disks.  The behavior varies a little, 
but they
all show this same overall trends.  And in general, the higher speed disk subsystems
(such as S/W RAID0 on U320 SCSI drives) show the problem most.


The following test were run against a set of 1 GByte partitions formatted with 
the indicated
 file systems.  The test, itself, does the following sequence:

        while loop count < 16
                start timer
                create file
                write file
                mark time
                fsync file
                close file
                stop timer

For example, on a HP wx8000 with 10KRPM U320 SCSI disks (running simple 
partitions),
Linux kernel 2.6.8-rc2-bk5 we see the following numbers:

>>> standard file system writes <<<<

ext2
performance numbers include fsync : 4.319 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 8.138 sec; 62.916 MB/s 53.97 % 
full
ext3
performance numbers include fsync : 6.067 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 8.996 sec; 56.917 MB/s 55.66 % 
full
xfs
performance numbers include fsync : 6.799 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 7.927 sec; 64.590 MB/s 53.34 % 
full
jfs
performance numbers include fsync : 2.940 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 8.067 sec; 63.465 MB/s 53.33 % 
full


>>> file system writes using O_DIRECT <<<

ext2
performance numbers include fsync : 0.166 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 7.926 sec; 64.594 MB/s 53.98 
% full		( 3% increase )
ext3
performance numbers include fsync : 0.003 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 8.298 sec; 61.705 MB/s 55.66 
% full		( 8% increase  )
xfs
performance numbers include fsync : 0.000 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 8.703 sec; 58.830 MB/s 53.34 
% full		( 9 % DECREASE )
jfs
performance numbers include fsync : 0.000 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 7.339 sec; 69.761 MB/s 53.33 
% full		(  10% increase )



In our original test case (S/W RAID0, U320 disks, Intel server motherboard), the 
performance
degradation was:

>>> standard file system writes <<<<

ext2
performance numbers include fsync : 0.829 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 2.174 sec; 235.460 MB/s 0.19 % 
full
xfs			(from a report summary)
	about 355 MB/s on 2.6.4
	about 345 MB/s on 2.6.7

>>> file system writes using O_DIRECT <<<

ext2
performance numbers include fsync : 0.046 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 1.484 sec; 345.120 MB/s 0.19 
% full		( 46 % increase )
xfs			(from a report summary)
	about 410 MB/s on 2.6.4						( 15% increase )
	about 180 MB/s on 2.6.7						( 47% DECREASE )


Obviously, the 410 MBytes/sec performance from the 2.6.4 kernel was what we had 
been counting on.  It
looke like S/W RAID + O_DIRECT really tanks XFS performance since 2.6.6 was 
released.

Does anyone know what has changed ?  And is there anything that can be cone to 
make the current kernels
behave better ?

Jim Foris


Has anyone esle