Re: XFS performance issues: O_DIRECT and Linux 2.6.6+

[James Foris <james.foris@xxxxxxxxxx>]

James Foris wrote:
> Nathan Scott wrote:
>
> > On Wed, Aug 11, 2004 at 03:39:44PM -0500, James Foris wrote:
> >
> > > I have been using XFS in both IRIX and Linux for many years now, and 
> > > overall I am
> > > quite happy with it - recommend it to everyone, run it on my home 
> > > systems, etc.
> > >
> > > But, recently I ran into something that I need some help in 
> > > understanding/explaining.
> > > The title says it all, really, but the details follow below.
> >
> >
> >
> > Hi there,
> >
> > Before I forget completely, from talking to Christoph a few
> > days back he suggested that the problems you were seeing may
> > have been resolved by some fixes that have gone into the tree
> > in the last week or two (they certainly touch this area) - if
> > you could retry your tests and let us know whether you still
> > see this degredation on Linus' current -bk tree, that would
> > be much appreciated!

More correctly, it happened between 2.6.5 and 2.6.5-bk1

From my test logs:
-----------------------

(first entry of each set is normal, second has O_DIRECT flag added)

Linux petpc1 2.6.5 #1 SMP Sun Sep 12 17:16:05 CDT 2004 i686 i686 i386 GNU/Linux


ext2
performance numbers include fsync : 6.690 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 7.978 sec; 64.176 MB/s 13.63 % 
full
performance numbers include fsync : 0.188 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 8.118 sec; 63.067 MB/s 13.63 % 
full
ext3
performance numbers include fsync : 6.541 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 9.015 sec; 56.794 MB/s 14.48 % 
full
performance numbers include fsync : 0.003 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 7.815 sec; 65.513 MB/s 14.48 % 
full
xfs
performance numbers include fsync : 6.539 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 7.740 sec; 66.147 MB/s 13.44 % 
full
performance numbers include fsync : 0.000 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 7.278 sec; 70.353 MB/s 13.44 % 
full
--------------------------------------------------------------------------------------------------^^
jfs
performance numbers include fsync : 6.517 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 8.016 sec; 63.871 MB/s 13.47 % 
full
performance numbers include fsync : 0.000 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 7.455 sec; 68.677 MB/s 13.47 % 
full
reiserfs
performance numbers include fsync : 7.412 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 8.798 sec; 58.195 MB/s 14.26 % 
full
performance numbers include fsync : 0.004 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 50.301 sec; 10.179 MB/s 14.26 
% full



Linux petpc1 2.6.5-bk1 #2 SMP Mon Sep 13 11:05:42 CDT 2004 i686 i686 i386 
GNU/Linux

ext2
performance numbers include fsync : 6.658 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 7.937 sec; 64.512 MB/s 13.63 % 
full
performance numbers include fsync : 0.151 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 8.142 sec; 62.885 MB/s 13.63 % 
full
ext3
performance numbers include fsync : 6.293 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 8.660 sec; 59.124 MB/s 14.48 % 
full
performance numbers include fsync : 0.003 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 7.849 sec; 65.232 MB/s 14.48 % 
full
xfs
performance numbers include fsync : 6.565 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 7.782 sec; 65.793 MB/s 13.44 % 
full
performance numbers include fsync : 0.000 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 8.545 sec; 59.917 MB/s 13.44 % 
full
--------------------------------------------------------------------------------------------------^^
jfs
performance numbers include fsync : 6.494 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 8.016 sec; 63.873 MB/s 13.47 % 
full
performance numbers include fsync : 0.000 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 7.396 sec; 69.227 MB/s 13.47 % 
full
reiserfs
performance numbers include fsync : 7.151 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 8.594 sec; 59.579 MB/s 14.26 % 
full
performance numbers include fsync : 0.004 sec
wrote  512.000 MB in  16 writes, 32.000 MB/write, 88.133 sec; 5.809 MB/s 14.26 % 
full


So..... something in the 2.6.5-bk1 patchset caused the change.  Any suggestions 
where
to begin looking (other than fs/direct_io.x) ?

Thanks,

Jim Foris

> Thanks again,
>
> Jim Foris
>
> > thanks.