[Top] [All Lists]

Re: xfs, 2.6.27=>.32 sync write 10 times slowdown [was: xfs, aacraid 2.6

To: Dave Chinner <david@xxxxxxxxxxxxx>
Subject: Re: xfs, 2.6.27=>.32 sync write 10 times slowdown [was: xfs, aacraid 2.6.27 => 2.6.32 results in 6 times slowdown]
From: Michael Tokarev <mjt@xxxxxxxxxx>
Date: Wed, 09 Jun 2010 10:43:37 +0400
Cc: Linux-kernel <linux-kernel@xxxxxxxxxxxxxxx>, xfs@xxxxxxxxxxx
In-reply-to: <20100608231845.GG7869@dastard>
Organization: Telecom Service, JSC
References: <4C0E13A7.20402@xxxxxxxxxxxxxxxx> <20100608122919.GC7869@dastard> <4C0EA938.9000104@xxxxxxxxxxxxxxxx> <20100608231845.GG7869@dastard>
User-agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv: Gecko/20100411 Icedove/3.0.4
09.06.2010 03:18, Dave Chinner wrote:
On Wed, Jun 09, 2010 at 12:34:00AM +0400, Michael Tokarev wrote:
Simple test doing random reads or writes of 4k blocks in a 1Gb
file located on an xfs filesystem, Mb/sec:

                      sync  direct
              read   write   write
2.6.27 xfs   1.17    3.69    3.80
2.6.32 xfs   1.26    0.52    5.10
2.6.32 ext3  1.19    4.91    5.02

Note the 10 times difference between O_SYNC and O_DIRECT writes
in 2.6.32.  This is, well, huge difference, and this is where
the original slowdown comes from, apparently.

Are you running on the raw block device, or on top of LVM/DM/MD to
split up the space on the RAID drive? DM+MD have grown barrier
support since 2.6.27, so it may be that barriers are now being
passed down to the raid hardware on 2.6.32 and they never were on
2.6.27. Can you paste the output of dmesg when the XFS filesystem in

That's why I asked how to tell if barriers are actually hitting the
device in question.

No, this is the only machine where DM/MD is _not_ used.  On all other
machines we use MD software raid, this machine comes with an onboard
raid controller that does not work in JBOD mode so I weren't able to
use linux software raid.  This is XFS on top of Adaptec RAID card,
nothing in-between.

Also, as I mentioned in the previous email, remounting with nobarrier
makes no difference whatsoever.

(Another side note here - I discovered that unknown options are
silently ignored in "remount mode" while correctly rejected in
"plain mount" mode, -- it looks like a kernel bug actually, but
it's entirely different issue).

question is mounted on both 2.6.27 and 2.6.32 so we can see if
there is a difference in the use of barriers?

Also, remember that O_DIRECT does not imply O_SYNC. O_DIRECT writes
only write data, while O_SYNC will also write metadata and/or the

I know this.  I also found osyncisosync and osyncisdsync mount
options, and when I try to use the latter, kernel tells it's the
default and hence deprecated.  I don't need metadata updates, but
it _looks_ like the system is doing such updates (with barriers
or flushes?) anyway even when mounted with -o osyncisdsync it behaves
the same: very slow.

I also experimented with both O_SYNC|O_DIRECT: it is as slow as
without O_DIRECT, i.e. O_SYNC makes whole thing slow regardless
of other options.

I looked at the dmesg outputs, and there's no relevant differences
related to block devices or usage of barriers.  For XFS it always
mounts like this:

 SGI XFS with ACLs, security attributes, large block/inode numbers, no debug 
 SGI XFS Quota Management subsystem
 XFS mounting filesystem sda6

and for the device in question, it is always like

 Adaptec aacraid driver 1.1-5[2456]-ms
 aacraid 0000:03:01.0: PCI INT A -> GSI 24 (level, low) -> IRQ 24
 AAC0: kernel 5.1-0[8832] Feb  1 2006
 AAC0: monitor 5.1-0[8832]
 AAC0: bios 5.1-0[8832]
 AAC0: serial 267BE0
 AAC0: Non-DASD support enabled.
 AAC0: 64bit support enabled.
 AAC0: 64 Bit DAC enabled
 scsi0 : aacraid
 scsi 0:0:0:0: Direct-Access     Adaptec  f0500            V1.0 PQ: 0 ANSI: 2
 sd 0:0:0:0: [sda] 286715904 512-byte hardware sectors (146799 MB)
 sd 0:0:0:0: [sda] Write Protect is off
 sd 0:0:0:0: [sda] Mode Sense: 06 00 10 00
 sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and 
  sda: sda1 sda2 sda3 < sda5 sda6 >

There are tons of other differences, but that is to be expected (like
format of CPU topology printing which is changed between .27 and .32).



<Prev in Thread] Current Thread [Next in Thread>