xfs
[Top] [All Lists]

Bad performance with XFS + 2.6.38 / 2.6.39

To: XFS group <xfs@xxxxxxxxxxx>
Subject: Bad performance with XFS + 2.6.38 / 2.6.39
From: Xupeng Yun <xupeng@xxxxxxxxx>
Date: Sun, 11 Dec 2011 20:45:14 +0800
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:from:date:x-google-sender-auth:message-id :subject:to:content-type; bh=Ap9jSj8YeirGKh0ufZN6qIIetrPWKF7RDtuPHn8Uoas=; b=WrxpBiSFDhhR5/UdjIPtMPZaHubWaiKmqqm1hYHLWCbnATE4KL7LMWikMtVhV5XceP 5R1GT3f6NUGwpyWb43faT3jR4AaBlAcXYC4+p8P19PyLl09ocOzdqQWoPpAqCvYsIZpw zn1cQa5+4TBGFM1C00daet3YFwY+9aQ59KRO0=
Sender: recordus@xxxxxxxxx
Hi,

I am using XFS + 2.6.29 on my MySQL servers, they perform great. 

I am testing XFS on SSD these days, due to the fact that FITRIM support of XFS was 
shipped with Linux kernel 2.6.38 or newer, I tested XFS + 2.6.38 and XFS + 2.6.39, but
it surprises me that the performance of XFS with these two versions of kernel drops so
much.

Here are the results of my tests with fio, all these two tests were taken on the same hardware
with same testing environment (except for different kernel version).

====== XFS + 2.6.29 ======

# mount | grep /mnt/xfs
/dev/sdc1 on /mnt/xfs type xfs (rw,noatime,nodiratime,nobarrier,logbufs=8)
# fio --filename=/mnt/xfs/test --direct=1 --rw=randrw --bs=16k --size=50G --numjobs=16 --runtime=120 --group_reporting --name=test --rwmixread=90 --thread --ioengine=psync
test: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1
...
test: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1
fio 1.58
Starting 16 threads
test: Laying out IO file(s) (1 file(s) / 51200MB)
Jobs: 16 (f=16): [mmmmmmmmmmmmmmmm] [100.0% done] [181.5M/21118K /s] [11.4K/1289 iops] [eta 00m:00s]
test: (groupid=0, jobs=16): err= 0: pid=8446
read : io=21312MB, bw=181862KB/s, iops=11366 , runt=120001msec
clat (usec): min=80 , max=146337 , avg=1369.72, stdev=1026.26
lat (usec): min=81 , max=146338 , avg=1370.87, stdev=1026.27
bw (KB/s) : min= 6998, max=13600, per=6.26%, avg=11376.13, stdev=499.42
write: io=2369.4MB, bw=20218KB/s, iops=1263 , runt=120001msec
clat (usec): min=67 , max=145760 , avg=268.28, stdev=894.06
lat (usec): min=67 , max=145761 , avg=269.46, stdev=894.09
bw (KB/s) : min= 509, max= 2166, per=6.26%, avg=1265.42, stdev=213.82
cpu : usr=11.09%, sys=44.83%, ctx=26015341, majf=0, minf=8396
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w/d: total=1363980/151635/0, short=0/0/0
lat (usec): 100=0.11%, 250=5.85%, 500=3.79%, 750=0.32%, 1000=5.51%
lat (msec): 2=80.06%, 4=1.26%, 10=3.07%, 20=0.01%, 50=0.01%
lat (msec): 100=0.01%, 250=0.01%

Run status group 0 (all jobs):
READ: io=21312MB, aggrb=181862KB/s, minb=186227KB/s, maxb=186227KB/s, mint=120001msec, maxt=120001msec
WRITE: io=2369.4MB, aggrb=20217KB/s, minb=20703KB/s, maxb=20703KB/s, mint=120001msec, maxt=120001msec

Disk stats (read/write):
sdc: ios=1361926/151423, merge=0/0, ticks=1793432/27812, in_queue=1820240, util=99.99%




====== XFS + 2.6.39 ======

# mount | grep /mnt/xfs
/dev/sdc1 on /mnt/xfs type xfs (rw,noatime,nodiratime,nobarrier,logbufs=8)
# fio --filename=/mnt/xfs/test --direct=1 --rw=randrw --bs=16k --size=50G --numjobs=16 --runtime=120 --group_reporting --name=test --rwmixread=90 --thread --ioengine=psync
test: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1
...
test: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1
fio 1.58
Starting 16 threads
test: Laying out IO file(s) (1 file(s) / 51200MB)
Jobs: 16 (f=16): [mmmmmmmmmmmmmmmm] [100.0% done] [58416K/6680K /s] [3565 /407 iops] [eta 00m:00s]
test: (groupid=0, jobs=16): err= 0: pid=26902
read : io=6507.1MB, bw=55533KB/s, iops=3470 , runt=120004msec
clat (usec): min=155 , max=356038 , avg=4562.52, stdev=4748.18
lat (usec): min=156 , max=356038 , avg=4562.69, stdev=4748.19
bw (KB/s) : min= 1309, max= 4864, per=6.26%, avg=3479.03, stdev=441.47
write: io=741760KB, bw=6181.2KB/s, iops=386 , runt=120004msec
clat (usec): min=71 , max=348236 , avg=390.11, stdev=3106.30
lat (usec): min=71 , max=348236 , avg=390.31, stdev=3106.30
bw (KB/s) : min= 28, max= 921, per=6.29%, avg=389.02, stdev=114.68
cpu : usr=3.43%, sys=11.12%, ctx=21598477, majf=0, minf=7762
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w/d: total=416508/46360/0, short=0/0/0
lat (usec): 100=2.65%, 250=0.98%, 500=6.58%, 750=31.88%, 1000=0.27%
lat (msec): 2=0.08%, 4=0.23%, 10=55.04%, 20=1.76%, 50=0.49%
lat (msec): 100=0.02%, 250=0.01%, 500=0.01%

Run status group 0 (all jobs):
READ: io=6507.1MB, aggrb=55532KB/s, minb=56865KB/s, maxb=56865KB/s, mint=120004msec, maxt=120004msec
WRITE: io=741760KB, aggrb=6181KB/s, minb=6329KB/s, maxb=6329KB/s, mint=120004msec, maxt=120004msec

Disk stats (read/write):
sdc: ios=416285/46351, merge=0/1, ticks=108136/8768, in_queue=116368, util=93.60%


as the tests result shows, the IOPS of XFS + 2.6.29 is about 12600, but it drops to about 3900
with XFS + 2.6.39.

I tried different XFS format options and different mount options, but it did not help.

Any thought?
<Prev in Thread] Current Thread [Next in Thread>