<div style="font:14px/1.5 'Lucida Grande', '微软雅黑';color:#333;"><p style="font:14px/1.5 'Lucida Grande';margin:0;">Hi,</p><p style="font:14px/1.5 'Lucida Grande';margin:0;">I am run some test with fio on XFS, and I found that buffer IO is very poor. These are some result:</p><p style="font:14px/1.5 'Lucida Grande';margin:0;"><br></p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>read(iops)<span class="Apple-tab-span" style="white-space:pre"> </span>write(iops)</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;">direct IO + ext3<span class="Apple-tab-span" style="white-space:pre"> </span>1848 <span class="Apple-tab-span" style="white-space:pre"> </span>1232<span class="Apple-tab-span" style="white-space:pre"> </span></p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;">buffer IO + ext3<span class="Apple-tab-span" style="white-space:pre"> </span>1976<span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span>1319<span class="Apple-tab-span" style="white-space:pre"> </span></p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;">direct IO + XFS<span class="Apple-tab-span" style="white-space:pre"> </span>1954<span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span>1304<span class="Apple-tab-span" style="white-space:pre"> </span></p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;">buffer IO + XFS<span class="Apple-tab-span" style="white-space:pre"> </span>307<span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span>203</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><br></p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;">I do not understand why such a big difference？ext3 is much better.</p><p class="MsoNormal"><font face="Lucida Grande">direct IO parameters:</font></p><p class="MsoNormal">fio --filename=/data1/fio.dat —direct=1 --thread --rw=randrw --rwmixread=60 --ioengine=libaio --runtime=300 --iodepth=1 --size=40G --numjobs=32 -name=test_rw --group_reporting --bs=16k —time_base </p><p class="MsoNormal"><br></p><p class="MsoNormal">buffer IO parametes:</p><p class="MsoNormal">fio --filename=/data1/fio.dat --direct=0 --thread --rw=randrw --rwmixread=60 --ioengine=libaio --runtime=300 --iodepth=1 --size=40G --numjobs=32 -name=test_rw --group_reporting --bs=16k —time_base </p><p class="MsoNormal"><br></p><p class="MsoNormal">the system I've used for my tests:</p><p class="MsoNormal">HW server: 4 cores (Intel), 32GB RAM, running RHEL 6.5</p><p class="MsoNormal">Kernel: 2.6.32-431.el6.x86_64</p><p class="MsoNormal">storage: 10disks RAID1+0, stripe size: 256KB </p><p class="MsoNormal"><br></p><p class="MsoNormal">XFS format parametes:</p><p class="MsoNormal">#mkfs.xfs -d su=256k,sw=5 /dev/sdb1</p><p class="MsoNormal">#cat /proc/mounts</p><p class="MsoNormal">/dev/sdb1 /data1 xfs rw,noatime,attr2,delaylog,nobarrier,logbsize=256k,sunit=512,swidth=2560,noquota 0 0</p><div>#fdisk -ul</div><div><div>Device Boot Start End Blocks Id System</div><div>/dev/sdb1 128 2929356359 1464678116 83 Linux</div></div><div><br></div><div><br></div><div><div># fio --filename=/data1/fio.dat --direct=0 --thread --rw=randrw --rwmixread=60 --ioengine=libaio --runtime=300 --iodepth=1 --size=40G --numjobs=32 -name=test_rw --group_reporting --bs=16k --time_base </div><div>test_rw: (g=0): rw=randrw, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=1</div><div>...</div><div>test_rw: (g=0): rw=randrw, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=1</div><div>fio-2.0.13</div><div>Starting 32 threads</div><div>Jobs: 32 (f=32): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [100.0% done] [5466K/3644K/0K /s] [341 /227 /0 iops] [eta 00m:00s]</div><div>test_rw: (groupid=0, jobs=32): err= 0: pid=5711: Wed Feb 11 15:26:30 2015</div><div> read : io=1442.2MB, bw=4922.3KB/s, iops=307 , runt=300010msec</div><div> slat (usec): min=7 , max=125345 , avg=5765.52, stdev=3741.61</div><div> clat (usec): min=0 , max=192 , avg= 2.72, stdev= 1.12</div><div> lat (usec): min=7 , max=125348 , avg=5770.09, stdev=3741.68</div><div> clat percentiles (usec):</div><div> | 1.00th=[ 1], 5.00th=[ 2], 10.00th=[ 2], 20.00th=[ 2],</div><div> | 30.00th=[ 2], 40.00th=[ 3], 50.00th=[ 3], 60.00th=[ 3],</div><div> | 70.00th=[ 3], 80.00th=[ 3], 90.00th=[ 3], 95.00th=[ 4],</div><div> | 99.00th=[ 4], 99.50th=[ 4], 99.90th=[ 14], 99.95th=[ 16],</div><div> | 99.99th=[ 20]</div><div> bw (KB/s) : min= 16, max= 699, per=3.22%, avg=158.37, stdev=85.79</div><div> write: io=978736KB, bw=3262.4KB/s, iops=203 , runt=300010msec</div><div> slat (usec): min=10 , max=577043 , avg=148215.93, stdev=125650.40</div><div> clat (usec): min=0 , max=198 , avg= 2.50, stdev= 1.26</div><div> lat (usec): min=11 , max=577048 , avg=148220.20, stdev=125650.94</div><div> clat percentiles (usec):</div><div> | 1.00th=[ 1], 5.00th=[ 1], 10.00th=[ 1], 20.00th=[ 2],</div><div> | 30.00th=[ 2], 40.00th=[ 2], 50.00th=[ 3], 60.00th=[ 3],</div><div> | 70.00th=[ 3], 80.00th=[ 3], 90.00th=[ 3], 95.00th=[ 3],</div><div> | 99.00th=[ 4], 99.50th=[ 6], 99.90th=[ 14], 99.95th=[ 14],</div><div> | 99.99th=[ 17]</div><div> bw (KB/s) : min= 25, max= 448, per=3.17%, avg=103.28, stdev=46.76</div><div> lat (usec) : 2=6.40%, 4=88.39%, 10=4.93%, 20=0.27%, 50=0.01%</div><div> lat (usec) : 100=0.01%, 250=0.01%</div><div> cpu : usr=0.00%, sys=0.13%, ctx=238853, majf=18446744073709551520, minf=18446744073709278371</div><div> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%</div><div> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%</div><div> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%</div><div> issued : total=r=92296/w=61171/d=0, short=r=0/w=0/d=0</div><div><br></div><div>Run status group 0 (all jobs):</div><div> READ: io=1442.2MB, aggrb=4922KB/s, minb=4922KB/s, maxb=4922KB/s, mint=300010msec, maxt=300010msec</div><div> WRITE: io=978736KB, aggrb=3262KB/s, minb=3262KB/s, maxb=3262KB/s, mint=300010msec, maxt=300010msec</div><div><br></div><div>Disk stats (read/write):</div><div> sdb: ios=89616/55141, merge=0/0, ticks=442611/171325, in_queue=613823, util=97.08%</div></div><div><br></div>
<p class="MsoNormal"><span lang="EN-US"><o:p></o:p></span></p><p class="MsoNormal"><span lang="EN-US"><br></span></p>
<p style="font:14px/1.5 'Lucida Grande';margin:0;"><span lang="EN-US" style="font-size:10.5pt;mso-bidi-font-size:
11.0pt;font-family:Calibri;mso-ascii-theme-font:minor-latin;mso-fareast-font-family:
宋体;mso-fareast-theme-font:minor-fareast;mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";mso-bidi-theme-font:minor-bidi;
mso-ansi-language:EN-US;mso-fareast-language:ZH-CN;mso-bidi-language:AR-SA"><br></span></p><p style="font:14px/1.5 'Lucida Grande';margin:0;"><br></p><p style="font:14px/1.5 'Lucida Grande';margin:0;"><br></p></div>