<div> Dave,</div><div>Thank you for your explanation. I got the reason, and I write some code to simulate the MySQL.It will reproduce the progress:<span id="_editor_bookmark_start_26" style="display: none; line-height: 0px;"></span></div><div><br></div><div>open file without direct flag</div><div>read file //cause kernel readahead 4 pages, and inode->i_mapping->nrpages > 0</div><div>close file</div><div><br></div><div>open file with direct flag</div><div>lseek 4*4096 // skip 4 readahead pages</div><div>read file //cause xfs_flushinval_pages to do nothing</div><div>...</div><div><br></div><div>I'd like to ask XFS how to resovle this problem?</div><div><br></div><div>Attach code：<span id="_editor_bookmark_start_28" style="display: none; line-height: 0px;"></span></div><div>
<p class="p1">/* gcc -o test_read test_read.c</p>
<p class="p1"> * dd if=/dev/zero of=/data1/fo.dat bs=4096 count=10</p>
<p class="p1"> * ./test_read /data1/fo.dat 2 direct</p>
<p class="p1"> * */</p>
<p class="p2">#define _GNU_SOURCE</p>
<p class="p2">#include <span class="s1"><stdio.h></span></p>
<p class="p3"><span class="s2">#include </span><unistd.h></p>
<p class="p3"><span class="s2">#include </span><sys/types.h></p>
<p class="p3"><span class="s2">#include </span><sys/stat.h></p>
<p class="p2">#include <span class="s1"><fcntl.h></span></p>
<p class="p2">#include <span class="s1"><errno.h></span></p>
<p class="p3"><span class="s2">#include </span><string.h></p>
<p class="p3"><span class="s2">#include </span><stdlib.h></p>
<p class="p4"><br></p>
<p class="p2">#define BUFSIZE <span class="s3">4096</span></p>
<p class="p4"><br></p>
<p class="p5"><span class="s4">int</span> read_count = <span class="s3">2</span>;</p>
<p class="p4"><br></p>
<p class="p5"><span class="s4">int</span> main(<span class="s4">int</span> argc, <span class="s4">char</span> *argv[]){</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="s4">if</span>(argc < <span class="s3">3</span>){</p>
<p class="p3"><span class="s5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>fprintf(stderr, </span>"usage: %s <file> <count> [buffer|direct]\n"<span class="s5">, argv[</span><span class="s3">0</span><span class="s5">]);</span></p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>exit(<span class="s3">1</span>);</p>
<p class="p5"><span class="Apple-tab-span"> </span>}</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="s4">char</span> *buf = memalign(BUFSIZE - <span class="s3">1</span>, BUFSIZE);</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="s4">char</span> *file = argv[<span class="s3">1</span>];</p>
<p class="p5"><span class="Apple-tab-span"> </span>read_count = atoi(argv[<span class="s3">2</span>]);</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="s4">int</span> ret = <span class="s3">0</span>,sum = <span class="s3">0</span>, i = <span class="s3">0</span>, fd = -<span class="s3">1</span>;</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="s4">if</span>(argc == <span class="s3">4</span> && strncmp(argv[<span class="s3">3</span>], <span class="s1">"direct"</span>,<span class="s3">6</span>) == <span class="s3">0</span>){</p>
<p class="p1"><span class="s5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span></span>//fd = open(file, O_RDONLY|O_DIRECT);</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>fd = open(file, O_RDONLY);</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="s4">if</span>(fd < <span class="s3">0</span>){</p>
<p class="p3"><span class="s5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>fprintf(stderr, </span>"open read only file failed\n"<span class="s5">);</span></p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>exit(<span class="s3">1</span>);</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>}</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>ret = read(fd, buf, BUFSIZE);</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="s4">if</span>(ret < <span class="s3">0</span>){</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>fprintf(stderr, <span class="s1">"buffer read error\n"</span>);</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>}</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>close(fd);</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>fd = open(file, O_RDWR);</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="s4">if</span>(fd < <span class="s3">0</span>){</p>
<p class="p3"><span class="s5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>fprintf(stderr, </span>"open read only file failed\n"<span class="s5">);</span></p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>exit(<span class="s3">1</span>);</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>}</p>
<p class="p4"><br></p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="s4">if</span> (fcntl(fd, F_SETFL, O_RDONLY|O_DIRECT) == -<span class="s3">1</span>) {</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>fprintf(stderr, <span class="s1">"set direct error\n"</span>);</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>exit(<span class="s3">1</span>);</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>}</p>
<p class="p4"><br></p>
<p class="p6"><span class="s5"><span class="Apple-tab-span"> </span>}</span>else<span class="s5">{</span></p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>fd = open(file, O_RDONLY);</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="s4">if</span>(fd < <span class="s3">0</span>){</p>
<p class="p3"><span class="s5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>fprintf(stderr, </span>"open buf file failed\n"<span class="s5">);</span></p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>exit(<span class="s3">1</span>);</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>}</p>
<p class="p5"><span class="Apple-tab-span"> </span>}</p>
<p class="p4"><br></p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="s4">while</span>(i++ < read_count){</p>
<p class="p1"><span class="s5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span></span>//memset(buf, 0, BUFSIZE);</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="s4">if</span>(buf == <span class="s4">NULL</span>){</p>
<p class="p3"><span class="s5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>fprintf(stderr, </span>"memory allocate failed\n"<span class="s5">);</span></p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>exit(<span class="s3">1</span>);</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>}</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="s4">if</span>(lseek(fd, <span class="s3">4</span>*<span class="s3">4096</span>, SEEK_SET) < <span class="s3">0</span>){</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>fprintf(stderr, <span class="s1">"seek error!\n"</span>);</p>
<p class="p6"><span class="s5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span></span>break<span class="s5">;</span></p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>}</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>ret = read(fd, buf, BUFSIZE);</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="s4">if</span>(ret > <span class="s3">0</span>){</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>sum += ret;</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>}<span class="s4">else</span> <span class="s4">if</span>(ret == <span class="s3">0</span>){</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>printf(<span class="s1">"read end\n"</span>);</p>
<p class="p6"><span class="s5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span></span>break<span class="s5">;</span></p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>}</p>
<p class="p6"><span class="s5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span></span>else<span class="s5">{</span></p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>printf(<span class="s1">"error:%d\n"</span>, errno);</p>
<p class="p6"><span class="s5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span></span>break<span class="s5">;</span></p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>}</p>
<p class="p5"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>sleep(<span class="s3">1</span>);</p>
<p class="p5"><span class="Apple-tab-span"> </span>}</p>
<p class="p3"><span class="s5"><span class="Apple-tab-span"> </span>printf(</span>"read sum: %d\n"<span class="s5">, sum);</span></p>
<p class="p5"><span class="Apple-tab-span"> </span>close(fd);</p>
<p class="p5"><span class="Apple-tab-span"> </span>free(buf);</p>
<p class="p6"><span class="s5"> </span>return<span class="s5"> </span><span class="s3">0</span><span class="s5">;</span></p>
<p class="p5">}<span id="_editor_bookmark_start_27" style="display: none; line-height: 0px;"></span></p></div><div><div><br></div><div><br></div><div style="font-size: 12px;font-family: Arial Narrow;padding:2px 0 2px 0;">------------------ 原始邮件 ------------------</div><div style="font-size: 12px;background:#efefef;padding:8px;"><div><b>发件人:</b> "Dave Chinner";<david@fromorbit.com>;</div><div><b>发送时间:</b> 2015年4月8日(星期三) 中午12:49</div><div><b>收件人:</b> "YeYin"<eyniy@qq.com>; <wbr></div><div><b>抄送:</b> "xfs"<xfs@oss.sgi.com>; <wbr></div><div><b>主题:</b> Re: XFS direct IO problem</div></div><div><br></div>On Wed, Apr 08, 2015 at 12:21:45PM +0800, YeYin wrote:<br>> Hi, About 2 months ago, I asked one problem in XFS, see<br>> here(http://oss.sgi.com/archives/xfs/2015-02/msg00197.html).<br>> <br>> <br>> After that, I use direct IO in MySQL, see<br>> here(https://dev.mysql.com/doc/refman/5.5/en/innodb-parameters.html#sysvar_innodb_flush_method).<br>> <br>> <br>> However, I found that MySQL performance is still poor sometimes. I<br>> use some tools(https://github.com/brendangregg/perf-tools) to<br>> trace the kernel, I found some problems:<br><br><snip><br><br>> This will cause bad performance, even direct IO. I still don't<br>> understand why not truncate_inode_page called?<br><br>Because the cached page must be outside the range of the direct IO<br>that is in progress - direct IO only tries to flush pages over the<br>range it is doing the IO over.<br><br>> Every time, after I run this: echo 1 > /proc/sys/vm/drop_caches<br>> <br>> Immediately enhance performance.<br><br>Because that flushes whatever page is in the cache. Can you identify<br>what offset that cached page is at? Tracing the xfs events will tell<br>you what pages that operation invalidates on each inode, and knowing<br>the offset may tell us why that page is not getting flushed.<br><br>Alternatively, write a simple C program that deomnstrates the same<br>problem so we can reproduce it easily, fix the problem and turn it<br>into a regression test....<br><br>Cheers,<br><br>Dave.<br>-- <br>Dave Chinner<br>david@fromorbit.com<br></div>