<div style="font:14px/1.5 'Lucida Grande', '微软雅黑';color:#333;"><p style="font:14px/1.5 'Lucida Grande';margin:0;">In function xfs_file_aio_read, will request <span style="line-height: 1.5;">XFS_IOLOCK_SHARED lock for both direct IO and buffered IO:</span></p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;">STATIC ssize_t</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;">xfs_file_aio_read(</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>struct kiocb<span class="Apple-tab-span" style="white-space:pre"> </span>*iocb,</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>const struct iovec<span class="Apple-tab-span" style="white-space:pre"> </span>*iovp,</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>unsigned long<span class="Apple-tab-span" style="white-space:pre"> </span>nr_segs,</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>loff_t<span class="Apple-tab-span" style="white-space:pre"> </span>pos)</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;">{</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;">...</p><p style="font:14px/1.5 'Lucida Grande';margin:0;">xfs_rw_ilock(ip, XFS_IOLOCK_SHARED);</p><p style="font:14px/1.5 'Lucida Grande';margin:0;"><br></p><p style="font:14px/1.5 'Lucida Grande';margin:0;"><a href="https://bitbucket.org/hustcat/kernel-2.6.32/src/786d720807052737bb17bc44da9da20554400039/fs/xfs/linux-2.6/xfs_file.c?at=master#cl-281">https://bitbucket.org/hustcat/kernel-2.6.32/src/786d720807052737bb17bc44da9da20554400039/fs/xfs/linux-2.6/xfs_file.c?at=master#cl-281</a></p><p style="font:14px/1.5 'Lucida Grande';margin:0;"><br></p><p style="font:14px/1.5 'Lucida Grande';margin:0;">so write will prevent read in XFS.</p><p style="font:14px/1.5 'Lucida Grande';margin:0;"><br></p><p style="font:14px/1.5 'Lucida Grande';margin:0;">However, in function generic_file_aio_read for ext3, will not lock inode->i_mutex, so write will not prevent read in ext3.</p><p style="font:14px/1.5 'Lucida Grande';margin:0;"><br></p><p style="font:14px/1.5 'Lucida Grande';margin:0;">I think this maybe the reason of poor performance for XFS. I do not know if this is a bug, or design flaws of XFS.</p><p style="font:14px/1.5 'Lucida Grande';margin:0;"><br></p><br><div><div style="font: 14px/1.5 'Lucida Grande';">Best regards,<div>yy</div></div></div><span style="font: 14px/1.5 'Lucida Grande';color:#333;"><br></span><div class="foxmail_blockquote_fromhere_element" style="font: 12px/1.5 'Lucida Grande';padding:2px 0 2px 0;"><span style="color:#333;text-decoration:line-through;white-space:pre-wrap;"> </span> 原始邮件 <span style="color:#333;text-decoration:line-through;white-space:pre-wrap;"> </span></div><div style="font: 12px/1.5 'Lucida Grande';background:#efefef;color:#666666;padding:8px;"><div><b style="color:#999;">发件人:</b> yy<yy@xspring.net></div><div><b style="color:#999;">收件人:</b> xfs<xfs@oss.sgi.com>; Eric Sandeen<sandeen@sandeen.net>; bfoster@redhat.com</div><div><b style="color:#999;">发送时间:</b> 2015年2月12日(周四) 13:30</div><div><b style="color:#999;">主题:</b> Re: XFS buffer IO performance is very poor</div></div><br><div class="mail_quote_5471F3E9861F421C94FD43CD6F17B524" style="font: 14px/1.5 'Lucida Grande';color:#333;"><div style="font:14px/1.5 'Lucida Grande', '微软雅黑';color:#333;"><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;">Brian and Eric,</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><br></p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;">Thanks very much for your replay.</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><br></p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;">I changed partition start position with 256K, but the performance is still poor, no change.</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"># fdisk -ul /dev/sdb </p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"> Device Boot Start End Blocks Id System</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;">/dev/sdb1 512 2929356359 1464677924 83 Linux</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><br></p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;">I checked the XFS’s code, I agree that XFS_IOLOCK_EXCL lock maybe the reason:</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><a href="https://bitbucket.org/hustcat/kernel-2.6.32/src/786d720807052737bb17bc44da9da20554400039/fs/xfs/linux-2.6/xfs_file.c?at=master#cl-714">https://bitbucket.org/hustcat/kernel-2.6.32/src/786d720807052737bb17bc44da9da20554400039/fs/xfs/linux-2.6/xfs_file.c?at=master#cl-714</a></p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;">STATIC ssize_t</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;">xfs_file_buffered_aio_write(</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>struct kiocb<span class="Apple-tab-span" style="white-space:pre"> </span>*iocb,</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>const struct iovec<span class="Apple-tab-span" style="white-space:pre"> </span>*iovp,</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>unsigned long<span class="Apple-tab-span" style="white-space:pre"> </span>nr_segs,</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>loff_t<span class="Apple-tab-span" style="white-space:pre"> </span>pos,</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>size_t<span class="Apple-tab-span" style="white-space:pre"> </span>ocount)</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;">{</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>struct file<span class="Apple-tab-span" style="white-space:pre"> </span>*file = iocb->ki_filp;</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>struct address_space<span class="Apple-tab-span" style="white-space:pre"> </span>*mapping = file->f_mapping;</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>struct inode<span class="Apple-tab-span" style="white-space:pre"> </span>*inode = mapping->host;</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>struct xfs_inode<span class="Apple-tab-span" style="white-space:pre"> </span>*ip = XFS_I(inode);</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>ssize_t<span class="Apple-tab-span" style="white-space:pre"> </span>ret;</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>int<span class="Apple-tab-span" style="white-space:pre"> </span>enospc = 0;</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>int<span class="Apple-tab-span" style="white-space:pre"> </span>iolock = XFS_IOLOCK_EXCL;</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>size_t<span class="Apple-tab-span" style="white-space:pre"> </span>count = ocount;</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><br></p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span><i><b>xfs_rw_ilock(ip, iolock);</b></i></p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><br></p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>ret = xfs_file_aio_write_checks(file, &pos, &count, &iolock);</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>if (ret)</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><br></p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><br></p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;">However，I found that EXT3 also have mutex when with buffered IO:</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><a href="https://bitbucket.org/hustcat/kernel-2.6.32/src/786d720807052737bb17bc44da9da20554400039/mm/filemap.c?at=master#cl-2642">https://bitbucket.org/hustcat/kernel-2.6.32/src/786d720807052737bb17bc44da9da20554400039/mm/filemap.c?at=master#cl-2642</a></p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;">ssize_t generic_file_aio_write(struct kiocb *iocb, const struct iovec *iov,</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>unsigned long nr_segs, loff_t pos)</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;">{</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>struct file *file = iocb->ki_filp;</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>struct inode *inode = file->f_mapping->host;</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>ssize_t ret;</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><br></p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>BUG_ON(iocb->ki_pos != pos);</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><br></p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>sb_start_write(inode->i_sb);</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span><i><b>mutex_lock(&inode->i_mutex);</b></i></p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>ret = __generic_file_aio_write(iocb, iov, nr_segs, &iocb->ki_pos);</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><span class="Apple-tab-span" style="white-space:pre"> </span>mutex_unlock(&inode->i_mutex);</p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;"><br></p><p style="line-height: 1.5; font-family: 'Lucida Grande'; margin: 0px;">I still don’t understand why ext3 does not have this problem with buffered IO. </p><br><div><div style="font: 14px/1.5 'Lucida Grande';">Best regards,<div>yy</div></div></div><span style="font: 14px/1.5 'Lucida Grande';color:#333;"><br></span><div class="foxmail_blockquote_fromhere_element" style="font: 12px/1.5 'Lucida Grande';padding:2px 0 2px 0;"><span style="color:#333;text-decoration:line-through;white-space:pre-wrap;"> </span> 原始邮件 <span style="color:#333;text-decoration:line-through;white-space:pre-wrap;"> </span></div><div style="font: 12px/1.5 'Lucida Grande';background:#efefef;color:#666666;padding:8px;"><div><b style="color:#999;">发件人:</b> Eric Sandeen<<a href="mailto:sandeen@sandeen.net" title="mailto:sandeen@sandeen.net">sandeen@sandeen.net</a>></div><div><b style="color:#999;">收件人:</b> yy<<a href="mailto:yy@xspring.net" title="mailto:yy@xspring.net">yy@xspring.net</a>>; xfs<<a href="mailto:xfs@oss.sgi.com" title="mailto:xfs@oss.sgi.com">xfs@oss.sgi.com</a>></div><div><b style="color:#999;">发送时间:</b> 2015年2月12日(周四) 00:08</div><div><b style="color:#999;">主题:</b> Re: XFS buffer IO performance is very poor</div></div><br><div class="mail_quote_469462ADD2E0441AB745D6686644F0DF" style="font: 14px/1.5 'Lucida Grande';color:#333;"><pre style="white-space:pre-wrap;">On 2/11/15 1:39 AM, yy wrote:
<snip>
(In addition to Brian's questions):
> XFS format parametes:
>
> #mkfs.xfs -d su=256k,sw=5 /dev/sdb1
>
> #cat /proc/mounts
>
> /dev/sdb1 /data1 xfs rw,noatime,attr2,delaylog,nobarrier,logbsize=256k,sunit=512,swidth=2560,noquota 0 0
>
> #fdisk -ul
> Device Boot Start End Blocks Id System
> /dev/sdb1 128 2929356359 1464678116 83 Linux
so 128*512 = 64k; your partition doesn't start on a 256k stripe unit boundary,
right? Shouldn't it do so?
-Eric
</pre></div></div></div></div>