<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">2014-08-26 21:13 GMT+08:00 Dave Chinner <span dir="ltr"><<a href="mailto:david@fromorbit.com" target="_blank">david@fromorbit.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div>On Tue, Aug 26, 2014 at 06:04:52PM +0800, Zhang Qiang wrote:<br>
> Thanks Dave/Greg for your analysis and suggestions.<br>
><br>
> I can summarize what I should do next:<br>
><br>
> - backup my data using xfsdump<br>
> - rebuilt filesystem using mkfs with options: agcount=32 for 2T disk<br>
> - mount filesystem with option inode64,nobarrier<br>
<br>
</div>Ok up to here.<br>
<div><br>
> - applied patches about adding free list inode on disk structure<br>
<br>
</div>No, don't do that. You're almost certain to get it wrong and corrupt<br>
your filesysetms and lose data.<br>
<div><br>
> As we have about ~100 servers need back up, so that will take much effort,<br>
> do you have any other suggestion?<br>
<br>
</div>Just remount them with inode64. Nothing else. Over time as you add<br>
and remove files the inodes will redistribute across all 4 AGs.<br></blockquote><div>OK.</div><div><br></div><div>How I can see the layout number of inodes on each AGs? Here's my checking steps:</div><div><br></div>
<div>1) Check unmounted file system first:</div><div><div>[root@fstest data1]# xfs_db -c "sb 0" -c "p" /dev/sdb1 |egrep 'icount|ifree'</div><div>icount = 421793920</div><div>ifree = 41</div><div>
[root@fstest data1]# xfs_db -c "sb 1" -c "p" /dev/sdb1 |egrep 'icount|ifree'</div><div>icount = 0</div><div>ifree = 0</div><div>[root@fstest data1]# xfs_db -c "sb 2" -c "p" /dev/sdb1 |egrep 'icount|ifree'</div>
<div>icount = 0</div><div>ifree = 0</div><div>[root@fstest data1]# xfs_db -c "sb 3" -c "p" /dev/sdb1 |egrep 'icount|ifree'</div><div>icount = 0</div><div>ifree = 0</div></div><div>2) mount it with inode64 and create many files:</div>
<div><br></div><div><div>[root@fstest /]# mount -o inode64,nobarrier /dev/sdb1 /data</div><div>[root@fstest /]# cd /data/tmp/</div><div>[root@fstest tmp]# fdtree.bash -d 16 -l 2 -f 100 -s 1</div></div><div>[root@fstest /]# umount /data<br>
</div><div><br></div><div>3) Check with xfs_db again:<br><br></div><div><div>[root@fstest data1]# xfs_db -f -c "sb 0" -c "p" /dev/sdb1 |egrep 'icount|ifree'</div><div>icount = 421821504</div>
<div>
ifree = 52</div><div>[root@fstest data1]# xfs_db -f -c "sb 1" -c "p" /dev/sdb1 |egrep 'icount|ifree'</div><div>icount = 0</div><div>ifree = 0</div></div><div><br></div><div>So, it seems that inodes only on first AG. Or icount/ifree is not the correct value to check, and how should I check how many inodes on each AGs?</div>
<div><br></div><div><br></div><div>I am finding a way to improve the performance based on current filesystem and kernel just remounting with inode64, I am trying how to make all inodes redistribute on all AGs averagely.</div>
<div><br></div><div>Is there any good way?, for example backup half of data to another device and remove it, then copy back it.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><br>
> What I am testing (ongoing):<br>
> - created a new 2T partition filesystem<br>
> - try to create small files and fill whole spaces then remove some of them<br>
> randomly<br>
> - check the performance of touch/cp files<br>
> - apply patches and verify it.<br>
><br>
> I have got more data from server:<br>
><br>
> 1) flush all cache(echo 3 > /proc/sys/vm/drop_caches), and umount filesystem<br>
> 2) mount filesystem and testing with touch command<br>
> * The first touch new file command take about ~23s<br>
> * second touch command take about ~0.1s.<br>
<br>
</div>So it's cache population that is your issue. You didn't say that<br>
first time around, which means the diagnosis was wrong. Again, it's having to<br>
search a btree with 220 million inodes in it to find the first free<br>
inode, and that btree has to be pulled in from disk and searched.<br>
Once it's cached, then each subsequent allocation will be much<br>
faster becaue the majority of the tree being searched will already<br>
be in cache...<br>
<div><br>
> I have compared the memory used, it seems that xfs try to load inode bmap<br>
> block for the first time, which take much time, is that the reason to take<br>
> so much time for the first touch operation?<br>
<br>
</div>No. reading the AGI btree to find the first free inode to allocate<br>
is what is taking the time. If you spread the inodes out over 4 AGs<br>
(using inode64) then the overhead of the first read will go down<br>
proportionally. Indeed, that is one of the reasons for using more<br>
AGs than 4 for filesystems lik ethis.<br></blockquote><div>OK, I see. </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Still, I can't help but wonder why you are using a filesystem to<br>
store hundreds of millions of tiny files, when a database is far<br>
better suited to storing and indexing this type and quantity of<br>
data....<br></blockquote><div><br></div><div>OK, this is a social networking website back end servers, actually the CDN infrastructure, and different server located different cities.</div><div>We have a global sync script to make all these 100 servers have the same data.<br>
</div><div><br></div><div>For each server we use RAID10 and XFS (CentOS6.3).</div><div><br></div><div>There are about 3M files (50K in size) generated every day, and we track the path of each files in database.</div><div>
<br></div><div>Do you have any suggestions to improve our solution?</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><div><br>
Cheers,<br>
<br>
Dave.<br>
--<br>
Dave Chinner<br>
<a href="mailto:david@fromorbit.com" target="_blank">david@fromorbit.com</a><br>
</div></div></blockquote></div><br></div></div>