<html><head><style type="text/css"><!-- DIV {margin:0px;} --></style></head><body><div style="font-family:times new roman, new york, times, serif;font-size:12pt"><DIV>Just to add if it helps- I find this logged by smart array controller:</DIV>
<DIV><SPAN lang=EN>
<P>Corrected ECC Error, Status=0x00000001 Addr=0x060f4e00</P>
<P> </P>
<P></SPAN><BR> </P></DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif">
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif"><FONT face=Tahoma size=2>
<HR SIZE=1>
<B><SPAN style="FONT-WEIGHT: bold">From:</SPAN></B> Leo Davis <leo1783@yahoo.com><BR><B><SPAN style="FONT-WEIGHT: bold">To:</SPAN></B> Dave Chinner <david@fromorbit.com><BR><B><SPAN style="FONT-WEIGHT: bold">Cc:</SPAN></B> xfs@oss.sgi.com<BR><B><SPAN style="FONT-WEIGHT: bold">Sent:</SPAN></B> Mon, April 25, 2011 9:55:02 AM<BR><B><SPAN style="FONT-WEIGHT: bold">Subject:</SPAN></B> Re: fs corruption<BR></FONT><BR>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif">
<DIV>Thank you for that :).</DIV>
<DIV> </DIV>
<DIV>However,I've run into another fs corruption issue on my other server. I just thought I would use the same thread rather than opening new.</DIV>
<DIV> </DIV>
<DIV>I was troublehooting a weird fiber channel issue ( logins going missing to my storage) when I noticed these backtraces in dmesg. </DIV>
<DIV> </DIV>
<DIV><SPAN lang=EN>
<P>Filesystem "cciss/c3d1p1": XFS internal error xfs_btree_check_lblock at line 186 of file fs/xfs/xfs_btree.c. Caller 0xffffffff881b92d6</P>
<P>Call Trace:</P>
<P>[<ffffffff881bce83>] :xfs:xfs_btree_check_lblock+0xf4/0xfe</P>
<P>[<ffffffff881b92d6>] :xfs:xfs_bmbt_lookup+0x159/0x420</P>
<P>[<ffffffff881b41cc>] :xfs:xfs_bmap_add_extent_delay_real+0x62a/0x103a</P>
<P>[<ffffffff881a8cfa>] :xfs:xfs_alloc_vextent+0x379/0x3ff</P>
<P>[<ffffffff881b543a>] :xfs:xfs_bmap_add_extent+0x1fb/0x390</P>
<P>[<ffffffff881b7f34>] :xfs:xfs_bmapi+0x895/0xe79</P>
<P>[<ffffffff881d4082>] :xfs:xfs_iomap_write_allocate+0x201/0x328</P>
<P>[<ffffffff881d4b09>] :xfs:xfs_iomap+0x22a/0x2a5</P>
<P>[<ffffffff881e9ae3>] :xfs:xfs_map_blocks+0x2d/0x65</P>
<P>[<ffffffff881ea723>] :xfs:xfs_page_state_convert+0x2af/0x544</P>
<P>[<ffffffff881eab04>] :xfs:xfs_vm_writepage+0xa7/0xdf</P>
<P>[<ffffffff8001cef2>] mpage_writepages+0x1bf/0x37d</P>
<P>[<ffffffff881eaa5d>] :xfs:xfs_vm_writepage+0x0/0xdf</P>
<P>[<ffffffff8005b1ea>] do_writepages+0x20/0x2f</P>
<P>[<ffffffff8005000e>] __filemap_fdatawrite_range+0x50/0x5b</P>
<P>[<ffffffff80050717>] do_fsync+0x2f/0xa4</P>
<P>[<ffffffff800e1ce9>] __do_fsync+0x23/0x36</P>
<P>[<ffffffff8005e116>] system_call+0x7e/0x83</P>
<P>Filesystem "cciss/c3d1p1": XFS internal error xfs_trans_cancel at line 1164 of file fs/xfs/xfs_trans.c. Caller 0xffffffff881d4186</P>
<P>Call Trace:</P>
<P>[<ffffffff881e1b37>] :xfs:xfs_trans_cancel+0x55/0xfa</P>
<P>[<ffffffff881d4186>] :xfs:xfs_iomap_write_allocate+0x305/0x328</P>
<P>[<ffffffff881d4b09>] :xfs:xfs_iomap+0x22a/0x2a5</P>
<P>[<ffffffff881e9ae3>] :xfs:xfs_map_blocks+0x2d/0x65</P>
<P>[<ffffffff881ea723>] :xfs:xfs_page_state_convert+0x2af/0x544</P>
<P>[<ffffffff881eab04>] :xfs:xfs_vm_writepage+0xa7/0xdf</P>
<P>[<ffffffff8001cef2>] mpage_writepages+0x1bf/0x37d</P>
<P>[<ffffffff881eaa5d>] :xfs:xfs_vm_writepage+0x0/0xdf</P>
<P>[<ffffffff8005b1ea>] do_writepages+0x20/0x2f</P>
<P>[<ffffffff8005000e>] __filemap_fdatawrite_range+0x50/0x5b</P>
<P>[<ffffffff80050717>] do_fsync+0x2f/0xa4</P>
<P>[<ffffffff800e1ce9>] __do_fsync+0x23/0x36</P>
<P>[<ffffffff8005e116>] system_call+0x7e/0x83</P>
<P>xfs_force_shutdown(cciss/c3d1p1,0x8) called from line 1165 of file fs/xfs/xfs_trans.c. Return address = 0xffffffff881e1b50</P>
<P>Filesystem "cciss/c3d1p1": Corruption of in-memory data detected. Shutting down filesystem: cciss/c3d1p1</P>
<P>Please umount the filesystem, and rectify the problem(s)</P>
<P>Filesystem "cciss/c3d1p1": xfs_log_force: error 5 returned.</P>
<P>Filesystem "cciss/c3d1p1": xfs_log_force: error 5 returned.</P>
<P> </P></SPAN></DIV>
<DIV> </DIV>
<DIV>Any thoughts on what the root cause might be?</DIV>
<DIV>- I've checked the underlying drives, array controller etc and all looks healthy; (indicating it is a fs issue for sure?)</DIV>
<DIV>I did the xfs_repair which corrected the issue but I'm worried as to how fs ended up in this state, this being a production box.</DIV>
<DIV> </DIV>
<DIV>Thanks in advance.<BR></DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif"><BR>
<DIV style="FONT-SIZE: 13px; FONT-FAMILY: arial, helvetica, sans-serif"><FONT face=Tahoma size=2>
<HR SIZE=1>
<B><SPAN style="FONT-WEIGHT: bold">From:</SPAN></B> Dave Chinner <david@fromorbit.com><BR><B><SPAN style="FONT-WEIGHT: bold">To:</SPAN></B> Leo Davis <leo1783@yahoo.com><BR><B><SPAN style="FONT-WEIGHT: bold">Cc:</SPAN></B> xfs@oss.sgi.com<BR><B><SPAN style="FONT-WEIGHT: bold">Sent:</SPAN></B> Tue, April 12, 2011 4:35:32 PM<BR><B><SPAN style="FONT-WEIGHT: bold">Subject:</SPAN></B> Re: fs corruption<BR></FONT><BR>On Tue, Apr 12, 2011 at 03:51:20AM -0700, Leo Davis wrote:<BR>> You have a corrupted free space btree.<BR>> <BR>> Err... apologies for my ignorance, but what is a free space btree?<BR><BR>A tree that indexes the free space in the filesystem. Every time you<BR>write a file or remove a file you are allocating or freeing space,<BR>and these tree keep track of that free space.<BR><BR>If you want to know - at a high level - how XFS is structured (good<BR>for understanding what a free space tree is), read this
paper:<BR><BR>http://oss.sgi.com/projects/xfs/papers/xfs_usenix/index.html<BR><BR>It's from 1996, but still correct on all the major structural<BR>details.<BR><BR>> I had serial trace from raid controller which i just checked and<BR>> it logged some 'Loose cabling', but this was months back..... not<BR>> sure whether that can be the cause of this.. strange if that is<BR>> the case since it's been a long time<BR><BR>it's possible that it took a couple of months to trip over a random<BR>metadata corruption. I've seen that before in directory trees and<BR>inode clusters where corruption is not detected until next time they<BR>are read from disk....<BR><BR>Cheers,<BR><BR>Dave.<BR>-- <BR>Dave Chinner<BR><A href="mailto:david@fromorbit.com" target=_blank rel=nofollow ymailto="mailto:david@fromorbit.com">david@fromorbit.com</A><BR></DIV></DIV></DIV></DIV></DIV></div></body></html>