<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">When I tried mounting the file system after running xfs_repair, a quota check was started. However it did not finish after a few hours like the quota check that was started when mounting the file system read only, so after a few days of waiting (with next to no disk activity) I restarted the server and mounted the file system without quotas. Soon afterwards I got the following error (note that it's a different line number in xfs_attr_leaf.c than the one I initially saw):</div><div class=""><br class=""></div><div class=""><div class=""><font face="Courier New" class="">[ 327.670974] ffff8802a273d000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00 ................</font></div><div class=""><font face="Courier New" class="">[ 327.678213] ffff8802a273d010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00 ..... ..........</font></div><div class=""><font face="Courier New" class="">[ 327.685878] ffff8802a273d020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................</font></div><div class=""><font face="Courier New" class="">[ 327.693950] ffff8802a273d030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................</font></div><div class=""><font face="Courier New" class="">[ 327.701567] XFS (dm-0): Internal error xfs_attr3_leaf_read_verify at line 246 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller 0xffffffffa00cb885</font></div><div class=""><font face="Courier New" class="">[ 327.718829] CPU: 1 PID: 2805 Comm: kworker/1:1H Not tainted 3.13.0-55-generic #94-Ubuntu</font></div><div class=""><font face="Courier New" class="">[ 327.718830] Hardware name: Dell Inc. PowerEdge R310/05XKKK, BIOS 1.8.2 08/17/2011</font></div><div class=""><font face="Courier New" class="">[ 327.718867] Workqueue: xfslogd xfs_buf_iodone_work [xfs]</font></div><div class=""><font face="Courier New" class="">[ 327.718869] 0000000000000001 ffff8800b82bfd68 ffffffff81723294 ffff8800368dd800</font></div><div class=""><font face="Courier New" class="">[ 327.718875] ffff8800b82bfd80 ffffffffa00ce6fb ffffffffa00cb885 ffff8800b82bfdb8</font></div><div class=""><font face="Courier New" class="">[ 327.718877] ffffffffa00ce755 000000f600203100 ffff8802a140ad00 ffff8800368dd800</font></div><div class=""><font face="Courier New" class="">[ 327.718879] Call Trace:</font></div><div class=""><font face="Courier New" class="">[ 327.718887] [<ffffffff81723294>] dump_stack+0x45/0x56</font></div><div class=""><font face="Courier New" class="">[ 327.718901] [<ffffffffa00ce6fb>] xfs_error_report+0x3b/0x40 [xfs]</font></div><div class=""><font face="Courier New" class="">[ 327.718911] [<ffffffffa00cb885>] ? xfs_buf_iodone_work+0x85/0xf0 [xfs]</font></div><div class=""><font face="Courier New" class="">[ 327.718921] [<ffffffffa00ce755>] xfs_corruption_error+0x55/0x80 [xfs]</font></div><div class=""><font face="Courier New" class="">[ 327.718935] [<ffffffffa00ebbdd>] xfs_attr3_leaf_read_verify+0x6d/0xf0 [xfs]</font></div><div class=""><font face="Courier New" class="">[ 327.718945] [<ffffffffa00cb885>] ? xfs_buf_iodone_work+0x85/0xf0 [xfs]</font></div><div class=""><font face="Courier New" class="">[ 327.718954] [<ffffffffa00cb885>] xfs_buf_iodone_work+0x85/0xf0 [xfs]</font></div><div class=""><font face="Courier New" class="">[ 327.718958] [<ffffffff81083b22>] process_one_work+0x182/0x450</font></div><div class=""><font face="Courier New" class="">[ 327.718961] [<ffffffff81084911>] worker_thread+0x121/0x410</font></div><div class=""><font face="Courier New" class="">[ 327.718963] [<ffffffff810847f0>] ? rescuer_thread+0x430/0x430</font></div><div class=""><font face="Courier New" class="">[ 327.718965] [<ffffffff8108b702>] kthread+0xd2/0xf0</font></div><div class=""><font face="Courier New" class="">[ 327.718967] [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0</font></div><div class=""><font face="Courier New" class="">[ 327.718970] [<ffffffff81733ca8>] ret_from_fork+0x58/0x90</font></div><div class=""><font face="Courier New" class="">[ 327.718972] [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0</font></div><div class=""><font face="Courier New" class="">[ 327.718973] XFS (dm-0): Corruption detected. Unmount and run xfs_repair</font></div><div class=""><font face="Courier New" class="">[ 327.729003] XFS (dm-0): metadata I/O error: block 0x157e84da0 ("xfs_trans_read_buf_map") error 117 numblks 8</font></div></div><div class=""><br class=""></div><div class="">I started another xfs_repair which did not report any errors. After mounting the file system again (still without quotas) I discovered that ls would write "Structure needs cleaning" whenever it listed a certain file (and the kernel would output error messages like those above). This was a file I didn't need, so I tried deleting it and running yet another xfs_repair:</div><div class=""><br class=""></div><div class=""><div class=""><font face="Courier New" class="">Phase 1 - find and verify superblock...</font></div><div class=""><font face="Courier New" class="">Phase 2 - using internal log</font></div><div class=""><font face="Courier New" class=""> - zero log...</font></div><div class=""><font face="Courier New" class=""> - scan filesystem freespace and inode maps...</font></div><div class=""><font face="Courier New" class="">agi unlinked bucket 11 is 2949684875 in ag 2 (inode=11539619467)</font></div><div class=""><font face="Courier New" class=""> - found root inode chunk</font></div><div class=""><font face="Courier New" class="">Phase 3 - for each AG...</font></div><div class=""><font face="Courier New" class=""> - scan and clear agi unlinked lists...</font></div><div class=""><font face="Courier New" class=""> - process known inodes and perform inode discovery...</font></div><div class=""><font face="Courier New" class=""> - agno = 0</font></div><div class=""><font face="Courier New" class="">doubling cache size to 591232</font></div><div class=""><font face="Courier New" class=""> - agno = 1</font></div><div class=""><font face="Courier New" class=""> - agno = 2</font></div><div class=""><font face="Courier New" class=""> - agno = 3</font></div></div><div class=""><font face="Courier New" class="">...</font></div><div class=""><div class=""><font face="Courier New" class="">Phase 6 - check inode connectivity...</font></div><div class=""><font face="Courier New" class=""> - resetting contents of realtime bitmap and summary inodes</font></div><div class=""><font face="Courier New" class=""> - traversing filesystem ...</font></div><div class=""><font face="Courier New" class=""> - traversal finished ...</font></div><div class=""><font face="Courier New" class=""> - moving disconnected inodes to lost+found ...</font></div><div class=""><font face="Courier New" class="">disconnected inode 11539619467, moving to lost+found</font></div><div class=""><font face="Courier New" class="">Phase 7 - verify and correct link counts...</font></div><div class=""><font face="Courier New" class="">done</font></div></div><div class=""><br class=""></div><div class="">The file then turns up in lost+found and when I "ls" it I get the same errors again. I've tried deleting it from lost+found, but then xfs_repair finds it again with exactly the same output as show above and puts it back.</div><div class=""><br class=""></div><div class="">Apart from that, everything apparently works fine.</div><div class=""><br class=""></div><div class="">Is there a way to permanently get rid of the file in lost+found? Its size is apparently 0 bytes.</div><div class=""><br class=""></div><div class="">Best,</div><div class=""><br class=""></div><div class="">Rasmus</div><div class=""><br class=""></div><br class=""><div apple-content-edited="true" class="">
<span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px;"><i class="">Intomics is a contract research organization specialized in deriving core biological insight from large scale data. We help our clients in the pharmaceutical industry develop tomorrow's medicines better, faster, and cheaper through optimized use of biomedical data.</i><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""><div class=""><font class="Apple-style-span" face="'Courier New'">-----------------------------------------------------------------</font></div><div class=""><font class="Apple-style-span" face="'Courier New'">Hansen, Rasmus Borup Intomics - from data to biology</font></div><div class=""><font class="Apple-style-span" face="'Courier New'">System Administrator Diplomvej 377</font></div><div class=""><font class="Apple-style-span" face="'Courier New'">Scientific Programmer DK-2800 Kgs. Lyngby</font></div><div class=""><font class="Apple-style-span" face="'Courier New'"> Denmark</font></div><div class=""><font class="Apple-style-span" face="'Courier New'">E: <a href="mailto:rbh@intomics.com" class="">rbh@intomics.com</a> W: <a href="http://www.intomics.com/" class="">http://www.intomics.com/</a></font></div><div class=""><font class="Apple-style-span" face="'Courier New'">P: +45 5167 7972 P: +45 8880 7979</font></div></div></span></span>
</div>
<br class=""><div><blockquote type="cite" class=""><div class="">On 26 Jun 2015, at 08:14, Rasmus Borup Hansen <<a href="mailto:rbh@intomics.com" class="">rbh@intomics.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">I tried mounting the file system read-only (which triggered a quota check – does this make sense when the file system is read-only?) and then I scanned the file system for files with the spurious inodes to see if I could find a pattern. This took quite a while, and I didn't find any patterns, except that all the files in directories with project quotas were affected (but there were also other files). I'm now running xfs_repair without -n (and I had to mount and unmount the file system before it would start). I'll report back when it has finished.</div><div class=""><br class=""></div><div class="">Best,</div><div class=""><br class=""></div><div class="">Rasmus</div><br class=""><div apple-content-edited="true" class="">
<span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-stroke-width: 0px;"><i class="">Intomics is a contract research organization specialized in deriving core biological insight from large scale data. We help our clients in the pharmaceutical industry develop tomorrow's medicines better, faster, and cheaper through optimized use of biomedical data.</i><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-stroke-width: 0px;"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""><div class=""><font class="Apple-style-span" face="'Courier New'">-----------------------------------------------------------------</font></div><div class=""><font class="Apple-style-span" face="'Courier New'">Hansen, Rasmus Borup Intomics - from data to biology</font></div><div class=""><font class="Apple-style-span" face="'Courier New'">System Administrator Diplomvej 377</font></div><div class=""><font class="Apple-style-span" face="'Courier New'">Scientific Programmer DK-2800 Kgs. Lyngby</font></div><div class=""><font class="Apple-style-span" face="'Courier New'"> Denmark</font></div><div class=""><font class="Apple-style-span" face="'Courier New'">E: <a href="mailto:rbh@intomics.com" class="">rbh@intomics.com</a> W: <a href="http://www.intomics.com/" class="">http://www.intomics.com/</a></font></div><div class=""><font class="Apple-style-span" face="'Courier New'">P: +45 5167 7972 P: +45 8880 7979</font></div></div></span></span>
</div>
<br class=""><div class=""><blockquote type="cite" class=""><div class="">On 25 Jun 2015, at 18:41, Emmanuel Florac <<a href="mailto:eflorac@intellique.com" class="">eflorac@intellique.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">Le Wed, 24 Jun 2015 09:39:45 +0200<br class="">Rasmus Borup Hansen <<a href="mailto:rbh@intomics.com" class="">rbh@intomics.com</a>> écrivait:<br class=""><br class=""><blockquote type="cite" class="">Only the first 20 lines are included. There are currently 250000+<br class="">more lines with "directory flags set on non-directory inode" and the<br class="">check is still running (the mostly small files take up around 30 TB,<br class="">so it'll probably take a while).<br class=""><br class="">I recently enabled user and project quota and updated from 3.13.0-53.<br class="">The file system has been heavily used for the last month or so.<br class=""><br class="">Does anyone have any thoughts on this? I'm tempted to stop using<br class="">quotas when the file system (hopefully) works again, as it's my<br class="">impression that project quotas are not widely used.<br class=""></blockquote><br class="">Did you first try remounting then unmounting the volume to clear the<br class="">log? That could clear out xfs_repair output.<br class=""><br class="">-- <br class="">------------------------------------------------------------------------<br class="">Emmanuel Florac | Direction technique<br class=""> | Intellique<br class=""> |<span class="Apple-tab-span" style="white-space:pre"> </span><<a href="mailto:eflorac@intellique.com" class="">eflorac@intellique.com</a>><br class=""> | +33 1 78 94 84 02<br class="">------------------------------------------------------------------------<br class=""><br class="">_______________________________________________<br class="">xfs mailing list<br class=""><a href="mailto:xfs@oss.sgi.com" class="">xfs@oss.sgi.com</a><br class=""><a href="http://oss.sgi.com/mailman/listinfo/xfs" class="">http://oss.sgi.com/mailman/listinfo/xfs</a><br class=""></div></blockquote></div><br class=""></div>_______________________________________________<br class="">xfs mailing list<br class=""><a href="mailto:xfs@oss.sgi.com" class="">xfs@oss.sgi.com</a><br class="">http://oss.sgi.com/mailman/listinfo/xfs<br class=""></div></blockquote></div><br class=""></body></html>