<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Hi! Yesterday I got the following error messages from the kernel (Ubuntu trusty, 3.13.0-55):</div><div class=""><br class=""></div><div class=""><div class=""><font face="Courier New" class="">[601201.817664] ffff88016e03e000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00 ................</font></div><div class=""><font face="Courier New" class="">[601201.818224] ffff88016e03e010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00 ..... ..........</font></div><div class=""><font face="Courier New" class="">[601201.818827] ffff88016e03e020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................</font></div><div class=""><font face="Courier New" class="">[601201.819429] ffff88016e03e030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................</font></div><div class=""><font face="Courier New" class="">[601201.820013] XFS (dm-0): <b class="">Internal error xfs_attr3_leaf_write_verify at line 216 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_attr_leaf.c.</b> Caller 0xffffffffa00996f0</font></div><div class=""><font face="Courier New" class="">[601201.820910] CPU: 1 PID: 421 Comm: xfsaild/dm-0 Not tainted 3.13.0-55-generic #92-Ubuntu</font></div><div class=""><font face="Courier New" class="">[601201.820913] Hardware name: Dell Inc. PowerEdge R310/05XKKK, BIOS 1.8.2 08/17/2011</font></div><div class=""><font face="Courier New" class="">[601201.820914] 0000000000000001 ffff880035cc3bd0 ffffffff81723294 ffff8802ac71d800</font></div><div class=""><font face="Courier New" class="">[601201.820918] ffff880035cc3be8 ffffffffa009d6fb ffffffffa00996f0 ffff880035cc3c20</font></div><div class=""><font face="Courier New" class="">[601201.820919] ffffffffa009d755 000000d800205500 ffff88001aa2a400 ffff880050acc2b8</font></div><div class=""><font face="Courier New" class="">[601201.820922] Call Trace:</font></div><div class=""><font face="Courier New" class="">[601201.820930] [<ffffffff81723294>] dump_stack+0x45/0x56</font></div><div class=""><font face="Courier New" class="">[601201.820964] [<ffffffffa009d6fb>] xfs_error_report+0x3b/0x40 [xfs]</font></div><div class=""><font face="Courier New" class="">[601201.820974] [<ffffffffa00996f0>] ? _xfs_buf_ioapply+0x70/0x3a0 [xfs]</font></div><div class=""><font face="Courier New" class="">[601201.820984] [<ffffffffa009d755>] xfs_corruption_error+0x55/0x80 [xfs]</font></div><div class=""><font face="Courier New" class="">[601201.820997] [<ffffffffa00bab50>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]</font></div><div class=""><font face="Courier New" class="">[601201.821007] [<ffffffffa00996f0>] ? _xfs_buf_ioapply+0x70/0x3a0 [xfs]</font></div><div class=""><font face="Courier New" class="">[601201.821016] [<ffffffffa009b3d5>] ? xfs_bdstrat_cb+0x55/0xb0 [xfs]</font></div><div class=""><font face="Courier New" class="">[601201.821026] [<ffffffffa00996f0>] _xfs_buf_ioapply+0x70/0x3a0 [xfs]</font></div><div class=""><font face="Courier New" class="">[601201.821030] [<ffffffff8109abc0>] ? wake_up_state+0x20/0x20</font></div><div class=""><font face="Courier New" class="">[601201.821040] [<ffffffffa009b3d5>] ? xfs_bdstrat_cb+0x55/0xb0 [xfs]</font></div><div class=""><font face="Courier New" class="">[601201.821050] [<ffffffffa009b336>] xfs_buf_iorequest+0x46/0x90 [xfs]</font></div><div class=""><font face="Courier New" class="">[601201.821060] [<ffffffffa009b3d5>] xfs_bdstrat_cb+0x55/0xb0 [xfs]</font></div><div class=""><font face="Courier New" class="">[601201.821070] [<ffffffffa009b56b>] __xfs_buf_delwri_submit+0x13b/0x210 [xfs]</font></div><div class=""><font face="Courier New" class="">[601201.821081] [<ffffffffa009c000>] ? xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs]</font></div><div class=""><font face="Courier New" class="">[601201.821100] [<ffffffffa00faaa0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]</font></div><div class=""><font face="Courier New" class="">[601201.821110] [<ffffffffa009c000>] xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs]</font></div><div class=""><font face="Courier New" class="">[601201.821127] [<ffffffffa00facd7>] xfsaild+0x237/0x5c0 [xfs]</font></div><div class=""><font face="Courier New" class="">[601201.821145] [<ffffffffa00faaa0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]</font></div><div class=""><font face="Courier New" class="">[601201.821148] [<ffffffff8108b702>] kthread+0xd2/0xf0</font></div><div class=""><font face="Courier New" class="">[601201.821150] [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0</font></div><div class=""><font face="Courier New" class="">[601201.821153] [<ffffffff81733ca8>] ret_from_fork+0x58/0x90</font></div><div class=""><font face="Courier New" class="">[601201.821155] [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0</font></div><div class=""><font face="Courier New" class="">[601201.821158] XFS (dm-0): <b class="">Corruption detected. Unmount and run xfs_repair</b></font></div><div class=""><font face="Courier New" class="">[601201.821581] XFS (dm-0): xfs_do_force_shutdown(0x8) called from line 1320 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_buf.c. Return address = 0xffffffffa009971c</font></div><div class=""><font face="Courier New" class="">[601201.832780] XFS (dm-0): <b class="">Corruption of in-memory data detected. Shutting down filesystem</b></font></div><div class=""><font face="Courier New" class="">[601201.833292] XFS (dm-0): Please umount the filesystem and rectify the problem(s)</font></div><div class=""><font face="Courier New" class="">[601201.843375] ffff880238b00000: 58 44 32 42 01 00 0e c0 00 00 00 00 00 00 00 00 XD2B............</font></div><div class=""><font face="Courier New" class="">[601201.843924] ffff880238b00010: 00 00 00 20 3d a7 0c 7b 01 2e 8d 9e 01 88 00 10 ... =..{........</font></div><div class=""><font face="Courier New" class="">[601201.844472] ffff880238b00020: 00 00 00 19 21 a9 2d 2f 02 2e 2e 9e 01 88 00 20 ....!.-/.......</font></div><div class=""><font face="Courier New" class="">[601201.845031] ffff880238b00030: 00 00 00 07 39 2b 68 20 18 4d 4f 42 31 41 5f 48 ....9+h .MOB1A_H</font></div><div class=""><font face="Courier New" class="">[601201.845585] XFS (dm-0): <b class="">Internal error xfs_da3_node_read_verify at line 240 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_da_btree.c.</b> Caller 0xffffffffa009a885</font></div><div class=""><font face="Courier New" class="">[601201.858885] CPU: 1 PID: 2806 Comm: kworker/1:1H Not tainted 3.13.0-55-generic #92-Ubuntu</font></div><div class=""><font face="Courier New" class="">[601201.858889] Hardware name: Dell Inc. PowerEdge R310/05XKKK, BIOS 1.8.2 08/17/2011</font></div><div class=""><font face="Courier New" class="">[601201.858938] Workqueue: xfslogd xfs_buf_iodone_work [xfs]</font></div><div class=""><font face="Courier New" class="">[601201.858941] 0000000000000001 ffff8802af9cbd68 ffffffff81723294 ffff8802ac71d800</font></div><div class=""><font face="Courier New" class="">[601201.858944] ffff8802af9cbd80 ffffffffa009d6fb ffffffffa009a885 ffff8802af9cbdb8</font></div><div class=""><font face="Courier New" class="">[601201.858945] ffffffffa009d755 000000f0810a0465 ffff880031b35b00 ffff880238b00000</font></div><div class=""><font face="Courier New" class="">[601201.858948] Call Trace:</font></div><div class=""><font face="Courier New" class="">[601201.858958] [<ffffffff81723294>] dump_stack+0x45/0x56</font></div><div class=""><font face="Courier New" class="">[601201.858969] [<ffffffffa009d6fb>] xfs_error_report+0x3b/0x40 [xfs]</font></div><div class=""><font face="Courier New" class="">[601201.858979] [<ffffffffa009a885>] ? xfs_buf_iodone_work+0x85/0xf0 [xfs]</font></div><div class=""><font face="Courier New" class="">[601201.858989] [<ffffffffa009d755>] xfs_corruption_error+0x55/0x80 [xfs]</font></div><div class=""><font face="Courier New" class="">[601201.859005] [<ffffffffa00d146c>] xfs_da3_node_read_verify+0x8c/0x180 [xfs]</font></div><div class=""><font face="Courier New" class="">[601201.859014] [<ffffffffa009a885>] ? xfs_buf_iodone_work+0x85/0xf0 [xfs]</font></div><div class=""><font face="Courier New" class="">[601201.859024] [<ffffffffa009a885>] xfs_buf_iodone_work+0x85/0xf0 [xfs]</font></div><div class=""><font face="Courier New" class="">[601201.859028] [<ffffffff81083b22>] process_one_work+0x182/0x450</font></div><div class=""><font face="Courier New" class="">[601201.859030] [<ffffffff81084911>] worker_thread+0x121/0x410</font></div><div class=""><font face="Courier New" class="">[601201.859032] [<ffffffff810847f0>] ? rescuer_thread+0x430/0x430</font></div><div class=""><font face="Courier New" class="">[601201.859035] [<ffffffff8108b702>] kthread+0xd2/0xf0</font></div><div class=""><font face="Courier New" class="">[601201.859037] [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0</font></div><div class=""><font face="Courier New" class="">[601201.859040] [<ffffffff81733ca8>] ret_from_fork+0x58/0x90</font></div><div class=""><font face="Courier New" class="">[601201.859042] [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0</font></div><div class=""><font face="Courier New" class="">[601201.859044] XFS (dm-0): <b class="">Corruption detected. Unmount and run xfs_repair</b></font></div><div class=""><font face="Courier New" class="">[601201.872288] XFS (dm-0): metadata I/O error: block 0x101ed393c0 ("xfs_trans_read_buf_map") error 117 numblks 8</font></div><div class=""><font face="Courier New" class="">[601201.942054] XFS (dm-0): xfs_imap_to_bp: xfs_trans_read_buf() returned error 5.</font></div><div class=""><font face="Courier New" class="">[601209.420268] XFS (dm-0): <b class="">xfs_log_force: error 5 returned.</b></font></div><div class=""><font face="Courier New" class="">[601239.529710] XFS (dm-0): xfs_log_force: error 5 returned.</font></div><div class=""><font face="Courier New" class="">[601269.639150] XFS (dm-0): xfs_log_force: error 5 returned.</font></div><div class=""><font face="Courier New" class="">[601299.748593] XFS (dm-0): xfs_log_force: error 5 returned.</font></div><div class=""><font face="Courier New" class="">[601329.858034] XFS (dm-0): xfs_log_force: error 5 returned.</font></div></div><div class=""><br class=""></div><div class="">I'm currently running <font face="Courier New" class="">xfs_repair -n</font> and so far I've seen the following output:</div><div class=""><br class=""></div><div class=""><div class="">Phase 1 - find and verify superblock...</div><div class="">Phase 2 - using internal log</div><div class=""> - scan filesystem freespace and inode maps...</div><div class=""> - found root inode chunk</div><div class="">Phase 3 - for each AG...</div><div class=""> - scan (but don't clear) agi unlinked lists...</div><div class=""> - process known inodes and perform inode discovery...</div><div class=""> - agno = 0</div><div class="">doubling cache size to 591200</div><div class="">directory flags set on non-directory inode 206624</div><div class="">directory flags set on non-directory inode 206625</div><div class="">directory flags set on non-directory inode 206626</div><div class="">directory flags set on non-directory inode 206627</div><div class="">directory flags set on non-directory inode 206628</div><div class="">directory flags set on non-directory inode 206629</div><div class="">directory flags set on non-directory inode 206630</div><div class="">directory flags set on non-directory inode 206631</div><div class="">directory flags set on non-directory inode 206632</div><div class="">directory flags set on non-directory inode 206633</div><div class="">directory flags set on non-directory inode 206634</div></div><div class=""><br class=""></div><div class="">Only the first 20 lines are included. There are currently 250000+ more lines with "directory flags set on non-directory inode" and the check is still running (the mostly small files take up around 30 TB, so it'll probably take a while).</div><div class=""><br class=""></div><div class="">I recently enabled user and project quota and updated from 3.13.0-53. The file system has been heavily used for the last month or so.</div><div class=""><br class=""></div><div class="">Does anyone have any thoughts on this? I'm tempted to stop using quotas when the file system (hopefully) works again, as it's my impression that project quotas are not widely used.</div><div class=""><br class=""></div><div class="">Best,</div><div class=""><br class=""></div><div class="">Rasmus</div><br class=""><div apple-content-edited="true" class="">
<span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px;"><i class="">Intomics is a contract research organization specialized in deriving core biological insight from large scale data. We help our clients in the pharmaceutical industry develop tomorrow's medicines better, faster, and cheaper through optimized use of biomedical data.</i><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""><div class=""><font class="Apple-style-span" face="'Courier New'">-----------------------------------------------------------------</font></div><div class=""><font class="Apple-style-span" face="'Courier New'">Hansen, Rasmus Borup Intomics - from data to biology</font></div><div class=""><font class="Apple-style-span" face="'Courier New'">System Administrator Diplomvej 377</font></div><div class=""><font class="Apple-style-span" face="'Courier New'">Scientific Programmer DK-2800 Kgs. Lyngby</font></div><div class=""><font class="Apple-style-span" face="'Courier New'"> Denmark</font></div><div class=""><font class="Apple-style-span" face="'Courier New'">E: <a href="mailto:rbh@intomics.com" class="">rbh@intomics.com</a> W: <a href="http://www.intomics.com/" class="">http://www.intomics.com/</a></font></div><div class=""><font class="Apple-style-span" face="'Courier New'">P: +45 5167 7972 P: +45 8880 7979</font></div></div></span></span>
</div>
<br class=""></body></html>