Re: "Internal error xfs_attr3_leaf_write_verify at line 216", "directory flags set on non-directory inode" and other errors

[Rasmus Borup Hansen <rbh@xxxxxxxxxxxx>]

When I tried mounting the file system after running xfs_repair, a quota check was started. However it did not finish after a few hours like the quota check that was started when mounting the file system read only, so after a few days of waiting (with next to no disk activity) I restarted the server and mounted the file system without quotas. Soon afterwards I got the following error (note that it's a different line number in xfs_attr_leaf.c than the one I initially saw):

[  327.670974] ffff8802a273d000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................

[  327.678213] ffff8802a273d010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........

[  327.685878] ffff8802a273d020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

[  327.693950] ffff8802a273d030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

[  327.701567] XFS (dm-0): Internal error xfs_attr3_leaf_read_verify at line 246 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_attr_leaf.c.  Caller 0xffffffffa00cb885

[  327.718829] CPU: 1 PID: 2805 Comm: kworker/1:1H Not tainted 3.13.0-55-generic #94-Ubuntu

[  327.718830] Hardware name: Dell Inc. PowerEdge R310/05XKKK, BIOS 1.8.2 08/17/2011

[  327.718867] Workqueue: xfslogd xfs_buf_iodone_work [xfs]

[  327.718869]  0000000000000001 ffff8800b82bfd68 ffffffff81723294 ffff8800368dd800

[  327.718875]  ffff8800b82bfd80 ffffffffa00ce6fb ffffffffa00cb885 ffff8800b82bfdb8

[  327.718877]  ffffffffa00ce755 000000f600203100 ffff8802a140ad00 ffff8800368dd800

[  327.718879] Call Trace:

[  327.718887]  [<ffffffff81723294>] dump_stack+0x45/0x56

[  327.718901]  [<ffffffffa00ce6fb>] xfs_error_report+0x3b/0x40 [xfs]

[  327.718911]  [<ffffffffa00cb885>] ? xfs_buf_iodone_work+0x85/0xf0 [xfs]

[  327.718921]  [<ffffffffa00ce755>] xfs_corruption_error+0x55/0x80 [xfs]

[  327.718935]  [<ffffffffa00ebbdd>] xfs_attr3_leaf_read_verify+0x6d/0xf0 [xfs]

[  327.718945]  [<ffffffffa00cb885>] ? xfs_buf_iodone_work+0x85/0xf0 [xfs]

[  327.718954]  [<ffffffffa00cb885>] xfs_buf_iodone_work+0x85/0xf0 [xfs]

[  327.718958]  [<ffffffff81083b22>] process_one_work+0x182/0x450

[  327.718961]  [<ffffffff81084911>] worker_thread+0x121/0x410

[  327.718963]  [<ffffffff810847f0>] ? rescuer_thread+0x430/0x430

[  327.718965]  [<ffffffff8108b702>] kthread+0xd2/0xf0

[  327.718967]  [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0

[  327.718970]  [<ffffffff81733ca8>] ret_from_fork+0x58/0x90

[  327.718972]  [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0

[  327.718973] XFS (dm-0): Corruption detected. Unmount and run xfs_repair

[  327.729003] XFS (dm-0): metadata I/O error: block 0x157e84da0 ("xfs_trans_read_buf_map") error 117 numblks 8

I started another xfs_repair which did not report any errors. After mounting the file system again (still without quotas) I discovered that ls would write "Structure needs cleaning" whenever it listed a certain file (and the kernel would output error messages like those above). This was a file I didn't need, so I tried deleting it and running yet another xfs_repair:

Phase 1 - find and verify superblock...

Phase 2 - using internal log

        - zero log...

        - scan filesystem freespace and inode maps...

agi unlinked bucket 11 is 2949684875 in ag 2 (inode=11539619467)

        - found root inode chunk

Phase 3 - for each AG...

        - scan and clear agi unlinked lists...

        - process known inodes and perform inode discovery...

        - agno = 0

doubling cache size to 591232

        - agno = 1

        - agno = 2

        - agno = 3

...

Phase 6 - check inode connectivity...

        - resetting contents of realtime bitmap and summary inodes

        - traversing filesystem ...

        - traversal finished ...

        - moving disconnected inodes to lost+found ...

disconnected inode 11539619467, moving to lost+found

Phase 7 - verify and correct link counts...

done

The file then turns up in lost+found and when I "ls" it I get the same errors again. I've tried deleting it from lost+found, but then xfs_repair finds it again with exactly the same output as show above and puts it back.

Apart from that, everything apparently works fine.

Is there a way to permanently get rid of the file in lost+found? Its size is apparently 0 bytes.

Best,

Rasmus

Intomics is a contract research organization specialized in deriving core biological insight from large scale data. We help our clients in the pharmaceutical industry develop tomorrow's medicines better, faster, and cheaper through optimized use of biomedical data.

-----------------------------------------------------------------

Hansen, Rasmus Borup              Intomics - from data to biology

System Administrator              Diplomvej 377

Scientific Programmer             DK-2800 Kgs. Lyngby

                                  Denmark

E: rbh@xxxxxxxxxxxx               W: http://www.intomics.com/

P: +45 5167 7972                  P: +45 8880 7979

On 26 Jun 2015, at 08:14, Rasmus Borup Hansen <rbh@xxxxxxxxxxxx> wrote:

I tried mounting the file system read-only (which triggered a quota check â does this make sense when the file system is read-only?) and then I scanned the file system for files with the spurious inodes to see if I could find a pattern. This took quite a while, and I didn't find any patterns, except that all the files in directories with project quotas were affected (but there were also other files). I'm now running xfs_repair without -n (and I had to mount and unmount the file system before it would start). I'll report back when it has finished.

Best,

Rasmus

Intomics is a contract research organization specialized in deriving core biological insight from large scale data. We help our clients in the pharmaceutical industry develop tomorrow's medicines better, faster, and cheaper through optimized use of biomedical data.

-----------------------------------------------------------------

Hansen, Rasmus Borup              Intomics - from data to biology

System Administrator              Diplomvej 377

Scientific Programmer             DK-2800 Kgs. Lyngby

                                  Denmark

E: rbh@xxxxxxxxxxxx               W: http://www.intomics.com/

P: +45 5167 7972                  P: +45 8880 7979

On 25 Jun 2015, at 18:41, Emmanuel Florac <eflorac@xxxxxxxxxxxxxx> wrote:

Le Wed, 24 Jun 2015 09:39:45 +0200
Rasmus Borup Hansen <rbh@xxxxxxxxxxxx> Ãcrivait:

Only the first 20 lines are included. There are currently 250000+
more lines with "directory flags set on non-directory inode" and the
check is still running (the mostly small files take up around 30 TB,
so it'll probably take a while).

I recently enabled user and project quota and updated from 3.13.0-53.
The file system has been heavily used for the last month or so.

Does anyone have any thoughts on this? I'm tempted to stop using
quotas when the file system (hopefully) works again, as it's my
impression that project quotas are not widely used.

Did you first try remounting then unmounting the volume to clear the
log? That could clear out xfs_repair output.

--
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                   |   Intellique
                   | <eflorac@xxxxxxxxxxxxxx>
                   |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@xxxxxxxxxxx
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@xxxxxxxxxxx
http://oss.sgi.com/mailman/listinfo/xfs