XFS File system in trouble

Thu Aug 13 01:21:16 CDT 2015

	The compressed tarball containing the dump file and the image are on my 
web site.

http://fletchergeek.com/images/metadump.tar.gz

	It's 22G in size.

On 8/9/2015 8:37 PM, Leslie Rhorer wrote:
> Well, nice try, but it doesn't wash for several reasons:
>
> 1. Power supply issues would be highly unlikely to be the cause of such
> a highly specific failure at always a very specific point in a process.
>   Problems would crop up all over the place, not just with one, very
> specific failure.  While I am thinking of it, I also ran memtest86+
> again on the new memory.  It passed all tests with flying colors.
>
> 2. The system has not been under a heavy load when this happens.  In
> fact, it's piddling.  Rsync and tar are single threaded, eating up at
> most 1 CPU core at a time.  I have processes that can regularly bang all
> 8 cores right to the wall with no errors.  The I/O stream is even more
> piddling.  Rsync is transferring nearly 120 MBps (it's a 1G link) during
> the process, and some portions of the tar process can bang out well over
> 2Gbps.  Creating a directory is nothing.
>
> 3.  All the power supply rails are nominal - I checked.
>
> 4. Most damning of all, I am able to reproduce the issue, now, on
> another machine.  I'm not entirely sure why creating the image on one
> partition and then copying it to the root or across the LAN stopped it
> from failing, but I took the 1.5T drive and moved it to the backup
> machine, which as I related earlier is nearly identical in hardware and
> highly similar in software to the primary system.  It's failing there
> repeatedly and consistently:
>
> RR274x/Driver/Freebsd/rr274x_3x-bsd-8.0-v1.0.10.0712.tgz
> RR274x/Driver/Linux/
> RR274x/Driver/Linux/Debian/
> tar: RR274x/Driver/Linux/Debian: Cannot mkdir: Structure needs cleaning
> RR274x/Driver/Linux/Debian/rr274x_3x-debian-5.0.1-i386/
> tar: RR274x/Driver/Linux/Debian: Cannot mkdir: Input/output error
> tar: RR274x/Driver/Linux/Debian/rr274x_3x-debian-5.0.1-i386: Cannot
> mkdir: No such file or directory
> RR274x/Driver/Linux/Debian/rr274x_3x-debian-5.0.1-i386/boot/
> tar: RR274x/Driver/Linux/Debian: Cannot mkdir: Input/output error
> tar: RR274x/Driver/Linux/Debian/rr274x_3x-debian-5.0.1-i386/boot: Cannot
> mkdir: No such file or directory
> RR274x/Driver/Linux/Debian/rr274x_3x-debian-5.0.1-i386/boot/rr274x_3x2.6.26-2-486i386.ko.gz
>
> tar: RR274x/Driver/Linux/Debian: Cannot mkdir: Input/output error
>
> gzip: stdin: Input/output error
> tar: Unexpected EOF in archive
> tar: RR274x/Driver/Linux: Cannot utime: Input/output error
> tar: RR274x/Driver/Linux: Cannot change ownership to uid 0, gid 1000:
> Input/output error
> tar: RR274x/Driver/Linux: Cannot change mode to rwxr-xr-x: Input/output
> error
> tar: RR274x/Driver: Cannot utime: Input/output error
> tar: RR274x/Driver: Cannot change ownership to uid 0, gid 1000:
> Input/output error
> tar: RR274x/Driver: Cannot change mode to rwxr-xr-x: Input/output error
> tar: RR274x: Cannot utime: Input/output error
> tar: RR274x: Cannot change ownership to uid 0, gid 1000: Input/output error
> tar: RR274x: Cannot change mode to rwxr-xr-x: Input/output error
> tar: Error is not recoverable: exiting now
>
>
> dmesg:
> [26743.775522] XFS (sdk): Mounting V4 Filesystem
> [26743.904281] XFS (sdk): Ending clean mount
> [26743.912614] Loading kernel module for a network device with
> CAP_SYS_MODULE (deprecated).  Use CAP_NET_ADMIN and alias netdev- instead.
>
> <repeats>
>
> [26772.528827] loop: module loaded
> [26772.601043] XFS (loop0): Mounting V4 Filesystem
> [26772.764360] XFS (loop0): Ending clean mount
> [26772.770627] Loading kernel module for a network device with
> CAP_SYS_MODULE (deprecated).  Use CAP_NET_ADMIN and alias netdev- instead.
>
> <repeats>
>
> [26899.019942] XFS (loop0): xfs_iread: validation failed for inode
> 124656869424 failed
> [26899.019952] ffff8800b473e000: 49 4e 00 00 03 02 00 00 00 30 00 70 00
> 00 03 e8  IN.......0.p....
> [26899.019957] ffff8800b473e010: 00 00 00 00 06 20 b0 6f 01 2e 00 00 00
> 00 00 16  ..... .o........
> [26899.019960] ffff8800b473e020: 01 57 37 fd 2b 5d 22 9e 1e 0a 61 8c 00
> 00 00 20  .W7.+]"...a....
> [26899.019964] ffff8800b473e030: ff ff 00 d2 1b f6 27 90 00 00 00 00 00
> 00 00 00  ......'.........
> [26899.019993] XFS (loop0): Internal error xfs_iread at line 392 of file
> /build/linux-u5KAtC/linux-3.16.7-ckt11/fs/xfs/xfs_inode_buf.c.  Caller
> xfs_iget+0x24b/0x690 [xfs]
> [26899.020000] CPU: 6 PID: 3756 Comm: tar Not tainted 3.16.0-4-amd64 #1
> Debian 3.16.7-ckt11-1+deb8u2
> [26899.020004] Hardware name: To be filled by O.E.M. To be filled by
> O.E.M./SABERTOOTH 990FX R2.0, BIOS 0803 08/15/2012
> [26899.020007]  0000000000000001 ffffffff8150b3d5 ffff8800065b9800
> ffffffffa06bd5cb
> [26899.020014]  0000018800000010 ffffffffa06c2f6b ffff88000a680400
> ffff8800065b9800
> [26899.020019]  0000000000000075 ffff88000527f140 ffffffffa0708b3a
> ffffffffa06c2f6b
> [26899.020024] Call Trace:
> [26899.020034]  [<ffffffff8150b3d5>] ? dump_stack+0x41/0x51
> [26899.020052]  [<ffffffffa06bd5cb>] ? xfs_corruption_error+0x5b/0x80 [xfs]
> [26899.020069]  [<ffffffffa06c2f6b>] ? xfs_iget+0x24b/0x690 [xfs]
> [26899.020090]  [<ffffffffa0708b3a>] ? xfs_iread+0xea/0x400 [xfs]
> [26899.020106]  [<ffffffffa06c2f6b>] ? xfs_iget+0x24b/0x690 [xfs]
> [26899.020124]  [<ffffffffa06c2f6b>] ? xfs_iget+0x24b/0x690 [xfs]
> [26899.020146]  [<ffffffffa0702de6>] ? xfs_ialloc+0xa6/0x500 [xfs]
> [26899.020192]  [<ffffffffa06d258e>] ? kmem_zone_alloc+0x6e/0xe0 [xfs]
> [26899.020215]  [<ffffffffa07032a2>] ? xfs_dir_ialloc+0x62/0x2a0 [xfs]
> [26899.020237]  [<ffffffffa06d11e5>] ? xfs_trans_reserve+0x1f5/0x200 [xfs]
> [26899.020261]  [<ffffffffa07039a9>] ? xfs_create+0x489/0x700 [xfs]
> [26899.020267]  [<ffffffff811b40ea>] ? kern_path_create+0xaa/0x190
> [26899.020286]  [<ffffffffa06c85ea>] ? xfs_generic_create+0xca/0x250 [xfs]
> [26899.020292]  [<ffffffff811b7ad0>] ? vfs_mkdir+0xb0/0x160
> [26899.020296]  [<ffffffff811b868b>] ? SyS_mkdirat+0xab/0xe0
> [26899.020303]  [<ffffffff8151158d>] ?
> system_call_fast_compare_end+0x10/0x15
> [26899.020307] XFS (loop0): Corruption detected. Unmount and run xfs_repair
> [26899.020337] XFS (loop0): Internal error xfs_trans_cancel at line 959
> of file /build/linux-u5KAtC/linux-3.16.7-ckt11/fs/xfs/xfs_trans.c.
> Caller xfs_create+0x2b2/0x700 [xfs]
> [26899.020342] CPU: 6 PID: 3756 Comm: tar Not tainted 3.16.0-4-amd64 #1
> Debian 3.16.7-ckt11-1+deb8u2
> [26899.020345] Hardware name: To be filled by O.E.M. To be filled by
> O.E.M./SABERTOOTH 990FX R2.0, BIOS 0803 08/15/2012
> [26899.020347]  000000000000000c ffffffff8150b3d5 ffff88000527f140
> ffffffffa06d1e07
> [26899.020354]  ffff88000a729800 ffff8800066e3ec8 ffff8800065b9800
> ffffffffa07037d2
> [26899.020359]  0000000000000001 ffff8800066e3e20 ffff8800066e3e1c
> ffff8800066e3eb0
> [26899.020364] Call Trace:
> [26899.020370]  [<ffffffff8150b3d5>] ? dump_stack+0x41/0x51
> [26899.020388]  [<ffffffffa06d1e07>] ? xfs_trans_cancel+0xc7/0xf0 [xfs]
> [26899.020409]  [<ffffffffa07037d2>] ? xfs_create+0x2b2/0x700 [xfs]
> [26899.020414]  [<ffffffff811b40ea>] ? kern_path_create+0xaa/0x190
> [26899.020432]  [<ffffffffa06c85ea>] ? xfs_generic_create+0xca/0x250 [xfs]
> [26899.020437]  [<ffffffff811b7ad0>] ? vfs_mkdir+0xb0/0x160
> [26899.020442]  [<ffffffff811b868b>] ? SyS_mkdirat+0xab/0xe0
> [26899.020447]  [<ffffffff8151158d>] ?
> system_call_fast_compare_end+0x10/0x15
> [26899.020454] XFS (loop0): xfs_do_force_shutdown(0x8) called from line
> 960 of file /build/linux-u5KAtC/linux-3.16.7-ckt11/fs/xfs/xfs_trans.c.
> Return address = 0xffffffffa06d1e20
> [26899.407181] XFS (loop0): Corruption of in-memory data detected.
> Shutting down filesystem
> [26899.407190] XFS (loop0): Please umount the filesystem and rectify the
> problem(s)
> [26923.319559] XFS (loop0): xfs_log_force: error 5 returned.
>
> <repeats>
>
> Xfs_repair still reports no faults.  I'm compressing the dump file and
> image file right now to be posted on http:/flethergeek.com/images when
> it is done, but it is taking a very long time.  I'll also try
> decompresssing the image to the other array to see if it still fails
> before I upload the file.  'No point in uploading if putting it through
> the compression process results in an image that does not fail.
>
> On 8/4/2015 5:42 PM, Dave Chinner wrote:
>> On Tue, Aug 04, 2015 at 02:52:33AM -0500, Leslie Rhorer wrote:
>>>     It's failing, again.  The rsync job failed and when I attempt to
>>> untar the file in the image mount, it fails there, as well.  See
>>> below.  I formatted a 1.5T drive as xfs and mounted it under /media.
>>> I then dumped the failing FS to a file on /media using xfs_metadump
>>> and used xfs_mdrestore to create an image of the FS.  I then mounted
>>> the image, copied over the tarball to its location, and ran tar to
>>> extract the files:
>>>
>>> [131874.545344] loop: module loaded
>>> [131874.549914] XFS (loop0): Mounting V4 Filesystem
>>                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>
>>> [131874.555540] XFS (loop0): Ending clean mount
>>> [132020.964431] XFS (loop0): xfs_iread: validation failed for inode
>>> 124656869424 failed
>>> [132020.964435] ffff88028b078000: 49 4e 00 00 03 02 00 00 00 30 00 70
>>> 00 00 03 e8  IN.......0.p....
>>> [132020.964437] ffff88028b078010: 00 00 00 00 06 20 b0 6f 01 2e 00 00
>>> 00 00 00 16  ..... .o........
>>> [132020.964438] ffff88028b078020: 01 57 37 fd 2b 5d 22 9e 1e 0a 61 8c
>>> 00 00 00 20  .W7.+]"...a....
>>> [132020.964440] ffff88028b078030: ff ff 00 d2 1b f6 27 90 00 00 00 00
>>> 00 00 00 00  ......'.........
>>> [132020.964454] XFS (loop0): Internal error xfs_iread at line 392 of
>>> file /build/linux-QZaPpC/linux-3.16.7-ckt11/fs/xfs/xfs_inode_buf.c.
>>> Caller xfs_iget+0x24b/0x690 [xfs]
>>
>> That's a different error to all the ones you've previously posted.
>> This is an inode allocation that has found a bad inode on disk.
>>
>> Decoding the 64 bytes above:
>>
>>     di_magic = 0x494e
>>     di_mode = 0
>>     di_version = 3            <<< That's *wrong*
>>     di_format = 2
>>     di_onlink = 0
>>     di_uid = 0x300070        <<< Looks unlikely
>>     di_gid = 0x3e8
>> ----
>>     di_nlink = 0
>>     di_projlo = 0x620        <<< should be zero
>>     di_projhi = 0xb06f        <<< should be zero
>>     di_pad[6] = 0x1 0x2e 0 0 0 0    <<< should be zero
>>     di_flushiter = 0x16        <<< should be zero for v3 inode
>> ---
>>     di_atime    <random>
>>     di_mtime    <random, should be similar to atime>
>>     di_ctime    <random, should be similar/same as mtime>
>>     di_size = 0x20ffff00d2        <<< should be zero
>> ----
>>     di_nblocks = 0x1bf6279000000000 <<< should be zero
>>     di_extsize = 0
>> ----
>>
>> You've just created and mounted a v4 filesystem, which means it is
>> using v2 inodes. This inode read back as a v3 inode, with lots of
>> crap in places where there should be zeros for either v2 or v3 inodes.
>>
>> This does not look like a filesystem problem - it's clear that what
>> has come from disk (or a cached memory buffer) is full of garbage
>> and contains invalid configuration, and the filesystem has quite
>> correctly detected the corruption and shut down. The filesystem
>> would give the same errors if it tried to *write* such a corrupt
>> block, so we know what was just been detected has not come from the
>> filesytem code...
>>
>> FWIW, I've occasionally seen this sort of thing happen when a power
>> supply had gone bad - it wasn't bad enough to make things fail, it
>> ust caused transient issues under load that manifest as corruptions
>> and crashes. Given that you've already found one set of hardware
>> problems and the corruption patterns are unlike any
>> filesystem/storage problem I've ever seen, I'd suggest that you
>> still have some kind of hardware issue...
>>
>> Cheers,
>>
>> Dave.
>>
>