XFS File system in trouble

Tue Aug 4 02:52:33 CDT 2015

	It's failing, again.  The rsync job failed and when I attempt to untar 
the file in the image mount, it fails there, as well.  See below.  I 
formatted a 1.5T drive as xfs and mounted it under /media.  I then 
dumped the failing FS to a file on /media using xfs_metadump and used 
xfs_mdrestore to create an image of the FS.  I then mounted the image, 
copied over the tarball to its location, and ran tar to extract the files:

RAID-Server:/# mount -o nouuid /media/md0.img /TEST

RAID-Server:/# cd "/TEST/Server-Main/Equipment/Drive 
Controllers/HighPoint Adapters/Rocket 2722/Driver"/

RAID-Server:/TEST/Server-Main/Equipment/Drive Controllers/HighPoint 
Adapters/Rocket 2722/Driver# cp "/RAID/Server-Main/Equipment/Drive 
Controllers/HighPoint Adapters/Rocket 2722/Driver/RR_27xx.tar.gz" ./

RAID-Server:/TEST/Server-Main/Equipment/Drive Controllers/HighPoint 
Adapters/Rocket 2722/Driver# tar -xzvf RR_27xx.tar.gz
DC7280/
DC7280/Linux/
DC7280/Linux/Opensource/
DC7280/Linux/Opensource/DC7280-linux-src-v1.0-110621-1313.tar.gz
DC7280/Windows/
DC7280/Windows/Vista-Win2008-Win7/
DC7280/Windows/Vista-Win2008-Win7/x32/
DC7280/Windows/Vista-Win2008-Win7/x32/dc7280.cat
DC7280/Windows/Vista-Win2008-Win7/x32/dc7280.inf
DC7280/Windows/Vista-Win2008-Win7/x32/dc7280.sys
DC7280/Windows/Vista-Win2008-Win7/x64/
DC7280/Windows/Vista-Win2008-Win7/x64/dc7280.cat
DC7280/Windows/Vista-Win2008-Win7/x64/dc7280.inf
DC7280/Windows/Vista-Win2008-Win7/x64/dc7280.sys
DC7280/Windows/Vista-Win2008-Win7/Readme.txt
DC7280/.ddinfo
R272x/
R272x/Linux/
R272x/Linux/Opensource/
R272x/Linux/Opensource/partial/
R272x/Linux/Opensource/partial/include/

...

RR274x/Driver/Linux/RHEL_CentOS/rr274x_3x-rhel_centos-4u8-i386/pcitable
RR274x/Driver/Linux/RHEL_CentOS/rr274x_3x-rhel_centos-4u8-i386/readme.txt
RR274x/Driver/Linux/RHEL_CentOS/rr274x_3x-rhel_centos-4u8-i386/rhdd
RR274x/Driver/Linux/RHEL_CentOS/rr274x_3x-rhel_centos-4u8-i386/rhel-install-step1.sh
RR274x/Driver/Linux/RHEL_CentOS/rr274x_3x-rhel_centos-4u8-i386/rhel-install-step2.sh
RR274x/Driver/Linux/RHEL_CentOS/rr274x_3x-rhel_centos-4u8-x86_64/
tar: RR274x/Driver/Linux/RHEL_CentOS/rr274x_3x-rhel_centos-4u8-x86_64: 
Cannot mkdir: Structure needs cleaning
RR274x/Driver/Linux/RHEL_CentOS/rr274x_3x-rhel_centos-4u8-x86_64/install.sh
tar: RR274x/Driver/Linux/RHEL_CentOS/rr274x_3x-rhel_centos-4u8-x86_64: 
Cannot mkdir: Input/output error
tar: 
RR274x/Driver/Linux/RHEL_CentOS/rr274x_3x-rhel_centos-4u8-x86_64/install.sh: 
Cannot open: No such file or directory
RR274x/Driver/Linux/RHEL_CentOS/rr274x_3x-rhel_centos-4u8-x86_64/installmethod.py
tar: RR274x/Driver/Linux/RHEL_CentOS/rr274x_3x-rhel_centos-4u8-x86_64: 
Cannot mkdir: Input/output error
tar: 
RR274x/Driver/Linux/RHEL_CentOS/rr274x_3x-rhel_centos-4u8-x86_64/installmethod.py: 
Cannot open: No such file or directory
RR274x/Driver/Linux/RHEL_CentOS/rr274x_3x-rhel_centos-4u8-x86_64/modinfo
tar: RR274x/Driver/Linux/RHEL_CentOS/rr274x_3x-rhel_centos-4u8-x86_64: 
Cannot mkdir: Input/output error
tar: 
RR274x/Driver/Linux/RHEL_CentOS/rr274x_3x-rhel_centos-4u8-x86_64/modinfo: Cannot 
open: No such file or directory
RR274x/Driver/Linux/RHEL_CentOS/rr274x_3x-rhel_centos-4u8-x86_64/modules.alias
tar: RR274x/Driver/Linux/RHEL_CentOS/rr274x_3x-rhel_centos-4u8-x86_64: 
Cannot mkdir: Input/output error
tar: 
RR274x/Driver/Linux/RHEL_CentOS/rr274x_3x-rhel_centos-4u8-x86_64/modules.alias: 
Cannot open: No such file or directory
RR274x/Driver/Linux/RHEL_CentOS/rr274x_3x-rhel_centos-4u8-x86_64/modules.cgz

gzip: tar: 
RR274x/Driver/Linux/RHEL_CentOS/rr274x_3x-rhel_centos-4u8-x86_64: Cannot 
mkdir: Input/output errorstdin: Input/output error

tar: Unexpected EOF in archive
tar: RR274x/Driver/Linux/RHEL_CentOS: Cannot utime: Input/output error
tar: RR274x/Driver/Linux/RHEL_CentOS: Cannot change ownership to uid 0, 
gid 1000: Input/output error
tar: RR274x/Driver/Linux/RHEL_CentOS: Cannot change mode to rwxr-xr-x: 
Input/output error
tar: RR274x/Driver/Linux: Cannot utime: Input/output error
tar: RR274x/Driver/Linux: Cannot change ownership to uid 0, gid 1000: 
Input/output error
tar: RR274x/Driver/Linux: Cannot change mode to rwxr-xr-x: Input/output 
error
tar: RR274x/Driver: Cannot utime: Input/output error
tar: RR274x/Driver: Cannot change ownership to uid 0, gid 1000: 
Input/output error
tar: RR274x/Driver: Cannot change mode to rwxr-xr-x: Input/output error
tar: RR274x: Cannot utime: Input/output error
tar: RR274x: Cannot change ownership to uid 0, gid 1000: Input/output error
tar: RR274x: Cannot change mode to rwxr-xr-x: Input/output error
tar: Error is not recoverable: exiting now


dmesg:
[131329.013475] XFS (md0): Mounting V4 Filesystem
[131329.918438] XFS (md0): Ending clean mount
[131499.357099] XFS (md0): Mounting V4 Filesystem
[131499.709248] XFS (md0): Ending clean mount
[131874.545344] loop: module loaded
[131874.549914] XFS (loop0): Mounting V4 Filesystem
[131874.555540] XFS (loop0): Ending clean mount
[132020.964431] XFS (loop0): xfs_iread: validation failed for inode 
124656869424 failed
[132020.964435] ffff88028b078000: 49 4e 00 00 03 02 00 00 00 30 00 70 00 
00 03 e8  IN.......0.p....
[132020.964437] ffff88028b078010: 00 00 00 00 06 20 b0 6f 01 2e 00 00 00 
00 00 16  ..... .o........
[132020.964438] ffff88028b078020: 01 57 37 fd 2b 5d 22 9e 1e 0a 61 8c 00 
00 00 20  .W7.+]"...a....
[132020.964440] ffff88028b078030: ff ff 00 d2 1b f6 27 90 00 00 00 00 00 
00 00 00  ......'.........
[132020.964454] XFS (loop0): Internal error xfs_iread at line 392 of 
file /build/linux-QZaPpC/linux-3.16.7-ckt11/fs/xfs/xfs_inode_buf.c. 
Caller xfs_iget+0x24b/0x690 [xfs]
[132020.964457] CPU: 2 PID: 21474 Comm: tar Not tainted 3.16.0-4-amd64 
#1 Debian 3.16.7-ckt11-1
[132020.964459] Hardware name: To be filled by O.E.M. To be filled by 
O.E.M./SABERTOOTH 990FX R2.0, BIOS 1503 01/11/2013
[132020.964460]  0000000000000001 ffffffff8150b405 ffff880424059800 
ffffffffa09115cb
[132020.964463]  0000018800000010 ffffffffa0916f6b ffff88030f5c6c00 
ffff880424059800
[132020.964465]  0000000000000075 ffff8800ad1afe98 ffffffffa095cb3a 
ffffffffa0916f6b
[132020.964467] Call Trace:
[132020.964471]  [<ffffffff8150b405>] ? dump_stack+0x41/0x51
[132020.964478]  [<ffffffffa09115cb>] ? xfs_corruption_error+0x5b/0x80 [xfs]
[132020.964483]  [<ffffffffa0916f6b>] ? xfs_iget+0x24b/0x690 [xfs]
[132020.964492]  [<ffffffffa095cb3a>] ? xfs_iread+0xea/0x400 [xfs]
[132020.964497]  [<ffffffffa0916f6b>] ? xfs_iget+0x24b/0x690 [xfs]
[132020.964503]  [<ffffffffa0916f6b>] ? xfs_iget+0x24b/0x690 [xfs]
[132020.964511]  [<ffffffffa0956de6>] ? xfs_ialloc+0xa6/0x500 [xfs]
[132020.964517]  [<ffffffffa092658e>] ? kmem_zone_alloc+0x6e/0xe0 [xfs]
[132020.964525]  [<ffffffffa09572a2>] ? xfs_dir_ialloc+0x62/0x2a0 [xfs]
[132020.964531]  [<ffffffffa09251e5>] ? xfs_trans_reserve+0x1f5/0x200 [xfs]
[132020.964538]  [<ffffffffa09579a9>] ? xfs_create+0x489/0x700 [xfs]
[132020.964541]  [<ffffffff811b40ea>] ? kern_path_create+0xaa/0x190
[132020.964548]  [<ffffffffa091c5ea>] ? xfs_generic_create+0xca/0x250 [xfs]
[132020.964550]  [<ffffffff811b7ad0>] ? vfs_mkdir+0xb0/0x160
[132020.964551]  [<ffffffff811b868b>] ? SyS_mkdirat+0xab/0xe0
[132020.964554]  [<ffffffff815115cd>] ? 
system_call_fast_compare_end+0x10/0x15
[132020.964555] XFS (loop0): Corruption detected. Unmount and run xfs_repair
[132020.964564] XFS (loop0): Internal error xfs_trans_cancel at line 959 
of file /build/linux-QZaPpC/linux-3.16.7-ckt11/fs/xfs/xfs_trans.c. 
Caller xfs_create+0x2b2/0x700 [xfs]
[132020.964566] CPU: 2 PID: 21474 Comm: tar Not tainted 3.16.0-4-amd64 
#1 Debian 3.16.7-ckt11-1
[132020.964567] Hardware name: To be filled by O.E.M. To be filled by 
O.E.M./SABERTOOTH 990FX R2.0, BIOS 1503 01/11/2013
[132020.964568]  000000000000000c ffffffff8150b405 ffff8800ad1afe98 
ffffffffa0925e07
[132020.964570]  ffff880002530800 ffff880079e03ec8 ffff880424059800 
ffffffffa09577d2
[132020.964571]  0000000000000001 ffff880079e03e20 ffff880079e03e1c 
ffff880079e03eb0
[132020.964573] Call Trace:
[132020.964575]  [<ffffffff8150b405>] ? dump_stack+0x41/0x51
[132020.964581]  [<ffffffffa0925e07>] ? xfs_trans_cancel+0xc7/0xf0 [xfs]
[132020.964588]  [<ffffffffa09577d2>] ? xfs_create+0x2b2/0x700 [xfs]
[132020.964590]  [<ffffffff811b40ea>] ? kern_path_create+0xaa/0x190
[132020.964596]  [<ffffffffa091c5ea>] ? xfs_generic_create+0xca/0x250 [xfs]
[132020.964598]  [<ffffffff811b7ad0>] ? vfs_mkdir+0xb0/0x160
[132020.964600]  [<ffffffff811b868b>] ? SyS_mkdirat+0xab/0xe0
[132020.964602]  [<ffffffff815115cd>] ? 
system_call_fast_compare_end+0x10/0x15
[132020.964604] XFS (loop0): xfs_do_force_shutdown(0x8) called from line 
960 of file /build/linux-QZaPpC/linux-3.16.7-ckt11/fs/xfs/xfs_trans.c. 
Return address = 0xffffffffa0925e20
[132021.196487] XFS (loop0): Corruption of in-memory data detected. 
Shutting down filesystem
[132021.196491] XFS (loop0): Please umount the filesystem and rectify 
the problem(s)
[132024.791456] XFS (loop0): xfs_log_force: error 5 returned.
[132054.854625] XFS (loop0): xfs_log_force: error 5 returned.
[132084.917775] XFS (loop0): xfs_log_force: error 5 returned.
[132114.980927] XFS (loop0): xfs_log_force: error 5 returned.
[132145.044086] XFS (loop0): xfs_log_force: error 5 returned.
[132175.107307] XFS (loop0): xfs_log_force: error 5 returned.
[132205.170404] XFS (loop0): xfs_log_force: error 5 returned.
[132235.233587] XFS (loop0): xfs_log_force: error 5 returned.


On 8/2/2015 3:24 PM, Leslie Rhorer wrote:
>
>      OK, this is goofy.  It seems to be working, now.  As usual, I've
> been doing some work on the server this weekend, but I can't think of
> anything I have done that would fix the issue.  I did replace the
> remaining good 4G RAM module with a pair of 8G RAM modules, but memtest
> reported the remaining 4G module as good, and I verified the removed
> module really was bad.  I also replaced the removable drive carrier and
> cables that were feeding the two SSDs, once of which was reporting
> failures as noted in the syslog.  It's hard for me to believe either of
> those things could have been causing the issue, though.
>
>      I attached a 1.5T external drive to the server and formatted it as
> XFS in preparation to continue troubleshooting.  To make sure of things,
> I tried decompressing the tarball, again, and this time it worked all
> the way to the end.  I then deleted the entire directory structure
> created by the tarball and decompressed the file again twice.  I'll see
> if the rsync process works.  That will take a couple of days.
>
> On 7/28/2015 5:11 PM, Brian Foster wrote:
>> On Tue, Jul 28, 2015 at 10:13:01AM -0500, Leslie Rhorer wrote:
>>> On 7/28/2015 7:33 AM, Brian Foster wrote:
>>>> On Tue, Jul 28, 2015 at 02:46:45AM -0500, Leslie Rhorer wrote:
>>>>> On 7/20/2015 6:17 AM, Brian Foster wrote:
>>>>>> On Sat, Jul 18, 2015 at 08:02:50PM -0500, Leslie Rhorer wrote:
>>>>>>>
>> ...
>>>>
>>>>>     I then copied both the tarball and the image over to the root,
>>>>> and while
>>>>> the system would not let me create the image on the root, it did
>>>>> let me copy
>>>>> the image to the root.  I then umounted the RAID array, mounted the
>>>>> image,
>>>>> and attempted to cd to the original directory in the image mount
>>>>> where the
>>>>> tarball was saved.  That failed with an I/O error:
>>>>>
>>>>
>>>> It sounds a bit strange for the mdrestore to fail on root but a cp of
>>>> the resulting image to work. Do the resulting images have the same file
>>>> size or is the rootfs copy truncated? If the latter, you could be
>>>> missing part of the fs and thus any of the following tests are probably
>>>> moot.
>>>
>>>     Well, it can't be as large as it is reported, let's put it that way,
>>> although the reported file size is the same.  Ls claims it to be 16T in
>>> size, which cannot be the case on a 100G partition.  I forgot to
>>> mention cp
>>> does complain:
>>>
>>> RAID-Server:/# cp /RAID/TEST/RAIDfile.img ./
>>> cp: cannot lseek ‘./RAIDfile.img’: Invalid argument
>>>
>>>     But it does the same thing on the backup server, and it works
>>> there.  I
>>> tried a cmp, and it seems to be hung.  It just may be taking a long
>>> time,
>>> however.
>>>
>>
>> Yeah, you can't really trust the resulting image. It doesn't take much
>> space to create a very large sparse file, but different filesystems have
>> different maximum file size limits. The problem here is that some
>> metadata near the beginning of the file might reference or depend on
>> something near the end, and I/Os beyond the end of the file will
>> probably result in errors.
>>
>> I'd probably try the nouuid approach since the hardware is similar as
>> well as some of the other interesting suggestions that have been made to
>> try and get the image on the rootfs and see what happens there too.
>>
>> Brian
>>
>>>> Brian
>>>>
>>>>> RAID-Server:/# cd "/media/Server-Main/Equipment/Drive
>>>>> Controllers/HighPoint
>>>>> Adapters/Rocket 2722/Driver/"
>>>>> bash: cd: /media/Server-Main/Equipment/Drive Controllers/HighPoint
>>>>> Adapters/Rocket 2722/Driver/: Input/output error
>>>>>
>>>>>     I changed directories to a point two directories above the
>>>>> previous attempt
>>>>> and did a long listing:
>>>>>
>>>>> RAID-Server:/# cd "/media/Server-Main/Equipment/Drive
>>>>> Controllers/HighPoint
>>>>> Adapters"
>>>>> RAID-Server:/media/Server-Main/Equipment/Drive Controllers/HighPoint
>>>>> Adapters# ll
>>>>> ls: cannot access RocketRAID 2722: Input/output error
>>>>> total 4
>>>>> drwxr-xr-x 6 root lrhorer 4096 Jul 18 19:26 Rocket 2722
>>>>> ?????????? ? ?    ?          ?            ? RocketRAID 2722
>>>>>
>>>>>     As you can see, Rocket 2722 is still there, but RocketRAID 2722
>>>>> is very
>>>>> sick.  Rocket 2722 is the parent of where the tarbal was, however,
>>>>> so I did
>>>>> a cd and an ll again:
>>>>>
>>>>> RAID-Server:/media/Server-Main/Equipment/Drive Controllers/HighPoint
>>>>> Adapters# cd "Rocket 2722"/
>>>>> RAID-Server:/media/Server-Main/Equipment/Drive Controllers/HighPoint
>>>>> Adapters/Rocket 2722# ll
>>>>> ls: cannot access BIOS: Input/output error
>>>>> ls: cannot access Driver: Input/output error
>>>>> ls: cannot access HighPoint RAID Management Software: Input/output
>>>>> error
>>>>> ls: cannot access Manual: Input/output error
>>>>> total 248
>>>>> -rwxr--r-- 1 root lrhorer 245760 Nov 20  2008 autorun.exe
>>>>> -rwxr--r-- 1 root lrhorer     51 Mar 21  2001 autorun.inf
>>>>> ?????????? ? ?    ?            ?            ? BIOS
>>>>> ?????????? ? ?    ?            ?            ? Driver
>>>>> ?????????? ? ?    ?            ?            ? HighPoint RAID
>>>>> Management
>>>>> Software
>>>>> ?????????? ? ?    ?            ?            ? Manual
>>>>> -rwxr--r-- 1 root lrhorer   1134 Feb  5  2012 readme.txt
>>>>>
>>>>>     So now, what?
>>>>>
>>>>> _______________________________________________
>>>>> xfs mailing list
>>>>> xfs at oss.sgi.com
>>>>> http://oss.sgi.com/mailman/listinfo/xfs
>>>>
>>>
>>> _______________________________________________
>>> xfs mailing list
>>> xfs at oss.sgi.com
>>> http://oss.sgi.com/mailman/listinfo/xfs
>>
>