> Last week one of my fileserver has rebooted due to power failure, and
found
> out there were corrupted files with the following error messages:
>
> Jan 13 08:01:17 fs5-11 kernel: Filesystem "sd(8,7)": corrupt dinode
> 101980910, extent total = 522866323, nblocks = 3608. Unmo
> unt and run xfs_repair.
> Jan 13 08:01:17 fs5-11 kernel: 0x0: 49 4e 81 80 01 02 00 01 00 00 03 e9 00
> 00 03 e9
> Jan 13 08:01:17 fs5-11 kernel: Filesystem "sd(8,7)": XFS internal error
> xfs_iformat(1) at line 472 of file xfs_inode.c. Calle
> r 0xe099535b
> Jan 13 08:01:17 fs5-11 kernel: dca71c68 e099423f e09db2b4 00000001
dde26800
> e09db292 000001d8 e099535b
> Jan 13 08:01:17 fs5-11 kernel: e099535b 1f2a3320 1f2a4e93 00000000
> 00000000 00000000 00000000 dde26800
> Jan 13 08:01:17 fs5-11 kernel: c6e03298 e099535b c6e03298 c386ae00
> c386ae00 c6e033c4 00000001 00000000
> Jan 13 08:01:17 fs5-11 kernel: Call Trace: [<e099423f>] xfs_iformat
[xfs]
> 0x20f (0xdca71c6c))
> Jan 13 08:01:17 fs5-11 kernel: [<e09db2b4>] .LC14 [xfs] 0x3d3
(0xdca71c70))
> Jan 13 08:01:17 fs5-11 kernel: [<e09db292>] .LC14 [xfs] 0x3b1
(0xdca71c7c))
> Jan 13 08:01:17 fs5-11 kernel: [<e099535b>] xfs_iread [xfs] 0xdb
> (0xdca71c84))
> Jan 13 08:01:17 fs5-11 kernel: [<e099535b>] xfs_iread [xfs] 0xdb
> (0xdca71c88))
> Jan 13 08:01:17 fs5-11 kernel: [<e099535b>] xfs_iread [xfs] 0xdb
> (0xdca71cac))
> Jan 13 08:01:17 fs5-11 kernel: [<e0992fb6>] xfs_iget_core [xfs] 0x1b6
> (0xdca71ce8))
> Jan 13 08:01:17 fs5-11 kernel: [<e099335d>] xfs_iget [xfs] 0x7d
> (0xdca71d38))
> Jan 13 08:01:17 fs5-11 kernel: [<e09a9f81>] xfs_dir_lookup_int [xfs] 0x61
> (0xdca71d7c))
> Jan 13 08:01:17 fs5-11 kernel: [<e09ae7be>] xfs_lookup [xfs] 0x3e
> (0xdca71db8))
> Jan 13 08:01:17 fs5-11 kernel: [<e09bb52f>] linvfs_lookup [xfs] 0x3f
> (0xdca71dec))
> Jan 13 08:01:17 fs5-11 kernel: [<c0156d70>] cached_lookup [kernel] 0x10
> (0xdca71e10))
> Jan 13 08:01:17 fs5-11 kernel: [<c0157e71>] lookup_hash [kernel] 0x91
> (0xdca71e24))
> Jan 13 08:01:17 fs5-11 kernel: [<c0157f09>] lookup_one_len [kernel] 0x59
> (0xdca71e40))
> Jan 13 08:01:17 fs5-11 kernel: [<e0a8a660>] nfsd_lookup [nfsd] 0x340
> (0xdca71e64))
> Jan 13 08:01:17 fs5-11 kernel: [<c011a5ef>] __wake_up [kernel] 0x4f
> (0xdca71ed0))
> Jan 13 08:01:17 fs5-11 kernel: [<e0a531a4>] svc_sock_enqueue [sunrpc]
0x184
> (0xdca71ef8))
> Jan 13 08:01:17 fs5-11 kernel: [<e0a53a43>] svc_udp_recvfrom [sunrpc]
0x2d3
> (0xdca71f10))
> Jan 13 08:01:17 fs5-11 kernel: [<e0a90418>] nfsd3_proc_lookup [nfsd] 0xd8
> (0xdca71f38))
> Jan 13 08:01:17 fs5-11 kernel: [<e0a99a8c>] nfsd_procedures3 [nfsd] 0x6c
> (0xdca71f58))
> Jan 13 08:01:17 fs5-11 kernel: [<e0a87567>] nfsd_dispatch [nfsd] 0xb7
> (0xdca71f64))
> Jan 13 08:01:17 fs5-11 kernel: [<e0a993d8>] nfsd_version3 [nfsd] 0x0
> (0xdca71f7c))
> Jan 13 08:01:17 fs5-11 kernel: [<e0a52e18>] svc_process_Rsmp_01d929dc
> [sunrpc] 0x368 (0xdca71f80))
> Jan 13 08:01:17 fs5-11 kernel: [<e0a99a8c>] nfsd_procedures3 [nfsd] 0x6c
> (0xdca71f9c))
> Jan 13 08:01:17 fs5-11 kernel: [<e0a993d8>] nfsd_version3 [nfsd] 0x0
> (0xdca71fa0))
> Jan 13 08:01:17 fs5-11 kernel: [<e0a993f8>] nfsd_program [nfsd] 0x0
> (0xdca71fa4))
> Jan 13 08:01:17 fs5-11 kernel: [<e0a8736a>] nfsd [nfsd] 0x1ca
(0xdca71fc0))
> Jan 13 08:01:17 fs5-11 kernel: [<e0a871a0>] nfsd [nfsd] 0x0 (0xdca71fe0))
> Jan 13 08:01:17 fs5-11 kernel: [<c0107275>] kernel_thread_helper [kernel]
> 0x5 (0xdca71ff0))
> Jan 13 08:01:17 fs5-11 kernel:
> Jan 13 08:01:17 fs5-11 kernel: nfsd: non-standard errno: -990
>
> Tried to run xfs_repair but failed with fatal error:
>
> Phase 1 - find and verify superblock...
> Phase 2 - using internal log
> - zero log...
> zero_log: head block 59683 tail block 59683
> - scan filesystem freespace and inode maps...
> - found root inode chunk
> Phase 3 - for each AG...
> - scan and clear agi unlinked lists...
> - process known inodes and perform inode discovery...
> - agno = 0
> corrupt block 0 in directory inode 158139
> will junk block
> no . entry for directory 158139
> no .. entry for directory 158139
> corrupt block 3 in directory inode 1231523
> will junk block
> .
> .
> .
> Phase 5 - rebuild AG headers and trees...
> - reset superblock...
> Phase 6 - check inode connectivity...
> - resetting contents of realtime bitmap and summary inodes
> - ensuring existence of lost+found directory
> - traversing filesystem starting at / ...
> corrupt block 8 in directory inode 838926683: junking block
> rebuilding directory inode 838926683
> corrupt block 5 in directory inode 170001259: junking block
> bad hash table for directory inode 170001259 (no data entry): rebuilding
> disconnected dir inode 2081119363, moving to lost+found
> disconnected dir inode 2116550700, moving to lost+found
> .
> .
> .
> Phase 7 - verify and correct link counts...
> resetting inode 158139 nlinks from 3 to 2
> resetting inode 16967337 nlinks from 3 to 2
> resetting inode 335545743 nlinks from 2806 to 2805
> fatal error -- couldn't map inode 2133733748, err = 22
>
> I found some other list posting with similar problem, did any fix being
> release regarding this issue?
>
> xfs_info as follow:
[root@fs5-11 fs5_11us4_2]# xfs_info /fs5_11us4_2/
meta-data=/fs5_11us4_2 isize=256 agcount=128, agsize=561334
blks
= sectsz=512
data = bsize=4096 blocks=71850704, imaxpct=25
= sunit=0 swidth=0 blks, unwritten=1
naming =version 2 bsize=4096
log =internal bsize=4096 blocks=32768, version=1
= sectsz=512 sunit=0 blks
realtime =none extsz=65536 blocks=0, rtextents=0
The system is now running but I feel there are something wrong with the file
system, moving 150GB+ data will be painful. Please help and see if any fixs
I can do to recover a clean file system.
Many thanks,
Eric Yu
Systems Engineer
Outblaze Ltd.
|