I've been having hard-to-reproduce sata_sil24 driver problems, and
the most recent lockup appears to have taken out my root file system.
(Hooray for emergency boot partitions!)
# mount -r /dev/md4 /mnt
mount: /dev/md4: can't read superblock
With the following appended to the kernel log:
XFS mounting filesystem md4
Starting XFS recovery on filesystem: md4 (dev: md4)
XFS: xlog_recover_process_data: bad clientid
XFS: log mount/recovery failed: error 5
XFS: log mount failed
I haven't put this server into production yet (due to aforementioned
libata issues), so I'm experiementing with XFS, but this failure isn't
confidence-inspiring.
It was running pcbackup, a disk-based backup system that generates lots
of files and lots of hard links. (It also prepends all file names with
"f" to prevent collisions with its own files.)
Emergency kernel not mounting is Linus 2.6.13.2 + libata patch
2.6.13-rc7-libata1.patch.bz2
Kernel not booting is libata development git based on 2.6.14-rc3.
md4 is a RAID10 array:
md4 : active raid10 sdf3[4] sde3[3] sdd3[2] sdc3[1] sdb3[0] sda3[5]
131837184 blocks 256K chunks 2 near-copies [6/6] [UUUUUU]
Unfortunately, being unable to mount the file system, I can't give xfs_info
output. xfs_db version gives
versionnum [0x3184] = V4,ALIGN,DALIGN,DIRV2,EXTFLG
xfs_logprint /dev/md4 produces:
xfs_logprint:
data device: 0x904
log device: 0x904 daddr: 131838464 length: 129024
Header 0x30c wanted 0xfeedbabe
**********************************************************************
* ERROR: header cycle=780 block=72440 *
**********************************************************************
Bad log record header
xfs_repair -n produces rather a lot of info. I'm not used to what sort
of errors are "harmless", so I'm not sure if it's a good idea to run
the recovery for real. The number of bad .. entries is a bit alarming.
I have compressed out ("[...]") runs of very similar lines with (what
appear to be) sequential block or inode numbers. (Perhaps xfs_check
could be amended to compress those lines down to one range?)
Anyway, can anyone suggest how to proceed? The file system isn't
cirtical, but it would be annoying to rebuild.
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- scan filesystem freespace and inode maps...
block (0,914473) multiply claimed by bno space tree, state - 1
[...]
block (0,914495) multiply claimed by bno space tree, state - 1
block (0,969351) already used, state 2
[...]
block (0,969407) already used, state 2
block (4,2557) already used, state 2
[...]
block (4,155647) already used, state 2
block (8,299639) multiply claimed by bno space tree, state - 1
block (10,1223238) already used, state 2
[...]
block (10,1223294) already used, state 2
- found root inode chunk
Phase 3 - for each AG...
- scan (but don't clear) agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
imap claims a free inode 15509544 is in use, would correct imap and clear inode
imap claims a free inode 15509545 is in use, would correct imap and clear inode
- agno = 1
imap claims a free inode 51346438 is in use, would correct imap and clear inode
- agno = 2
data fork in ino 69496886 claims free block 4352910
- agno = 3
imap claims a free inode 129571903 is in use, would correct imap and clear inode
- agno = 4
data fork in ino 134219837 claims free block 8391159
[...]
data fork in ino 134219837 claims free block 8544227
imap claims a free inode 155289632 is in use, would correct imap and clear inode
[...]
imap claims a free inode 155289649 is in use, would correct imap and clear inode
- agno = 5
imap claims in-use inode 193486864 is free, would correct imap
imap claims in-use inode 193486865 is free, would correct imap
imap claims a free inode 193852455 is in use, would correct imap and clear inode
- agno = 6
- agno = 7
data fork in ino 248475688 claims free block 15529735
data fork in ino 248475689 claims free block 15529725
- agno = 8
imap claims a free inode 287203328 is in use, would correct imap and clear inode
[...]
imap claims a free inode 287203343 is in use, would correct imap and clear inode
imap claims in-use inode 287203344 is free, would correct imap
- agno = 9
imap claims in-use inode 313304070 is free, would correct imap
- agno = 10
imap claims a free inode 349614096 is in use, would correct imap and clear inode
[...]
imap claims a free inode 349614109 is in use, would correct imap and clear inode
- agno = 11
imap claims in-use inode 386031670 is free, would correct imap
data fork in ino 386031671 claims free block 24126990
imap claims in-use inode 386031671 is free, would correct imap
- agno = 12
imap claims a free inode 424609824 is in use, would correct imap and clear inode
[...]
imap claims a free inode 424609839 is in use, would correct imap and clear inode
- agno = 13
imap claims a free inode 448496640 is in use, would correct imap and clear inode
[...]
imap claims a free inode 448496654 is in use, would correct imap and clear inode
- agno = 14
imap claims a free inode 486567939 is in use, would correct imap and clear inode
- agno = 15
imap claims in-use inode 523361317 is free, would correct imap
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
entry ".." at block 0 offset 32 in directory inode 15509541 references free
inode 448496649
would clear inode number in entry at offset 32...
entry ".." at block 0 offset 32 in directory inode 15509542 references free
inode 424609837
would clear inode number in entry at offset 32...
- agno = 1
entry "fimkr6_1" at block 0 offset 168 in directory inode 51345453 references
free inode 287203337
would clear inode number in entry at offset 168...
entry ".." at block 0 offset 32 in directory inode 51346433 references free
inode 448496647
would clear inode number in entry at offset 32...
entry ".." at block 0 offset 32 in directory inode 51346435 references free
inode 448496649
would clear inode number in entry at offset 32...
- agno = 2
entry "fbasic_ofstream" at block 0 offset 264 in directory inode 84868115
references free inode 287203330
would clear inode number in entry at offset 264...
entry "fbasic_ostringstream" at block 0 offset 328 in directory inode 84868115
references free inode 448496640
would clear inode number in entry at offset 328...
entry "fbasic_streambuf" at block 0 offset 360 in directory inode 84868115
references free inode 155289638
would clear inode number in entry at offset 360...
entry "fbasic_stringstream" at block 0 offset 424 in directory inode 84868115
references free inode 448496644
would clear inode number in entry at offset 424...
entry "fios_base" at block 0 offset 472 in directory inode 84868115 references
free inode 155289642
would clear inode number in entry at offset 472...
entry "fmanipulators" at block 0 offset 496 in directory inode 84868115
references free inode 424609831
would clear inode number in entry at offset 496...
entry "fpod" in shortform directory 84868120 references free inode 155289633
would have junked entry "fpod" in directory inode 84868120
entry ".." at block 0 offset 32 in directory inode 84868136 references free
inode 424609838
would clear inode number in entry at offset 32...
- agno = 3
entry "fchar" in shortform directory 129571885 references free inode 155289632
would have junked entry "fchar" in directory inode 129571885
entry "fchar" in shortform directory 129571887 references free inode 155289634
would have junked entry "fchar" in directory inode 129571887
entry "frdbuf" in shortform directory 129571888 references free inode 155289635
would have junked entry "frdbuf" in directory inode 129571888
entry "fchar" in shortform directory 129571893 references free inode 155289640
would have junked entry "fchar" in directory inode 129571893
entry "fchar" in shortform directory 129571894 references free inode 155289641
would have junked entry "fchar" in directory inode 129571894
entry "fchar" in shortform directory 129571896 references free inode 155289643
would have junked entry "fchar" in directory inode 129571896
entry ".." at block 0 offset 32 in directory inode 129571901 references free
inode 448496649
would clear inode number in entry at offset 32...
entry ".." at block 0 offset 32 in directory inode 129571902 references free
inode 349614102
would clear inode number in entry at offset 32...
- agno = 4
- agno = 5
entry "finf" at block 0 offset 3536 in directory inode 193483808 references
free inode 349614101
would clear inode number in entry at offset 3536...
entry "fres" in shortform directory 193486858 references free inode 349614100
would have junked entry "fres" in directory inode 193486858
entry ".." at block 0 offset 32 in directory inode 193486859 references free
inode 287203340
would clear inode number in entry at offset 32...
entry ".." at block 0 offset 32 in directory inode 193486860 references free
inode 424609836
would clear inode number in entry at offset 32...
- agno = 6
entry "fzlib" at block 0 offset 1304 in directory inode 225607728 references
free inode 287203340
would clear inode number in entry at offset 1304...
entry "fpod" in shortform directory 225609755 references free inode 287203332
would have junked entry "fpod" in directory inode 225609755
- agno = 7
entry "fchar" in shortform directory 248475672 references free inode 287203329
would have junked entry "fchar" in directory inode 248475672
entry "fchar" in shortform directory 248475676 references free inode 287203333
would have junked entry "fchar" in directory inode 248475676
entry "fchar" in shortform directory 248475678 references free inode 287203335
would have junked entry "fchar" in directory inode 248475678
entry "fchar" in shortform directory 248475679 references free inode 287203336
would have junked entry "fchar" in directory inode 248475679
entry ".." at block 0 offset 32 in directory inode 248475686 references free
inode 349614102
would clear inode number in entry at offset 32...
entry ".." at block 0 offset 32 in directory inode 248475688 references free
inode 424609838
would clear inode number in entry at offset 32...
- agno = 8
- agno = 9
entry ".." at block 0 offset 32 in directory inode 313303087 references free
inode 287203328
would clear inode number in entry at offset 32...
entry "fchar" in shortform directory 313303095 references free inode 349614096
would have junked entry "fchar" in directory inode 313303095
entry ".." at block 0 offset 32 in directory inode 313304068 references free
inode 287203342
would clear inode number in entry at offset 32...
entry ".." at block 0 offset 32 in directory inode 313304069 references free
inode 287203343
would clear inode number in entry at offset 32...
- agno = 10
entry "fwchar_t" in shortform directory 349614091 references free inode
424609825
would have junked entry "fwchar_t" in directory inode 349614091
entry "fimbue" at block 0 offset 144 in directory inode 349614094 references
free inode 448496642
would clear inode number in entry at offset 144...
entry "fsgetn" at block 0 offset 336 in directory inode 349614094 references
free inode 448496643
would clear inode number in entry at offset 336...
- agno = 11
entry "fchar" in shortform directory 386031657 references free inode 424609824
would have junked entry "fchar" in directory inode 386031657
entry "fchar" in shortform directory 386031659 references free inode 424609826
would have junked entry "fchar" in directory inode 386031659
entry "fchar" in shortform directory 386031661 references free inode 424609828
would have junked entry "fchar" in directory inode 386031661
entry "fchar" in shortform directory 386031662 references free inode 424609829
would have junked entry "fchar" in directory inode 386031662
entry "fchar" in shortform directory 386031663 references free inode 424609830
would have junked entry "fchar" in directory inode 386031663
entry ".." at block 0 offset 32 in directory inode 386031665 references free
inode 349614098
would clear inode number in entry at offset 32...
entry ".." at block 0 offset 32 in directory inode 386031667 references free
inode 287203340
would clear inode number in entry at offset 32...
entry "fasm386" at block 0 offset 80 in directory inode 386031667 references
free inode 424609834
would clear inode number in entry at offset 80...
entry "fasm586" at block 0 offset 104 in directory inode 386031667 references
free inode 448496648
would clear inode number in entry at offset 104...
entry "funtgz" at block 0 offset 272 in directory inode 386031667 references
free inode 155289646
would clear inode number in entry at offset 272...
entry "fbuild" in shortform directory 386031668 references free inode 424609836
would have junked entry "fbuild" in directory inode 386031668
entry "fdebian" in shortform directory 386031668 references free inode 287203342
would have junked entry "fdebian" in directory inode 386031668
entry "fsrc" in shortform directory 386031668 references free inode 349614102
would have junked entry "fsrc" in directory inode 386031668
entry ".." at block 0 offset 32 in directory inode 386031669 references free
inode 349614102
would clear inode number in entry at offset 32...
entry ".." at block 0 offset 32 in directory inode 386031671 references free
inode 424609838
would clear inode number in entry at offset 32...
- agno = 12
entry "fgcc-arm-linux_3.3.3-1.dsc.asc" at block 0 offset 2584 in directory
inode 424606730 references free inode 424609835
would clear inode number in entry at offset 2584...
entry "fgcc-m68k-elf-3.4.1" at block 0 offset 2944 in directory inode 424606730
references free inode 287203341
would clear inode number in entry at offset 2944...
- agno = 13
- agno = 14
entry "fconfig" at block 0 offset 432 in directory inode 486566933 references
free inode 287203338
would clear inode number in entry at offset 432...
entry "fdemangle" at block 0 offset 472 in directory inode 486566933 references
free inode 349614098
would clear inode number in entry at offset 472...
entry "flibstdc++-dg" at block 0 offset 528 in directory inode 486566933
references free inode 424609833
would clear inode number in entry at offset 528...
entry "fperformance" at block 0 offset 552 in directory inode 486566933
references free inode 448496647
would clear inode number in entry at offset 552...
entry "fstdio_filebuf" at block 0 offset 312 in directory inode 486566973
references free inode 155289644
would clear inode number in entry at offset 312...
entry ".." at block 0 offset 32 in directory inode 486567936 references free
inode 448496649
would clear inode number in entry at offset 32...
entry ".." at block 0 offset 32 in directory inode 486567937 references free
inode 424609837
would clear inode number in entry at offset 32...
- agno = 15
entry "fextractors_other" at block 0 offset 208 in directory inode 523361301
references free inode 287203328
would clear inode number in entry at offset 208...
entry "fends" at block 0 offset 144 in directory inode 523361304 references
free inode 155289636
would clear inode number in entry at offset 144...
entry "fflush" at block 0 offset 184 in directory inode 523361304 references
free inode 287203331
would clear inode number in entry at offset 184...
entry "fseekp" at block 0 offset 320 in directory inode 523361304 references
free inode 155289637
would clear inode number in entry at offset 320...
entry ".." at block 0 offset 32 in directory inode 523361316 references free
inode 424609837
would clear inode number in entry at offset 32...
entry ".." at block 0 offset 32 in directory inode 523361317 references free
inode 424609838
would clear inode number in entry at offset 32...
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
- traversing filesystem starting at / ...
entry "fgcc-arm-linux_3.3.3-1.dsc.asc" in directory inode 424606730 points to
free inode 424609835, would junk entry
entry "fgcc-m68k-elf-3.4.1" in directory inode 424606730 points to free inode
287203341, would junk entry
entry "fbuild" in shortform directory inode 386031668 points to free inode
424609836
would junk entry "fbuild"
entry "fdebian" in shortform directory inode 386031668 points to free inode
287203342
would junk entry "fdebian"
entry "fsrc" in shortform directory inode 386031668 points to free inode
349614102
would junk entry "fsrc"
entry "fzlib" in directory inode 225607728 points to free inode 287203340,
would junk entry
entry "fconfig" in directory inode 486566933 points to free inode 287203338,
would junk entry
entry "fdemangle" in directory inode 486566933 points to free inode 349614098,
would junk entry
entry "flibstdc++-dg" in directory inode 486566933 points to free inode
424609833, would junk entry
entry "fperformance" in directory inode 486566933 points to free inode
448496647, would junk entry
entry "fstdio_filebuf" in directory inode 486566973 points to free inode
155289644, would junk entry
entry "fbasic_ofstream" in directory inode 84868115 points to free inode
287203330, would junk entry
entry "fbasic_ostringstream" in directory inode 84868115 points to free inode
448496640, would junk entry
entry "fbasic_streambuf" in directory inode 84868115 points to free inode
155289638, would junk entry
entry "fbasic_stringstream" in directory inode 84868115 points to free inode
448496644, would junk entry
entry "fios_base" in directory inode 84868115 points to free inode 155289642,
would junk entry
entry "fmanipulators" in directory inode 84868115 points to free inode
424609831, would junk entry
entry "fchar" in shortform directory inode 129571896 points to free inode
155289643
would junk entry "fchar"
entry "fimbue" in directory inode 349614094 points to free inode 448496642,
would junk entry
entry "fsgetn" in directory inode 349614094 points to free inode 448496643,
would junk entry
entry "fchar" in shortform directory inode 386031663 points to free inode
424609830
would junk entry "fchar"
entry "fchar" in shortform directory inode 313303095 points to free inode
349614096
would junk entry "fchar"
entry "fchar" in shortform directory inode 248475679 points to free inode
287203336
would junk entry "fchar"
entry "fchar" in shortform directory inode 129571894 points to free inode
155289641
would junk entry "fchar"
entry "fchar" in shortform directory inode 386031662 points to free inode
424609829
would junk entry "fchar"
entry "fchar" in shortform directory inode 248475678 points to free inode
287203335
would junk entry "fchar"
entry "fchar" in shortform directory inode 129571893 points to free inode
155289640
would junk entry "fchar"
entry "fchar" in shortform directory inode 386031661 points to free inode
424609828
would junk entry "fchar"
entry "fends" in directory inode 523361304 points to free inode 155289636,
would junk entry
entry "fflush" in directory inode 523361304 points to free inode 287203331,
would junk entry
entry "fseekp" in directory inode 523361304 points to free inode 155289637,
would junk entry
entry "fchar" in shortform directory inode 386031659 points to free inode
424609826
would junk entry "fchar"
entry "fpod" in shortform directory inode 225609755 points to free inode
287203332
would junk entry "fpod"
entry "fwchar_t" in shortform directory inode 349614091 points to free inode
424609825
would junk entry "fwchar_t"
entry "frdbuf" in shortform directory inode 129571888 points to free inode
155289635
would junk entry "frdbuf"
entry "fextractors_other" in directory inode 523361301 points to free inode
287203328, would junk entry
entry "fchar" in shortform directory inode 248475672 points to free inode
287203329
would junk entry "fchar"
entry "fchar" in shortform directory inode 129571887 points to free inode
155289634
would junk entry "fchar"
entry "fpod" in shortform directory inode 84868120 points to free inode
155289633
would junk entry "fpod"
entry "fchar" in shortform directory inode 129571885 points to free inode
155289632
would junk entry "fchar"
entry "finf" in directory inode 193483808 points to free inode 349614101, would
junk entry
entry "fimkr6_1" in directory inode 51345453 points to free inode 287203337,
would junk entry
entry "fres" in shortform directory inode 193486858 points to free inode
349614100
would junk entry "fres"
[Whoops... missed a bit... history is full of "would have resent inode X nlinks
from Y to Z"
messages. It ends with:]
would have reset inode 515105802 nlinks from 46 to 45
would have reset inode 515270693 nlinks from 6 to 5
would have reset inode 515270694 nlinks from 6 to 5
would have reset inode 515270695 nlinks from 6 to 5
would have reset inode 515270696 nlinks from 6 to 5
would have reset inode 515270697 nlinks from 4 to 3
would have reset inode 515270710 nlinks from 4 to 3
would have reset inode 515272704 nlinks from 4 to 3
would have reset inode 515272705 nlinks from 4 to 3
would have reset inode 523361301 nlinks from 17 to 16
would have reset inode 523361304 nlinks from 15 to 12
No modify flag set, skipping filesystem flush and exiting.
|