Some background. I'm running a distributed Web crawler on 16
machines, each with 2 TB of disk space. The machines were all
formated with XFS. This is the second machine that I've had
to roll back to ext3 due to filesystem corruption.
Here's my attempt to characterize the latest problem:
Hardware:
Tyan S2882G3NR Thunder K8S Pro Motherboard
Dual AMD Opteron processors
3ware 9500S-8 RAID Controller
8 Western Digital 2500SD 250GB disk drives
The drives are all tied together into a single RAID 5 unit.
The array is partitioned as follows:
100 MB /boot partition (ext3)
16 GB swap partition
the rest is the root (/) partition formatted XFS
OS:
Fedora Core 3 - 2.6.9 SMP kernel.
The crawler application, which is built on top of MySQL,
generates the following type of load (iostat):
avg-cpu: %user %nice %sys %iowait %idle
7.25 0.00 1.50 48.75 42.50
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 19.40 224.88 32.34 4195.02 2824.88 2097.51 1412.44
27.29 8.74 41.56 3.87 99.50
sda 0.00 0.00 59.00 89.00 1424.00 4354.50 712.00 2177.25 39.04
6.41 41.63 6.76 100.05
sda 0.00 33.83 108.96 62.69 2288.56 3866.17 1144.28 1933.08
35.86 5.16 31.30 5.80 99.50
I came in one morning about 4 weeks ago and noticed that
the filesystem on one machine (wds17) had shut down. Rebooting
seemed to fix the problem. A few weeks later, I tried greping
through /var/log/messages for XFS error messages and here's what
I saw:
/var/log# grep -i xfs messages*
messages:Jan 26 04:02:08 wds17 kernel: XFS internal error
XFS_WANT_CORRUPTED_GOTO at line 4340 of file fs/xfs/xfs_bmap.c. Caller
0xffffffff8020a125
messages:Jan 26 04:02:08 wds17 kernel: Call
Trace:<ffffffff801e73c3>{xfs_bmap_read_extents+899}
<ffffffff8020a125>{xfs_iread_extents+197}
messages:Jan 26 04:02:08 wds17 kernel:
<ffffffff801f6220>{xfs_dir2_put_dirent64_direct+0}
messages:Jan 26 04:02:08 wds17 kernel:
<ffffffff801e6f6f>{xfs_bmap_last_offset+159}
<ffffffff801f5a0f>{xfs_dir2_isblock+31}
messages:Jan 26 04:02:08 wds17 kernel:
<ffffffff801f5d80>{xfs_dir2_getdents+192} <ffffffff80225e26>{xfs_readdir+102}
[...]
These messages were being generated once daily at 4:02am which
is when the daily cron job gets run. I then tried listing the
/var/log directory and here's what I got:
/var/log# ls -l
total 5112
?--------- ? ? ? ? ? acp
-rw-r----- 1 root root 80 Jan 31 17:15 acpid
?--------- ? ? ? ? ? ama
?--------- ? ? ? ? ? ana
?--------- ? ? ? ? ? anaconda.log
?--------- ? ? ? ? ? boo
?--------- ? ? ? ? ? boo
-rw------- 1 root root 3566 Feb 2 19:05 boot.log
-rw------- 1 root root 105 Jan 23 04:02 boot.log.1
-rw------- 1 root root 8265 Jan 16 04:02 boot.log.2
drwxr-xr-x 2 canna canna 6 Sep 27 23:28 canna
?--------- ? ? ? ? ? cron
-rw------- 1 root root 397138 Jan 23 04:02 cron.1
-rw------- 1 root root 382145 Jan 16 04:02 cron.2
-rw------- 1 root root 396661 Jan 9 04:02 cron.3
-rw------- 1 root root 397080 Jan 2 04:04 cron.4
drwxr-xr-x 2 lp sys 4096 Jan 23 04:02 cups
-rw-r--r-- 1 root root 12936 Jan 31 17:14 dmesg
drwxr-x--- 2 exim exim 6 Oct 7 03:00 exim
drwxr-xr-x 2 root root 6 Jun 15 2004 fax
drwxr-xr-x 2 root root 79 Dec 20 16:19 gdm
?--------- ? ? ? ? ? httpd
?--------- ? ? ? ? ? iii
drwx------ 2 root root 24 Dec 16 13:29 iptraf
?--------- ? ? ? ? ? las
?--------- ? ? ? ? ? mai
?--------- ? ? ? ? ? mai
?--------- ? ? ? ? ? mai
?--------- ? ? ? ? ? mai
drwxr-xr-x 2 root root 23 Dec 16 12:31 mail
-rw------- 1 root root 13318 Feb 7 04:07 maillog
-rw------- 1 root root 203188 Jan 9 04:02 maillog.3
drwxrwsr-x 2 root mailman 6 Oct 18 10:54 mailman
?--------- ? ? ? ? ? mes
-rw------- 1 root root 1312545 Feb 7 10:21 messages
-rw------- 1 root root 532217 Jan 23 04:02 messages.1
?--------- ? ? ? ? ? messages.2
-rw------- 1 root root 535803 Jan 2 04:04 messages.4
-rw-r----- 1 mysql mysql 0 Jan 23 04:02 mysqld.log
?--------- ? ? ? ? ? mysqld.log.
?--------- ? ? ? ? ? mysqld.log.
?--------- ? ? ? ? ? mysqld.log.
?--------- ? ? ? ? ? mysqld.log.
drwxr-xr-x 3 news news 65 Dec 16 13:29 news
?--------- ? ? ? ? ? pgsql
drwx------ 2 root root 6 Nov 2 07:04 ppp
-rw-r--r-- 1 root root 74215 Feb 7 04:03 prelink.log
drwxr-xr-x 2 privoxy privoxy 6 Sep 9 15:19 privoxy
?--------- ? ? ? ? ? quagga
drwx------ 3 radiusd radiusd 4096 Jan 1 04:04 radius
drwxr-xr-x 2 root root 6 Jun 15 2004 routed
-rw-r--r-- 1 root root 57666 Feb 7 04:04 rpmpkgs
-rw-r--r-- 1 root root 57666 Jan 22 04:10 rpmpkgs.1
-rw-r--r-- 1 root root 57666 Jan 15 04:21 rpmpkgs.2
-rw-r--r-- 1 root root 57586 Jan 8 04:05 rpmpkgs.3
-rw-r--r-- 1 root root 57586 Jan 1 04:08 rpmpkgs.4
drwxr-xr-x 2 root root 4096 Feb 7 00:00 sa
drwx------ 2 root root 6 Oct 19 07:53 samba
?--------- ? ? ? ? ? scr
-rw------- 1 root root 114845 Feb 7 10:21 secure
?--------- ? ? ? ? ? secure.1
-rw------- 1 root root 9806 Jan 14 16:35 secure.2
-rw------- 1 root root 23535 Jan 9 00:11 secure.3
-rw------- 1 root root 14844 Jan 1 00:02 secure.4
?--------- ? ? ? ? ? spo
?--------- ? ? ? ? ? spo
-rw------- 1 root root 0 Jan 23 04:02 spooler
-rw------- 1 root root 0 Jan 9 04:02 spooler.2
?--------- ? ? ? ? ? spooler.4
drwxr-x--- 2 squid squid 6 Oct 18 14:19 squid
-rw-r--r-- 1 root root 149 Jan 25 23:25 tdm_aen_0.bin
-rw-rw-r-- 1 root root 0 Jan 23 04:02 up2date
-rw-rw-r-- 1 root root 0 Jan 16 04:02 up2date.1
-rw-rw-r-- 1 root root 0 Jan 12 04:02 up2date.2
-rw-rw-r-- 1 root root 69 Jan 11 12:32 up2date.3
drwxr-xr-x 2 uucp uucp 40 Dec 16 13:30 uucp
drwxr-xr-x 2 root root 6 Oct 5 07:53 vbox
-rw-rw-r-- 1 root utmp 86784 Feb 7 10:20 wtmp
?--------- ? ? ? ? ? wtmp.1
-rw-r--r-- 1 root root 43112 Jan 11 11:12 Xorg.0.log
?--------- ? ? ? ? ? Xorg.0.log.
One thing to keep in mind is that I have not noticed
any corruption in my database, which comprises a fixed
200 GB InnoDB tablespace file and a few other fixed
MyISAM table files (e.g. these files do not move or grow
they just get updated in place).
This morning, I shut the machine down and it hung while
it was trying to unmount the filesystems. There was no disk
activity so I did a hard reset. I then booted with a recovery
CD and did an xfs_check and xfs_repair -n:
# xfs_check /dev/sda3
dir 3758096515 entry LC_MESSAGES bad offset 0
dir 3758096516 entry LC_MESSAGES bad offset 0
dir 3758096516 entry charset bad offset 0
dir 3758096517 entry LC_MESSAGES bad offset 0
dir 3758096518 entry LC_MESSAGES bad offset 0
dir 3758096519 entry LC_MESSAGES bad offset 0
dir 3758096520 entry entry.desktop bad offset 0
dir 3758096521 entry LC_MESSAGES bad offset 0
dir 3758096524 entry README.bonding bad offset 0
dir 3758096527 block 0 extra leaf entry d3a74ed 2f
dir 3758096527 block 0 extra leaf entry 1c7c34e2 65
dir 3758096527 block 0 extra leaf entry 3c2df83b b1
dir 3758096527 block 0 extra leaf entry 3c2df83c 14f
dir 3758096527 block 0 extra leaf entry 3c2df83d 152
dir 3758096527 block 0 extra leaf entry 3c2df83e 10b
dir 3758096527 block 0 extra leaf entry 3e9a84e8 a
dir 3758096527 block 0 extra leaf entry 56ed8872 29
dir 3758096527 block 0 extra leaf entry 83781ccf ca
dir 3758096527 block 0 extra leaf entry 8ca5f608 5a
dir 3758096527 block 0 extra leaf entry 995b180b 6d
dir 3758096527 block 0 extra leaf entry 9d9a80e9 129
dir 3758096527 block 0 extra leaf entry a03a7052 76
dir 3758096527 block 0 extra leaf entry a03a7054 126
dir 3758096527 block 0 extra leaf entry a03a7057 9a
dir 3758096527 block 0 extra leaf entry a2260673 179
dir 3758096527 block 0 extra leaf entry a2260674 17c
dir 3758096527 block 0 extra leaf entry bf5d284d 16a
dir 3758096527 block 0 extra leaf entry bf5d284f 170
dir 3758096527 block 0 extra leaf entry dc3bb16f 3c
dir ino 3758096527 missing leaf entry for 8363dccf/ca
dir ino 3758096527 missing leaf entry for 1c7c0062/65
dir ino 3758096527 missing leaf entry for 501d8872/29
dir ino 3758096527 missing leaf entry for 995ad40b/6d
dir ino 3758096527 missing leaf entry for d3a7480/2f
dir ino 3758096527 missing leaf entry for 3c2df80f/b1
dir ino 3758096527 missing leaf entry for 301a84e8/a
dir ino 3758096527 missing leaf entry for 3c2df80f/10b
dir ino 3758096527 missing leaf entry for 3c2df80f/14f
dir ino 3758096527 missing leaf entry for dc20316f/3c
dir ino 3758096527 missing leaf entry for 3c2df80f/152
dir ino 3758096527 missing leaf entry for 901a80e9/129
dir ino 3758096527 missing leaf entry for a03a7334/126
dir ino 3758096527 missing leaf entry for bf5d2b37/170
dir ino 3758096527 missing leaf entry for 8ca99608/5a
dir ino 3758096527 missing leaf entry for bf5d2b35/16a
dir ino 3758096527 missing leaf entry for a227d674/17c
dir ino 3758096527 missing leaf entry for a227d673/179
dir ino 3758096527 missing leaf entry for a03a7332/76
dir ino 3758096527 missing leaf entry for a03a7337/9a
bad magic # 0x7bfaf5bb in inode 3758096532 bmbt block 0/8389223
expected level 0 got 30859 in inode 3758096532 bmbt block 0/8389223
block 0/8389223 expected type unknown got data
block 0/8389223 claimed by inode 3758096532, previous inum 2953261983
bad btree nrecs (24735, min=127, max=254) in inode 3758096532 bmap block 8389223
extent count for ino 3758096532 data fork too low (0) for file format
bad nblocks 87 for inode 3758096532, counted 2
bad nextents 66 for inode 3758096532, counted 0
no . entry for directory 3758096532
no .. entry for directory 3758096532
dir 3758096534 entry Center bad offset 0
block 14/612 type unknown not expected
block 14/721 type unknown not expected
block 14/841 type unknown not expected
block 14/956 type unknown not expected
block 14/2416 type unknown not expected
block 14/2561 type unknown not expected
block 14/2726 type unknown not expected
block 14/2880 type unknown not expected
block 14/3047 type unknown not expected
block 14/3210 type unknown not expected
block 14/3375 type unknown not expected
block 14/3540 type unknown not expected
block 14/4267 type unknown not expected
block 14/5147 type unknown not expected
block 14/10899 type unknown not expected
block 14/11252 type unknown not expected
block 14/11375 type unknown not expected
block 14/11518 type unknown not expected
block 14/11660 type unknown not expected
block 14/11799 type unknown not expected
block 14/11937 type unknown not expected
block 14/12522 type unknown not expected
block 14/12626 type unknown not expected
block 14/12751 type unknown not expected
block 14/12860 type unknown not expected
block 14/12972 type unknown not expected
block 14/13062 type unknown not expected
block 14/13161 type unknown not expected
block 14/13269 type unknown not expected
block 14/13388 type unknown not expected
block 14/13504 type unknown not expected
block 14/13631 type unknown not expected
block 14/13796 type unknown not expected
block 14/13964 type unknown not expected
block 14/14111 type unknown not expected
block 14/15983 type unknown not expected
block 14/16112 type unknown not expected
block 14/17304 type unknown not expected
block 14/17578 type unknown not expected
block 14/17700 type unknown not expected
block 14/17819 type unknown not expected
block 14/17942 type unknown not expected
block 14/18071 type unknown not expected
block 14/18192 type unknown not expected
block 14/18326 type unknown not expected
block 14/19971 type unknown not expected
block 14/20068 type unknown not expected
block 14/20186 type unknown not expected
block 14/20315 type unknown not expected
block 14/23700 type unknown not expected
block 14/24648 type unknown not expected
block 14/24977 type unknown not expected
block 14/25001 type unknown not expected
block 14/25023 type unknown not expected
block 14/25040 type unknown not expected
block 14/25056 type unknown not expected
block 14/25185 type unknown not expected
block 14/25227 type unknown not expected
block 14/25257 type unknown not expected
block 14/25635 type unknown not expected
block 14/26936 type unknown not expected
block 14/27218 type unknown not expected
block 14/8389220 type unknown not expected
block 14/8389221 type unknown not expected
block 14/8389222 type unknown not expected
block 14/8389223 type unknown not expected
block 14/8389224 type unknown not expected
block 14/8389225 type unknown not expected
block 14/8389226 type unknown not expected
block 14/8389227 type unknown not expected
block 14/8389228 type unknown not expected
block 14/8389229 type unknown not expected
block 14/8389230 type unknown not expected
block 14/8389231 type unknown not expected
block 14/8389232 type unknown not expected
block 14/8389233 type unknown not expected
block 14/8389234 type unknown not expected
block 14/8389235 type unknown not expected
block 14/8389236 type unknown not expected
block 14/8389237 type unknown not expected
block 14/8389238 type unknown not expected
block 14/8389239 type unknown not expected
block 14/8389240 type unknown not expected
block 14/8389241 type unknown not expected
block 14/8389242 type unknown not expected
block 14/8389243 type unknown not expected
link count mismatch for inode 353681 (name ?), nlink 0, counted 1
link count mismatch for inode 128 (name ?), nlink 26, counted 27
link count mismatch for inode 437972 (name ?), nlink 0, counted 1
link count mismatch for inode 49624 (name ?), nlink 0, counted 1
link count mismatch for inode 18052 (name ?), nlink 0, counted 1
link count mismatch for inode 438996 (name ?), nlink 0, counted 1
link count mismatch for inode 448011 (name ?), nlink 0, counted 1
link count mismatch for inode 448015 (name ?), nlink 0, counted 1
link count mismatch for inode 262530 (name ?), nlink 0, counted 1
link count mismatch for inode 16990 (name ?), nlink 0, counted 1
link count mismatch for inode 368079 (name ?), nlink 0, counted 1
link count mismatch for inode 805306496 (name ?), nlink 25, counted 24
link count mismatch for inode 1073741960 (name ?), nlink 40, counted 39
link count mismatch for inode 1610662360 (name ?), nlink 2, counted 1
link count mismatch for inode 1879310722 (name ?), nlink 2, counted 1
disconnected inode 3758386284, nlink 1
disconnected inode 3758316708, nlink 1
disconnected inode 3758148566, nlink 1
disconnected inode 3758386285, nlink 1
disconnected inode 3758316709, nlink 1
disconnected inode 3758148567, nlink 1
disconnected inode 3758386286, nlink 1
disconnected inode 3758316710, nlink 1
disconnected inode 3758148568, nlink 1
disconnected inode 3758450065, nlink 1
disconnected inode 3758386287, nlink 1
disconnected inode 3758316711, nlink 1
disconnected inode 3758148569, nlink 1
disconnected inode 3758386288, nlink 1
disconnected inode 3758316712, nlink 1
disconnected inode 3758148570, nlink 1
disconnected inode 3758386289, nlink 1
disconnected inode 3758316713, nlink 1
disconnected inode 3758148571, nlink 1
disconnected inode 3758386290, nlink 1
[ ... 6325 of these disconnected lines total ...]
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- scan filesystem freespace and inode maps...
- found root inode chunk
Phase 3 - for each AG...
- scan (but don't clear) agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
entry contains offset out of order in shortform dir 3758096515
would have corrected entry offsets in directory 3758096515
entry contains offset out of order in shortform dir 3758096516
entry contains offset out of order in shortform dir 3758096516
would have corrected entry offsets in directory 3758096516
entry contains offset out of order in shortform dir 3758096517
would have corrected entry offsets in directory 3758096517
entry contains offset out of order in shortform dir 3758096518
would have corrected entry offsets in directory 3758096518
entry contains offset out of order in shortform dir 3758096519
would have corrected entry offsets in directory 3758096519
entry contains offset out of order in shortform dir 3758096520
would have corrected entry offsets in directory 3758096520
entry contains offset out of order in shortform dir 3758096521
would have corrected entry offsets in directory 3758096521
entry contains illegal character in shortform dir 3758096523
would have junked entry "BuiltInSes" in directory inode 3758096523
entry contains offset out of order in shortform dir 3758096524
would have corrected entry offsets in directory 3758096524
entry at block 0 offset 80 in directory inode 3758096527 has illegal name "
las": would clear entry
entry at block 0 offset 328 in directory inode 3758096527 has illegal name "
scr": would clear entry
entry at block 0 offset 376 in directory inode 3758096527 has illegal name "
iii": would clear entry
entry at block 0 offset 480 in directory inode 3758096527 has illegal name "
ama": would clear entry
entry at block 0 offset 720 in directory inode 3758096527 has illegal name "
ana": would clear entry
entry at block 0 offset 808 in directory inode 3758096527 has illegal name "
acp": would clear entry
entry at block 0 offset 872 in directory inode 3758096527 has illegal name "
mes": would clear entry
entry at block 0 offset 944 in directory inode 3758096527 has illegal name "
mai": would clear entry
entry at block 0 offset 1232 in directory inode 3758096527 has illegal name "
mai": would clear entry
entry at block 0 offset 1416 in directory inode 3758096527 has illegal name "
mysqld.log.": would clear entry
entry at block 0 offset 1616 in directory inode 3758096527 has illegal name "
Xorg.0.log.": would clear entry
entry at block 0 offset 2136 in directory inode 3758096527 has illegal name "
mysqld.log.": would clear entry
entry at block 0 offset 2352 in directory inode 3758096527 has illegal name "
mai": would clear entry
entry at block 0 offset 2376 in directory inode 3758096527 has illegal name "
mai": would clear entry
entry at block 0 offset 2680 in directory inode 3758096527 has illegal name "
mysqld.log.": would clear entry
entry at block 0 offset 2704 in directory inode 3758096527 has illegal name "
mysqld.log.": would clear entry
entry at block 0 offset 2896 in directory inode 3758096527 has illegal name "
spo": would clear entry
entry at block 0 offset 2944 in directory inode 3758096527 has illegal name "
spo": would clear entry
entry at block 0 offset 3016 in directory inode 3758096527 has illegal name "
boo": would clear entry
entry at block 0 offset 3040 in directory inode 3758096527 has illegal name "
boo": would clear entry
bad magic # 0x7bfaf5bb in inode 3758096532 (data fork) bmbt block 8389223
bad data fork in inode 3758096532
would have cleared inode 3758096532
entry contains offset out of order in shortform dir 3758096534
would have corrected entry offsets in directory 3758096534
- agno = 15
- agno = 16
- agno = 17
- agno = 18
- agno = 19
- agno = 20
- agno = 21
- agno = 22
- agno = 23
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- agno = 30
- agno = 31
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
entry "man3" at block 0 offset 96 in directory inode 1073741960 references free
inode 3758096532
would clear inode number in entry at offset 96...
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
entry contains offset out of order in shortform dir 3758096515
would have corrected entry offsets in directory 3758096515
entry contains offset out of order in shortform dir 3758096516
entry contains offset out of order in shortform dir 3758096516
would have corrected entry offsets in directory 3758096516
entry contains offset out of order in shortform dir 3758096517
entry "charset" in shortform directory 3758096517 references non-existent inode
368079
would have junked entry "charset" in directory inode 3758096517
would have corrected entry offsets in directory 3758096517
entry contains offset out of order in shortform dir 3758096518
would have corrected entry offsets in directory 3758096518
entry contains offset out of order in shortform dir 3758096519
would have corrected entry offsets in directory 3758096519
entry contains offset out of order in shortform dir 3758096520
would have corrected entry offsets in directory 3758096520
entry contains offset out of order in shortform dir 3758096521
would have corrected entry offsets in directory 3758096521
entry contains illegal character in shortform dir 3758096523
would have junked entry "BuiltInSes" in directory inode 3758096523
entry contains offset out of order in shortform dir 3758096524
would have corrected entry offsets in directory 3758096524
entry "httpd" at block 0 offset 280 in directory inode 3758096527 references
non-existent inode 49624
would clear inode number in entry at offset 280...
entry "quagga" at block 0 offset 408 in directory inode 3758096527 references
non-existent inode 262530
would clear inode number in entry at offset 408...
entry "pgsql" at block 0 offset 544 in directory inode 3758096527 references
non-existent inode 353681
would clear inode number in entry at offset 544...
entry "anaconda.log" at block 0 offset 696 in directory inode 3758096527
references non-existent inode 437972
would clear inode number in entry at offset 696...
entry "wtmp.1" at block 0 offset 1736 in directory inode 3758096527 references
non-existent inode 18052
would clear inode number in entry at offset 1736...
entry "messages.2" at block 0 offset 1824 in directory inode 3758096527
references non-existent inode 448011
would clear inode number in entry at offset 1824...
entry "cron" at block 0 offset 2496 in directory inode 3758096527 references
non-existent inode 438996
would clear inode number in entry at offset 2496...
entry "secure.1" at block 0 offset 2824 in directory inode 3758096527
references non-existent inode 16990
would clear inode number in entry at offset 2824...
entry "spooler.4" at block 0 offset 2872 in directory inode 3758096527
references non-existent inode 448015
would clear inode number in entry at offset 2872...
bad magic # 0x7bfaf5bb in inode 3758096532 (data fork) bmbt block 8389223
bad data fork in inode 3758096532
would have cleared inode 3758096532
entry contains offset out of order in shortform dir 3758096534
would have corrected entry offsets in directory 3758096534
- agno = 15
- agno = 16
- agno = 17
- agno = 18
- agno = 19
- agno = 20
- agno = 21
- agno = 22
- agno = 23
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- agno = 30
- agno = 31
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
- traversing filesystem starting at / ...
entry "man3" in directory inode 1073741960 points to free inode 3758096532,
would junk entry
entry "charset" in shortform directory 3758096517 references non-existent inode
368079
would junk entry
entry "charset" in shortform directory 3758096517 references non-existent inode
368079
would junk entry
entry "log" in dir inode 805306496 inconsistent with .. value (128) in ino
3758096527,
would clear entry "log"
- traversal finished ...
- traversing all unattached subtrees ...
entry "httpd" in directory inode 3758096527 points to non-existent inode, would
junk entry
entry "quagga" in directory inode 3758096527 points to non-existent inode,
would junk entry
entry "pgsql" in directory inode 3758096527 points to non-existent inode, would
junk entry
entry "anaconda.log" in directory inode 3758096527 points to non-existent
inode, would junk entry
entry "wtmp.1" in directory inode 3758096527 points to non-existent inode,
would junk entry
entry "messages.2" in directory inode 3758096527 points to non-existent inode,
would junk entry
entry "cron" in directory inode 3758096527 points to non-existent inode, would
junk entry
entry "secure.1" in directory inode 3758096527 points to non-existent inode,
would junk entry
entry "spooler.4" in directory inode 3758096527 points to non-existent inode,
would junk entry
bad hash table for directory inode 3758096527 (hash value mismatch): would
rebuild
- traversals finished ...
- moving disconnected inodes to lost+found ...
disconnected dir inode 1610662360, would move to lost+found
disconnected dir inode 1879310722, would move to lost+found
disconnected dir inode 3758096527, would move to lost+found
disconnected inode 3758104518, would move to lost+found
disconnected inode 3758105657, would move to lost+found
disconnected inode 3758105658, would move to lost+found
disconnected inode 3758105659, would move to lost+found
disconnected inode 3758105660, would move to lost+found
disconnected inode 3758105661, would move to lost+found
disconnected inode 3758105662, would move to lost+found
disconnected inode 3758105663, would move to lost+found
disconnected inode 3758105664, would move to lost+found
disconnected inode 3758105665, would move to lost+found
disconnected inode 3758105666, would move to lost+found
disconnected inode 3758105667, would move to lost+found
disconnected inode 3758105668, would move to lost+found
disconnected inode 3758105669, would move to lost+found
disconnected inode 3758105670, would move to lost+found
disconnected inode 3758105671, would move to lost+found
disconnected inode 3758105672, would move to lost+found
[...]
|