Looks like we're having similar problems with EXT-3. Also, it happened
again on the same two machines. We're starting to suspect hardware.
- Doug
On Tue, 2005-02-08 at 14:39, Douglass Judd wrote:
> Some background. I'm running a distributed Web crawler on 16
> machines, each with 2 TB of disk space. The machines were all
> formated with XFS. This is the second machine that I've had
> to roll back to ext3 due to filesystem corruption.
>
> Here's my attempt to characterize the latest problem:
>
> Hardware:
> Tyan S2882G3NR Thunder K8S Pro Motherboard
> Dual AMD Opteron processors
> 3ware 9500S-8 RAID Controller
> 8 Western Digital 2500SD 250GB disk drives
>
> The drives are all tied together into a single RAID 5 unit.
>
> The array is partitioned as follows:
> 100 MB /boot partition (ext3)
> 16 GB swap partition
> the rest is the root (/) partition formatted XFS
>
> OS:
> Fedora Core 3 - 2.6.9 SMP kernel.
>
> The crawler application, which is built on top of MySQL,
> generates the following type of load (iostat):
>
> avg-cpu: %user %nice %sys %iowait %idle
> 7.25 0.00 1.50 48.75 42.50
>
>
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
> avgrq-sz avgqu-sz await svctm %util
> sda 0.00 19.40 224.88 32.34 4195.02 2824.88 2097.51 1412.44
> 27.29 8.74 41.56 3.87 99.50
> sda 0.00 0.00 59.00 89.00 1424.00 4354.50 712.00 2177.25
> 39.04 6.41 41.63 6.76 100.05
> sda 0.00 33.83 108.96 62.69 2288.56 3866.17 1144.28 1933.08
> 35.86 5.16 31.30 5.80 99.50
>
> I came in one morning about 4 weeks ago and noticed that
> the filesystem on one machine (wds17) had shut down. Rebooting
> seemed to fix the problem. A few weeks later, I tried greping
> through /var/log/messages for XFS error messages and here's what
> I saw:
>
> /var/log# grep -i xfs messages*
>
> messages:Jan 26 04:02:08 wds17 kernel: XFS internal error
> XFS_WANT_CORRUPTED_GOTO at line 4340 of file fs/xfs/xfs_bmap.c. Caller
> 0xffffffff8020a125
> messages:Jan 26 04:02:08 wds17 kernel: Call
> Trace:<ffffffff801e73c3>{xfs_bmap_read_extents+899}
> <ffffffff8020a125>{xfs_iread_extents+197}
> messages:Jan 26 04:02:08 wds17 kernel:
> <ffffffff801f6220>{xfs_dir2_put_dirent64_direct+0}
> messages:Jan 26 04:02:08 wds17 kernel:
> <ffffffff801e6f6f>{xfs_bmap_last_offset+159}
> <ffffffff801f5a0f>{xfs_dir2_isblock+31}
> messages:Jan 26 04:02:08 wds17 kernel:
> <ffffffff801f5d80>{xfs_dir2_getdents+192} <ffffffff80225e26>{xfs_readdir+102}
> [...]
>
> These messages were being generated once daily at 4:02am which
> is when the daily cron job gets run. I then tried listing the
> /var/log directory and here's what I got:
>
> /var/log# ls -l
> total 5112
> ?--------- ? ? ? ? ? acp
> -rw-r----- 1 root root 80 Jan 31 17:15 acpid
> ?--------- ? ? ? ? ? ama
> ?--------- ? ? ? ? ? ana
> ?--------- ? ? ? ? ? anaconda.log
> ?--------- ? ? ? ? ? boo
> ?--------- ? ? ? ? ? boo
> -rw------- 1 root root 3566 Feb 2 19:05 boot.log
> -rw------- 1 root root 105 Jan 23 04:02 boot.log.1
> -rw------- 1 root root 8265 Jan 16 04:02 boot.log.2
> drwxr-xr-x 2 canna canna 6 Sep 27 23:28 canna
> ?--------- ? ? ? ? ? cron
> -rw------- 1 root root 397138 Jan 23 04:02 cron.1
> -rw------- 1 root root 382145 Jan 16 04:02 cron.2
> -rw------- 1 root root 396661 Jan 9 04:02 cron.3
> -rw------- 1 root root 397080 Jan 2 04:04 cron.4
> drwxr-xr-x 2 lp sys 4096 Jan 23 04:02 cups
> -rw-r--r-- 1 root root 12936 Jan 31 17:14 dmesg
> drwxr-x--- 2 exim exim 6 Oct 7 03:00 exim
> drwxr-xr-x 2 root root 6 Jun 15 2004 fax
> drwxr-xr-x 2 root root 79 Dec 20 16:19 gdm
> ?--------- ? ? ? ? ? httpd
> ?--------- ? ? ? ? ? iii
> drwx------ 2 root root 24 Dec 16 13:29 iptraf
> ?--------- ? ? ? ? ? las
> ?--------- ? ? ? ? ? mai
> ?--------- ? ? ? ? ? mai
> ?--------- ? ? ? ? ? mai
> ?--------- ? ? ? ? ? mai
> drwxr-xr-x 2 root root 23 Dec 16 12:31 mail
> -rw------- 1 root root 13318 Feb 7 04:07 maillog
> -rw------- 1 root root 203188 Jan 9 04:02 maillog.3
> drwxrwsr-x 2 root mailman 6 Oct 18 10:54 mailman
> ?--------- ? ? ? ? ? mes
> -rw------- 1 root root 1312545 Feb 7 10:21 messages
> -rw------- 1 root root 532217 Jan 23 04:02 messages.1
> ?--------- ? ? ? ? ? messages.2
> -rw------- 1 root root 535803 Jan 2 04:04 messages.4
> -rw-r----- 1 mysql mysql 0 Jan 23 04:02 mysqld.log
> ?--------- ? ? ? ? ? mysqld.log.
> ?--------- ? ? ? ? ? mysqld.log.
> ?--------- ? ? ? ? ? mysqld.log.
> ?--------- ? ? ? ? ? mysqld.log.
> drwxr-xr-x 3 news news 65 Dec 16 13:29 news
> ?--------- ? ? ? ? ? pgsql
> drwx------ 2 root root 6 Nov 2 07:04 ppp
> -rw-r--r-- 1 root root 74215 Feb 7 04:03 prelink.log
> drwxr-xr-x 2 privoxy privoxy 6 Sep 9 15:19 privoxy
> ?--------- ? ? ? ? ? quagga
> drwx------ 3 radiusd radiusd 4096 Jan 1 04:04 radius
> drwxr-xr-x 2 root root 6 Jun 15 2004 routed
> -rw-r--r-- 1 root root 57666 Feb 7 04:04 rpmpkgs
> -rw-r--r-- 1 root root 57666 Jan 22 04:10 rpmpkgs.1
> -rw-r--r-- 1 root root 57666 Jan 15 04:21 rpmpkgs.2
> -rw-r--r-- 1 root root 57586 Jan 8 04:05 rpmpkgs.3
> -rw-r--r-- 1 root root 57586 Jan 1 04:08 rpmpkgs.4
> drwxr-xr-x 2 root root 4096 Feb 7 00:00 sa
> drwx------ 2 root root 6 Oct 19 07:53 samba
> ?--------- ? ? ? ? ? scr
> -rw------- 1 root root 114845 Feb 7 10:21 secure
> ?--------- ? ? ? ? ? secure.1
> -rw------- 1 root root 9806 Jan 14 16:35 secure.2
> -rw------- 1 root root 23535 Jan 9 00:11 secure.3
> -rw------- 1 root root 14844 Jan 1 00:02 secure.4
> ?--------- ? ? ? ? ? spo
> ?--------- ? ? ? ? ? spo
> -rw------- 1 root root 0 Jan 23 04:02 spooler
> -rw------- 1 root root 0 Jan 9 04:02 spooler.2
> ?--------- ? ? ? ? ? spooler.4
> drwxr-x--- 2 squid squid 6 Oct 18 14:19 squid
> -rw-r--r-- 1 root root 149 Jan 25 23:25 tdm_aen_0.bin
> -rw-rw-r-- 1 root root 0 Jan 23 04:02 up2date
> -rw-rw-r-- 1 root root 0 Jan 16 04:02 up2date.1
> -rw-rw-r-- 1 root root 0 Jan 12 04:02 up2date.2
> -rw-rw-r-- 1 root root 69 Jan 11 12:32 up2date.3
> drwxr-xr-x 2 uucp uucp 40 Dec 16 13:30 uucp
> drwxr-xr-x 2 root root 6 Oct 5 07:53 vbox
> -rw-rw-r-- 1 root utmp 86784 Feb 7 10:20 wtmp
> ?--------- ? ? ? ? ? wtmp.1
> -rw-r--r-- 1 root root 43112 Jan 11 11:12 Xorg.0.log
> ?--------- ? ? ? ? ? Xorg.0.log.
>
> One thing to keep in mind is that I have not noticed
> any corruption in my database, which comprises a fixed
> 200 GB InnoDB tablespace file and a few other fixed
> MyISAM table files (e.g. these files do not move or grow
> they just get updated in place).
>
> This morning, I shut the machine down and it hung while
> it was trying to unmount the filesystems. There was no disk
> activity so I did a hard reset. I then booted with a recovery
> CD and did an xfs_check and xfs_repair -n:
>
> # xfs_check /dev/sda3
> dir 3758096515 entry LC_MESSAGES bad offset 0
> dir 3758096516 entry LC_MESSAGES bad offset 0
> dir 3758096516 entry charset bad offset 0
> dir 3758096517 entry LC_MESSAGES bad offset 0
> dir 3758096518 entry LC_MESSAGES bad offset 0
> dir 3758096519 entry LC_MESSAGES bad offset 0
> dir 3758096520 entry entry.desktop bad offset 0
> dir 3758096521 entry LC_MESSAGES bad offset 0
> dir 3758096524 entry README.bonding bad offset 0
> dir 3758096527 block 0 extra leaf entry d3a74ed 2f
> dir 3758096527 block 0 extra leaf entry 1c7c34e2 65
> dir 3758096527 block 0 extra leaf entry 3c2df83b b1
> dir 3758096527 block 0 extra leaf entry 3c2df83c 14f
> dir 3758096527 block 0 extra leaf entry 3c2df83d 152
> dir 3758096527 block 0 extra leaf entry 3c2df83e 10b
> dir 3758096527 block 0 extra leaf entry 3e9a84e8 a
> dir 3758096527 block 0 extra leaf entry 56ed8872 29
> dir 3758096527 block 0 extra leaf entry 83781ccf ca
> dir 3758096527 block 0 extra leaf entry 8ca5f608 5a
> dir 3758096527 block 0 extra leaf entry 995b180b 6d
> dir 3758096527 block 0 extra leaf entry 9d9a80e9 129
> dir 3758096527 block 0 extra leaf entry a03a7052 76
> dir 3758096527 block 0 extra leaf entry a03a7054 126
> dir 3758096527 block 0 extra leaf entry a03a7057 9a
> dir 3758096527 block 0 extra leaf entry a2260673 179
> dir 3758096527 block 0 extra leaf entry a2260674 17c
> dir 3758096527 block 0 extra leaf entry bf5d284d 16a
> dir 3758096527 block 0 extra leaf entry bf5d284f 170
> dir 3758096527 block 0 extra leaf entry dc3bb16f 3c
> dir ino 3758096527 missing leaf entry for 8363dccf/ca
> dir ino 3758096527 missing leaf entry for 1c7c0062/65
> dir ino 3758096527 missing leaf entry for 501d8872/29
> dir ino 3758096527 missing leaf entry for 995ad40b/6d
> dir ino 3758096527 missing leaf entry for d3a7480/2f
> dir ino 3758096527 missing leaf entry for 3c2df80f/b1
> dir ino 3758096527 missing leaf entry for 301a84e8/a
> dir ino 3758096527 missing leaf entry for 3c2df80f/10b
> dir ino 3758096527 missing leaf entry for 3c2df80f/14f
> dir ino 3758096527 missing leaf entry for dc20316f/3c
> dir ino 3758096527 missing leaf entry for 3c2df80f/152
> dir ino 3758096527 missing leaf entry for 901a80e9/129
> dir ino 3758096527 missing leaf entry for a03a7334/126
> dir ino 3758096527 missing leaf entry for bf5d2b37/170
> dir ino 3758096527 missing leaf entry for 8ca99608/5a
> dir ino 3758096527 missing leaf entry for bf5d2b35/16a
> dir ino 3758096527 missing leaf entry for a227d674/17c
> dir ino 3758096527 missing leaf entry for a227d673/179
> dir ino 3758096527 missing leaf entry for a03a7332/76
> dir ino 3758096527 missing leaf entry for a03a7337/9a
> bad magic # 0x7bfaf5bb in inode 3758096532 bmbt block 0/8389223
> expected level 0 got 30859 in inode 3758096532 bmbt block 0/8389223
> block 0/8389223 expected type unknown got data
> block 0/8389223 claimed by inode 3758096532, previous inum 2953261983
> bad btree nrecs (24735, min=127, max=254) in inode 3758096532 bmap block
> 8389223
> extent count for ino 3758096532 data fork too low (0) for file format
> bad nblocks 87 for inode 3758096532, counted 2
> bad nextents 66 for inode 3758096532, counted 0
> no . entry for directory 3758096532
> no .. entry for directory 3758096532
> dir 3758096534 entry Center bad offset 0
> block 14/612 type unknown not expected
> block 14/721 type unknown not expected
> block 14/841 type unknown not expected
> block 14/956 type unknown not expected
> block 14/2416 type unknown not expected
> block 14/2561 type unknown not expected
> block 14/2726 type unknown not expected
> block 14/2880 type unknown not expected
> block 14/3047 type unknown not expected
> block 14/3210 type unknown not expected
> block 14/3375 type unknown not expected
> block 14/3540 type unknown not expected
> block 14/4267 type unknown not expected
> block 14/5147 type unknown not expected
> block 14/10899 type unknown not expected
> block 14/11252 type unknown not expected
> block 14/11375 type unknown not expected
> block 14/11518 type unknown not expected
> block 14/11660 type unknown not expected
> block 14/11799 type unknown not expected
> block 14/11937 type unknown not expected
> block 14/12522 type unknown not expected
> block 14/12626 type unknown not expected
> block 14/12751 type unknown not expected
> block 14/12860 type unknown not expected
> block 14/12972 type unknown not expected
> block 14/13062 type unknown not expected
> block 14/13161 type unknown not expected
> block 14/13269 type unknown not expected
> block 14/13388 type unknown not expected
> block 14/13504 type unknown not expected
> block 14/13631 type unknown not expected
> block 14/13796 type unknown not expected
> block 14/13964 type unknown not expected
> block 14/14111 type unknown not expected
> block 14/15983 type unknown not expected
> block 14/16112 type unknown not expected
> block 14/17304 type unknown not expected
> block 14/17578 type unknown not expected
> block 14/17700 type unknown not expected
> block 14/17819 type unknown not expected
> block 14/17942 type unknown not expected
> block 14/18071 type unknown not expected
> block 14/18192 type unknown not expected
> block 14/18326 type unknown not expected
> block 14/19971 type unknown not expected
> block 14/20068 type unknown not expected
> block 14/20186 type unknown not expected
> block 14/20315 type unknown not expected
> block 14/23700 type unknown not expected
> block 14/24648 type unknown not expected
> block 14/24977 type unknown not expected
> block 14/25001 type unknown not expected
> block 14/25023 type unknown not expected
> block 14/25040 type unknown not expected
> block 14/25056 type unknown not expected
> block 14/25185 type unknown not expected
> block 14/25227 type unknown not expected
> block 14/25257 type unknown not expected
> block 14/25635 type unknown not expected
> block 14/26936 type unknown not expected
> block 14/27218 type unknown not expected
> block 14/8389220 type unknown not expected
> block 14/8389221 type unknown not expected
> block 14/8389222 type unknown not expected
> block 14/8389223 type unknown not expected
> block 14/8389224 type unknown not expected
> block 14/8389225 type unknown not expected
> block 14/8389226 type unknown not expected
> block 14/8389227 type unknown not expected
> block 14/8389228 type unknown not expected
> block 14/8389229 type unknown not expected
> block 14/8389230 type unknown not expected
> block 14/8389231 type unknown not expected
> block 14/8389232 type unknown not expected
> block 14/8389233 type unknown not expected
> block 14/8389234 type unknown not expected
> block 14/8389235 type unknown not expected
> block 14/8389236 type unknown not expected
> block 14/8389237 type unknown not expected
> block 14/8389238 type unknown not expected
> block 14/8389239 type unknown not expected
> block 14/8389240 type unknown not expected
> block 14/8389241 type unknown not expected
> block 14/8389242 type unknown not expected
> block 14/8389243 type unknown not expected
> link count mismatch for inode 353681 (name ?), nlink 0, counted 1
> link count mismatch for inode 128 (name ?), nlink 26, counted 27
> link count mismatch for inode 437972 (name ?), nlink 0, counted 1
> link count mismatch for inode 49624 (name ?), nlink 0, counted 1
> link count mismatch for inode 18052 (name ?), nlink 0, counted 1
> link count mismatch for inode 438996 (name ?), nlink 0, counted 1
> link count mismatch for inode 448011 (name ?), nlink 0, counted 1
> link count mismatch for inode 448015 (name ?), nlink 0, counted 1
> link count mismatch for inode 262530 (name ?), nlink 0, counted 1
> link count mismatch for inode 16990 (name ?), nlink 0, counted 1
> link count mismatch for inode 368079 (name ?), nlink 0, counted 1
> link count mismatch for inode 805306496 (name ?), nlink 25, counted 24
> link count mismatch for inode 1073741960 (name ?), nlink 40, counted 39
> link count mismatch for inode 1610662360 (name ?), nlink 2, counted 1
> link count mismatch for inode 1879310722 (name ?), nlink 2, counted 1
> disconnected inode 3758386284, nlink 1
> disconnected inode 3758316708, nlink 1
> disconnected inode 3758148566, nlink 1
> disconnected inode 3758386285, nlink 1
> disconnected inode 3758316709, nlink 1
> disconnected inode 3758148567, nlink 1
> disconnected inode 3758386286, nlink 1
> disconnected inode 3758316710, nlink 1
> disconnected inode 3758148568, nlink 1
> disconnected inode 3758450065, nlink 1
> disconnected inode 3758386287, nlink 1
> disconnected inode 3758316711, nlink 1
> disconnected inode 3758148569, nlink 1
> disconnected inode 3758386288, nlink 1
> disconnected inode 3758316712, nlink 1
> disconnected inode 3758148570, nlink 1
> disconnected inode 3758386289, nlink 1
> disconnected inode 3758316713, nlink 1
> disconnected inode 3758148571, nlink 1
> disconnected inode 3758386290, nlink 1
> [ ... 6325 of these disconnected lines total ...]
>
> Phase 1 - find and verify superblock...
> Phase 2 - using internal log
> - scan filesystem freespace and inode maps...
> - found root inode chunk
> Phase 3 - for each AG...
> - scan (but don't clear) agi unlinked lists...
> - process known inodes and perform inode discovery...
> - agno = 0
> - agno = 1
> - agno = 2
> - agno = 3
> - agno = 4
> - agno = 5
> - agno = 6
> - agno = 7
> - agno = 8
> - agno = 9
> - agno = 10
> - agno = 11
> - agno = 12
> - agno = 13
> - agno = 14
> entry contains offset out of order in shortform dir 3758096515
> would have corrected entry offsets in directory 3758096515
> entry contains offset out of order in shortform dir 3758096516
> entry contains offset out of order in shortform dir 3758096516
> would have corrected entry offsets in directory 3758096516
> entry contains offset out of order in shortform dir 3758096517
> would have corrected entry offsets in directory 3758096517
> entry contains offset out of order in shortform dir 3758096518
> would have corrected entry offsets in directory 3758096518
> entry contains offset out of order in shortform dir 3758096519
> would have corrected entry offsets in directory 3758096519
> entry contains offset out of order in shortform dir 3758096520
> would have corrected entry offsets in directory 3758096520
> entry contains offset out of order in shortform dir 3758096521
> would have corrected entry offsets in directory 3758096521
> entry contains illegal character in shortform dir 3758096523
> would have junked entry "BuiltInSes" in directory inode 3758096523
> entry contains offset out of order in shortform dir 3758096524
> would have corrected entry offsets in directory 3758096524
> entry at block 0 offset 80 in directory inode 3758096527 has illegal name "
> las": would clear entry
> entry at block 0 offset 328 in directory inode 3758096527 has illegal name "
> scr": would clear entry
> entry at block 0 offset 376 in directory inode 3758096527 has illegal name "
> iii": would clear entry
> entry at block 0 offset 480 in directory inode 3758096527 has illegal name "
> ama": would clear entry
> entry at block 0 offset 720 in directory inode 3758096527 has illegal name "
> ana": would clear entry
> entry at block 0 offset 808 in directory inode 3758096527 has illegal name "
> acp": would clear entry
> entry at block 0 offset 872 in directory inode 3758096527 has illegal name "
> mes": would clear entry
> entry at block 0 offset 944 in directory inode 3758096527 has illegal name "
> mai": would clear entry
> entry at block 0 offset 1232 in directory inode 3758096527 has illegal name "
> mai": would clear entry
> entry at block 0 offset 1416 in directory inode 3758096527 has illegal name "
> mysqld.log.": would clear entry
> entry at block 0 offset 1616 in directory inode 3758096527 has illegal name "
> Xorg.0.log.": would clear entry
> entry at block 0 offset 2136 in directory inode 3758096527 has illegal name "
> mysqld.log.": would clear entry
> entry at block 0 offset 2352 in directory inode 3758096527 has illegal name "
> mai": would clear entry
> entry at block 0 offset 2376 in directory inode 3758096527 has illegal name "
> mai": would clear entry
> entry at block 0 offset 2680 in directory inode 3758096527 has illegal name "
> mysqld.log.": would clear entry
> entry at block 0 offset 2704 in directory inode 3758096527 has illegal name "
> mysqld.log.": would clear entry
> entry at block 0 offset 2896 in directory inode 3758096527 has illegal name "
> spo": would clear entry
> entry at block 0 offset 2944 in directory inode 3758096527 has illegal name "
> spo": would clear entry
> entry at block 0 offset 3016 in directory inode 3758096527 has illegal name "
> boo": would clear entry
> entry at block 0 offset 3040 in directory inode 3758096527 has illegal name "
> boo": would clear entry
> bad magic # 0x7bfaf5bb in inode 3758096532 (data fork) bmbt block 8389223
> bad data fork in inode 3758096532
> would have cleared inode 3758096532
> entry contains offset out of order in shortform dir 3758096534
> would have corrected entry offsets in directory 3758096534
> - agno = 15
> - agno = 16
> - agno = 17
> - agno = 18
> - agno = 19
> - agno = 20
> - agno = 21
> - agno = 22
> - agno = 23
> - agno = 24
> - agno = 25
> - agno = 26
> - agno = 27
> - agno = 28
> - agno = 29
> - agno = 30
> - agno = 31
> - process newly discovered inodes...
> Phase 4 - check for duplicate blocks...
> - setting up duplicate extent list...
> - check for inodes claiming duplicate blocks...
> - agno = 0
> - agno = 1
> - agno = 2
> - agno = 3
> - agno = 4
> entry "man3" at block 0 offset 96 in directory inode 1073741960 references
> free inode 3758096532
> would clear inode number in entry at offset 96...
> - agno = 5
> - agno = 6
> - agno = 7
> - agno = 8
> - agno = 9
> - agno = 10
> - agno = 11
> - agno = 12
> - agno = 13
> - agno = 14
> entry contains offset out of order in shortform dir 3758096515
> would have corrected entry offsets in directory 3758096515
> entry contains offset out of order in shortform dir 3758096516
> entry contains offset out of order in shortform dir 3758096516
> would have corrected entry offsets in directory 3758096516
> entry contains offset out of order in shortform dir 3758096517
> entry "charset" in shortform directory 3758096517 references non-existent
> inode 368079
> would have junked entry "charset" in directory inode 3758096517
> would have corrected entry offsets in directory 3758096517
> entry contains offset out of order in shortform dir 3758096518
> would have corrected entry offsets in directory 3758096518
> entry contains offset out of order in shortform dir 3758096519
> would have corrected entry offsets in directory 3758096519
> entry contains offset out of order in shortform dir 3758096520
> would have corrected entry offsets in directory 3758096520
> entry contains offset out of order in shortform dir 3758096521
> would have corrected entry offsets in directory 3758096521
> entry contains illegal character in shortform dir 3758096523
> would have junked entry "BuiltInSes" in directory inode 3758096523
> entry contains offset out of order in shortform dir 3758096524
> would have corrected entry offsets in directory 3758096524
> entry "httpd" at block 0 offset 280 in directory inode 3758096527 references
> non-existent inode 49624
> would clear inode number in entry at offset 280...
> entry "quagga" at block 0 offset 408 in directory inode 3758096527 references
> non-existent inode 262530
> would clear inode number in entry at offset 408...
> entry "pgsql" at block 0 offset 544 in directory inode 3758096527 references
> non-existent inode 353681
> would clear inode number in entry at offset 544...
> entry "anaconda.log" at block 0 offset 696 in directory inode 3758096527
> references non-existent inode 437972
> would clear inode number in entry at offset 696...
> entry "wtmp.1" at block 0 offset 1736 in directory inode 3758096527
> references non-existent inode 18052
> would clear inode number in entry at offset 1736...
> entry "messages.2" at block 0 offset 1824 in directory inode 3758096527
> references non-existent inode 448011
> would clear inode number in entry at offset 1824...
> entry "cron" at block 0 offset 2496 in directory inode 3758096527 references
> non-existent inode 438996
> would clear inode number in entry at offset 2496...
> entry "secure.1" at block 0 offset 2824 in directory inode 3758096527
> references non-existent inode 16990
> would clear inode number in entry at offset 2824...
> entry "spooler.4" at block 0 offset 2872 in directory inode 3758096527
> references non-existent inode 448015
> would clear inode number in entry at offset 2872...
> bad magic # 0x7bfaf5bb in inode 3758096532 (data fork) bmbt block 8389223
> bad data fork in inode 3758096532
> would have cleared inode 3758096532
> entry contains offset out of order in shortform dir 3758096534
> would have corrected entry offsets in directory 3758096534
> - agno = 15
> - agno = 16
> - agno = 17
> - agno = 18
> - agno = 19
> - agno = 20
> - agno = 21
> - agno = 22
> - agno = 23
> - agno = 24
> - agno = 25
> - agno = 26
> - agno = 27
> - agno = 28
> - agno = 29
> - agno = 30
> - agno = 31
> No modify flag set, skipping phase 5
> Phase 6 - check inode connectivity...
> - traversing filesystem starting at / ...
> entry "man3" in directory inode 1073741960 points to free inode 3758096532,
> would junk entry
> entry "charset" in shortform directory 3758096517 references non-existent
> inode 368079
> would junk entry
> entry "charset" in shortform directory 3758096517 references non-existent
> inode 368079
> would junk entry
> entry "log" in dir inode 805306496 inconsistent with .. value (128) in ino
> 3758096527,
> would clear entry "log"
> - traversal finished ...
> - traversing all unattached subtrees ...
> entry "httpd" in directory inode 3758096527 points to non-existent inode,
> would junk entry
> entry "quagga" in directory inode 3758096527 points to non-existent inode,
> would junk entry
> entry "pgsql" in directory inode 3758096527 points to non-existent inode,
> would junk entry
> entry "anaconda.log" in directory inode 3758096527 points to non-existent
> inode, would junk entry
> entry "wtmp.1" in directory inode 3758096527 points to non-existent inode,
> would junk entry
> entry "messages.2" in directory inode 3758096527 points to non-existent
> inode, would junk entry
> entry "cron" in directory inode 3758096527 points to non-existent inode,
> would junk entry
> entry "secure.1" in directory inode 3758096527 points to non-existent inode,
> would junk entry
> entry "spooler.4" in directory inode 3758096527 points to non-existent inode,
> would junk entry
> bad hash table for directory inode 3758096527 (hash value mismatch): would
> rebuild
> - traversals finished ...
> - moving disconnected inodes to lost+found ...
> disconnected dir inode 1610662360, would move to lost+found
> disconnected dir inode 1879310722, would move to lost+found
> disconnected dir inode 3758096527, would move to lost+found
> disconnected inode 3758104518, would move to lost+found
> disconnected inode 3758105657, would move to lost+found
> disconnected inode 3758105658, would move to lost+found
> disconnected inode 3758105659, would move to lost+found
> disconnected inode 3758105660, would move to lost+found
> disconnected inode 3758105661, would move to lost+found
> disconnected inode 3758105662, would move to lost+found
> disconnected inode 3758105663, would move to lost+found
> disconnected inode 3758105664, would move to lost+found
> disconnected inode 3758105665, would move to lost+found
> disconnected inode 3758105666, would move to lost+found
> disconnected inode 3758105667, would move to lost+found
> disconnected inode 3758105668, would move to lost+found
> disconnected inode 3758105669, would move to lost+found
> disconnected inode 3758105670, would move to lost+found
> disconnected inode 3758105671, would move to lost+found
> disconnected inode 3758105672, would move to lost+found
> [...]
>
>
>
|