Hello,
I'm using XFS on a large 8 disk RAID-5 partition. I was running a MySQL
process that was doing some random access (reads and writes) on a couple
of files. I then kicked off an scp job that was copying a few gigabytes
worth of data in about 5000 files to this machine when I lost the
filesystem. Any attempt to read resulted in input/output errors:
[doug@wds12 webcache]$ ls
ls: reading directory .: Input/output error
Something similar happened before, so I upgraded to the 2.6.10 kernel
after reading the thread "RE: Unknown Issue"
(http://oss.sgi.com/archives/linux-xfs/2004-12/msg00048.html) hoping
that it would solve my problems.
The 3dm daemon reported no errors with the disk array and
/var/log/messages had the following lines in it:
Jan 28 10:47:40 wds12 kernel: xfs_force_shutdown(sda1,0x8) called from
line 1091 of file fs/xfs/xfs_trans.c. Return address =
0xffffffff8021cb48
Jan 28 10:47:40 wds12 kernel: Filesystem "sda1": Corruption of in-memory
data detected. Shutting down filesystem: sda1
Jan 28 10:47:40 wds12 kernel: Please umount the filesystem, and rectify
the problem(s)
Here's my system configuration:
Hardware:
Tyan S2882G3NR Thunder K8S Pro Motherboard
2 - AMD Opteron processors
3ware 9500S-8 RAID Controller
8 - Western Digital 2500SD 250GB disk drives
OS:
2.6.10
Kernel, built with these options ...
- Processor Family: AMD Opteron/Athalon64
- Symmetric multi-processing support
- Preemptible kernel
I ran xfs_check and here's what it reported:
[root@wds12 log]# xfs_check /dev/sda1
bad format 2 for inode 1208 type 0
bad format 2 for inode 1209 type 0
bad format 2 for inode 1210 type 0
bad format 2 for inode 1211 type 0
bad format 2 for inode 1212 type 0
bad format 2 for inode 1213 type 0
bad format 2 for inode 1214 type 0
bad format 2 for inode 1215 type 0
ir_freecount/free mismatch, inode chunk 0/1152, freecount 64 nfree 56
bad format 2 for inode 268436760 type 0
bad format 2 for inode 268436761 type 0
bad format 2 for inode 268436762 type 0
bad format 2 for inode 268436763 type 0
bad format 2 for inode 268436767 type 0
ir_freecount/free mismatch, inode chunk 1/1248, freecount 52 nfree 47
bad format 2 for inode 268454488 type 0
bad format 2 for inode 268454489 type 0
bad format 2 for inode 268454490 type 0
bad format 2 for inode 268454491 type 0
bad format 2 for inode 268454492 type 0
bad format 2 for inode 268454494 type 0
bad format 2 for inode 268454495 type 0
ir_freecount/free mismatch, inode chunk 1/18976, freecount 64 nfree 57
bad format 2 for inode 536872216 type 0
bad format 2 for inode 536872217 type 0
bad format 2 for inode 536872218 type 0
bad format 2 for inode 536872219 type 0
bad format 2 for inode 536872220 type 0
bad format 2 for inode 536872221 type 0
bad format 2 for inode 536872222 type 0
bad format 2 for inode 536872223 type 0
ir_freecount/free mismatch, inode chunk 2/1248, freecount 8 nfree 0
bad format 2 for inode 536889880 type 0
bad format 2 for inode 536889881 type 0
bad format 2 for inode 536889883 type 0
bad format 2 for inode 536889886 type 0
bad format 2 for inode 536889887 type 0
ir_freecount/free mismatch, inode chunk 2/18912, freecount 64 nfree 59
bad format 2 for inode 805325336 type 0
bad format 2 for inode 805325337 type 0
bad format 2 for inode 805325338 type 0
bad format 2 for inode 805325339 type 0
bad format 2 for inode 805325340 type 0
bad format 2 for inode 805325341 type 0
bad format 2 for inode 805325342 type 0
bad format 2 for inode 805325343 type 0
ir_freecount/free mismatch, inode chunk 3/18912, freecount 64 nfree 56
bad format 2 for inode 1073743128 type 0
bad format 2 for inode 1073743129 type 0
bad format 2 for inode 1073743130 type 0
bad format 2 for inode 1073743131 type 0
bad format 2 for inode 1073743132 type 0
bad format 2 for inode 1073743133 type 0
bad format 2 for inode 1073743134 type 0
bad format 2 for inode 1073743135 type 0
ir_freecount/free mismatch, inode chunk 4/1248, freecount 64 nfree 56
bad format 2 for inode 1073760824 type 0
bad format 2 for inode 1073760825 type 0
bad format 2 for inode 1073760826 type 0
bad format 2 for inode 1073760827 type 0
ir_freecount/free mismatch, inode chunk 4/18944, freecount 64 nfree 60
bad format 2 for inode 1342178584 type 0
bad format 2 for inode 1342178585 type 0
bad format 2 for inode 1342178586 type 0
bad format 2 for inode 1342178587 type 0
bad format 2 for inode 1342178588 type 0
bad format 2 for inode 1342178589 type 0
bad format 2 for inode 1342178590 type 0
bad format 2 for inode 1342178591 type 0
ir_freecount/free mismatch, inode chunk 5/1248, freecount 59 nfree 51
bad format 2 for inode 1342196280 type 0
bad format 2 for inode 1342196281 type 0
bad format 2 for inode 1342196282 type 0
bad format 2 for inode 1342196283 type 0
bad format 2 for inode 1342196284 type 0
bad format 2 for inode 1342196285 type 0
bad format 2 for inode 1342196286 type 0
bad format 2 for inode 1342196287 type 0
ir_freecount/free mismatch, inode chunk 5/18944, freecount 64 nfree 56
bad format 2 for inode 1610614011 type 0
bad format 2 for inode 1610614012 type 0
bad format 2 for inode 1610614013 type 0
bad format 2 for inode 1610614014 type 0
ir_freecount/free mismatch, inode chunk 6/1216, freecount 64 nfree 60
bad format 2 for inode 1610631704 type 0
bad format 2 for inode 1610631705 type 0
bad format 2 for inode 1610631706 type 0
bad format 2 for inode 1610631707 type 0
bad format 2 for inode 1610631708 type 0
bad format 2 for inode 1610631709 type 0
bad format 2 for inode 1610631710 type 0
bad format 2 for inode 1610631711 type 0
ir_freecount/free mismatch, inode chunk 6/18nk count
allocated inode 1213 has 0 link count
allocated inode 1214 has 0 link count
allocated inode 1215 has 0 link count
allocated inode 268436760 has 0 link count
allocated inode 268436761 has 0 link count
allocated inode 268436762 has 0 link count
allocated inode 268436763 has 0 link count
[... 150 lines like this ...]
I haven't run xfs_repair yet. Is there anything more I can do to
diagnose this problem?
- Doug
P.S. I'm here in Mountain View, CA so if someone want's to swing by and
have a look, feel free! :)
|