Hardware is: 2P Xeon 2.8GHz 2GB memory JBOD with 13 disks 3 md volumes: md0: 7 disks RAID0 md1: 4 disks RAID5 md2: 2 disks RAID0 Adaptec AIC-7899 U160 SCSI controller Broadcom BCM-5701 gigabit network interface XFS setup: $ xfs_info /dev/md0 meta-data=/mnt/vol102 isize=256 agcount=60, agsize=1048576 blks = sectsz=512 data = bsize=4096 blocks=62229552, imaxpct=25 = sunit=8 swidth=56 blks, unwritten=1 naming =version 2 bsize=4096 log =internal bsize=4096 blocks=30392, version=1 = sectsz=512 sunit=0 blks realtime =none extsz=229376 blocks=0, rtextents=0 $ xfs_info /dev/md1 meta-data=/mnt/vol101 isize=256 agcount=103, agsize=1048576 blks = sectsz=512 data = bsize=4096 blocks=107528976, imaxpct=25 = sunit=8 swidth=24 blks, unwritten=1 naming =version 2 bsize=4096 log =internal bsize=4096 blocks=32768, version=1 = sectsz=512 sunit=0 blks realtime =none extsz=98304 blocks=0, rtextents=0 $ xfs_info /dev/md2 meta-data=/mnt/vol55 isize=256 agcount=17, agsize=1048568 blks = sectsz=512 data = bsize=4096 blocks=17779872, imaxpct=25 = sunit=8 swidth=16 blks, unwritten=1 naming =version 2 bsize=4096 log =internal bsize=4096 blocks=8688, version=1 = sectsz=512 sunit=0 blks realtime =none extsz=65536 blocks=0, rtextents=0 /var/log/messages snippets: Linux 2.4.22 SMP + xfs-2.4.22 patches ===================================== Dec 24 12:44:49 foobar kernel: xfs_inotobp: xfs_imap() returned an error 22 on md(9,1). Returning error. Dec 24 12:44:49 foobar kernel: xfs_iunlink_remove: xfs_inotobp() returned an error 22 on md(9,1). Returning error. Dec 24 12:44:49 foobar kernel: xfs_inactive:^Ixfs_ifree() returned an error = 22 on md(9,1) Dec 24 12:44:49 foobar kernel: xfs_force_shutdown(md(9,1),0x1) called from line 1873 of file xfs_vnodeops.c. Return address = 0xf89f5ee6 Dec 24 12:44:49 foobar kernel: Filesystem "md(9,1)": I/O Error Detected. Shutting down filesystem: md(9,1) Dec 24 12:44:49 foobar kernel: Please umount the filesystem, and rectify the problem(s) Jan 6 16:00:05 foobar kernel: Filesystem "md(9,0)": xfs_log_write: reservation ran out. Need to up reservation Jan 6 16:00:05 foobar kernel: xfs_force_shutdown(md(9,0),0x8) called from line 1715 of file xfs_log.c. Return address = 0xf89f5ee6 Jan 6 16:00:05 foobar kernel: Filesystem "md(9,0)": Corruption of in-memory data detected. Shutting down filesystem: md(9,0) Jan 6 16:00:05 foobar kernel: Please umount the filesystem, and rectify the problem(s) Jan 6 16:00:05 foobar kernel: xfs_force_shutdown(md(9,0),0x2) called from line 1297 of file xfs_log.c. Return address = 0xf89f5ee6 Jan 9 09:40:27 foobar kernel: XFS internal error XFS_WANT_CORRUPTED_GOTO at line 866 of file xfs_ialloc.c. Caller 0xf89ca7d1 Jan 9 09:40:27 foobar kernel: d4b55b8c f89c47b9 f8a11f75 00000001 00000000 f8a11f68 00000362 f89ca7d1 Jan 9 09:40:27 foobar kernel: 0803ae90 00000000 00000001 00000000 00001000 00000001 00000018 00000000 Jan 9 09:40:27 foobar kernel: 00000005 000000f0 0000002f 00000008 0003ae90 0000a205 00000000 00000000 Jan 9 09:40:27 foobar kernel: Call Trace: [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] Jan 9 09:40:27 foobar kernel: nfsd: non-standard errno: -990 Jan 9 11:57:24 foobar kernel: XFS internal error XFS_WANT_CORRUPTED_GOTO at line 866 of file xfs_ialloc.c. Caller 0xf89ca7d1 Jan 9 11:57:24 foobar kernel: d4af7b8c f89c47b9 f8a11f75 00000001 00000000 f8a11f68 00000362 f89ca7d1 Jan 9 11:57:24 foobar kernel: 0803ae90 00000000 00000001 00000000 d1466000 00000000 00000018 c0259824 Jan 9 11:57:24 foobar kernel: f6c11de0 c4d12580 0000002f 00000008 0003ae90 00000000 d1466000 00000000 Jan 9 11:57:24 foobar kernel: Call Trace: [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] Jan 9 11:57:24 foobar kernel: nfsd: non-standard errno: -990 Jan 9 14:11:25 foobar kernel: XFS internal error XFS_WANT_CORRUPTED_RETURN at line 295 of file xfs_alloc.c. Caller 0xf899559d Jan 9 14:11:25 foobar kernel: c44ada00 f89947bd f8a11a09 00000001 00000000 f8a119fd 00000127 f899559d Jan 9 14:11:25 foobar kernel: 00000000 00000000 00000000 0000000c ca23789c ca237920 00091e18 f899559d Jan 9 14:11:25 foobar kernel: ca23789c ca237920 00091e18 00000010 00091e1c 0000000c 00000001 0009203c Jan 9 14:11:25 foobar kernel: Call Trace: [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] Jan 12 08:08:08 foobar kernel: XFS internal error XFS_WANT_CORRUPTED_GOTO at line 866 of file xfs_ialloc.c. Caller 0xf89ca7d1 Jan 12 08:08:08 foobar kernel: d4b91b8c f89c47b9 f8a11f75 00000001 00000000 f8a11f68 00000362 f89ca7d1 Jan 12 08:08:08 foobar kernel: 21000c16 00000000 00000001 00000000 c025e407 43011dac 00000018 00000000 Jan 12 08:08:08 foobar kernel: 00000018 00000006 00000056 00000021 00000c16 d5122c20 00000296 00000000 Jan 12 08:08:08 foobar kernel: Call Trace: [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] Jan 12 08:08:08 foobar kernel: nfsd: non-standard errno: -990 Added ikeep mount option. Linux 2.4.24 SMP + xfs-2.4.23 patches ===================================== Jan 20 08:50:54 foobar kernel: XFS internal error XFS_WANT_CORRUPTED_GOTO at line 866 of file xfs_ialloc.c. Caller 0xf89cb7d1 Jan 20 08:50:54 foobar kernel: f5497b8c f89c57b9 f8a11e55 00000001 00000000 f8a11e48 00000362 f89cb7d1 Jan 20 08:50:54 foobar kernel: 4d164a94 00000000 00000001 00000000 00000000 00000001 00000018 ffffffff Jan 20 08:50:54 foobar kernel: 00000000 00000000 00000047 0000004d 00164a94 c2908400 039d3000 00000000 Jan 20 08:50:54 foobar kernel: Call Trace: [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] Jan 20 08:50:54 foobar kernel: nfsd: non-standard errno: -990 Jan 20 09:21:26 foobar kernel: XFS internal error XFS_WANT_CORRUPTED_GOTO at line 866 of file xfs_ialloc.c. Caller 0xf89cb7d1 Jan 20 09:21:26 foobar kernel: f549db8c f89c57b9 f8a11e55 00000001 00000000 f8a11e48 00000362 f89cb7d1 Jan 20 09:21:26 foobar kernel: 4b0035b5 00000000 00000001 00000000 00000000 f5a29400 00000018 00000000 Jan 20 09:21:26 foobar kernel: 00000000 00000001 00000056 0000004b 000035b5 000001c4 f89c9ad3 00000000 Jan 20 09:21:26 foobar kernel: Call Trace: [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] Jan 20 09:21:26 foobar kernel: nfsd: non-standard errno: -990 Jan 20 16:23:34 foobar kernel: XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1634 of file xfs_alloc.c. Caller 0xf8997fe2 Jan 20 16:23:34 foobar kernel: e67e5d28 f8997289 f8a11903 00000001 00000000 f8a118dd 00000662 f8997fe2 Jan 20 16:23:34 foobar kernel: f55fb800 e6cf2a6c e6cf285c 00091e88 000000ab 00000001 00091e18 00000010 Jan 20 16:23:34 foobar kernel: 00000000 00000001 f35c2da0 e6cf8c04 e6cf8c04 e6cf5f0c f8997fe2 e6cf8c04 Jan 20 16:23:34 foobar kernel: Call Trace: [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] Jan 20 16:23:34 foobar kernel: xfs_force_shutdown(md(9,0),0x8) called from line 4046 of file xfs_bmap.c. Return address = 0xf89f6fc6 Jan 20 16:23:34 foobar kernel: Filesystem "md(9,0)": Corruption of in-memory data detected. Shutting down filesystem: md(9,0) Jan 20 16:23:34 foobar kernel: Please umount the filesystem, and rectify the problem(s) Jan 20 21:05:01 foobar kernel: XFS internal error XFS_WANT_CORRUPTED_GOTO at line 866 of file xfs_ialloc.c. Caller 0xf89cb7d1 Jan 20 21:05:01 foobar kernel: f5d93b8c f89c57b9 f8a11e55 00000001 00000000 f8a11e48 00000362 f89cb7d1 Jan 20 21:05:01 foobar kernel: 4b0035b5 00000000 00000001 00000000 c0251659 f76af0c0 00000018 c033f450 Jan 20 21:05:01 foobar kernel: 00000020 00000042 0000003f 0000004b 000035b5 f76af0c0 c607b2d4 00000000 Jan 20 21:05:01 foobar kernel: Call Trace: [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] Jan 20 21:05:01 foobar kernel: nfsd: non-standard errno: -990 Jan 20 21:16:06 foobar kernel: XFS internal error XFS_WANT_CORRUPTED_GOTO at line 866 of file xfs_ialloc.c. Caller 0xf89cb7d1 Jan 20 21:16:06 foobar kernel: f5d3fb8c f89c57b9 f8a11e55 00000001 00000000 f8a11e48 00000362 f89cb7d1 Jan 20 21:16:06 foobar kernel: 48be1c1e 00000000 00000001 00000000 00001000 00000001 00000018 00000000 Jan 20 21:16:06 foobar kernel: 00000005 000000f0 00000047 00000048 00be1c1e 0000a205 00000000 00000000 Jan 20 21:16:06 foobar kernel: Call Trace: [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] Jan 20 21:16:06 foobar kernel: nfsd: non-standard errno: -990 Jan 20 21:42:26 foobar kernel: XFS internal error XFS_WANT_CORRUPTED_GOTO at line 866 of file xfs_ialloc.c. Caller 0xf89cb7d1 Jan 20 21:42:26 foobar kernel: f5d8fb8c f89c57b9 f8a11e55 00000001 00000000 f8a11e48 00000362 f89cb7d1 Jan 20 21:42:26 foobar kernel: 48be1c1e 00000000 00000001 00000000 00000000 f5fab000 00000018 00000000 Jan 20 21:42:26 foobar kernel: 00000000 00000001 00000056 00000048 00be1c1e 000001c4 f89df7f2 00000000 Jan 20 21:42:26 foobar kernel: Call Trace: [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] Jan 20 21:42:26 foobar kernel: nfsd: non-standard errno: -990 Jan 21 01:04:41 foobar kernel: XFS internal error XFS_WANT_CORRUPTED_GOTO at line 866 of file xfs_ialloc.c. Caller 0xf89cb7d1 Jan 21 01:04:41 foobar kernel: f5d59b8c f89c57b9 f8a11e55 00000001 00000000 f8a11e48 00000362 f89cb7d1 Jan 21 01:04:41 foobar kernel: 4b0035b5 00000000 00000001 00000000 00000000 f5fab000 00000018 00000000 Jan 21 01:04:41 foobar kernel: 00000000 e02b24ec 0000003f 0000004b 000035b5 c02633f4 ca0458a0 00000000 Jan 21 01:04:41 foobar kernel: Call Trace: [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] Jan 21 01:04:41 foobar kernel: nfsd: non-standard errno: -990 Jan 21 01:08:02 foobar kernel: XFS internal error XFS_WANT_CORRUPTED_GOTO at line 866 of file xfs_ialloc.c. Caller 0xf89cb7d1 Jan 21 01:08:02 foobar kernel: f5d55b8c f89c57b9 f8a11e55 00000001 00000000 f8a11e48 00000362 f89cb7d1 Jan 21 01:08:02 foobar kernel: 4cd07e81 00000000 00000001 00000000 00000000 f5fab000 00000018 00000000 Jan 21 01:08:02 foobar kernel: 00000000 00000001 00000047 0000004c 00d07e81 000001c4 f89c9ad3 00000000 Jan 21 01:08:02 foobar kernel: Call Trace: [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] Jan 21 01:08:02 foobar kernel: nfsd: non-standard errno: -990 Jan 21 01:23:46 foobar kernel: XFS internal error XFS_WANT_CORRUPTED_GOTO at line 866 of file xfs_ialloc.c. Caller 0xf89cb7d1 Jan 21 01:23:46 foobar kernel: f5877b8c f89c57b9 f8a11e55 00000001 00000000 f8a11e48 00000362 f89cb7d1 Jan 21 01:23:46 foobar kernel: 4cd07e81 00000000 00000001 00000000 00001000 00000001 00000018 00000000 Jan 21 01:23:46 foobar kernel: 00000005 000000f0 00000056 0000004c 00d07e81 0000a205 00000000 00000000 Jan 21 01:23:46 foobar kernel: Call Trace: [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] Jan 21 01:23:46 foobar kernel: nfsd: non-standard errno: -990 Jan 21 06:23:18 foobar kernel: XFS internal error XFS_WANT_CORRUPTED_GOTO at line 866 of file xfs_ialloc.c. Caller 0xf89ca7d1 Jan 21 06:23:18 foobar kernel: f603bb8c f89c47b9 f8a11f75 00000001 00000000 f8a11f68 00000362 f89ca7d1 Jan 21 06:23:18 foobar kernel: 4cd07e81 00000000 00000001 00000000 f5c938c0 f7fcba40 00000018 f7fcba40 Jan 21 06:23:18 foobar kernel: c0264300 f603bca8 0000003f 0000004c 00d07e81 00000040 f5c938c0 00000000 Jan 21 06:23:18 foobar kernel: Call Trace: [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] Jan 21 06:23:18 foobar kernel: nfsd: non-standard errno: -990 Jan 21 06:33:41 foobar kernel: XFS internal error XFS_WANT_CORRUPTED_GOTO at line 866 of file xfs_ialloc.c. Caller 0xf89ca7d1 Jan 21 06:33:41 foobar kernel: f6015b8c f89c47b9 f8a11f75 00000001 00000000 f8a11f68 00000362 f89ca7d1 Jan 21 06:33:41 foobar kernel: 4cd07e81 00000000 00000001 00000000 c2dcd800 c2dcd800 00000018 c2dcd800 Jan 21 06:33:41 foobar kernel: c0264300 f6015ca8 00000047 0000004c 00d07e81 c47ab308 e9207acc 00000000 Jan 21 06:33:41 foobar kernel: Call Trace: [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] Jan 21 06:33:41 foobar kernel: nfsd: non-standard errno: -990