XFS internel errors (oops?)

[Michael Loftis <mloftis@xxxxxxxxx>]

OK at about 3AM this morning under heavy-ish load one of the remaining NFS 
fileservers with XFS quipped the below...

It's still..running....I don't know if that's a good thing or not.
We're looking at about 160GB of storage, and this sysem is extremely 
downtime sensitive, I can't just take the thing offline...

0x0: 58 46 53 42 00 00 10 00 00 00 00 00 02 14 ac 20
Filesystem "dac960(48,3)": XFS internal error xfs_itobp at line 382 of file 
xfs_inode.c.  Caller 0xf889b40b
f6f8fc94 f889a164 f88de7ea 00000001 f773c400 f88de7d2 0000017e f889b40b
      f889b40b 00000000 00000000 00a80200 cf7d4140 f7bb8524 cf7d4140 
e07920a0
      00000001 00000000 00000001 cf7d4140 f88eea90 00000000 00000000 
00000010
Call Trace:
[<f889a164>] xfs_itobp+0x1a4/0x1f0 [xfs]
[<f88de7ea>] .rodata.str1.1+0x5ea/0x190c [xfs]
[<f88de7d2>] .rodata.str1.1+0x5d2/0x190c [xfs]
[<f889b40b>] xfs_iread+0x5b/0x1c0 [xfs]
[<f889b40b>] xfs_iread+0x5b/0x1c0 [xfs]
[<f88eea90>] pbhash+0xc30/0x1000 [xfs]
[<f88aeb5d>] xfs_trans_brelse+0x4d/0xf0 [xfs]
[<f889b40b>] xfs_iread+0x5b/0x1c0 [xfs]
[<f889916a>] xfs_iget_core+0x14a/0x480 [xfs]
[<f889951d>] xfs_iget+0x7d/0x150 [xfs]
[<f88b19db>] xfs_vget+0x4b/0xba [xfs]
[<f88c3bc1>] vfs_vget+0x21/0x30 [xfs]
[<f88c368e>] linvfs_fh_to_dentry+0x6e/0x100 [xfs]
[<f89f2f58>] nfsd_get_dentry+0x28/0xb0 [nfsd]
[<f89f34ad>] find_fh_dentry+0x14d/0x360 [nfsd]
[<f89f3931>] fh_verify+0x271/0x470 [nfsd]
[<f89d4174>] svc_sock_enqueue+0x184/0x200 [sunrpc]
[<f89f4cec>] nfsd_access+0x1c/0xe0 [nfsd]
[<f89fa4c7>] nfsd3_proc_access+0xb7/0xd0 [nfsd]
[<f8a031b0>] nfsd_procedures3+0x90/0x360 [nfsd]
[<f8a031b0>] nfsd_procedures3+0x90/0x360 [nfsd]
[<f89f15b7>] nfsd_dispatch+0xb7/0x17b [nfsd]
[<f89d3e07>] svc_process_Rsmp_cec9731f+0x347/0x518 [sunrpc]
[<f8a02ad8>] nfsd_version3+0x0/0x10 [nfsd]
[<f8a02af8>] nfsd_program+0x0/0x28 [nfsd]
[<f89f13ac>] nfsd+0x20c/0x360 [nfsd]
[<f89f11a0>] nfsd+0x0/0x360 [nfsd]
[<c0107296>] arch_kernel_thread+0x26/0x30 [kernel]
[<f89f11a0>] nfsd+0x0/0x360 [nfsd]

0x0: 58 46 53 42 00 00 10 00 00 00 00 00 02 14 ac 20
Filesystem "dac960(48,3)": XFS internal error xfs_itobp at line 382 of file 
xfs_inode.c.  Caller 0xf889b40b
f6f8fc94 f889a164 f88de7ea 00000001 f773c400 f88de7d2 0000017e f889b40b
      f889b40b 00000000 00000000 00000004 cf7d4140 d6fbd320 00000000 
c03c6ae0
      00000001 00000080 ddae5ea4 ddae5e90 00000000 00000000 00000000 
00000010
Call Trace:
[<f889a164>] xfs_itobp+0x1a4/0x1f0 [xfs]
[<f88de7ea>] .rodata.str1.1+0x5ea/0x190c [xfs]
[<f88de7d2>] .rodata.str1.1+0x5d2/0x190c [xfs]
[<f889b40b>] xfs_iread+0x5b/0x1c0 [xfs]
[<f889b40b>] xfs_iread+0x5b/0x1c0 [xfs]
[<c01f3c70>] output_maybe_reroute+0x0/0x10 [kernel]
[<f889b40b>] xfs_iread+0x5b/0x1c0 [xfs]
[<c01f3c7b>] output_maybe_reroute+0xb/0x10 [kernel]
[<f889916a>] xfs_iget_core+0x14a/0x480 [xfs]
[<f889951d>] xfs_iget+0x7d/0x150 [xfs]
[<f88b19db>] xfs_vget+0x4b/0xba [xfs]
[<f88c3bc1>] vfs_vget+0x21/0x30 [xfs]
[<f88c368e>] linvfs_fh_to_dentry+0x6e/0x100 [xfs]
[<f89f2f58>] nfsd_get_dentry+0x28/0xb0 [nfsd]
[<f89f34ad>] find_fh_dentry+0x14d/0x360 [nfsd]
[<c02153c5>] inet_sendmsg+0x35/0x40 [kernel]
[<f89f3931>] fh_verify+0x271/0x470 [nfsd]
[<f89f4cec>] nfsd_access+0x1c/0xe0 [nfsd]
[<f89fa4c7>] nfsd3_proc_access+0xb7/0xd0 [nfsd]
[<f8a031b0>] nfsd_procedures3+0x90/0x360 [nfsd]
[<f8a031b0>] nfsd_procedures3+0x90/0x360 [nfsd]
[<f89f15b7>] nfsd_dispatch+0xb7/0x17b [nfsd]
[<f89d3e07>] svc_process_Rsmp_cec9731f+0x347/0x518 [sunrpc]
[<f8a02ad8>] nfsd_version3+0x0/0x10 [nfsd]
[<f8a02af8>] nfsd_program+0x0/0x28 [nfsd]
[<f89f13ac>] nfsd+0x20c/0x360 [nfsd]
[<f89f11a0>] nfsd+0x0/0x360 [nfsd]
[<c0107296>] arch_kernel_thread+0x26/0x30 [kernel]
[<f89f11a0>] nfsd+0x0/0x360 [nfsd]

0x0: 58 46 53 42 00 00 10 00 00 00 00 00 02 14 ac 20
Filesystem "dac960(48,3)": XFS internal error xfs_itobp at line 382 of file 
xfs_inode.c.  Caller 0xf889b40b
f6f8dc94 f889a164 f88de7ea 00000001 f773c400 f88de7d2 0000017e f889b40b
      f889b40b 00000000 00000000 00a80200 f0858900 f7bb8524 c0359dac 
c0359cec
      c0359cdc 00000001 c01db013 c0359cec f6f8dcf0 00000000 00000000 
00000010
Call Trace:
[<f889a164>] xfs_itobp+0x1a4/0x1f0 [xfs]
[<f88de7ea>] .rodata.str1.1+0x5ea/0x190c [xfs]
[<f88de7d2>] .rodata.str1.1+0x5d2/0x190c [xfs]
[<f889b40b>] xfs_iread+0x5b/0x1c0 [xfs]
[<f889b40b>] xfs_iread+0x5b/0x1c0 [xfs]
[<c01db013>] net_rx_action+0xb3/0x170 [kernel]
[<f889b40b>] xfs_iread+0x5b/0x1c0 [xfs]
[<f889916a>] xfs_iget_core+0x14a/0x480 [xfs]
[<f889951d>] xfs_iget+0x7d/0x150 [xfs]
[<f88b19db>] xfs_vget+0x4b/0xba [xfs]
[<f88c3bc1>] vfs_vget+0x21/0x30 [xfs]
[<f88c368e>] linvfs_fh_to_dentry+0x6e/0x100 [xfs]
[<f89f2f58>] nfsd_get_dentry+0x28/0xb0 [nfsd]
[<f89f34ad>] find_fh_dentry+0x14d/0x360 [nfsd]
[<f89f3931>] fh_verify+0x271/0x470 [nfsd]
[<f89d4174>] svc_sock_enqueue+0x184/0x200 [sunrpc]
[<f89f4cec>] nfsd_access+0x1c/0xe0 [nfsd]
[<f89fa4c7>] nfsd3_proc_access+0xb7/0xd0 [nfsd]
[<f8a031b0>] nfsd_procedures3+0x90/0x360 [nfsd]
[<f8a031b0>] nfsd_procedures3+0x90/0x360 [nfsd]
[<f89f15b7>] nfsd_dispatch+0xb7/0x17b [nfsd]
[<f89d3e07>] svc_process_Rsmp_cec9731f+0x347/0x518 [sunrpc]
[<f8a02ad8>] nfsd_version3+0x0/0x10 [nfsd]
[<f8a02af8>] nfsd_program+0x0/0x28 [nfsd]
[<f89f13ac>] nfsd+0x20c/0x360 [nfsd]
[<f89f11a0>] nfsd+0x0/0x360 [nfsd]
[<c0107296>] arch_kernel_thread+0x26/0x30 [kernel]
[<f89f11a0>] nfsd+0x0/0x360 [nfsd]



--
Undocumented Features quote of the moment...
"It's not the one bullet with your name on it that you
have to worry about; it's the twenty thousand-odd rounds
labeled `occupant.'"
  --Murphy's Laws of Combat