XFS corruption under 2.6.15.1

[Justin Piszcz <jpiszcz@xxxxxxxxxxxxxxx>]

After > 70 days of uptime a FS on one my machines became corrupted.

When I unmounted and remounted, at [6388318.818916], the problem persists.

Any idea what happened here?

[6371204.029520] XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1563 
of file fs/xfs/xfs_alloc.c.  Caller 0xc021e3c9
[6371204.030411]  [<c021d176>] xfs_free_ag_extent+0x376/0x750
[6371204.030523]  [<c021e3c9>] xfs_free_extent+0xe9/0xf0
[6371204.030567]  [<c021e3c9>] xfs_free_extent+0xe9/0xf0
[6371204.030611]  [<c02738a8>] xfs_trans_get_efd+0x38/0x50
[6371204.030661]  [<c0231110>] xfs_bmap_finish+0x140/0x1e0
[6371204.030730]  [<c025ab54>] xfs_itruncate_finish+0x214/0x440
[6371204.030783]  [<c027a753>] xfs_inactive+0x533/0x590
[6371204.030825]  [<c028a8b4>] linvfs_clear_inode+0x84/0xa0
[6371204.030873]  [<c016f35a>] clear_inode+0xba/0xe0
[6371204.030917]  [<c01703b0>] generic_delete_inode+0x100/0x110
[6371204.030959]  [<c01705d2>] iput+0x62/0x80
[6371204.030997]  [<c0165c0a>] sys_unlink+0xfa/0x130
[6371204.031043]  [<c0168470>] filldir64+0x0/0x110
[6371204.031053]  [<c0168628>] sys_getdents64+0xa8/0xb2
[6371204.031061]  [<c0168470>] filldir64+0x0/0x110
[6371204.031069]  [<c0102c21>] syscall_call+0x7/0xb
[6371204.031092] xfs_force_shutdown(loop6,0x8) called from line 4125 of 
file fs/xfs/xfs_bmap.c.  Return address = 0xc028b70c
[6371204.032677] Filesystem "loop6": Corruption of in-memory data 
detected.  Shutting down filesystem: loop6
[6371204.032748] Please umount the filesystem, and rectify the problem(s)
[6388287.970034] xfs_force_shutdown(loop6,0x1) called from line 339 of 
file fs/xfs/xfs_rw.c.  Return address = 0xc028b70c
[6388318.818916] XFS mounting filesystem loop6
[6388318.929946] Starting XFS recovery on filesystem: loop6 (logdev: 
internal)
[6388318.970850] XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1563 
of file fs/xfs/xfs_alloc.c.  Caller 0xc021e3c9
[6388318.971186]  [<c021d176>] xfs_free_ag_extent+0x376/0x750
[6388318.971264]  [<c021e3c9>] xfs_free_extent+0xe9/0xf0
[6388318.971308]  [<c021e3c9>] xfs_free_extent+0xe9/0xf0
[6388318.971350]  [<c02738a8>] xfs_trans_get_efd+0x38/0x50
[6388318.971400]  [<c026a367>] xlog_recover_process_efi+0x187/0x200
[6388318.971442]  [<c026a436>] xlog_recover_process_efis+0x56/0xb0
[6388318.971483]  [<c026bb90>] xlog_recover_finish+0x20/0xd0
[6388318.971524]  [<c026265e>] xfs_log_mount_finish+0x3e/0x40
[6388318.971563]  [<c026d5ec>] xfs_mountfs+0xa3c/0xf50
[6388318.971604]  [<c0282ee0>] pagebuf_rele+0x20/0xc0
[6388318.971647]  [<c026c886>] xfs_readsb+0x196/0x200
[6388318.971686]  [<c025dd46>] xfs_ioinit+0x26/0x50
[6388318.971746]  [<c0275963>] xfs_mount+0x3e3/0x660
[6388318.971756]  [<c028b434>] vfs_mount+0x34/0x40
[6388318.971769]  [<c028b247>] linvfs_fill_super+0x97/0x1f0
[6388318.971778]  [<c02c2897>] snprintf+0x27/0x30
[6388318.971796]  [<c018c871>] disk_name+0xb1/0xc0
[6388318.971809]  [<c015ba95>] sget+0x175/0x1b0
[6388318.971825]  [<c015cbce>] sb_set_blocksize+0x2e/0x60
[6388318.971834]  [<c015c53b>] get_sb_bdev+0xfb/0x140
[6388318.971843]  [<c028b3cf>] linvfs_get_sb+0x2f/0x60
[6388318.971851]  [<c028b1b0>] linvfs_fill_super+0x0/0x1f0
[6388318.971859]  [<c015c79f>] do_kern_mount+0x5f/0xe0
[6388318.971867]  [<c0173685>] do_new_mount+0xa5/0xf0
[6388318.971881]  [<c0173d59>] do_mount+0x199/0x200
[6388318.971890]  [<c03e7cb6>] iret_exc+0x3b2/0x6f7
[6388318.971906]  [<c0173b78>] copy_mount_options+0x78/0xc0
[6388318.971914]  [<c0174155>] sys_mount+0xa5/0xf0
[6388318.971921]  [<c0102c21>] syscall_call+0x7/0xb
[6388319.427361] Ending XFS recovery on filesystem: loop6 (logdev: 
internal)

Thanks,

Justin.