I have a virtual server hosted in a XEN environment. Kernel is
2.6.16.13-XenU and I can't change it. :( Host provider is not responsive
to getting us a newer kernel.
Anyway, during my nightly rsync backup, I started seeing the following:
Jul 18 12:51:01 hosted kernel: 0x0: 00 00 00 00 01 00 00 00 23 00 00 00
00 00 00 00
Jul 18 12:51:01 hosted kernel: Filesystem "sda1": XFS internal error
xfs_da_do_buf(2) at line 2221 of file fs/xfs/xfs_da_btree.c. Caller
0xc02288fd
Jul 18 12:51:01 hosted kernel: [<c0237b71>] xfs_corruption_error+0xbb/0xc6
Jul 18 12:51:01 hosted kernel: [<c02288fd>] xfs_da_read_buf+0x25/0x2a
Jul 18 12:51:01 hosted kernel: [<c025e4e5>] xfs_buf_read_flags+0x1f/0x82
Jul 18 12:51:01 hosted kernel: [<c025b1aa>] kmem_zone_alloc+0x43/0x88
Jul 18 12:51:01 hosted kernel: [<c02287bf>] xfs_da_do_buf+0x5df/0x6a9
Jul 18 12:51:01 hosted kernel: [<c02288fd>] xfs_da_read_buf+0x25/0x2a
Jul 18 12:51:01 hosted kernel: [<c0169d88>] dput+0x1b/0x11c
Jul 18 12:51:01 hosted kernel: [<c02288fd>] xfs_da_read_buf+0x25/0x2a
Jul 18 12:51:01 hosted kernel: [<c022cc12>]
xfs_dir2_block_getdents+0xe0/0x2e4
Jul 18 12:51:01 hosted kernel: [<c022cc12>]
xfs_dir2_block_getdents+0xe0/0x2e4
Jul 18 12:51:01 hosted kernel: [<c013e7ee>]
get_page_from_freelist+0xa8/0x34b
Jul 18 12:51:01 hosted kernel: [<c0219919>] xfs_bmap_last_offset+0xd5/0xe8
Jul 18 12:51:01 hosted kernel: [<c022c537>]
xfs_dir2_put_dirent64_direct+0x0/0x8b
Jul 18 12:51:01 hosted kernel: [<c022c17e>] xfs_dir2_isblock+0x1c/0x65
Jul 18 12:51:01 hosted kernel: [<c022c537>]
xfs_dir2_put_dirent64_direct+0x0/0x8b
Jul 18 12:51:01 hosted kernel: [<c022c452>] xfs_dir2_getdents+0xb0/0xe4
Jul 18 12:51:01 hosted kernel: [<c022c537>]
xfs_dir2_put_dirent64_direct+0x0/0x8b
Jul 18 12:51:01 hosted kernel: [<c02559d4>] xfs_readdir+0x3f/0x51
Jul 18 12:51:01 hosted kernel: [<c025ea5e>] linvfs_readdir+0xfd/0x1d2
Jul 18 12:51:01 hosted kernel: [<c0165d46>] vfs_readdir+0x42/0x6f
Jul 18 12:51:01 hosted kernel: [<c0165e59>] filldir64+0x0/0xc6
Jul 18 12:51:01 hosted kernel: [<c0165f82>] sys_getdents64+0x63/0xa9
Jul 18 12:51:01 hosted kernel: [<c0165e59>] filldir64+0x0/0xc6
Jul 18 12:51:01 hosted kernel: [<c0104869>] syscall_call+0x7/0xb
I've narrowed this down to a single directory which cannot be removed:
/usr/src/linux-2.6.16-xen/sound/pci/au88x0
any attempt to get the contents of the directory results in an error and
the trace:
hosted:/usr/src/linux-2.6.16-xen/sound/pci# ls -l au88x0/
ls: reading directory au88x0/: Unknown error 990
Everything is on / so I can't easily get it unmounted to run xfs_repair.
I try booting into single user mode, mount -o remount ,ro /. xfs_repair
-d /dev/sda1 still complains about a mounted writable partition.
===
On boot, mount shows:
/dev/sda1 on / type xfs (rw)
# mount -o remount,ro /
after the remount, mount still shows / as rw
# mount
/dev/sda1 on / type xfs (rw)
but attempts to modify it result in:
hosted:~# touch foo
touch: cannot touch `foo': Read-only file system
Ok, so try xfs_repair:
hosted:~# xfs_repair -d /dev/sda1
xfs_repair: /dev/sda1 contains a mounted and writable filesystem
fatal error -- couldn't initialize XFS library
xfs_repair thinks we are writable.
I tried doing this as early as I could in the boot process by putting a
script in
/etc/rcS.d/S00fixroot
but that is not early enough.
Any thoughts?
-Rupa
|