> on EXT2, EXT3 and XFS filesystems. For the EXT2 and EXT3 filesystems the
> tests went okay. But I got stack trace on XFS filesystem and the machine
> went down.
Fantastic, thanks for running these tests.
> kernel BUG at kernel/workqueue.c:113!
> EIP is at queue_work+0x86/0x90
We were able to set the pending bit but then found that list_empty()
failed on the work queue's entry list_head. Let's call this memory
corruption of some kind.
> [<c02b43a2>] xfs_finish_ioend+0x20/0x22
> [<c02b5e2f>] xfs_end_io_direct+0x3c/0x68
> [<c018e77a>] dio_complete+0xe3/0xfe
> [<c018e82d>] dio_bio_end_aio+0x98/0xb1
> [<c016e889>] bio_endio+0x4e/0x78
> [<c02cdc89>] __end_that_request_first+0xcd/0x416
It was completing an AIO request.
ret = blockdev_direct_IO_own_locking(rw, iocb, inode,
iomap.iomap_target->bt_bdev,
iov, offset, nr_segs,
xfs_get_blocks_direct,
xfs_end_io_direct);
if (unlikely(ret <= 0 && iocb->private))
xfs_destroy_ioend(iocb->private);
It looks like xfs_vm_direct_io() is destroying the ioend in the case
where direct IO is returning -EIOCBQUEUED. Later the AIO will complete
and try to call queue_work on the freed ioend. This wasn't a problem
before when blkdev_direct_IO_*() would just return the number of bytes
in the op that was in flight. That test should be
if (unlikely(ret != -EIOCBQUEUED && iocb->private))
I'll update the patch set and send it out.
This makes me worry that XFS might have other paths that need to know
about the magical -EIOCBQUEUED case which actually means that a AIO DIO
is in flight.
Could I coerce some XFS guys into investigating if we might have other
problems with trying to bubble -EIOCBQUEUED up from
blockdev_direct_IO_own_locking() up through to xfs_file_aio_write()'s
caller before calling xfs_end_io_direct()?
- z
|