On Tue, Aug 30, 2016 at 12:39:05PM +1000, Dave Chinner wrote:
> On Mon, Aug 29, 2016 at 06:37:54PM +0800, Eryu Guan wrote:
> > Hi,
> >
> > I've hit an XFS internal error then filesystem shutdown with 4.8-rc3
> > kernel but not with 4.8-rc2
> .....
> > I attached a script too to reproduce it. Please note that the XFS
> > partition needs about 40G frees space, and it may take hours to finish
> > based on your memory setup on your host.
>
> Ugh. can you try to narrow the cause so it takes less time to
> reproduce? This is almost certainly one of two things:
>
> 1) a ENOSPC issue where an AG is almost-but-not-quite full,
> but fixing up the freelist results in there being not enough
> blocks left to allocate the data extent; or
>
> 2) we've split a delalloc extent so many times that we've
> run out of indirect block reservation and we hit ENOSPC as a
> result.
[Sorry for getting back on this after so long time..]
>
> For the latter, I suspect a test case where we take a large delalloc
> range and use sync_file_range to do single page writeback to "binary
> split" the delalloc range. i.e. start with a 128MB delalloc, then
> sync a 4k block at offset 64MB, then 4k at 32MB, then 16MB, then
> 8MB, ... all the way down to writing the first block in the file,
> and also all the way up to the final block in the file.
>
> Then write every second 4k block to cause worse case growth of the
> bmbt and hopefully then exhaust the indirect block reservation for
> that delalloc region...
Seems it's only reproducible on certain hosts, and I haven't been able
to work out an efficient & reliable reproducer. I tried to tuned the
parameters of bash-shared-mapping run, but failed to find a efficient
parameter conbination. Also tried to write a script (attached, not sure
if it's correct) based on the second case above, but still cannot
reproduce it.
By adding debug logs, I got the following stack trace, it's
xfs_bmbt_alloc_block() returns first ENOSPC:
if (!args.wasdel && args.tp->t_blk_res == 0) {
error = -ENOSPC;
goto error0;
}
[ 783.154400] xfs_bmbt_alloc_block: set -ENOSPC and returned
[ 783.183011] __xfs_btree_split: cur->bc_ops->alloc_block returned -28
[ 783.214758] xfs_btree_split: workqueue result returned -28
[ 783.239319] xfs_btree_make_block_unfull: xfs_btree_split returned -28
[ 783.268190] xfs_btree_insrec: xfs_btree_make_block_unfull returned -28
[ 783.297550] xfs_btree_insert: xfs_btree_insrec returned -28
[ 783.322672] xfs_bmap_add_extent_hole_real: case 0 xfs_btree_insert returned
-28
[ 783.355441] xfs_bmapi_allocate: xfs_bmap_add_extent_hole_real returned -28
[ 783.386272] xfs_bmapi_write: xfs_bmapi_allocate returned -28
[ 783.411332] xfs_iomap_write_allocate: goto trans_cancel after
xfs_bmapi_write, error=-28
Not sure if this helps.
Thanks,
Eryu
|