xfs
[Top] [All Lists]

Re: Infinite loop in xfssyncd on full file system

To: David Chinner <dgc@xxxxxxx>, Luciano Chavez <lnx1138@xxxxxxxxxx>
Subject: Re: Infinite loop in xfssyncd on full file system
From: Stephane Doyon <sdoyon@xxxxxxxxx>
Date: Tue, 29 Aug 2006 09:25:47 -0400 (EDT)
Cc: linux-xfs@xxxxxxxxxxx
In-reply-to: <1156794030.5848.3.camel@localhost>
References: <Pine.LNX.4.64.0608221318300.3139@xxxxxxxxxxxxxxxxxxxxx> <20060823040218.GC807872@xxxxxxxxxxxxxxxxx> <20060823044829.GD807872@xxxxxxxxxxxxxxxxx> <Pine.LNX.4.64.0608231056370.3139@xxxxxxxxxxxxxxxxxxxxx> <1156360259.5368.7.camel@localhost> <Pine.LNX.4.64.0608221318300.3139@xxxxxxxxxxxxxxxxxxxxx> <20060823040218.GC807872@xxxxxxxxxxxxxxxxx> <20060823044829.GD807872@xxxxxxxxxxxxxxxxx> <Pine.LNX.4.64.0608231056370.3139@xxxxxxxxxxxxxxxxxxxxx> <20060823231429.GF807872@xxxxxxxxxxxxxxxxx> <20060828072343.GJ807872@xxxxxxxxxxxxxxxxx> <1156794030.5848.3.camel@localhost>
Sender: xfs-bounce@xxxxxxxxxxx
On Mon, 28 Aug 2006, Luciano Chavez wrote:

On Mon, 2006-08-28 at 17:23 +1000, David Chinner wrote:
On Thu, Aug 24, 2006 at 09:14:29AM +1000, David Chinner wrote:
On Wed, Aug 23, 2006 at 11:00:43AM -0400, Stephane Doyon wrote:
On Wed, 23 Aug 2006, David Chinner wrote:

On Wed, Aug 23, 2006 at 02:02:18PM +1000, David Chinner wrote:
On Tue, Aug 22, 2006 at 04:01:10PM -0400, Stephane Doyon wrote:
I'm seeing what appears to be an infinite loop in xfssyncd. It is
triggered when writing to a file system that is full or nearly full. I
have pinpointed the change that introduced this problem: it's

   "TAKE 947395 - Fixing potential deadlock in space allocation and
   freeing due to ENOSPC"

git commit d210a28cd851082cec9b282443f8cc0e6fc09830.

.....

Now we know what patch introduces the problem, we know where to look.
Stay tuned...

I've had a quick look at the above commit. I'm not yet certain that
everything is correct in terms of the semantics laid down in the
change or that enough blocks are reserved for btree splits , but I

I actually tried, naively, to bump up SET_ASIDE_BLOCKS from 8 to 32. I
won't claim to understand half of what's going on but I wondered whether
that might make the problem noticeably harder to reproduce at least, but
it had no effect ;-).

That was going to be my next question. ;)

At least that rules out a small error in the block reservation decision,
so I'm going to have  analyse all the code paths the mod introduced
and work out what is going wrong.

You know, if you had of buumped it up just a bit higher, the problem might
have gone away. With a fielsystem that only has 8 AGs in it, if you bumped
it to 33, then problem would have disappeared....

What we have here is a small error in the block reservation code. Basically,
all the logic is correct except for one critical detail - while we need to
reserve 4 blocks for the AG freelist so a minimum allocation can succeed,
we need to reserve 4 blocks in _every AG_ so that when every AG is empty
we will fail with ENOSPC instead of trying to allocate a block when we
have an AG with less thaan 4 free blocks in it.

So, it's not 4 blocks filesystem wide we need to reserve, it's 4 blocks per AG
we need to reserve.

Stephane and Luciano, can you try the patch attæched below - it fixes the
100% repeatable test case (while [ 1 ]; dd to enospc; done) on my test
machine.


Dave,

The latest patch seems to work for me running bonnie++ on a small 2GB
XFS filesystem. bonnie++ gets an ENOSPC on a write() and ends plus I
don't see the softwatchdog timer dump the kernel stack or xfssyncd
looping. Thanks!

Works here too. Tried on three different file system configurations (sizes and numbers of AGs). I ran only simple tests though, but at least it fixes the obvious test case. Ideally I'd like to run some stress test involving NFS service, but I won't be able to make time for that in the very near future.

Thanks!
<Prev in Thread] Current Thread [Next in Thread>