On Mon, Oct 23, 2006 at 01:03:30PM -0700, bugzilla-daemon@xxxxxxxxxxx wrote:
> [root@link-07 tmp]# mkfs.xfs -f -dfile,name=fsfile,size=100m
So a 100MB filesystem, and you are....
> copy a bunch of 4k files to it in a loop, things start failing....
What i see when things start failing:
budgie:/mnt/scratch # xfs_db -r -c "freesp -s" -f /mnt/scratch/fsfile
from to extents blocks pct
1 1 26 26 0.20
1024 2047 1 1973 15.10
2048 4096 3 11064 84.70
total free extents 30
total free blocks 13063
average free extent size 435.433
And then for each AG:
udgie:/mnt/scratch # xfs_db -r -c "freesp -s -a 0" -f /mnt/scratch/fsfile
from to extents blocks pct
1 1 5 5 100.00
total free extents 5
total free blocks 5
average free extent size 1
budgie:/mnt/scratch # xfs_db -r -c "freesp -s -a 1" -f /mnt/scratch/fsfile
from to extents blocks pct
1 1 5 5 100.00
total free extents 5
total free blocks 5
average free extent size 1
budgie:/mnt/scratch # xfs_db -r -c "freesp -s -a 2" -f /mnt/scratch/fsfile
from to extents blocks pct
1 1 4 4 0.20
1024 2047 1 1973 99.80
total free extents 5
total free blocks 1977
average free extent size 395.4
budgie:/mnt/scratch # xfs_db -r -c "freesp -s -a 3" -f /mnt/scratch/fsfile
from to extents blocks pct
1 1 4 4 0.14
2048 4096 1 2888 99.86
total free extents 5
total free blocks 2892
average free extent size 578.4
budgie:/mnt/scratch # xfs_db -r -c "freesp -s -a 4" -f /mnt/scratch/fsfile
from to extents blocks pct
1 1 4 4 0.10
2048 4096 1 4088 99.90
total free extents 5
total free blocks 4092
average free extent size 818.4
budgie:/mnt/scratch # xfs_db -r -c "freesp -s -a 5" -f /mnt/scratch/fsfile
from to extents blocks pct
1 1 4 4 0.10
2048 4096 1 4088 99.90
total free extents 5
total free blocks 4092
average free extent size 818.4
So, in AG 0 and 1 we have one free 4k block we can use for allocation. In the
other AGs, we have a single extent remaining with 4 free single block extents.
They are full (might be able to allocate one more block out of them).
The other AGs have a single large(-ish) extent plus 4 smaller extents.
So we definitely have space available....
> [root@link-07 tmp]# for I in `seq 1 20000`; do cp 4kfile mnt/$I; done
> ...
> cp: cannot create regular file `mnt/5186': No space left on device
> cp: cannot create regular file `mnt/5187': No space left on device
> cp: cannot create regular file `mnt/5188': No space left on device
So that's an inode allocation failing (needs 16k to allocate an
inode chunk of 256 byte inodes)....
> cp: writing `mnt/5869': No space left on device
> cp: writing `mnt/5870': No space left on device
And that's a data write failing....
I just reran the copy over the top of the previous one, and it
copied more data into the filesystem (without an umount/remount)
and left the state:
budgie:/mnt/scratch # for i in 0 1 2 3 4 5; do xfs_db -r -c "freesp -a $i" -f
/mnt/scratch/fsfile; done
from to extents blocks pct
1 1 5 5 100.00
from to extents blocks pct
1 1 5 5 100.00
from to extents blocks pct
1 1 5 5 100.00
from to extents blocks pct
1 1 4 4 0.59
512 1023 1 670 99.41
from to extents blocks pct
1 1 4 4 0.10
2048 4096 1 4088 99.90
from to extents blocks pct
1 1 4 4 0.10
2048 4096 1 4088 99.90
So we filled up AG 2 this time before failing.....
And a third time:
udgie:/mnt/scratch # for i in 0 1 2 3 4 5; do xfs_db -r -c "freesp -a $i" -f
/mnt/scratch/fsfile; done
from to extents blocks pct
1 1 5 5 100.00
from to extents blocks pct
1 1 5 5 100.00
from to extents blocks pct
1 1 5 5 100.00
from to extents blocks pct
1 1 5 5 100.00
from to extents blocks pct
1 1 4 4 0.21
1024 2047 1 1914 99.79
from to extents blocks pct
1 1 4 4 0.10
2048 4096 1 4088 99.90
We've filled up AG 3 now. This is almost like we are failing to
search all the AGs for space. Strange.....
Interesting thing to note:
budgie:/mnt/scratch # for i in 0 1 2 3 4 5; do xfs_db -r -c "agi $i" -c "p
freecount" -f /mnt/scratch/fsfile; done
freecount = 0
freecount = 0
freecount = 0
freecount = 0
freecount = 0
freecount = 0
There's no free inodes anywhere.
budgie:/mnt/scratch # echo alfred > /mnt/data/alfred
budgie:/mnt/scratch # for i in 0 1 2 3 4 5; do xfs_db -r -c "agi $i" -c "p
freecount" -f /mnt/scratch/fsfile; done
freecount = 0
freecount = 0
freecount = 0
freecount = 0
freecount = 63
freecount = 0
but we can still allocate new inodes in the AGs that are not full.
So we have some transient condition that is causing ENOSPC errors
and it seems to be related to async (delalloc) writes - mounting
-o loop,sync makes the problem go away.
> This probably isn't behaving as expected...? I suppose I should test it on
> non-loopback too.
Not as expected and I doubt it has anything to do with loopback
mounts. Thanks for the bug report, Eric - I'll keep looking into
this...
Cheers,
Dave.
--
Dave Chinner
Principal Engineer
SGI Australian Software Group
|