On Fri, Mar 07, 2008 at 12:19:28PM +0100, Christian Røsnes wrote:
> > Actually, a single mkdir command is enough to trigger the filesystem
> > shutdown when its 99% full (according to df -k):
> >
> > /data# mkdir test
> > mkdir: cannot create directory `test': No space left on device
Ok, that's helpful ;)
So, can you dump the directory inode with xfs_db? i.e.
# ls -ia /data
The directory inode is the inode at ".", and if this is the root of
the filesystem it will probably be 128.
Then run:
# xfs_db -r -c 'inode 128' -c p /dev/sdb1
> > --------
> > meta-data=/dev/sdb1 isize=512 agcount=16, agsize=4476752
> > blks
> > = sectsz=512 attr=0
> > data = bsize=4096 blocks=71627792, imaxpct=25
> > = sunit=16 swidth=32 blks, unwritten=1
> > naming =version 2 bsize=4096
> > log =internal bsize=4096 blocks=32768, version=2
> > = sectsz=512 sunit=16 blks, lazy-count=0
> > realtime =none extsz=65536 blocks=0, rtextents=0
> >
> > xfs_db -r -c 'sb 0' -c p /dev/sdb1
> > ----------------------------------
.....
> > fdblocks = 847484
Apparently there are still lots of free blocks. I wonder if you are out of
space in the metadata AGs.
Can you do this for me:
-------
#!/bin/bash
for i in `seq 0 1 15`; do
echo freespace histogram for AG $i
xfs_db -r -c "freesp -bs -a $i" /dev/sdb1
done
------
> Instrumenting the code, I found that this occurs on my system when I
> do a 'mkdir /data/test' on the partition in question:
>
> in xfs_mkdir (xfs_vnodeops.c):
>
> error = xfs_dir_ialloc(&tp, dp, mode, 2,
> 0, credp, prid, resblks > 0,
> &cdp, NULL);
>
> if (error) {
> if (error == ENOSPC)
> goto error_return; <=== this is hit and then
> execution jumps to error_return
> goto abort_return;
> }
Ah - you can ignore my last email, then. You're already one step ahead
of me ;)
This does not appear to be the case I was expecting, though I can
see how we can get an ENOSPC here with plenty of blocks free - none
are large enough to allocate an inode chunk. What would be worth
knowing is the value of resblks when this error is reported.
This tends to imply we are returning an ENOSPC with a dirty
transaction. Right now I can't see how that would occur, though
the fragmented free space is something I can try to reproduce with.
> Is this the correct behavior for this type of situation: mkdir command
> fails due to no available space on filesystem,
> and xfs_mkdir goes to label error_return ? (And after this the
> filesystem is shutdown)
The filesystem should not be shutdown here. We need to trace through
further to the source of the error....
Cheers,
Dave.
--
Dave Chinner
Principal Engineer
SGI Australian Software Group
|