Thank you both for informative replies, much appreciated.<br><br>I ended up doing the tedious job of shuffling data around to get a more reasonable distance to ENOSPC, as there was no way I could free up the needed contiguous free space with my dataset. <div>
<br></div><div><br></div><div>Thanks,</div><div><br></div><div>André<br><br><div class="gmail_quote">On 20 June 2012 08:07, Dave Chinner <span dir="ltr"><<a href="mailto:david@fromorbit.com" target="_blank">david@fromorbit.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Tue, Jun 19, 2012 at 07:36:24AM -0500, Geoffrey Wehrman wrote:<br>
> On Tue, Jun 19, 2012 at 02:05:34PM +0200, André Øien Langvand wrote:<br>
> | Hi,<br>
> |<br>
> | I know there are quite a few posts regarding similar issues around, but I<br>
> | can't seem to find a solution or at least an answer to why this is<br>
> | happening in my case, so I thought I'd try the mailing list and I hope<br>
> | that's okay.<br>
> |<br>
> | We have 2 file servers with identical hardware and identical configuration<br>
> | (Dell R610's, H800 controllers, MD1200 DAS, RAID-5) set up with rsync to<br>
> | mirror the contents. The content is music in several formats (from PCM WAV<br>
> | to 64kbit AAC previews), which means file sizes of about 1 - 40mb. Both<br>
> | systems running SLES 11 SP1. Same kernel (2.6.32.59-0.3-default), same<br>
> | xfsprogs version (xfsprogs-3.1.1-0.1.36).<br>
> |<br>
> | My example partition on the source now has 9.9G (of 9.1T) available space<br>
> | and still doesn't report the drive as full. On the destination, however, it<br>
> | wont allow me to use any of the remaining 51G. This is obviously a problem<br>
> | when trying to do mirroring.<br>
> |<br>
> | Both file systems have been mounted with inode64 option since first mount,<br>
> | there are plenty of inodes available and I've also verified that there are<br>
> | noe sparse files (find -type f -printf "%S\t%p\n" 2>/dev/null | gawk '{if<br>
> | ($1 < 1.0) print $1 $2}'), just in case.<br>
> |<br>
> | I have tried repairing (xfs_repair), defragmenting (xfs_fsr) and alter<br>
> | imaxpct without any luck. Rsync is run like this: # ionice -c3 rsync -rv<br>
> | --size-only --progress --delete-before --inplace.<br>
> |<br>
> |<br>
> | More detailed information on source file system:<br>
> |<br>
> | # df -k | grep sdg1<br>
> | /dev/sdg1 9762777052 9752457156 10319896 100% /content/raid31<br>
> |<br>
> | # df -i | grep sdg1<br>
> | /dev/sdg1 7471884 2311914 5159970 31% /content/raid31<br>
> |<br>
> | # xfs_info /dev/sdg1<br>
> | meta-data=/dev/sdg1 isize=2048 agcount=10, agsize=268435424<br>
> | blks<br>
> | = sectsz=512 attr=2<br>
> | data = bsize=4096 blocks=2441215991, imaxpct=5<br>
> | = sunit=16 swidth=80 blks<br>
> | naming =version 2 bsize=4096 ascii-ci=0<br>
> | log =internal bsize=4096 blocks=521728, version=2<br>
> | = sectsz=512 sunit=16 blks, lazy-count=1<br>
> | realtime =none extsz=4096 blocks=0, rtextents=0<br>
> |<br>
> | # xfs_db -r "-c freesp -s" /dev/sdg1<br>
> | from to extents blocks pct<br>
> | 1 1 69981 69981 2.99<br>
> | 2 3 246574 559149 23.86<br>
> | 4 7 315038 1707929 72.88<br>
> | 8 15 <a href="tel:561%20%20%20%206374%20%20%200.27" value="+15616374027">561 6374 0.27</a><br>
> | total free extents 632154<br>
> | total free blocks 2343433<br>
> | average free extent size 3.70706<br>
> |<br>
> |<br>
> |<br>
> | More detailed information on destination file system:<br>
> |<br>
> | # df -k | grep sdj1<br>
> | /dev/sdj1 9762777052 9710148076 52628976 100% /content/sg08/vd08<br>
> |<br>
> | # df -i | grep sdj1<br>
> | /dev/sdj1 28622264 2307776 26314488 9% /content/sg08/vd08<br>
> |<br>
> | # xfs_info /dev/sdj1<br>
> | meta-data=/dev/sdj1 isize=2048 agcount=10, agsize=268435424<br>
> | blks<br>
> | = sectsz=512 attr=2<br>
> | data = bsize=4096 blocks=2441215991, imaxpct=5<br>
> | = sunit=16 swidth=80 blks<br>
> | naming =version 2 bsize=4096 ascii-ci=0<br>
> | log =internal bsize=4096 blocks=521728, version=2<br>
> | = sectsz=512 sunit=16 blks, lazy-count=1<br>
> | realtime =none extsz=4096 blocks=0, rtextents=0<br>
> |<br>
> | # xfs_db -r "-c freesp -s" /dev/sdj1<br>
> | from to extents blocks pct<br>
> | 1 1 81761 81761 0.62<br>
> | 2 3 530258 1147719 8.73<br>
> | 4 7 675864 3551039 27.01<br>
> | 8 15 743089 8363043 63.62<br>
> | 16 31 102 1972 0.02<br>
> | total free extents 2031074<br>
> | total free blocks 13145534<br>
> | average free extent size 6.47221<br>
> |<br>
> |<br>
> | I would be grateful if anyone could shed some light on why this is<br>
> | happening or maybe even provide a solution.<br>
><br>
> You are using 2 KiB inodes, so an inode cluster (64 inodes) requires<br>
> 128 KiB of contiguous space on disk. The freesp output above shows that<br>
> the largest possible contiguous free space chunk available is 31 * 4 KiB<br>
> or 4 KiB short of 128 KiB. You don't have enough contiguous space to<br>
> create a new inode cluster, and your existing inodes are likely all<br>
> used. This can be verified using xfs_db:<br>
> xfs_db -r -c "sb" -c "p ifree" /dev/sdj1<br>
><br>
> xfs_fsr does not defragment free space, it only makes the problem worse.<br>
> A possible solution:<br>
> 1. mount the filesystem with the ikeep mount option<br>
> 2. delete a few large files to free up some contiguous space<br>
<br>
</div></div>large -contiguous- files. It's likely any files written recently<br>
will be as fragmented as the free space....<br>
<div class="im"><br>
> 3. create a few thousand files to "preallocate" inodes<br>
> 4. delete the newly created files<br>
<br>
</div>That will work for a while, but it's really just a temporary<br>
workaround until those "preallocated" inodes are exhausted. Normally<br>
to recover from this situation you need to free 15-20% of the disk<br>
space to allow sufficiently large contiguous free space extents to<br>
reform naturally and allow the allocator to work at full efficiency<br>
again....<br>
<div class="im"><br>
> The ikeep mount option will prevent the space for inodes from being<br>
> reused for other purposes.<br>
<br>
</div>The problem with using ikeep is that the remaining empty inode<br>
chunks prevent free space from defragmenting itself fully as you<br>
remove files from the filesystem.<br>
<br>
Realistically, I think the problem is that you are running your<br>
filesystems at near ENOSPC for extended periods of time. That is<br>
guaranteed to fragment free space and any files that are written<br>
when the filesytem is in this condition. As Geoffrey has said -<br>
xfs_fsr will not fix your problems - only changing the way you use<br>
your storage will prevent the problem from occurring again.<br>
<br>
Cheers,<br>
<br>
Dave.<br>
<span class="HOEnZb"><font color="#888888">--<br>
Dave Chinner<br>
<a href="mailto:david@fromorbit.com">david@fromorbit.com</a><br>
</font></span></blockquote></div><br></div>