[Top] [All Lists]

Re: Q: about xfs pre-allocation

To: LA Walsh <xfs@xxxxxxxxx>, xfs-oss <xfs@xxxxxxxxxxx>
Subject: Re: Q: about xfs pre-allocation
From: Brian Foster <bfoster@xxxxxxxxxx>
Date: Tue, 10 Dec 2013 20:49:24 -0500
Delivered-to: xfs@xxxxxxxxxxx
In-reply-to: <52A75820.9090004@xxxxxxxxx>
References: <52A75820.9090004@xxxxxxxxx>
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.1.0
On 12/10/2013 01:06 PM, LA Walsh wrote:
> Could someone comment on my [mis-]understanding in regards to
> what the note below said that was posted by someone else
> to another list.  The pre-allocation behavior for XFS that the
> note describes doesn't jive w/what I thought happened and I
> was wondering if my brain was out of date or something (at
> least in regards to this topic! ;-)).  Names elided from
> Original Message, below for no great reason...
> I thought file space pre-allocation ended when you closed the file??

XFS normally frees post-eof preallocation on close, but there is the
possibility for the preallocated space to hang around longer if a file
is opened, written and closed repeatedly. You could observe this by
repeatedly doing an open/write/close cycle on a file with xfs_io. After
a few cycles, you'll see the blocks allocated to the file exceed the
size of the file.

> But this note from the open-suse list indicates that, at least
> with ext2, a kernel thread removes this later. 
> I thought the FS-space allocator gave *preference* to having the
> next file start at least "filesize%('allocsize || 64K')" from
> the end of the previous, BUT, if needed it will allocate space
> from the end of the previous file (rounded to fs-blocksize) if
> space is really that tight.

XFS has a background scanning thread for this purpose as of 3.8. It runs
on a 5 minute interval by default and trims the speculative
preallocation from inodes that haven't been written to recently.


> -------- Original Message --------
> Modern filesystems use preallocation.
> Per <http://ext2.sourceforge.net/2005-ols/paper-html/node6.html> ext2
> already had it by 2005.
> That means when a file is created and written to they automatically
> allocate a unused tail at the end of each file.
> Then some time later (hours / days) they have a background kernel
> thread that scavenges any tails that are still unused.
> The positive is that files (like logs) growing slowly over time won't
> get fragmented so badly.
> The bad is that for highspeed filesystem filling tasks like a massive
> rsync, the disk usage is anomalously high for a while (hours / days).
> With XFS you can disable pre-allocation via the allocsize mount
> parameter.  (That parameter has been around many years. so yes 11.4
> has preallocation for XFS at a minimum and ext3/ext4 I think.)
> allocsize=size
> Sets the buffered I/O end-of-file preallocation size when doing
> delayed allocation writeout (default size is 64KiB). Valid values for
> this option are page size (typically 4KiB) through to 1GiB, inclusive,
> in power-of-2 increments.
> size = 0 disables preallocation and is probably smart on your
> distination backup disk.
> I assume other filesystems have a way to disable preallocation as well.
> FYI: I don't know how to determine the total amount of preallocation
> space on a filesystem.  I'm sure it can be done somehow.
> gggg
> ----
> To unsubscribe, e-mail: opensuse+unsubscribe@xxxxxxxxxxxx
> To contact the owner, e-mail: opensuse+owner@xxxxxxxxxxxx
> _______________________________________________
> xfs mailing list
> xfs@xxxxxxxxxxx
> http://oss.sgi.com/mailman/listinfo/xfs

<Prev in Thread] Current Thread [Next in Thread>