On 10.05.2011 23:17, Dave Chinner wrote:
> On Tue, May 10, 2011 at 12:57:00PM +0200, Matthias Schniedermeyer wrote:
> > Hi
> > Since a few weeks i'm experiencing an annoying 'thing' where files are
> > often too big in `du` and directory totals are to high in `ls -l`.
> > I appears that files, which are in the process of beeing
> > copied/downloaded/whatever, grow in large chunks ahead of time, while
> > the actual file-content is beeing copied into the files.
> It's supposed to work like this. It's called speculative allocation
> beyond end of file. XFS has always done this, but we've recently
> made it more aggressive to prevent excessive fragmentation on
> concurrent large file workloads when there is lots of disk space
> > And then it
> > appears that the last chunk isn't shrunk after the process is finished.
> It should be truncated away when the file descriptor is closed and
> the last reference goes away.
> > Neither xfs_bmap (Version 3.1.5) nor filefrag show anything beyond the
> > extent that compromises the actual file-content.
> what is the output of xfs_bmap -vvp on a file that apparently hasn't
> been shrunk? How do you know it hasn't been shrunk? Does it persist
> forever in this state, or does doing something like dropping caches
> (echo 3 > /proc/sys/vm/drop_caches) cause the specualtive
> preallocation to disappear?
sync ; echo 3 > /proc/sys/vm/drop_caches
At least in several tries the `du` output shrunk to the size of the
> > Any idea how to debug this, or is this a known bug and waiting a few
> > days for 2.6.39 should fix this?
> It doesn't appear to be doing anything wrong from your description.
> Remember that XFS is optimised for high end storage and server
> configurations and workloads, not typical desktop usage...
I would call it a regression.
I reguarly follow copying/downloading with `du`, the speculative
preallocation makes that more or less useless. Especially downloading
someting big from the internet which @ 231kb/s isn't exactly fast and
shows identical `du`s for increasingly longer periods of time.
(Or "--apparent-size" should be made default, but that falls short with
IMHO `du`/`ls -l` should not be able to 'see' the speculative
Real Programmers consider "what you see is what you get" to be just as
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated,
cryptic, powerful, unforgiving, dangerous.