Files appear too big in `du`

[Matthias Schniedermeyer]

On 10.05.2011 23:17, Dave Chinner wrote:
> On Tue, May 10, 2011 at 12:57:00PM +0200, Matthias Schniedermeyer wrote:
> > Hi
> > 
> > 
> > Since a few weeks i'm experiencing an annoying 'thing' where files are 
> > often too big in `du` and directory totals are to high in `ls -l`.
> > 
> > I appears that files, which are in the process of beeing 
> > copied/downloaded/whatever, grow in large chunks ahead of time, while 
> > the actual file-content is beeing copied into the files.
> 
> It's supposed to work like this. It's called speculative allocation
> beyond end of file. XFS has always done this, but we've recently
> made it more aggressive to prevent excessive fragmentation on
> concurrent large file workloads when there is lots of disk space
> free.

OK.

> > And then it 
> > appears that the last chunk isn't shrunk after the process is finished.
> 
> It should be truncated away when the file descriptor is closed and
> the last reference goes away.
>
> > Neither xfs_bmap (Version 3.1.5) nor filefrag show anything beyond the 
> > extent that compromises the actual file-content.
> 
> what is the output of xfs_bmap -vvp on a file that apparently hasn't
> been shrunk? How do you know it hasn't been shrunk? Does it persist

du

> forever in this state, or does doing something like dropping caches
> (echo 3 > /proc/sys/vm/drop_caches) cause the specualtive
> preallocation to disappear?

This works:
sync ; echo 3 > /proc/sys/vm/drop_caches

At least in several tries the `du` output shrunk to the size of the 
original.

> > Any idea how to debug this, or is this a known bug and waiting a few 
> > days for 2.6.39 should fix this?
> 
> It doesn't appear to be doing anything wrong from your description.
> Remember that XFS is optimised for high end storage and server
> configurations and workloads, not typical desktop usage...

I would call it a regression.
I reguarly follow copying/downloading with `du`, the speculative
preallocation makes that more or less useless. Especially downloading 
someting big from the internet which @ 231kb/s isn't exactly fast and 
shows identical `du`s for increasingly longer periods of time.
(Or "--apparent-size" should be made default, but that falls short with 
sparse-files)

IMHO `du`/`ls -l` should not be able to 'see' the speculative 
preallocation.




Bis denn

-- 
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.