[Top] [All Lists]

Re: Files appear too big in `du`

To: Dave Chinner <david@xxxxxxxxxxxxx>
Subject: Re: Files appear too big in `du`
From: Matthias Schniedermeyer <ms@xxxxxxx>
Date: Tue, 10 May 2011 17:33:00 +0200
Cc: xfs@xxxxxxxxxxx
In-reply-to: <20110510131705.GE19446@dastard>
References: <20110510105700.GA20307@xxxxxxx> <20110510131705.GE19446@dastard>
User-agent: Mutt/1.5.21 (2010-09-15)
On 10.05.2011 23:17, Dave Chinner wrote:
> On Tue, May 10, 2011 at 12:57:00PM +0200, Matthias Schniedermeyer wrote:
> > Hi
> > 
> > 
> > Since a few weeks i'm experiencing an annoying 'thing' where files are 
> > often too big in `du` and directory totals are to high in `ls -l`.
> > 
> > I appears that files, which are in the process of beeing 
> > copied/downloaded/whatever, grow in large chunks ahead of time, while 
> > the actual file-content is beeing copied into the files.
> It's supposed to work like this. It's called speculative allocation
> beyond end of file. XFS has always done this, but we've recently
> made it more aggressive to prevent excessive fragmentation on
> concurrent large file workloads when there is lots of disk space
> free.


> > And then it 
> > appears that the last chunk isn't shrunk after the process is finished.
> It should be truncated away when the file descriptor is closed and
> the last reference goes away.
> > Neither xfs_bmap (Version 3.1.5) nor filefrag show anything beyond the 
> > extent that compromises the actual file-content.
> what is the output of xfs_bmap -vvp on a file that apparently hasn't
> been shrunk? How do you know it hasn't been shrunk? Does it persist


> forever in this state, or does doing something like dropping caches
> (echo 3 > /proc/sys/vm/drop_caches) cause the specualtive
> preallocation to disappear?

This works:
sync ; echo 3 > /proc/sys/vm/drop_caches

At least in several tries the `du` output shrunk to the size of the 

> > Any idea how to debug this, or is this a known bug and waiting a few 
> > days for 2.6.39 should fix this?
> It doesn't appear to be doing anything wrong from your description.
> Remember that XFS is optimised for high end storage and server
> configurations and workloads, not typical desktop usage...

I would call it a regression.
I reguarly follow copying/downloading with `du`, the speculative
preallocation makes that more or less useless. Especially downloading 
someting big from the internet which @ 231kb/s isn't exactly fast and 
shows identical `du`s for increasingly longer periods of time.
(Or "--apparent-size" should be made default, but that falls short with 

IMHO `du`/`ls -l` should not be able to 'see' the speculative 

Bis denn

Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.

<Prev in Thread] Current Thread [Next in Thread>