[Top] [All Lists]

Re: Insane file system overhead on large volume

To: Martin Steigerwald <Martin@xxxxxxxxxxxx>
Subject: Re: Insane file system overhead on large volume
From: Eric Sandeen <sandeen@xxxxxxxxxxx>
Date: Sat, 28 Jan 2012 09:35:14 -0600
Cc: xfs@xxxxxxxxxxx, Manny <dermaniac@xxxxxxxxx>
In-reply-to: <201201281555.22179.Martin@xxxxxxxxxxxx>
References: <CAEBWcAT2zfDskgDjFr0KcnfsT2A65r04AM1cv2-TfnNJTB1__Q@xxxxxxxxxxxxxx> <4F22EB3C.6020106@xxxxxxxxxxx> (sfid-20120127_225414_630806_3564B30F) <201201281555.22179.Martin@xxxxxxxxxxxx>
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0) Gecko/20111222 Thunderbird/9.0.1
On 1/28/12 8:55 AM, Martin Steigerwald wrote:
> Am Freitag, 27. Januar 2012 schrieb Eric Sandeen:
>> On 1/27/12 1:50 AM, Manny wrote:
>>> Hi there,
>>> I'm not sure if this is intended behavior, but I was a bit stumped
>>> when I formatted a 30TB volume (12x3TB minus 2x3TB for parity in RAID
>>> 6) with XFS and noticed that there were only 22 TB left. I just
>>> called mkfs.xfs with default parameters - except for swith and sunit
>>> which match the RAID setup.
>>> Is it normal that I lost 8TB just for the file system? That's almost
>>> 30% of the volume. Should I set the block size higher? Or should I
>>> increase the number of allocation groups? Would that make a
>>> difference? Whats the preferred method for handling such large
>>> volumes?
>> If it was 12x3TB I imagine you're confusing TB with TiB, so
>> perhaps your 30T is really only 27TiB to start with.
>> Anyway, fs metadata should not eat much space:
>> # mkfs.xfs -dfile,name=fsfile,size=30t
>> # ls -lh fsfile
>> -rw-r--r-- 1 root root 30T Jan 27 12:18 fsfile
>> # mount -o loop fsfile  mnt/
>> # df -h mnt
>> Filesystem            Size  Used Avail Use% Mounted on
>> /tmp/fsfile            30T  5.0M   30T   1% /tmp/mnt
>> So Christoph's question was a good one; where are you getting
>> your sizes?

To solve your original problem, can you answer the above question?
Adding your actual raid config output (/proc/mdstat maybe) would help

> An academic question:
> Why is it that I get
> merkaba:/tmp> mkfs.xfs -dfile,name=fsfile,size=30t
> meta-data=fsfile                 isize=256    agcount=30, agsize=268435455 
> blks
>          =                       sectsz=512   attr=2, projid32bit=0
> data     =                       bsize=4096   blocks=8053063650, imaxpct=5
>          =                       sunit=0      swidth=0 blks
> naming   =version 2              bsize=4096   ascii-ci=0
> log      =Internes Protokoll     bsize=4096   blocks=521728, version=2
>          =                       sectsz=512   sunit=0 blks, lazy-count=1
> realtime =keine                  extsz=4096   blocks=0, rtextents=0
> merkaba:/tmp> mount -o loop fsfile /mnt/zeit
> merkaba:/tmp> df -hT /mnt/zeit
> Dateisystem    Typ  Größe Benutzt Verf. Verw% Eingehängt auf
> /dev/loop0     xfs    30T     33M   30T    1% /mnt/zeit
> merkaba:/tmp> LANG=C df -hT /mnt/zeit
> Filesystem     Type  Size  Used Avail Use% Mounted on
> /dev/loop0     xfs    30T   33M   30T   1% /mnt/zeit
> 33MiB used on first mount instead of 5?

Not sure offhand, differences in xfsprogs version mkfs defaults perhaps.


> Hmmm, but creating the file on Ext4 does not work:

ext4 is not designed to handle very large files, so anything
above 16T will fail.

> fallocate instead of sparse file?

no, you just ran into file offset limits on ext4.
> And on BTRFS as well as XFS it appears to try to create a 30T file for 
> real, i.e. by writing data - I stopped it before it could do too much 
> harm.

Why do you say that it appears to create a 30T file for real?  It
should not...
> Where did you create that hugish XFS file?

On XFS.  Of course.  :)

> Ciao,

<Prev in Thread] Current Thread [Next in Thread>