128TB filesystem limit?

[david at lang.hm]

On Fri, 26 Mar 2010, Dave Chinner wrote:

> On Thu, Mar 25, 2010 at 04:15:42PM -0700, david at lang.hm wrote:
>> I'm working with a raid 0 (md) array on top of 10 16x1TB raid 6
>> hardware arrays.
>>
>> fdisk -l shows me 10 drives like
>>
>> WARNING: GPT (GUID Partition Table) detected on '/dev/sdk'! The util
>> fdisk doesn't support GPT. Use GNU Parted.
>>
>>
>> Disk /dev/sdk: 13999.9 GB, 13999999025152 bytes
>> 255 heads, 63 sectors/track, 1702069 cylinders
>> Units = cylinders of 16065 * 512 = 8225280 bytes
>> Disk identifier: 0x00000000
>>
>>    Device Boot      Start         End      Blocks   Id  System
>> /dev/sdk1               1      267350  2147483647+  ee  EFI GPT
>>
>> and then the md0 device as
>>
>> Disk /dev/md0: 139999.9 GB, 139999989596160 bytes
>> 2 heads, 4 sectors/track, -1 cylinders
>> Units = cylinders of 8 * 512 = 4096 bytes
>> Disk identifier: 0x00000000
>>
>> Disk /dev/md0 doesn't contain a valid partition table
>>
>>
>> I then did mkfs.xfs /dev/md0
>>
>> but a df is showing me 128TB
>
> What is in /proc/partitions?

# cat /proc/partitions
major minor  #blocks  name

    8        0  292542464 sda
    8        1    2048287 sda1
    8        2    2048287 sda2
    8        3    2048287 sda3
    8        4  286390755 sda4
    8       16 13671874048 sdb
    8       17 13671874014 sdb1
    8       32 13671874048 sdc
    8       33 13671874014 sdc1
    8       48 13671874048 sdd
    8       49 13671874014 sdd1
    8       64 13671874048 sde
    8       65 13671874014 sde1
    8       80 13671874048 sdf
    8       81 13671874014 sdf1
    8       96 13671874048 sdg
    8       97 13671874014 sdg1
    8      112 13671874048 sdh
    8      113 13671874014 sdh1
    8      128 13671874048 sdi
    8      129 13671874014 sdi1
    8      144 13671874048 sdj
    8      145 13671874014 sdj1
    8      160 13671874048 sdk
    8      161 13671874014 sdk1
    9        0 136718739840 md0

>> is this just rounding error combined with the 1000=1k vs 1024=1k
>> marketing stuff,
>
> Probably.
>
>> or is there some limit I am bumping into here.
>
> Unlikely to be an XFS limit - I was doing some "what happens if"
> testing on multi-PB sized XFS filesystems hosted on sparse files
> a couple of days ago....

Ok, 128TB is a suspiciously round (in computer terms) number, especially 
when the math is 10 sets of 14 drives (each 1TB), so I figured I'd double 
check.

David Lang