XFS over LVM over md RAID

[Richard Scobie]

Hi Dave,

Dave Chinner wrote:
> On Fri, Sep 10, 2010 at 10:58:22AM +1200, Richard Scobie wrote:
>> Using the latest, stable versions of LVM2 and xfsprogs and the
>> 2.6.35.4 kernel, I am setting up lvm on a 16 drive, 256k chunk md
>> RAID6, which has been used to date with XFS directly on the RAID.
>>
>> mkfs.xfs directly on the RAID gives:
>>
>> meta-data=/dev/md8               isize=256    agcount=32,
>> agsize=106814656 blks
>>           =                       sectsz=4096  attr=2
>> data     =                       bsize=4096   blocks=3418068864, imaxpct=5
>>           =                       sunit=64     swidth=896 blks
>> naming   =version 2              bsize=4096   ascii-ci=0
>>
>> which gives the correct sunit and swidth values for the array.
>>
>> Creating an lv which uses the entire array and mkfs.xfs on that, gives:
>>
>> meta-data=/dev/vg_local/Storage  isize=256    agcount=13,
>> agsize=268435455 blks
>>           =                       sectsz=512   attr=2
>> data     =                       bsize=4096   blocks=3418067968, imaxpct=5
>>           =                       sunit=0      swidth=0 blks
>> naming   =version 2              bsize=4096   ascii-ci=0
>
> Hmmm - it's treating MD very differently to the LVM volume -
> different numbers of AGs, different sunit/swdith. Did you
> build xfsprogs yourself? Is it linked against libblkid or libdisk?

I should clarify - the first set was created with xfsprogs 3.0.0 and 
second was done with xfsprogs 3.1.3, so I wondered if the default ag 
count had changed.

Given a 12TB array, would I be better using 32?

I did build 3.1.3. Is libblkid preferable?  I note it is defaulted off 
in configure and I used the default configuration.

l appear to have e2fsprogs and devel packages installed (Fedora 11), for 
libblkid, but when I enable and try to build, it fails:

     [CC]     xfs_mkfs.o
xfs_mkfs.c: In function ‘check_overwrite’:
xfs_mkfs.c:298: error: ‘blkid_probe’ undeclared (first use in this function)
xfs_mkfs.c:298: error: (Each undeclared identifier is reported only once
xfs_mkfs.c:298: error: for each function it appears in.)
xfs_mkfs.c:298: error: expected ‘;’ before ‘pr’
xfs_mkfs.c:321: error: ‘pr’ undeclared (first use in this function)
xfs_mkfs.c:321: warning: implicit declaration of function 
‘blkid_new_probe_from_filename’
xfs_mkfs.c:325: warning: implicit declaration of function 
‘blkid_probe_enable_partitions’
xfs_mkfs.c:329: warning: implicit declaration of function 
‘blkid_do_fullprobe’
xfs_mkfs.c:345: warning: implicit declaration of function 
‘blkid_probe_lookup_value’
xfs_mkfs.c:362: warning: implicit declaration of function ‘blkid_free_probe’
xfs_mkfs.c: In function ‘blkid_get_topology’:
xfs_mkfs.c:372: error: ‘blkid_topology’ undeclared (first use in this 
function)
xfs_mkfs.c:372: error: expected ‘;’ before ‘tp’
xfs_mkfs.c:373: error: ‘blkid_probe’ undeclared (first use in this function)
xfs_mkfs.c:373: error: expected ‘;’ before ‘pr’
xfs_mkfs.c:381: error: ‘pr’ undeclared (first use in this function)
xfs_mkfs.c:385: error: ‘tp’ undeclared (first use in this function)
xfs_mkfs.c:385: warning: implicit declaration of function 
‘blkid_probe_get_topology’
xfs_mkfs.c:397: warning: implicit declaration of function 
‘blkid_topology_get_minimum_io_size’
xfs_mkfs.c:400: warning: implicit declaration of function 
‘blkid_topology_get_optimal_io_size’
xfs_mkfs.c:403: warning: implicit declaration of function 
‘blkid_probe_get_sectorsize’
xfs_mkfs.c:406: warning: implicit declaration of function 
‘blkid_topology_get_alignment_offset’
gmake[2]: *** [xfs_mkfs.o] Error 1
gmake[1]: *** [mkfs] Error 2
make: *** [default] Error 2


> Or it might be that LVM is not exporting the characteristic of the
> underlying volume. Can you check if there are different parameter
> values exported by the two devices in /sys/block/<dev>/queue?

They look the same.

>> Limited testing using dd and bonnie++ shows no difference in write
>> performance whether I use sunit=64/swidth=896 or sunit=0/swidth=0 on
>> the lv.
>
> These benchmarks won't realy show any difference on an empty
> filesystem. It will have an impact on how the filesystems age and
> how well aligned the IO will be to the underlying device under more
> complex workloads...

I figured I'd go with the geometry specified.

Regards,

Richard