xfs
[Top] [All Lists]

Re: XFS over LVM over md RAID

To: Dave Chinner <david@xxxxxxxxxxxxx>
Subject: Re: XFS over LVM over md RAID
From: Richard Scobie <richard@xxxxxxxxxxx>
Date: Fri, 10 Sep 2010 14:29:42 +1200
Cc: xfs@xxxxxxxxxxx
In-reply-to: <20100910013026.GA24409@dastard>
References: <4C89668E.6010800@xxxxxxxxxxx> <20100910013026.GA24409@dastard>
User-agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100317 Lightning/1.0b1 SeaMonkey/2.0.4
Hi Dave,

Dave Chinner wrote:
On Fri, Sep 10, 2010 at 10:58:22AM +1200, Richard Scobie wrote:
Using the latest, stable versions of LVM2 and xfsprogs and the
2.6.35.4 kernel, I am setting up lvm on a 16 drive, 256k chunk md
RAID6, which has been used to date with XFS directly on the RAID.

mkfs.xfs directly on the RAID gives:

meta-data=/dev/md8               isize=256    agcount=32,
agsize=106814656 blks
          =                       sectsz=4096  attr=2
data     =                       bsize=4096   blocks=3418068864, imaxpct=5
          =                       sunit=64     swidth=896 blks
naming   =version 2              bsize=4096   ascii-ci=0

which gives the correct sunit and swidth values for the array.

Creating an lv which uses the entire array and mkfs.xfs on that, gives:

meta-data=/dev/vg_local/Storage  isize=256    agcount=13,
agsize=268435455 blks
          =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=3418067968, imaxpct=5
          =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0

Hmmm - it's treating MD very differently to the LVM volume -
different numbers of AGs, different sunit/swdith. Did you
build xfsprogs yourself? Is it linked against libblkid or libdisk?

I should clarify - the first set was created with xfsprogs 3.0.0 and second was done with xfsprogs 3.1.3, so I wondered if the default ag count had changed.

Given a 12TB array, would I be better using 32?

I did build 3.1.3. Is libblkid preferable? I note it is defaulted off in configure and I used the default configuration.

l appear to have e2fsprogs and devel packages installed (Fedora 11), for libblkid, but when I enable and try to build, it fails:

    [CC]     xfs_mkfs.o
xfs_mkfs.c: In function ‘check_overwrite’:
xfs_mkfs.c:298: error: ‘blkid_probe’ undeclared (first use in this function)
xfs_mkfs.c:298: error: (Each undeclared identifier is reported only once
xfs_mkfs.c:298: error: for each function it appears in.)
xfs_mkfs.c:298: error: expected ‘;’ before ‘pr’
xfs_mkfs.c:321: error: ‘pr’ undeclared (first use in this function)
xfs_mkfs.c:321: warning: implicit declaration of function ‘blkid_new_probe_from_filename’ xfs_mkfs.c:325: warning: implicit declaration of function ‘blkid_probe_enable_partitions’ xfs_mkfs.c:329: warning: implicit declaration of function ‘blkid_do_fullprobe’ xfs_mkfs.c:345: warning: implicit declaration of function ‘blkid_probe_lookup_value’
xfs_mkfs.c:362: warning: implicit declaration of function ‘blkid_free_probe’
xfs_mkfs.c: In function ‘blkid_get_topology’:
xfs_mkfs.c:372: error: ‘blkid_topology’ undeclared (first use in this function)
xfs_mkfs.c:372: error: expected ‘;’ before ‘tp’
xfs_mkfs.c:373: error: ‘blkid_probe’ undeclared (first use in this function)
xfs_mkfs.c:373: error: expected ‘;’ before ‘pr’
xfs_mkfs.c:381: error: ‘pr’ undeclared (first use in this function)
xfs_mkfs.c:385: error: ‘tp’ undeclared (first use in this function)
xfs_mkfs.c:385: warning: implicit declaration of function ‘blkid_probe_get_topology’ xfs_mkfs.c:397: warning: implicit declaration of function ‘blkid_topology_get_minimum_io_size’ xfs_mkfs.c:400: warning: implicit declaration of function ‘blkid_topology_get_optimal_io_size’ xfs_mkfs.c:403: warning: implicit declaration of function ‘blkid_probe_get_sectorsize’ xfs_mkfs.c:406: warning: implicit declaration of function ‘blkid_topology_get_alignment_offset’
gmake[2]: *** [xfs_mkfs.o] Error 1
gmake[1]: *** [mkfs] Error 2
make: *** [default] Error 2


Or it might be that LVM is not exporting the characteristic of the
underlying volume. Can you check if there are different parameter
values exported by the two devices in /sys/block/<dev>/queue?

They look the same.

Limited testing using dd and bonnie++ shows no difference in write
performance whether I use sunit=64/swidth=896 or sunit=0/swidth=0 on
the lv.

These benchmarks won't realy show any difference on an empty
filesystem. It will have an impact on how the filesystems age and
how well aligned the IO will be to the underlying device under more
complex workloads...

I figured I'd go with the geometry specified.

Regards,

Richard

<Prev in Thread] Current Thread [Next in Thread>