XFS/LVM/Multipath on a single RAID volume

[Dave Hall]

Dave Hall
Binghamton University
kdhall at binghamton.edu
607-760-2328 (Cell)
607-777-4641 (Office)


On 02/23/2015 06:18 AM, Emmanuel Florac wrote:
> Le Sun, 22 Feb 2015 18:35:19 -0500
> Dave Hall<kdhall at binghamton.edu>  écrivait:
>
>    
>> So since I have a fresh array that's not in production yet I was
>> hoping to get some pointers on how to configure it to maximize XFS
>> performance.  In particular, I've seen a suggestion that a
>> multipathed array should be sliced up into logical drives and pasted
>> back together with LVM.  Wondering also about putting the journal in
>> a separate logical drive on the same array.
>>      
> What's the hardware configuration like? before multipathing, you need
> to know if your RAID controller and disks can actually saturate your
> link. Generally SAS-attached enclosures are driven through a 4 way
> SFF-8088 cable, with a bandwidth of 4x 6Gbps (maximum throughput per
> link: 3 GB/s) or 4 x 12 Gbps (max thruput: 6 GB/s).
>
>    
The new hardware is an Infortrend with 16 x 2TB 6Gbps SAS drives.  It 
has one controller with dual 6Gbps SAS ports.  The server currently has 
two 3Gbps SAS HBAs.

On an existing array based on similar but slightly slower hardware, I'm 
getting miserable performance.  The bottleneck seems to be on the server 
side.  For specifics, the array is laid out as a single 26TB volume and 
attached by a single 3Gbps SAS.  The server is quad 8-core Xeon with 
128GB RAM.  The networking is all 10GB.  The application is rsnapshot 
which is essentially a series of rsync copies where the unchanged files 
are hard-linked from one snapshot to the next.  CPU utilization is very 
low and only a few cores seem to be active.  Yet the operation is taking 
hours to complete.

The premise that was presented to me by someone in the storage business 
is that with 'many' proccessor cores one should slice a large array up 
into segments, multipath the whole deal, and then mash the segments back 
together with LVM (or MD).  Since the kernel would ultimately see a 
bunch of smaller storage segments that were all getting activity, it 
should dispatch a set of cores for each storage segment and get the job 
done faster.  I think in theory this would even work to some extent on a 
single-path SAS connection.
>> I am able to set up a 2-way multipath right now, and I might be able
>> to justify adding a second controller to the array to get a 4-way
>> multipath going.
>>      
> A multipath can double the throughput, provided that you have enough
> drives: you'll need about 24 7k RPM drives to saturate _one_ 4x6Gbps
> SAS link. If you have only 12 drives, dual attachment probably won't
> yield much.
>
>    
>> Even if the LVM approach is the wrong one, I clearly have a rare
>> chance to set this array up the right way.  Please let me know if you
>> have any suggestions.
>>      
> In my experience, software RAID-0 with md gives slightly better
> performance than LVM, though not much.
>
>
>    
MD RAID-0 seems as likely as LVM, so I'd probably try that first.  The 
big question is how to size the slices of the array to make XFS happy 
and then how to make sure XFS knows about it.  Secondly, there is the 
question of the log volume.  Seems that with multipath there might be 
some possible advantage to putting this in it's on slice on the array so 
that log writes could be in an I/O stream that is managed separately 
from the rest.