<div dir="ltr"><div>Hi Stan,<br><br></div>once again, thanks for your answer.<br><br><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="">
> Hi Stan,<br>
><br>
> thanks for your answer.<br>
><br>
> Everything begins and ends with the workload.<br>
>><br>
>> On 5/7/2014 7:43 AM, Marc Caubet wrote:<br>
>>> Hi all,<br>
>>><br>
>>> I am trying to setup a storage pool with correct disk alignment and I<br>
>> hope<br>
>>> somebody can help me to understand some unclear parts to me when<br>
>>> configuring XFS over LVM2.<br>
>><br>
>> I'll try. But to be honest, after my first read of your post, a few<br>
>> things jump out as breaking traditional rules.<br>
>><br>
>> The first thing you need to consider is your workload and the type of<br>
>> read/write patterns it will generate. This document is unfinished, and<br>
>> unformatted, but reading what is there should be informative:<br>
>><br>
>> <a href="http://www.hardwarefreak.com/xfs/storage-arch.txt" target="_blank">http://www.hardwarefreak.com/xfs/storage-arch.txt</a><br>
>><br>
><br>
> Basically we are moving a lot of data :) It means, parallel large files<br>
> (GBs) are being written and read all the time. Basically we have a batch<br>
> farm with 3,5k cores processing jobs that are constantly reading and<br>
> writing to the storage pools (4PBs). Only few pools (~5% of the total)<br>
> contain small files (and only small files).<br>
<br>
</div>And these pools are tied together with? Gluster? Ceph?<br></blockquote><div><br></div><div>We are using dCache (<a href="http://www.dcache.org/">http://www.dcache.org/</a>), where a file is written in a single pool instead of spreading parts among pools as Ceph or Hadoop do. So large files go entirely to a pool.<br>
</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="">
>>> Actually we have few storage pools with the following settings each:<br>
>>><br>
>>> - LSI Controller with 3xRAID6<br>
>>> - Each RAID6 is configured with 10 data disks + 2 for double-parity.<br>
>>> - Each disk has a capacity of 4TB, 512e and physical sector size of 4K.<br>
>><br>
>> 512e drives may cause data loss. See:<br>
>> <a href="http://docs.oracle.com/cd/E26502_01/html/E28978/gmkgj.html#gmlfz" target="_blank">http://docs.oracle.com/cd/E26502_01/html/E28978/gmkgj.html#gmlfz</a><br>
>><br>
><br>
> Haven't experienced this yet. But good to know thanks :) On the other<br>
> hand, we do not use zfs<br>
<br>
</div>This problem affects all filesystems. If the drive loses power during<br>
an RMW cycle the physical sector is corrupted. As noted, not all 512e<br>
drives may have this problem. And for the bulk of your workload this<br>
shouldn't be an issue. If you have sufficient and properly functioning<br>
UPS it shouldn't be an issue either.<br></blockquote><div><br></div><div> Actually all LSI controllers have batteries so I hope it will not happen. This problems is good to have this in mind when we purchase new storage machines so thanks :)<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class=""><br>
>>> - 3x(10+2) configuration was considered in order to gain best performance<br>
>>> and data safety (less disks per RAID less probability of data corruption)<br>
>><br>
>> RAID6 is the worst performer of all the RAID levels but gives the best<br>
>> resilience to multiple drive failure. The reason for using fewer drives<br>
>> per array has less to do with probability of corruption, but<br>
>><br>
>> 1. Limiting RMW operations to as few drives as possible, especially for<br>
>> controllers that do full stripe scrubbing on RMW<br>
>><br>
>> 2. Lowering bandwidth and time required to rebuild a dead drive, fewer<br>
>> drives tied up during a rebuild<br>
>><br>
><br>
>>> From the O.S. side we see:<br>
>>><br>
>>> [root@stgpool01 ~]# fdisk -l /dev/sda /dev/sdb /dev/sdc<br>
>> ...<br>
>><br>
>> You omitted crucial information. What is the stripe unit size of each<br>
>> RAID6?<br>
>><br>
><br>
> Actually the stripe size for each RAID6 is 256KB but we plan to increase<br>
> some pools to 1MB for all their RAIDs. It will be in order to compare<br>
> performance for pools containing large files and if this improves, we will<br>
> apply it to the other systems in the future.<br>
<br>
</div>So currently you have a 2.5MB stripe width per RAID6 and you plan to<br>
test with a 10MB stripe width.<br></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class=""><br>
>>> The idea is to aggregate the above devices and show only 1 storage space.<br>
>>> We did as follows:<br>
>>><br>
>>> vgcreate dcvg_a /dev/sda /dev/sdb /dev/sdc<br>
>>> lvcreate -i 3 -I 4096 -n dcpool -l 100%FREE -v dcvg_a<br>
>><br>
>> You've told LVM that its stripe unit is 4MB, and thus the stripe width<br>
>> of each RAID6 is 4MB. This is not possible with 10 data spindles.<br>
>> Again, show the RAID geometry from the LSI tools.<br>
>><br>
> When creating a nested stripe, the stripe unit of the outer stripe (LVM)<br>
>> must equal the stripe width of eachinner stripe (RAID6).<br>
>><br>
><br>
> Great. Hence, if the RAID6 stripe size is 256k then the LVM should be<br>
> defined with 256k as well, isn't it?<br>
<br>
</div>No. And according to lvcreate(8) you cannot use LVM for the outer<br>
stripe because you have 10 data spindles per RAID6. "StripeSize" is<br>
limited to power of 2 values. Your RAID6 stripe width is 2560 KB which<br>
is not a power of 2 value. So you must use md. See mdadm(8).<br></blockquote><div><br></div><div>Great thanks, this is exactly what I needed and I think I am starting to understand then :) So a RAID6 of 16+2 disks, stripe width of 256KB will have a stripe width of 256*16=4096 which is a power of 2. Then in this case LVM2 can be used. Am I correct? Then seems clear to me that new purchases will go in this way (we have planned a new purchase in the next month and I am trying to understand this)<br>
</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
And be careful with terminology. "Stripe unit" is per disk, called<br>
"chunk" by mdadm. "Stripe width" is per array. "Stripe size" is ambiguous.<br></blockquote><div><br></div><div>Yes correct, sorry for the wrong terminology is something that I don't use to manage :)<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
When nesting stripes, the "stripe width" of the RAID6 becomes the<br>
"stripe unit" of the outer stripe of the resulting RAID60. In essence,<br>
each RAID6 is treated as a "drive" in the outer stripe. For example:<br>
<br>
RAID6 stripe unit = 256 KB<br>
RAID6 stripe width = 2560 KB<br>
RAID60 stripe unit = 2560 KB<br>
RAID60 stripe width = 7680 KB<br>
<br>
For RAID6 w/1MB stripe unit<br>
<br>
RAID6 stripe unit = 1 MB<br>
RAID6 stripe width = 10 MB<br>
RAID60 stripe unit = 10 MB<br>
RAID60 stripe width = 30 MB<br>
<br>
This is assuming your stated configuration of 12 drives per RAID6, 10<br>
data spindles, and 3 RAID6 arrays per nested stripe.<br>
<div class=""><br>
>> Hence, stripe of the 3 RAID6 in a LV.<br>
>><br>
>> Each RAID6 has ~1.3GB/s of throughput. By striping the 3 arrays into a<br>
>> nested RAID60 this suggests you need single file throughput greater than<br>
>> 1.3GB/s and that all files are very large. If not, you'd be better off<br>
>> using a concatenation, and using md to accomplish that instead of LVM.<br>
>><br>
>>> And here is my first question: How can I check if the storage and the LV<br>
>>> are correctly aligned?<br>
>><br>
>> Answer is above. But the more important question is whether your<br>
>> workload wants a stripe or a concatenation.<br>
>><br>
>>> On the other hand, I have formatted XFS as follows:<br>
>>><br>
>>> mkfs.xfs -d su=256k,sw=10 -l size=128m,lazy-count=1 /dev/dcvg_a/dcpool<br>
<br>
</div>lazy-count=1 is the default. No need to specify it.<br></blockquote><div><br></div><div>Ok thanks :)<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class=""><br>
>> This alignment is not correct. XFS must be aligned to the LVM stripe<br>
>> geometry. Here you apparently aligned XFS to the RAID6 geometry<br>
>> instead. Why are you manually specifying a 128M log? If you knew your<br>
>> workload that well, you would not have made these other mistakes.<br>
>><br>
><br>
> We receive several parallel writes all the time, and afaik filesystems with<br>
> such write load benenfit from a larger log. 128M is the maximum log size.<br>
<br>
</div>Metadata is journaled, file data is not. Filesystems experiencing a<br>
large amount of metadata modification may benefit from a larger journal<br>
log, however writing many large files in parallel typically doesn't<br>
generate much metadata modification. In addition, with delayed logging<br>
now the default, the amount of data written to the journal is much less<br>
than it used to be. So specifying a log size should not be necessary<br>
with your workload.<br></blockquote><div><br></div><div>Ok. Then I'll try to remove that.<br> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="">
> So how XFS should be formatted then? As you specify, should be aligned with<br>
> the LVM stripe, as we have a LV with 3 stripes then 256k*3 and sw=30?<br>
<br>
</div>It must be aligned to the outer stripe in the nest, which would be the<br>
LVM geometry if it could work. However, as stated, it appears you<br>
cannot use lvcreate to make the outer stripe because it does not allow a<br>
2560 KiB StripeSize. Destroy the LVM volume and create an md RAID0<br>
device of the 3 RAID6 devices, eg:<br>
<br>
$ mdadm -C /dev/md0 --raid_devices=3 --chunk=2560 --level=0 /dev/sd[abc]<br>
<br>
For making the filesystem and aligning it to the md nested stripe<br>
RAID60, this is all that is required:<br>
<br>
$ mkfs.xfs -d su=2560k,sw=3 /dev/md0<br></blockquote><div><br></div><div>Perfect! I'll try this with the current server having 3xRAID6(10+2). You really helped me with that.<br><br></div><div>Just one final question, if I had 3*RAID6(16+2) the Stripe Width should be 4096 (256KB*16) and when applying this to LVM2 should be:<br>
<br> lvcreate -i 3 -I 4096 -n dcpool -l 100%FREE -v dcvg_a<br><br></div><div></div><div>And then the XFS format should be:<br><br>mkfs.xfs -d su=4096k, sw=3 /dev/dcvg_a/dcpool<br><br></div><div>Is it correct?<br></div>
</div><br></div><div class="gmail_extra">Thanks a lot for your help,<br>-- <br>Marc Caubet Serrabou<br>PIC (Port d'Informació Científica)<br>Campus UAB, Edificio D<br>E-08193 Bellaterra, Barcelona<br>Tel: +34 93 581 33 22<br>
Fax: +34 93 581 41 10<br><a href="http://www.pic.es" target="_blank">http://www.pic.es</a><br>Avis - Aviso - Legal Notice: <a href="http://www.ifae.es/legal.html" target="_blank">http://www.ifae.es/legal.html</a>
</div></div>