<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Sat, Dec 7, 2013 at 3:12 AM, Stan Hoeppner <span dir="ltr"><<a href="mailto:stan@hardwarefreak.com" target="_blank">stan@hardwarefreak.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On 12/6/2013 4:14 PM, Mike Dacre wrote:<br>
> On Fri, Dec 6, 2013 at 12:58 AM, Stan Hoeppner <<a href="mailto:stan@hardwarefreak.com">stan@hardwarefreak.com</a>>wrote:<br>
</div>...<br>
<div class="im">> UUID=a58bf1db-0d64-4a2d-8e03-aad78dbebcbe /science xfs<br>
> defaults,inode64 1 0<br>
<br>
</div>Your RAID card has persistent write cache (BBWC) and we know it's<br>
enabled from tool your output. By default XFS assumes BBWC is not<br>
present, and uses write barriers to ensure order/consistency. Using<br>
barriers on top of BBWC will be detrimental to write performance, for a<br>
couple of reasons:<br>
<br>
1. Prevents the controller from optimizing writeback patterns<br>
2. A portion, or all of, the write cache is frequently flushed<br>
<br>
Add 'nobarrier' to your mount options to avoid this problem. It should<br>
speed up many, if not all, write operations considerably, which will in<br>
turn decrease seek contention amongst jobs. Currently your write cache<br>
isn't working nearly as well as it should, and in fact could be<br>
operating horribly.<br>
<div class="im"><br>
> On the slave nodes, I managed to reduce the demand on the disks by adding<br>
> the actimeo=60 mount option. Prior to doing this I would sometimes see the<br>
> disk being negatively affected by enormous numbers of getattr requests.<br>
> Here is the fstab mount on the nodes:<br>
><br>
> 192.168.2.1:/science /science nfs<br>
> defaults,vers=3,nofail,actimeo=60,bg,hard,intr,rw 0 0<br>
<br>
</div>One minute attribute cache lifetime seems maybe a little high for a<br>
compute cluster. But if you've had no ill effects and it squelched the<br>
getattr flood this is good.<br>
<br>
...<br>
<div class="im">> Correct, I am not consciously aligning the XFS to the RAID geometry, I<br>
> actually didn't know that was possible.<br>
<br>
</div>XFS alignment is not something to worry about in this case.<br>
<br>
...<br>
<div class="im">>> So it's a small compute cluster using NFS over Infiniband for shared<br>
>> file access to a low performance RAID6 array. The IO resource sharing<br>
>> is automatic. But AFAIK there's no easy way to enforce IO quotas on<br>
>> users or processes, if at all. You may simply not have sufficient IO to<br>
>> go around. Let's ponder that.<br>
><br>
> I have tried a few things to improve IO allocation. BetterLinux have a<br>
> cgroup control suite that allow on-the-fly user-level IO adjustments,<br>
> however I found them to be quite cumbersome.<br>
<br>
</div>This isn't going to work well because a tiny IO stream can seek the<br>
disks to death, such as a complex find command, ls -R, etc. A single<br>
command such as these can generate thousands of seeks. Shaping/limiting<br>
user IO won't affect this.<br>
<br>
...<br>
<div class="im">>> Looking at the math, you currently have approximately 14*150=2100<br>
>> seeks/sec capability with 14x 7.2k RPM data spindles. That's less than<br>
>> 100 seeks/sec per compute node, i.e. each node is getting about 2/3rd of<br>
>> the performance of a single SATA disk from this array. This simply<br>
>> isn't sufficient for servicing a 23 node cluster, unless all workloads<br>
>> are compute bound, and none IO/seek bound. Given the overload/crash<br>
>> that brought you to our attention, I'd say some of your workloads are<br>
>> obviously IO/seek bound. I'd say you probably need more/faster disks.<br>
>> Or you need to identify which jobs are IO/seek heavy and schedule them<br>
>> so they're not running concurrently.<br>
><br>
> Yes, this is a problem. We sadly lack the resources to do much better than<br>
> this, we have recently been adding extra storage by just chaining together<br>
> USB3 drives with RAID and LVM... which is cumbersome and slow, but cheaper.<br>
<br>
</div>USB disk is generally a recipe for disaster. Plenty of horror stories<br>
on both this list and linux-raid regarding USB connected drives,<br>
enclosures, etc. I pray you don't run into those problems.<br>
<div class="im"><br>
> My current solution is to be on the alert for high IO jobs, and to move<br>
> them to a specific torque queue that limits the number of concurrent jobs.<br>
> This works, but I have not found a way to do it automatically.<br>
> Thankfully, with a 12 member lab, it is actually not terribly complex to<br>
> handle, but I would definitely prefer a more comprehensive solution. I<br>
> don't doubt that the huge IO and seek demands we put on these disks will<br>
> cause more problems in the future.<br>
<br>
</div>Your LSI 9260 controller supports using SSDs for read/write flash cache.<br>
LSI charges $279 for it. It's called CacheCade Pro:<br>
<br>
<a href="http://www.lsi.com/products/raid-controllers/pages/megaraid-cachecade-pro-software.aspx" target="_blank">http://www.lsi.com/products/raid-controllers/pages/megaraid-cachecade-pro-software.aspx</a>.<br>
<br>
<br>
Connect two good quality fast SSDs to the controller, such as:<br>
<a href="http://www.newegg.com/Product/Product.aspx?Item=N82E16820147192" target="_blank">http://www.newegg.com/Product/Product.aspx?Item=N82E16820147192</a><br>
<br>
Two SSDs, mirrored, to prevent cached writes from being lost if a single<br>
SSD fails. You now have a ~90K IOPS, 128GB, 500MB/s low latency<br>
read/write cache in front of your RAID6 array. This should go a long<br>
way toward eliminating your bottlenecks. You can accomplish this for<br>
~$550 assuming you have two backplane drive slots free for the SSDs. If<br>
not, you add one of these for $279:<br>
<br>
<a href="http://www.newegg.com/Product/Product.aspx?Item=N82E16816117207" target="_blank">http://www.newegg.com/Product/Product.aspx?Item=N82E16816117207</a><br>
<br>
This is an Intel 24 port SAS expander, the same device as in your drive<br>
backplane. SAS expanders can be daisy chained many deep. You can drop<br>
it into a PCIe x4 or greater slot from which it only draws power--no<br>
data pins are connected. Or if no slots are available you can mount it<br>
to the side wall of your rack server chassis and power it via the 4 pin<br>
Molex plug. This requires a drill, brass or plastic standoffs, and DIY<br>
skills. I use this option as it provides a solid mount for un/plugging<br>
the SAS cables, and being side mounted neither it nor the cables<br>
interfere with airflow.<br>
<br>
You'll plug the 9260-4i into one port of the Intel expander. You'll<br>
need another SFF-8087 cable for this:<br>
<br>
<a href="http://www.newegg.com/Product/Product.aspx?Item=N82E16812652015" target="_blank">http://www.newegg.com/Product/Product.aspx?Item=N82E16812652015</a><br>
<br>
You will plug your drive backplane cable into another of the 6 SFF-8087<br>
ports on the Intel. Into a 3rd port you will plug an SFF-8087 breakout<br>
cable to give you 4 individual drive connections. You will plug two of<br>
these into your two SSDs.<br>
<br>
<a href="http://www.newegg.com/Product/Product.aspx?Item=N82E16816116097" target="_blank">http://www.newegg.com/Product/Product.aspx?Item=N82E16816116097</a><br>
<br>
If you have no internal 2.5/3.5" drive brackets free for the SSDs and<br>
you'd prefer not to drill (more) holes in the chassis to directly mount<br>
them or a new cage for them, simply use some heavy duty Velcro squares,<br>
2" is fine.<br>
<br>
Worst case scenario you're looking at less than $1000 to cure your IO<br>
bottlenecks, or at the very least mitigate them to a minor annoyance<br>
instead of a show stopper. And if you free up some money for some<br>
external JBODs and drives in the future, you can route 2 of the unused<br>
SFF-8087 connectors of the Intel Expander out the back panel to attach<br>
expander JBOD enclosures, using one of these and 2 more of the 8087<br>
cables up above:<br>
<br>
<a href="http://www.ebay.com/itm/8-Port-SAS-SATA-6G-Dual-SFF-8088-mini-SAS-to-SFF-8087-PCIe-Adapter-w-LP-Bracket-/390508767029" target="_blank">http://www.ebay.com/itm/8-Port-SAS-SATA-6G-Dual-SFF-8088-mini-SAS-to-SFF-8087-PCIe-Adapter-w-LP-Bracket-/390508767029</a><br>
<br>
I'm sure someone makes a 3 port model but 10 minutes of searching didn't<br>
turn one up. These panel adapters are application specific. Most are<br>
made to be mounted in a disk enclosure where the HBA/RAID card is on the<br>
outside of the chassis, on the other end of the 8088 cable. This two<br>
port model is designed to be inside a server chassis, where the HBA<br>
connects to the internal 8087 ports. Think Ethernet x-over cable.<br>
<br>
The 9260-4i supports up to 128 drives. This Intel expander and a panel<br>
connector allow you to get there with external JBODs. The only caveat<br>
being that you're limited to "only" 4.8 GB/s to/from all the disks.<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Stan<br>
</font></span></blockquote></div><br></div><div class="gmail_extra">Hi Stan,</div><div class="gmail_extra"><br></div><div class="gmail_extra">Thanks for the great advice, I think you are on to something there. I will look into doing this in the next week or so when I have more time. I added 'nobarrier' to my mount options.</div>
<div class="gmail_extra"><br></div><div class="gmail_extra">Thanks again, I will let you know how it goes after I have upgraded.</div><div class="gmail_extra"><br></div><div class="gmail_extra">Best,</div><div class="gmail_extra">
<br></div><div class="gmail_extra">Mike</div></div>