<html><body><div style="color:#000; background-color:#fff; font-family:HelveticaNeue-Light, Helvetica Neue Light, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px"><div id="yui_3_16_0_1_1436454217132_37949">Hello,</div><div id="yui_3_16_0_1_1436454217132_37949"><br></div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr">Recently we encountered a previously-reported issue regarding write amplification with MySQL replication and XFS when used with certain RAID controllers (In our case, HP P420). That issue exactly matches our issue and was documented by someone else here - <a href="http://oss.sgi.com/archives/xfs/2013-03/msg00133.html" id="yui_3_16_0_1_1436454217132_38091">http://oss.sgi.com/archives/xfs/2013-03/msg00133.html</a> - but I don't see any resolution. I will say that the problem *does not* exist when mkfs.xfs 2.9.6 is used to format the filesystem on RHEL6 as that sets sunit=0 and swidth=0 instead of setting based on minimum_io_size and optimal_io_size.</div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr"><br></div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr">We have systems that are identical in how they are built and configured, we can take a RHEL6 box that has the MySQL partition formatted with mkfs.xfs v3.1.1 and reproduce the write amplification problem with MySQL replication every single time. If we take the same box and format the MySQL partition with mkfs.xfs 2.9.6, then bring up MySQL with the exact same configuration there is no problem. I've included the working and broken settings below. If it's not the sunit/swidth settings then what will cause 7-10MB/s worth of writes to the XFS partition to become over 200MB/s downstream? The actual data change on the disks is not 200MB/s, but because the write ops are truly being amplified and not just being misreported our MySQL slaves with the bad XFS settings cannot keep up and the lag steadily increases with no hope of ever becoming current.</div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr"><br></div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr">I am happy to try some other settings/options with the RHEL6 mkfs.xfs to see if replication performance is able to match that of systems formatted with mkfs.xfs 2.9.6, but the values set by 3.1.1 with the P420 RAID do not work for MySQL replication. We have ruled out everything else as a possible cause, the absolute only difference on these systems is what values are set by mkfs.xfs.</div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr"><br></div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr">============================================================ </div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr">Working RHEL6 XFS partition:</div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr"><br></div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr" class="">meta-data=/dev/mapper/sys-home isize=256 agcount=4, agsize=71271680 blks</div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr" class=""> = sectsz=512 attr=2, projid32bit=0</div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr" class="">data = bsize=4096 blocks=285086720, imaxpct=5</div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr" class=""> = sunit=0 swidth=0 blks</div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr" class="">naming =version 2 bsize=4096 ascii-ci=0</div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr" class="">log =internal bsize=4096 blocks=32768, version=2</div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr" class=""> = sectsz=512 sunit=0 blks, lazy-count=0</div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr" class="">realtime =none extsz=4096 blocks=0, rtextents=0</div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr" class=""><br></div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr" class="">============================================================ <br></div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr" class="">Broken RHEL6 XFS partition:</div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr" class=""><br></div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr" class="">meta-data=/dev/mapper/sys-home isize=256 agcount=32, agsize=8908992 blks</div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr" class=""> = sectsz=512 attr=2, projid32bit=0</div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr" class="">data = bsize=4096 blocks=285086720, imaxpct=5</div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr" class=""> = sunit=64 swidth=128 blks</div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr" class="">naming =version 2 bsize=4096 ascii-ci=0</div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr" class="">log =internal bsize=4096 blocks=139264, version=2</div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr" class=""> = sectsz=512 sunit=64 blks, lazy-count=1</div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr" class="">realtime =none extsz=4096 blocks=0, rtextents=0</div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr"><br></div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr">============================================================ <br></div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr"><br></div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr">Thanks!</div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr"><br></div><div id="yui_3_16_0_1_1436454217132_37949" dir="ltr">-Hogan</div></div></body></html>