The issue isn't random small I/O being sent to disk and the disks maxing out
IOPS, it's this particular workload created by MySQL replication and the XFS
options that trigger something bad to happen on the controller. I can run disk
I/O tests at the same time that replication is choking and see perfectly fine
throughput and response time. Yeah, it's weird.
As for the noalign option, that would be great to have but it does not exist in
version 3.1.1 which RHEL6 uses. It sounds like not having it in previous 3.x
versions was enough of an issue that it was added in 3.2.x, which is great. I
can probably work with this and get our ramdisks used for cloning updated with
a stable 3.2.x since that would be more desirable than reverting back to 2.x.x.
Thanks again for your help, now I have a fix that doesn't involve using an old
version of mkfs.xfs.
-Hogan
----- Original Message -----
From: Dave Chinner <david@xxxxxxxxxxxxx>
To: Hogan Whittall <whittalh@xxxxxxxxxxxxx>
Cc: "xfs@xxxxxxxxxxx" <xfs@xxxxxxxxxxx>
Sent: Friday, July 10, 2015 5:42 PM
Subject: Re: Issue with RHEL6 mkfs.xfs (3.1.1+), HP P420 RAID, and MySQL
replication
On Fri, Jul 10, 2015 at 03:59:48PM +0000, Hogan Whittall wrote:
> Hi Dave,
>
> Thanks for the reply, we can certainly try with the smaller log,
> but IIRC the performance hit wasn't because the disks were busy,
> it was the controller itself trying to determine what changed and
> then write that to disk.
That makes no sense to me - the controller is almost never the IO
limitation in a hardware RAID when random small IO is being issued
by the host.
> Smaller anything should help the
> controller be able to cope better, but that's not really a
> solution.
>
> Doing disk write performance tests on these systems produce very
> different results, they are capable of much more I/O than what was
> being triggered with this issue.
>
> Back to why I think this should be considered a bug, by 2.9.6
> setting 0 as the default for sunit/swidth and 3.1.1 having no way
> to set 0 for sunit/swidth the newer versions behave differently
False:
# man mkfs.xfs
....
noalign
This option disables automatic geometry detection
and creates the filesystem without stripe geometry
alignment even if the underlying storage device
provides this information.
IOWs:
# mkfs.xfs -d noalign ....
Will do exactly what you want. Or alternatively:
# mkfs.xfs -d sunit=0,swidth=0 ....
Or perhaps just turning of log stripe unit alignment will be enough:
# mkfs.xfs -l sunit=1 ....
Cheers,
Dave.
--
Dave Chinner
david@xxxxxxxxxxxxx
|