> 150MB/s isn't correct. Should be closer to 450MB/s. This makes it
> appear that you're writing all these files to a single directory. If
> you're writing them fairly evenly to 3 directories or a multiple of 3,
> you should see close to 450MB/s, if using mdraid linear over 3 P400
> RAID1 pairs. If this is what you're doing then something seems wrong
> somewhere. Try unpacking a kernel tarball. Lots of subdirectories to
> exercise all 3 AGs thus all 3 spindles.
The spindles were exercised; I watched it with iostat. Maybe I could
have reached more with more parallelism, but that wasn’t my goal at
all. Although, over the course of these experiments, I got to doubt
that the controller could even handle this data rate.
>> simple copy of the tar onto the XFS file system yields the same linear
>> performance, the same as with ext4, btw. So 150 MB/sec seems to be the
>> best these disks can do, meaning that theoretically, with 3 AGs, it
>> should be able to reach 450 MB/sec under optimal conditions.
> The optimal condition, again, requires writing 3 of this file to 3
> directories to hit ~450MB/s, which you should get close to if using
> mdraid linear over RAID1 pairs. XFS is a filesystem after all, so it's
> parallelism must come from manipulating usage of filesystem structures.
> I thought I explained all of this previously when I introduced the "XFS
> concat" into this thread.
The optimal condition would be 3 parallel writes of huge files, which
can be easily written linearly. Not thousands of tiny files.
>> But then I guess I’m back to ext4 land. XFS just doesn’t offer enough
>> benefits in this case to justify the hassle.
> If you were writing to only one directory I can understand this
> sentiment. Again, if you were writing 3 directories fairly evenly, with
> the md concat, then your sentiment here should be quite different.
Haha, I made a U-turn on this one. XFS is back on the table (and on
the disks now) ;). When I thought I was done, I wanted to restore a
few large KVM images which were on the disks prior to the RAID
reconfiguration. With ext4, I watched iostat writing at 130MB/s for a
while. After 2 or 3 minutes, it broke down completely and languished
at 30-40MB/s for many minutes, even after I had SIGSTOPed the writing
process, during which it was nearly impossible to use vim to edit a
file on the ext4 partition. It would pause for tens of seconds all the
time. It’s not even clear why it broke down so badly. From another
seekwatcher sample I took, it looked like fairly linear writing.
So I threw XFS back in, restarted the restore, and it went very
smoothly while still providing acceptable interactivity.
XFS is not a panacea (obviously), and it may be a bit slower in many
cases, and doesn’t seem to cope well with fragmented free space (which
is what this entire thread is really about), but overall it feels more
well-rounded. After all, I don’t really care how much it writes per
time unit, as long as it’s not ridiculously little and it doesn’t
bring everything else to a halt.