Re: XFS use within multi-threaded apps

[Angelo McComis <angelo@xxxxxxxxxxx>]

Stewart, and others:

On Tue, Oct 19, 2010 at 12:24 AM, Stewart Smith <stewart@xxxxxxxxxxxxxxxx> wrote:

On Mon, 18 Oct 2010 09:42:04 -0400, Angelo McComis <angelo@xxxxxxxxxxx> wrote:
> I have a use case where I'd like to forward the use of XFS. This is for
> large (multi-GB, say anywhere from 5GB to 300GB) individual files, such as
> what you'd see under a database's data file / tablespace.

The general advice from not only those of us who hack on database
systems for a living (and hobby), but those that also run it in
production on more systems than you'll ever be able to count is this for
database system performance tuning (i.e. after making your SQL not
completely nuts)

Step 1) Use XFS.

Nothing, and I do mean nothing comes close to reliability and consistent
performance.

We've seen various benchmarks where X was faster.... most of the
time. Then suddenly your filesystem takes a mutex for 15 seconds and
you're database performance goes down the crapper.

I have been running iozone benchmarks, both head to head ext3 versus XFS, single LUN, default mkfs options, etc.  And the short answer is XFS wins hands down on writes and random writes.  Ext3 wins a couple of the other tests, but not by nearly the margin that XFS wins on the other ones.  That was the KB/sec tests. The IOPS test was even more telling, and showed XFS winning by orders of magnitudes on a few tests, and being close or a tie on the ones that it didn't win.  I took two SAN luns and ran local FS versus SAN-presented (this is FC, attached to IBM DS8k storage), and ran the tests there.  When done with the head to head tests, I concatenated the LUNS to make a RAID 0 / simple 2-stripe set, and ran the tests some more.

I can't say that the numbers make a lot of sense, since it's a 4gbit FC connection
 

> My database vendor (who, coincidentally markets their own filesystems and
> operating systems) says that there are certain problems under XFS with
> specific mention of corruption issues, if a single root or the metadata
> become corrupted, the entire filesystem is gone, and it has performance
> issues on a multi-threaded workload, caused by the single root filesystem
> for metadata becoming a bottleneck.

XFS has anything but performance problems on multithreaded
workloads. It is *the* best of the Linux filesystems
(actually... possibly any file system anywhere) for multithreaded
IO. You can either benchmark it or go and read the source - check out
the direct IO codepaths and what locks get taken (or rather, what locks
aren't taken).

Generally speaking, most DBMSs don't do much filesystem metadata
operations, the most common being extending the data file. So what you
really care about is multithreaded direct IO performance, scalability
and reliability.

> This feedback from the vendor is surely taken with a grain of salt as they
> have marketing motivations of their own product to consider.

If the vendor is who I suspect, and the filesystem being pushed is
starting with two letters down the alphabet than XFS... I
wouldn't. While a great file system for a number of applications, it is
nowhere near ready for big database IO loads - to the extent that last I
heard it still wasn't being recommended for the various DBs I care about
(at least by the DB support guys).

Well -  I mentioned it above. Their current recommendation for Linux is to stick with ext3... and for big file/big IO operations, switch to ext4. And we had a meeting wherein we had a discussion that went like "well, ext3 has problems whenever the kernel journal thread wakes up to flush under heavy I/O, and ext4
is not available to us..."  and Dave Chinner on an earlier post regarding the maturity level of ext4 and its present status.

I have been able to schedule a meeting with the folks at my vendor, on the database software side. Aside from the questions I have, points to make, etc., I'm curious if there's anything else, based on anyone here's input, I should be asking them? This is a pretty grand opportunity to sit down and grill them.

Thanks to everyone who participates on this list - you are all a great resource and a perfect example of what the open source community is all about.

Regards,
Angelo