XFS use within multi-threaded apps

[Angelo McComis]

On Sat, Oct 23, 2010 at 3:56 PM, Peter Grandi <pg_xf2 at xf2.for.sabi.co.uk>wrote:

> >>> I have a use case where I'd like to forward the use of XFS. This is for
> >>> large (multi-GB, say anywhere from 5GB to 300GB) individual files, such
> as
> >>> what you'd see under a database's data file / tablespace.
>
> >> Step 1) Use XFS.
> >> Nothing, and I do mean nothing comes close to reliability and consistent
> >> performance.
>
> > I have been running iozone benchmarks, [ ... ]
>
> I think that it is exceptionally difficult to get useful results
> out of Iozone...
>
>
True - the benchmarks themselves don't tell a complete story. Specific to
iozone, I was basically comparing XFS to EXT3, and showing the results
(various record sizes, various file sizes, and various worker thread
counts)... The only true benchmark is to run the application in the way that
is characteristic of how it will be used. Database benchmarks themselves
would vary greatly between use cases: from generic looking up data (random
reads), data warehouse analytics (sequential reads), ETL (sequential reads,
sequential writes), etc.


> >>> My database vendor (who, coincidentally markets their own
> >>> filesystems and operating systems) says that there are
> >>> certain problems under XFS with specific mention of
> >>> corruption issues, if a single root or the metadata become
> >>> corrupted, the entire filesystem is gone,
>
> If that's bad enough it applies to any file system out there
> except FAT and Reiser, as they store some metadata with each
> block. ZFS and BTRFS may have something similar. But it is not
> an issue.
>
> >>> and it has performance issues on a multi-threaded workload,
> >>> caused by the single root filesystem for metadata becoming a
> >>> bottleneck.
>
> That's actually more of a problem with Lustre, in extreme cases.
>
> >> XFS has anything but performance problems on multithreaded
> >> workloads. It is *the* best of the Linux filesystems
> >> (actually... possibly any file system anywhere) for
> >> multithreaded IO.
>
> That's actually multithreaded IO to the same file, for
> multithreaded IO to different files JFS (and allegedly 'ext4') are
> also fairly good.
>
> > Well - I mentioned it above. Their current recommendation for
> > Linux is to stick with ext3... and for big file/big IO
> > operations, switch to ext4.
>
> That's just about because those are the file systems that are
> "qualified", and 'ext3' defaults give the lowest risks in case the
> application environment is misdesigned and relies on 'O_PONIES'.
>
> > [ ... ] "well, ext3 has problems whenever the kernel journal
> > thread wakes up to flush under heavy I/O,
>
> That actually happens with every file system, and it is one of
> several naive misdesigns in the Linux IO subsystem. The default
> Linux page cache flusher parameters are often too "loose" by a 1-2
> orders of magnitude, and this can cause serious problems. Nedver
> mind that the page cache
>
>
> In any case the Linux page cache itself is also a bit of a joke, a
> (hopefully) a DBMS will not use it anyhow, but use direct IO, and
> XFS is targeted at direct IO, large file, multistreaming loads.
>
>
Peter, and others:

Thanks for this great discussion. I appreciate the thought that went into
all of the replies.

In the end, we had a sit down discussion with our vendor.  They admitted
that they "support" XFS, but have very few customers using it (said they can
count them on one hand), and when I pressed them on if it's a technology
limitation, they threw down the gauntlet and said "look, we're giving you
our frank recommendation here. EXT3."  They quoted as having 10+TB databases
running OLTP transactions on XFS, with 4-5GB/sec sustained throughput to the
disk system.  And 20-30TB for data warehouse type operations.  When pressed
about the cache flush issue, they mentioned they use direct IO under ext3,
and it's not an issue in that case.

In doing my research, I searched for references of other Fortune nn-sized
companies who use this DB and use XFS underneath it. I came up empty
handed...  I searched my network for large-ish companies using XFS, and how
they were using it.  I'm not sure if we're bordering on "secret sauce" type
stuff here, but I had an extremely difficult time getting enterprise
references to back up the research I've done.

For our use, we had to opt to follow the vendor recommendation, and it came
down to not wanting to be one of those that they can count on one hand using
XFS with their product.

I'm still confounded by why - when XFS is technically superior in these
cases - is it so obscure?  Are Enterprise Linux guys just not looking this
deep under the covers to uncover performance enhancements like this? Is it
because RedHat didn't to include the XFS tools in the distro until recently,
causing XFS to not be a choice part of it? Are other Linux folks "next,
next, finish..." people when it comes to how they install?  I really don't
get it.

Thanks for all the discussion folks. I hope to put forth other use cases as
the surface.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://oss.sgi.com/pipermail/xfs/attachments/20101023/1219ab88/attachment.htm>