[Top] [All Lists]

Re: 30 TB RAID6 + XFS slow write performance

To: xfs@xxxxxxxxxxx
Subject: Re: 30 TB RAID6 + XFS slow write performance
From: Michael Monnerie <michael.monnerie@xxxxxxxxxxxxxxxxxxx>
Date: Fri, 22 Jul 2011 08:10:00 +0200
Cc: Dave Chinner <david@xxxxxxxxxxxxx>, John Bokma <contact@xxxxxxxxxxxxx>, Stan Hoeppner <stan@xxxxxxxxxxxxxxxxx>
In-reply-to: <20110721064838.GA13963@dastard>
Organization: it-management http://it-management.at
References: <4E24907F.6020903@xxxxxxxxxxxxx> <201107210820.01019@xxxxxx> <20110721064838.GA13963@dastard>
User-agent: KMail/1.13.6 (Linux/; KDE/4.6.0; x86_64; ; )
On Donnerstag, 21. Juli 2011 Dave Chinner wrote:
> If you are writing files that grow like this, then you are doing
> something wrong. If the app can't do it's IO differently, then this
> is exactly the reason we have userspace-controlled preallocation
> interfaces.
> Filesystems cannot prevent user stupidity from screwing something
> up....

This can happen if you copy a syslog server over to a new disk, then let 
it start it's work again. Many files that start small and grow. Luckily, 
the logs are rotated latest monthly, so it shouldn't be too bad.
> > And files >64KiB are immediately fragmented
> > then. At this time, there are only 16384 * 2KiB = 32MiB used, which
> > is 3,125% of the disk. I can't believe my numbers, are they true?
> No, because most filesystems have a 4k block size. 

I just meant pure disk usage. Of 1GB, only 32MB are used, and this worst 
case example hits us badly.

> Not to mention
> that fragmentation is likely to be limited to the single AG the files
> in the directory belong to. i.e. even if we can't allocation a sunit
> aligned chunk in an AG, we won't switch to another AG just to do
> sunit aligned allocation.

This is good to know also, thanks.
> > OK, this is a worst case scenario, and as you've said before, any
> > filesystem can be considered full at 85% fill grade. But it's
> > incredible how quickly you could fuck up a filesystem when using
> > su/sw and writing small files.
> Well, don't use a filesystem that is optimised for storing large
> sizes, large files and high bandwidth for storing lots of small
> files, then.  Indeed, the point of not packing the files is so they
> -don't fragemnt as they grow-. XFS is not designed to be optimal
> for small filesystems or small files. In most cases it will deal
> with them just fine, so in reality your concerns are mostly
> unfounded...

Yes, I just wanted to know about the corner cases, and how XFS behaves. 
Actually, we're changing over to using NetApps, and with their WAFL 
anyway I should drop all su/sw usage and just use 4KB blocks.

And even when XFS is optimized for large files, there are often small 
ones. Think of a mysql server with hundreds of DBs and 
innodb_file_per_table set. Even when some DBs are large, there are many 
small files.

But this thread has drifted a bit. XFS does great work, and now I 
understand the background a bit more. Thanks, Dave.

mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services: Protéger
http://proteger.at [gesprochen: Prot-e-schee]
Tel: +43 660 / 415 6531

// Haus zu verkaufen: http://zmi.at/langegg/

Attachment: signature.asc
Description: This is a digitally signed message part.

<Prev in Thread] Current Thread [Next in Thread>