[Top] [All Lists]

Re: raid10n2/xfs setup guidance on write-cache/barrier

To: Linux RAID <linux-raid@xxxxxxxxxxxxxxx>, Linux fs XFS <xfs@xxxxxxxxxxx>
Subject: Re: raid10n2/xfs setup guidance on write-cache/barrier
From: Jessie Evangelista <jessie.evangelista@xxxxxxxxx>
Date: Fri, 16 Mar 2012 00:18:41 +0800
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; bh=PjLXKbaoOGBjQcYnlEKNHHBrhdPsF9+qaEEpnp7cxtc=; b=Mf6jZYs2dyXDRd07iEU13KBzJxVsIDtX/tr0PUeh5PeeXY27Q88A6IE/kfD7hfgEi5 XuPtQYXclGVG8hGeH5Kphh4NZJafQGTp24zD6Ye4dPBUHzhoaWx2LoimbpOuBSw7mxZY jvrhS+S1J+2OoNVFHJ00zlfT8pT8FgMUvjiRuYLOjBAMhgrqSuEiHXBmqya1OfNtGGbn ic4JU7e7DIjl3WBouBog6HPNB5MOfQmQRkqHD7gwOEO9INCMGi86qPMWVSbntvmukC9R ++VJEISuEYopkvVNgu5Z4Y87KMaPAzj/NuXRRkpdrmc9BD289E1iySXSdbLjzg57dgk5 Forg==
In-reply-to: <20321.63389.586851.689070@xxxxxxxxxxxxxxxxxx>
References: <CAA8mOyDKrWg0QUEHxcD4ocXXD42nJu0TG+sXjC4j2RsigHTcmw@xxxxxxxxxxxxxx> <4F61803A.60009@xxxxxxxxxxxxxxxxx> <CAA8mOyCzs36YD_QUMq25HQf8zuq1=tmSTPjYdoFJwy2Oq9sLmw@xxxxxxxxxxxxxx> <20321.63389.586851.689070@xxxxxxxxxxxxxxxxxx>
Hey Peter,

On Thu, Mar 15, 2012 at 10:07 PM, Peter Grandi <pg@xxxxxxxxxxxxxxxxxxx> wrote:
>>>> I want to create a raid10,n2 using 3 1TB SATA drives.
>>>> I want to create an xfs filesystem on top of it. The
>>>> filesystem will be used as NFS/Samba storage.
> Consider also an 'o2' layout (it is probably the same thing for a
> 3 drive RAID10) or even a RAID5, as 3 drives and this usage seems
> one of the few cases where RAID5 may be plausible.

Thanks for reminding me about raid5. I'll probably give it a try and
do some benchmarks. I'd also like to try raid10f2.

>> [ ... ] I've run some benchmarks with dd trying the different
>> chunks and 256k seems like the sweetspot.  dd if=/dev/zero
>> of=/dev/md0 bs=64k count=655360 oflag=direct
> That's for bulk sequential transfers. Random-ish, as in a
> fileserver perhaps with many smaller files, may not be the same,
> but probably larger chunks are good.
>>> [ ... ] What kernel version?  This can make a significant
>>> difference in XFS metadata performance.
> As an aside, that's a myth that has been propagandaized by DaveC
> in his entertaining presentation not long ago.
> There have been decent but no major improvements in XFS metadata
> *performance*, but weaker implicit *semantics* have been made an
> option, and these have a different safety/performance tradeoff
> (less implicit safety, somewhat more performance), not "just"
> better performance.
> http://lwn.net/Articles/476267/
>  «In other words, instead of there only being a maximum of 2MB of
>  transaction changes not written to the log at any point in time,
>  there may be a much greater amount being accumulated in memory.
>  Hence the potential for loss of metadata on a crash is much
>  greater than for the existing logging mechanism.
>  It should be noted that this does not change the guarantee that
>  log recovery will result in a consistent filesystem.
>  What it does mean is that as far as the recovered filesystem is
>  concerned, there may be many thousands of transactions that
>  simply did not occur as a result of the crash.
>  This makes it even more important that applications that care
>  about their data use fsync() where they need to ensure
>  application level data integrity is maintained.»
>>>  Your NFS/Samba workload on 3 slow disks isn't sufficient to
>>> need that much in memory journal buffer space anyway.
> That's probably true, but does no harm.
>>>  XFS uses relatime which is equivalent to noatime WRT IO
>>> reduction performance, so don't specify 'noatime'.
> Uhm, not so sure, and 'noatime' does not hurt either.
>> I just wanted to be explicit about it so that I know what is
>> set just in case the defaults change
> That's what I do as well, because relying on remembering exactly
> what the defaults are can cause sometimes confusion. But it is a
> matter of taste to a large degree, like 'noatime'.
>>> In fact, it appears you don't need to specify anything in
>>> mkfs.xfs or fstab, but just use the defaults.  Fancy that.
> For NFS/Samba, especially with ACLs (SMB protocol), and
> especially if one expects largish directories, and in general I
> would recommend a larger inode size, at least 1024B, if not even
> 2048B.

thanks for this tip. will look into adjusting inode size.

> Also, as a rule I want to make sure that the sector size is set
> to 4096B, for future proofing (and recent drives not only have
> 4096B sectors but usually lie).

it seems the 1TB drivers that I have still have 512byte sectors

>>>  And the one thing that might actually increase your
>>> performance a little bit you didn't specify--sunit/swidth.
> Especially 'sunit', as XFS ideally would align metadata on chunk
> boundaries.
>>>  However, since you're using mdraid, mkfs.xfs will calculate
>>> these for you (which is nice as mdraid10 with odd disk count
>>> can be a tricky calculation).
> Ambiguous more than tricky, and not very useful, except the chunk
> size.
>>>> Will my files be safe even on sudden power loss?
> The answer is NO, if you mean "absolutely safe". But see the
> discussion at the end.
>>> [ ... ]  Application write behavior does play a role.
> Indeed, see the discussion at the end and ways to mitigate.
>>>  UPS with shutdown scripts, and persistent write cache prevent
>>> this problem. [ ... ]
> There is always the problem of system crashes that don't depend
> on power....
>>>> Is barrier=1 enough?  Do i need to disable the write cache?
>>>> with: hdparm -W0 /dev/sdb /dev/sdc /dev/sdd
>>> Disabling drive write caches does decrease the likelihood of
>>> data loss.
>>>> I tried it but performance is horrendous.
>>> And this is why you should leave them enabled and use
>>> barriers.  Better yet, use a RAID card with BBWC and disable
>>> the drive caches.
>> Budget does not allow for RAID card with BBWC
> You'd be surprised by how cheap you can get one. But many HW host
> adapters with builtin cache have bad performance or horrid bugs,
> so you'd have to be careful.

could you please suggest a hardware raid card with BBU that's cheap?

> In any case that's not the major problem you have.
>>>> Am I better of with ext4? Data safety/integrity is the
>>>> priority and optimization affecting it is not acceptable.
> XFS is the filesystem of the future ;-). I would choose it over
> 'ext4' in every plausible case.
>> nightly backups will be stored on an external USB disk
> USB is an unreliable, buggy transport, and slow, eSATA is
> enormously better and faster.
>> is xfs going to be prone to more data loss in case the
>> non-redundant power supply goes out?
> That's the wrong question entirely. Data loss can happen for many
> other reasons, and XFS is probably one of the safest designs, if
> properly used and configured. The problems are elsewhere.

Can you please elaborate how xfs can be properly used and configured?
>> I just updated the kernel to 3.0.0-16.  Did they take out
>> barrier support in mdraid? or was the implementation replaced
>> with FUA?  Is there a definitive test to determine if the off
>> the shelf consumer sata drives honor barrier or cache flush
>> requests?
> Usually they do, but that's the least of your worries. Anyhow a
> test that occurs to me is to write a know pattern to a file,
> let's say 1GiB, then 'fsync', and as soon as 'fsync' completes,
> power off. Then check whether the whole 1GiB is the known pattern.
>> I think I'd like to go with device cache turned ON and barrier
>> enabled.
> That's how it is supposed to work.
> As to general safety issues, there seem to be some misunderstanding,
> and I'll try to be more explicit than "lob the grenade" notion.
> It matters a great deal what "safety" means in your mind and that
> of your users. As a previous comment pointed out, that usually
> involves backups, that is data that has already been stored.
> But your insistence on power off and disk caches etc. seems to
> indicate that "safety" in your mind means "when I click the
> 'Save' button it is really saved and not partially".
let me define safety as needed by the usecase:
fileA is a 2MB open office document file already existing on the file system.
userA opens fileA locally, modifies a lot of lines and attempts to save it.
as the saving operation is proceeding, the PSU goes haywire and power
is cut abruptly.
When the system is turned on, i expect some sort of recovery process
to bring the filesystem to a consistent state.
I expect fileA should be as it was before the save operation and
should not be corrupted in anyway.
Am I asking/expecting too much?

> As to that there quite a lot of qualifiers:
>  * Most users don't understand that even in the best scenario a
>    file is really saved not when they *click* the 'Save' button,
>    but when they get the "Saved!" message. In between anything
>    can happen. Also, work in progress (not yet saved explicitly)
>    is fair game.
>  * "Really saved" is an *application* concern first and foremost.
>    The application *must* say (via 'fsync') that it wants the
>    data really saved. Unfortunately most applications don't do
>    that because "really saved" is a very expensive operation, and
>    usually sytems don't crash, so the application writer looks
>    like a genius if he has an "optimistic" attitude. If you do a
>    web search look for various O_PONIES discussions. Some intros:
>      http://lwn.net/Articles/351422/
>      http://lwn.net/Articles/322823/
>  * XFS (and to a point 'ext4') is designed for applications that
>    work correctly and issue 'fsync' appropriately, and if they do
>    it is very safe, because it tries hard to ensure that either
>    'fsync' means "really saved" or you know that it does not. XFS
>    takes advantage of the assumption that applications do the
>    right thing to do various latency-based optimizations between
>    calls to 'fsync'.
>  * Unfortunately most GUI applications don't do the right thing,
>    but fortunately you can compensate for that. The key here is
>    to make sure that the flusher's parameter are set for rather
>    more frequent flushing than the default, which is equivalent
>    to issuing 'fsync' systemwide fairly frequently. Ideally set
>    'vm/dirty_bytes' to something like 1-3 seconds of IO transfer
>    rate (and in reversal on some of my previous advice leave
>    'vm/dirty_background_bytes' to something quite large unless
>    you *really* want safety), and to shorten significantly
>    'vm/dirty_expire_centisecs', 'vm/dirty_writeback_centisecs'.
>    This defeats some XFS optimizations, but that's inevitable.
>  * In any case you are using NFS/Samba, and that opens a much
>    bigger set of issues, because caching happens on the clients
>    too: http://www.sabi.co.uk/0707jul.html#070701b
> Then Von Neuman help you if your users or you decide to store lots
> of messages in MH/Maildir style mailstores, or VM images on
> "growable" virtual disks.

what's wrong with VM images on "growable" virtual disks. are you
saying not to rely on lvm2 volumes?

> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

<Prev in Thread] Current Thread [Next in Thread>