On Wed, Jul 30, 2014 at 01:44:49PM +0200, Frank . wrote:
> Indeed, I turned sync and wsync flags on. As excpected, I had terribly low
> performance (1MB/s for write operations). So I decided to turn them back off.
> (I got my 100 MB/s write throughput back).
> I just wanted to reduce as much as possible unnecessary cache between my VM's
> and my physcal hard drives knowing that there are up to 8 write cache levels.
> I'm getting off the subject a bit but here is the list. This is only my
> conclusion. I don't know if I'm right.
> - Guest page cache.
> - Virtual disk drive write cache. (off KVM cache=directsync)
> - Host page cache. (off KVM cache=directsync)
Pretty normal. I tend to use cache=none rather than cache=directsync
because cache=none behaves exactly like a normal disk, including
write cache behaviour. So as long as you use barriers in your guest
filesystems (xfs, ext4, btrfs all do by default) then it is
no different to running the guest on a real disk with a small
volatile write cache.
i.e. when your app/database issues a fsync() in the guest, the guest
filesystem issues a flush/fua sequence and KVM then guarantees that
it only returns when all the previously written data to that file is
on stable storage. As long as all the layers below KVM provide this
same guarantee, then you don't need to turn caches off at all.
> - GlusterFS cache. (off)
> - NAS page cache. (?)
> - XFS cache (filesystem).
The gluster client side cache is being avoided due to KVM direct IO
config, the gluster server/NAS page cache/XFS cache are all the same
thing from a data perspective (i.e. 1 layer, not 3). AFAIK this is
all buffered IO, and so the only way to get data in the backing XFS
filesystem consistent on disk is for the application to issue a
fsync() on the file at the gluster client side. This comes from the
guest via KVM translating flush/fua operations or via the KVM IO
mechanism - gluster then takes care of the rest.
If KVM never issues a fsync() operation, then lower level caches
will never be flushed correctly regardless of whether you turn off
all caching or not. IOWs, fsync() is required at the XFS level to
synchronise allocation transactions with data writes, and the only
way to have that happen for the layer above xfs to issue
f[data]sync() on the relevant XFS file(s)...
Hence you need to keep in mind that turning off high level caches
does not guarantee that low level caching behaviour will behave as
you expect - even with high level caching turned off you still need
those layers to propagate the data integrity directives from the top
of the stack to the bottom so that every layer can do the right
thing regardless of whether they are caching data or not.
i.e. caching doesn't cause data loss - it's the incorrect
propagation or non-existant use of application level data
synchronisation primitives that cause data loss....
> - RAID controller write cache. (off)
There's no benefit to turning this off if it's battery backed - all
turning it off will do is cause performance to be horrible,
especially when you turn off all the other layers of caching above
the RAID controller.
> - Physical hard drive write cache. (off)
Right, those definitely need to be off so that the RAID controller
doesn't have internal consistency problems when power fails.