xfs
[Top] [All Lists]

Writeback Stalls

To: "xfs@xxxxxxxxxxx" <xfs@xxxxxxxxxxx>
Subject: Writeback Stalls
From: Markus Stockhausen <stockhausen@xxxxxxxxxxx>
Date: Wed, 19 Sep 2012 18:36:11 +0000
Accept-language: de-DE, en-US
Thread-index: Ac2Wj1HT6pr2HR9YR5m1/8HFWskN8g==
Thread-topic: Writeback Stalls
Hello,

I'm not sure if you can help me with the following problem but I must start 
somewhere.

We have an small sized VMWare infrastructure. 8 hosts with round about 50 VMs. 
Their images are hosted on 2 Ubuntu 12.04 NFS storage servers with each of them
having a 14 disk RAID 6 array. On top of the array runs a single XFS filesystem 
with 
round about 10TB of disk space.

>From time to time we see stalls in the infrastructure. The machines become 
>unresponsive
and hang for a few seconds. The controller was the first item to be under 
suspect. 
But after a lot examination we created a very artifical setup that shows the 
real reasons
for the stalls: Writeback handling. The parameters are:

- set NFS to async
- disable controller and disk writeback cache
- enable cgroup
- set dirty_background_bytes to some very high value (4GB)
- set dirty_bytes to some very high value (4GB)
- set dirty_expire_centisecs to to some very high value (60 secs)
- set blkio.throttle.write_bps_device to a very low value (2MB/sek)

Now generate some write load on a Windows VM. During the test we observe what
is happening on the storage and the VM. The result is:

- dirty pages are increasing
- writeback is constantly 0
- VM is working well

At some point writeback is kicking in and all of a sudden the VM stalls. During
this time the setup shows

- most of the dirty pages are transfered to writeback pages
- writeback is done at the above set limit (2MB/sek)
- VM is not responding 

After writeback has finished everything goes back to normal. Additional remark:
VMs DO NOT hang if I create heavy writes on the XFS into non-VM related files.

We are interested in this kind of setup for several reasons:

1. Keep VMs reponsive

2. Allow VMs to generate short spikes of write I/Os at a higher rate than the 
disk subsystem is capable of. 

3. Write this data back to the disk in the background over a longer period
of time. Ensure that a limited writeback rate keeps enough headroom so that
read I/Os are not delayed too much.

Can you explain me if there is some blocking going on in XFS in the time
between start and end of writeback? If yes what can I do to achieve my
goals and stop the writeback stalls. 

For you help I thank you in advance.

Markus










Attachment: InterScan_Disclaimer.txt
Description: Text document

<Prev in Thread] Current Thread [Next in Thread>