xfs
[Top] [All Lists]

pdflush not flushing all dirty_expire_centisecs

To: Linux xfs mailing list <linux-xfs@xxxxxxxxxxx>
Subject: pdflush not flushing all dirty_expire_centisecs
From: Alexander Bergolth <leo@xxxxxxxxxxxxxxxxxxxx>
Date: Fri, 17 Mar 2006 16:06:44 +0100
Sender: linux-xfs-bounce@xxxxxxxxxxx
User-agent: Mozilla Thunderbird 1.0.7-1.1.fc3 (X11/20050929)
Hi!

On my FC3 systems, I'm experiencing the problem that dirty buffers are
not periodically flushed under certain conditions.

I noticed that on several occasions:

* Unplugged a memory stick and shut down the system a few minutes later
(using a normal shutdown -h now). Unplugging the memory-stick caused the
hal-daemon to rewrite fstab via fstab-sync. On reboot, /etc/fstab was
filled with zeroes.
* Upgraded several RPMs via apt-get upgrade. About one hour later, the
system crashed and all files that had been upgraded were filled with zeroes.

I don't think that the problem is XFS-specific as it does also happen on
boxes with ext3 filesystems but maybe anyone has a hint...

According to the kernel-docs, pdflush should wake up all
dirty_writeback_centisecs (=500 on my system) and write all buffers that
have been dirty for more than 30 seconds
(/proc/sys/vm/dirty_expire_centisecs = 3000) to disk.

However, on my systems this periodic flushing doesn't seem to be done:

* flushing seems to be only triggered by the amount of dirty memory, not
at regular intervals
* even the sync command does only flush a small amount of the dirty buffers
* a freshly booted system doesn't show these symptoms

I've written a small test-script to visualize the behavior. (Attached.)

The script creates a 200MB file, monitors nr_dirty in /proc/vmstat and
executes sync after some time.

The output looks like that:

-------------------- snip! bad: --------------------
Linux slime.wu-wien.ac.at 2.6.14-1.1653_FC4smp #1 SMP Tue Dec 13
21:46:01 EST 2005 i686 i686 i386 GNU/Linux
 12:22:46 up 10 days, 18:12, 37 users,  load average: 0.17, 0.15, 0.09
12:22:46 start: head -c 200000000 /dev/zero
>/var/tmp/dirty-buffers.EFYFF13399 # nr_dirty 1076
12:22:46 # nr_dirty 1805
12:22:47 end: head -c 200000000 /dev/zero
>/var/tmp/dirty-buffers.EFYFF13399 # nr_dirty 31061
12:22:51 # nr_dirty 25671
12:22:56 # nr_dirty 25724
12:23:01 # nr_dirty 25724
12:23:06 # nr_dirty 25724
12:23:11 # nr_dirty 25724
12:23:16 # nr_dirty 25724
12:23:21 # nr_dirty 25724
12:23:26 # nr_dirty 25724
12:23:31 # nr_dirty 25724
12:23:36 # nr_dirty 25724
12:23:41 # nr_dirty 25724
12:23:47 # nr_dirty 25725
12:23:52 # nr_dirty 25726
12:23:57 # nr_dirty 25728
12:24:02 # nr_dirty 25728
12:24:07 # nr_dirty 25728
12:24:12 # nr_dirty 25728
12:24:12 # nr_dirty 25728
12:24:12 start: sync # nr_dirty 25728
12:24:12 end: sync # nr_dirty 23566
12:24:17 # nr_dirty 23566
12:24:22 # nr_dirty 23582
12:24:27 # nr_dirty 23583
12:24:32 # nr_dirty 23583
12:24:37 # nr_dirty 23583
12:24:42 # nr_dirty 23583
12:24:47 # nr_dirty 23583
12:24:52 # nr_dirty 23583
12:24:57 # nr_dirty 23583
-------------------- snip! --------------------

While writing the temp-file, some buffers are flushed. (31061->25671)
But after writing is completed, the 25000 buffers remain dirty and are
not flushed after 30 secs, as I would expect. The sync causes the dirty
buffers to shrink from 25728 to 23566 but I'd expect that sync should
cause them to become near 0.

Here is the output of another system with a lower uptime that doesn't
show that behavior yet:

-------------------- snip! good: --------------------
Linux roaster.wu-wien.ac.at 2.6.12-1.1376_FC3.stk16smp #1 SMP Mon Aug 29
16:41:37 EDT 2005 i686 i686 i386 GNU/Linux
 12:44:54 up 3 days,  1:50,  2 users,  load average: 0.00, 0.16, 0.14
12:44:54 start: head -c 200000000 /dev/zero
>/tmp/dirty-buffers.cgRFjZ1720 # nr_dirty 2
12:44:54 # nr_dirty 2
12:44:55 end: head -c 200000000 /dev/zero >/tmp/dirty-buffers.cgRFjZ1720
# nr_dirty 31257
12:44:59 # nr_dirty 22239
12:45:04 # nr_dirty 22239
12:45:09 # nr_dirty 22239
12:45:14 # nr_dirty 22240
12:45:19 # nr_dirty 22240
12:45:24 # nr_dirty 22240
12:45:29 # nr_dirty 4830
12:45:34 # nr_dirty 1
12:45:39 # nr_dirty 1
12:45:44 # nr_dirty 2
12:45:49 # nr_dirty 2
12:45:54 # nr_dirty 2
12:45:59 # nr_dirty 2
12:46:04 # nr_dirty 1
12:46:09 # nr_dirty 1
12:46:14 # nr_dirty 1
12:46:19 # nr_dirty 1
12:46:19 # nr_dirty 1
12:46:19 start: sync # nr_dirty 1
12:46:19 end: sync # nr_dirty 0
12:46:24 # nr_dirty 0
12:46:29 # nr_dirty 0
12:46:34 # nr_dirty 0
12:46:39 # nr_dirty 0
12:46:44 # nr_dirty 0
12:46:49 # nr_dirty 0
12:46:54 # nr_dirty 0
12:46:59 # nr_dirty 1
12:47:04 # nr_dirty 1
-------------------- snip! --------------------

Any help is greatly appreciated.

Cheers,
--leo
-- 
-----------------------------------------------------------------------
Alexander.Bergolth@xxxxxxxxxxxxx                Fax: +43-1-31336-906050
Zentrum fuer Informatikdienste - Wirtschaftsuniversitaet Wien - Austria

Attachment: dirty-buffers.sh
Description: application/shellscript

<Prev in Thread] Current Thread [Next in Thread>