xfs
[Top] [All Lists]

Re: A huge latency in ext4 and xfs because of stable page write

To: Dave Chinner <david@xxxxxxxxxxxxx>
Subject: Re: A huge latency in ext4 and xfs because of stable page write
From: Zheng Liu <gnehzuil.liu@xxxxxxxxx>
Date: Wed, 12 Dec 2012 12:18:03 +0800
Cc: linux-ext4@xxxxxxxxxxxxxxx, xfs@xxxxxxxxxxx, linux-fsdevel@xxxxxxxxxxxxxxx, tytso@xxxxxxx, hch@xxxxxxxxxxxxx, darrick.wong@xxxxxxxxxx
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=cezgh0yUeU5xR3aABYeX2XzmOifJUOJTGWdI0lbVXk0=; b=aBfu+YAHEw8GAEDXCMLbZSIdwC3qhfnRYhAMKv7+nISETKbPUr6vDnwy0KKD7I6dWN MJk9r03fgdwidvI60Sd/o3nFGM/y/bRMMioLkrK014G28PpICo7NW1zOdqk8JDe8s9Yb insJxW7tXDjuYmhkNQ7p2MK3Ed8fYQkw1JZDDmCKPCUf8Wu77zQJ68VJqFiJLA6MX7eS //2imAA94lpD8hc5LHT3I2N9x4V4CS5Z3kjhwtticEWjLFkmpyx+TjI8bQa2aNpwBljF HtHUdcw+z9pJxogNdqUUEpjI67gs97WiIdfjk/6VH8hu14bBD0JVsAgdhasTHyI+eYWz zW/A==
In-reply-to: <20121211111707.GU15784@dastard>
Mail-followup-to: Dave Chinner <david@xxxxxxxxxxxxx>, linux-ext4@xxxxxxxxxxxxxxx, xfs@xxxxxxxxxxx, linux-fsdevel@xxxxxxxxxxxxxxx, tytso@xxxxxxx, hch@xxxxxxxxxxxxx, darrick.wong@xxxxxxxxxx
References: <20121211084520.GA13277@xxxxxxxxx> <20121211111707.GU15784@dastard>
User-agent: Mutt/1.5.21 (2010-09-15)
On Tue, Dec 11, 2012 at 10:17:07PM +1100, Dave Chinner wrote:
> On Tue, Dec 11, 2012 at 04:45:21PM +0800, Zheng Liu wrote:
> > Hi all,
> > 
> > At Tao Bao we meet a problem in our product system which causes a huge 
> > latency
> > because of stable page write.  This problem is easy to reproduce in a 
> > testing
> > environment, and I can reproduce it in my desktop with a SATA disk.  Here 
> > is the
> > fio config file that is used to reproduce this problem.
> > 
> > config file
> > -----------
> > [global]
> > iodepth=1
> > directory=/mnt/sda3
> > direct=0
> > group_reporting
> > thread
> > fallocate=0
> > runtime=120
> > 
> > [log-append]
> > ioengine=sync
> > rw=write
> > bs=1k
> 
> Sub-page sized IO. That's guaranteed to have noticable IO latency
> anomalies, regardless of stable data pages. If you are just
> doing appending writes, then you can easily buffer them till you
> have a page of data to write and avoid the problem altogether.

Hi Dave,

Thanks for your reply.  I agree with you that the application still
could be improved.  But, unfortunately, the real life is that we cannot
change the application.  We couldn't convince the application developers
that they need to change their program to adapt to the latest kernel
because the latest kernel has a new feature that is not useful for them.
That is unacceptable for them. :-(

> 
> > Hence, I wonder whether or not we could revert stable page write 
> > temporarily.
> > After it is improved, we could add it back again.
> 
> The plan is to turn it off for filesystems/devices that don't
> require it. That list of devices will grow in future, so you
> probably should plan to handle latencies in the application
> properly...

I wonder whether we can provide a sysctl to turn on/off stable page
write.  At least we need to give sysadmin an opportunity to control it.

Regards,
                                                - Zheng

> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@xxxxxxxxxxxxx
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

<Prev in Thread] Current Thread [Next in Thread>