[Top] [All Lists]

Re: page fault scalability (ext3, ext4, xfs)

To: Andy Lutomirski <luto@xxxxxxxxxxxxxx>
Subject: Re: page fault scalability (ext3, ext4, xfs)
From: Dave Chinner <david@xxxxxxxxxxxxx>
Date: Thu, 15 Aug 2013 17:11:42 +1000
Cc: Theodore Ts'o <tytso@xxxxxxx>, Dave Hansen <dave.hansen@xxxxxxxxx>, Dave Hansen <dave.hansen@xxxxxxxxxxxxxxx>, Linux FS Devel <linux-fsdevel@xxxxxxxxxxxxxxx>, xfs@xxxxxxxxxxx, "linux-ext4@xxxxxxxxxxxxxxx" <linux-ext4@xxxxxxxxxxxxxxx>, Jan Kara <jack@xxxxxxx>, LKML <linux-kernel@xxxxxxxxxxxxxxx>, Tim Chen <tim.c.chen@xxxxxxxxxxxxxxx>, Andi Kleen <ak@xxxxxxxxxxxxxxx>
Delivered-to: xfs@xxxxxxxxxxx
In-reply-to: <CALCETrUF+dGhE3qv4LoYmc7A=a+ry93u-d-GgHSAwHXvYN+VNw@xxxxxxxxxxxxxx>
References: <520BB9EF.5020308@xxxxxxxxxxxxxxx> <20130814194359.GA22316@xxxxxxxxx> <520BED7A.4000903@xxxxxxxxx> <20130814230648.GD22316@xxxxxxxxx> <CALCETrVaRQ3WQ5++Uu_0JTaVnjUugAaAhqQK__7r5YWvLxpAhw@xxxxxxxxxxxxxx> <20130815011101.GA3572@xxxxxxxxx> <20130815021028.GM6023@dastard> <CALCETrUfuzgG9U=+eSzCGvbCx-ZskWw+MhQ-qmEyWZK=XWNVmg@xxxxxxxxxxxxxx> <20130815060149.GP6023@dastard> <CALCETrUF+dGhE3qv4LoYmc7A=a+ry93u-d-GgHSAwHXvYN+VNw@xxxxxxxxxxxxxx>
User-agent: Mutt/1.5.21 (2010-09-15)
On Wed, Aug 14, 2013 at 11:14:37PM -0700, Andy Lutomirski wrote:
> On Wed, Aug 14, 2013 at 11:01 PM, Dave Chinner <david@xxxxxxxxxxxxx> wrote:
> > On Wed, Aug 14, 2013 at 09:32:13PM -0700, Andy Lutomirski wrote:
> >> On Wed, Aug 14, 2013 at 7:10 PM, Dave Chinner <david@xxxxxxxxxxxxx> wrote:
> >> > On Wed, Aug 14, 2013 at 09:11:01PM -0400, Theodore Ts'o wrote:
> >> >> On Wed, Aug 14, 2013 at 04:38:12PM -0700, Andy Lutomirski wrote:
> >> >> > > It would be better to write zeros to it, so we aren't measuring the
> >> >> > > cost of the unwritten->written conversion.
> >> >> >
> >> >> > At the risk of beating a dead horse, how hard would it be to defer
> >> >> > this part until writeback?
> >> >>
> >> >> Part of the work has to be done at write time because we need to
> >> >> update allocation statistics (i.e., so that we don't have ENOSPC
> >> >> problems).  The unwritten->written conversion does happen at writeback
> >> >> (as does the actual block allocation if we are doing delayed
> >> >> allocation).
> >> >>
> >> >> The point is that if the goal is to measure page fault scalability, we
> >> >> shouldn't have this other stuff happening as the same time as the page
> >> >> fault workload.
> >> >
> >> > Sure, but the real problem is not the block mapping or allocation
> >> > path - even if the test is changed to take that out of the picture,
> >> > we still have timestamp updates being done on every single page
> >> > fault. ext4, XFS and btrfs all do transactional timestamp updates
> >> > and have nanosecond granularity, so every page fault is resulting in
> >> > a transaction to update the timestamp of the file being modified.
> >>
> >> I have (unmergeable) patches to fix this:
> >>
> >> http://comments.gmane.org/gmane.linux.kernel.mm/92476
> >
> > The big problem with this approach is that not doing the
> > timestamp update on page faults is going to break the inode change
> > version counting because for ext4, btrfs and XFS it takes a
> > transaction to bump that counter. NFS needs to know the moment a
> > file is changed in memory, not when it is written to disk. Also, NFS
> > requires the change to the counter to be persistent over server
> > failures, so it needs to be changed as part of a transaction....
> I've been running a kernel that has the file_update_time call
> commented out for over a year now, and the only problem I've seen is
> that the timestamp doesn't get updated :)
> I think I must be misunderstanding you (or vice versa).  I'm currently

Yup, you are.

> redoing the patches, and this time I'll do it for just the mm core and
> ext4.  The only change I'm proposing to ext4's page_mkwrite is to
> remove the file_update_time call.

Right. Where does that end up? All the way down in
ext4_mark_iloc_dirty(), and that does:

        if (IS_I_VERSION(inode))

The XFS transaction code is the same - deep inside it where an inode
is marked as dirty in the transaction, it bumps the same counter and
adds it to the transaction.

If a filesystem is providing an i_version value, then NFS uses it to
determine whether client side caches are still consistent with the
server state. If the filesystem does not provide an i_version, then
NFS falls back to checking c/mtime for changes. If files on the
server are being modified without either the tiemstamps or i_version
changing, then it's likely that there will be problems with client
side cache consistency....

> Instead, ext4 will call
> file_update_time on munmap, exit, MS_ASYNC, and at the end of
> writepages.  Unless I'm missing something, there's no need to
> unconditionally start a transaction on page_mkwrite (and there had
> better not be, because file_update_time won't start a transaction if
> the time doesn't change).

Right, there's no unconditional need for a transaction except if the
filesystem is providing the inode version change feature for NFS.
ext4, btrfs and XFS all do this unconditionally, and so therefore
those filesystem have a need for an inode change transaction on
every page fault, just like they do for every write(2) call.


Dave Chinner

<Prev in Thread] Current Thread [Next in Thread>