xfs
[Top] [All Lists]

Re: page fault scalability (ext3, ext4, xfs)

To: Andy Lutomirski <luto@xxxxxxxxxxxxxx>
Subject: Re: page fault scalability (ext3, ext4, xfs)
From: Dave Chinner <david@xxxxxxxxxxxxx>
Date: Fri, 16 Aug 2013 07:39:28 +1000
Cc: Jan Kara <jack@xxxxxxx>, Theodore Ts'o <tytso@xxxxxxx>, Dave Hansen <dave.hansen@xxxxxxxxx>, Dave Hansen <dave.hansen@xxxxxxxxxxxxxxx>, Linux FS Devel <linux-fsdevel@xxxxxxxxxxxxxxx>, xfs@xxxxxxxxxxx, "linux-ext4@xxxxxxxxxxxxxxx" <linux-ext4@xxxxxxxxxxxxxxx>, LKML <linux-kernel@xxxxxxxxxxxxxxx>, Tim Chen <tim.c.chen@xxxxxxxxxxxxxxx>, Andi Kleen <ak@xxxxxxxxxxxxxxx>
Delivered-to: xfs@xxxxxxxxxxx
In-reply-to: <CALCETrU=Ag1bvmc=9Wo8K66gOSYtCyncveYEycYdTd_1T9z-JA@xxxxxxxxxxxxxx>
References: <CALCETrVaRQ3WQ5++Uu_0JTaVnjUugAaAhqQK__7r5YWvLxpAhw@xxxxxxxxxxxxxx> <20130815011101.GA3572@xxxxxxxxx> <20130815021028.GM6023@dastard> <CALCETrUfuzgG9U=+eSzCGvbCx-ZskWw+MhQ-qmEyWZK=XWNVmg@xxxxxxxxxxxxxx> <20130815060149.GP6023@dastard> <CALCETrUF+dGhE3qv4LoYmc7A=a+ry93u-d-GgHSAwHXvYN+VNw@xxxxxxxxxxxxxx> <20130815071141.GQ6023@dastard> <20130815074531.GA2147@xxxxxxxxxxxxx> <20130815212826.GS6023@dastard> <CALCETrU=Ag1bvmc=9Wo8K66gOSYtCyncveYEycYdTd_1T9z-JA@xxxxxxxxxxxxxx>
User-agent: Mutt/1.5.21 (2010-09-15)
On Thu, Aug 15, 2013 at 02:31:14PM -0700, Andy Lutomirski wrote:
> On Thu, Aug 15, 2013 at 2:28 PM, Dave Chinner <david@xxxxxxxxxxxxx> wrote:
> > On Thu, Aug 15, 2013 at 09:45:31AM +0200, Jan Kara wrote:
> >> On Thu 15-08-13 17:11:42, Dave Chinner wrote:
> >> > On Wed, Aug 14, 2013 at 11:14:37PM -0700, Andy Lutomirski wrote:
> >> > > On Wed, Aug 14, 2013 at 11:01 PM, Dave Chinner <david@xxxxxxxxxxxxx> 
> >> > > wrote:
> >> > > > On Wed, Aug 14, 2013 at 09:32:13PM -0700, Andy Lutomirski wrote:
> >> > > >> On Wed, Aug 14, 2013 at 7:10 PM, Dave Chinner <david@xxxxxxxxxxxxx> 
> >> > > >> wrote:
> >> > > >> > On Wed, Aug 14, 2013 at 09:11:01PM -0400, Theodore Ts'o wrote:
> >> > > >> >> On Wed, Aug 14, 2013 at 04:38:12PM -0700, Andy Lutomirski wrote:
> >> > > >> >> > > It would be better to write zeros to it, so we aren't 
> >> > > >> >> > > measuring the
> >> > > >> >> > > cost of the unwritten->written conversion.
> >> > > >> >> >
> >> > > >> >> > At the risk of beating a dead horse, how hard would it be to 
> >> > > >> >> > defer
> >> > > >> >> > this part until writeback?
> >> > > >> >>
> >> > > >> >> Part of the work has to be done at write time because we need to
> >> > > >> >> update allocation statistics (i.e., so that we don't have ENOSPC
> >> > > >> >> problems).  The unwritten->written conversion does happen at 
> >> > > >> >> writeback
> >> > > >> >> (as does the actual block allocation if we are doing delayed
> >> > > >> >> allocation).
> >> > > >> >>
> >> > > >> >> The point is that if the goal is to measure page fault 
> >> > > >> >> scalability, we
> >> > > >> >> shouldn't have this other stuff happening as the same time as 
> >> > > >> >> the page
> >> > > >> >> fault workload.
> >> > > >> >
> >> > > >> > Sure, but the real problem is not the block mapping or allocation
> >> > > >> > path - even if the test is changed to take that out of the 
> >> > > >> > picture,
> >> > > >> > we still have timestamp updates being done on every single page
> >> > > >> > fault. ext4, XFS and btrfs all do transactional timestamp updates
> >> > > >> > and have nanosecond granularity, so every page fault is resulting 
> >> > > >> > in
> >> > > >> > a transaction to update the timestamp of the file being modified.
> >> > > >>
> >> > > >> I have (unmergeable) patches to fix this:
> >> > > >>
> >> > > >> http://comments.gmane.org/gmane.linux.kernel.mm/92476
> >> > > >
> >> > > > The big problem with this approach is that not doing the
> >> > > > timestamp update on page faults is going to break the inode change
> >> > > > version counting because for ext4, btrfs and XFS it takes a
> >> > > > transaction to bump that counter. NFS needs to know the moment a
> >> > > > file is changed in memory, not when it is written to disk. Also, NFS
> >> > > > requires the change to the counter to be persistent over server
> >> > > > failures, so it needs to be changed as part of a transaction....
> >> > >
> >> > > I've been running a kernel that has the file_update_time call
> >> > > commented out for over a year now, and the only problem I've seen is
> >> > > that the timestamp doesn't get updated :)
> >> > >
> >> > > I think I must be misunderstanding you (or vice versa).  I'm currently
> >> >
> >> > Yup, you are.
> >> >
> >> > > redoing the patches, and this time I'll do it for just the mm core and
> >> > > ext4.  The only change I'm proposing to ext4's page_mkwrite is to
> >> > > remove the file_update_time call.
> >> >
> >> > Right. Where does that end up? All the way down in
> >> > ext4_mark_iloc_dirty(), and that does:
> >> >
> >> >         if (IS_I_VERSION(inode))
> >> >             inode_inc_iversion(inode);
> >> >
> >> > The XFS transaction code is the same - deep inside it where an inode
> >> > is marked as dirty in the transaction, it bumps the same counter and
> >> > adds it to the transaction.
> >>   Yeah, I'd just add that ext4 maintains i_version only if it has been
> >> mounted with i_version mount option. But then NFS server would depend on
> >> c/mtime update so it won't help you much - you still should update at least
> >> one of i_version, ctime, mtime on page fault. OTOH if the filesystem isn't
> >> exported, you could avoid this relatively expensive dance and defer things
> >> as Andy suggests.
> >
> > The problem with "not exported, don't update" is that files can be
> > modified on server startup (e.g. after a crash) or in short
> > maintenance periods when the NFS service is down. When the server is
> > started back up, the change number needs to indicate the file has
> > been modified so that clients reconnecting to the server see the
> > change.
> >
> > IOWs, even if the NFS server is not up or the filesystem not
> > exported we still need to update change counts whenever a file
> > changes if we are going to tell the NFS server that we keep them...
> 
> This will keep working as long as the clients are willing to wait for
> writeback (or msync, munmap, or exit) on the server.

I don't follow you - what will keep working? If we don't record
changes while the filesystem is not exported, then NFS clients can't
determine if files have changed while the server was down for a
period....

Cheers,

Dave.
-- 
Dave Chinner
david@xxxxxxxxxxxxx

<Prev in Thread] Current Thread [Next in Thread>