Hi,
What we were actually getting bitten by was much worse than this, but
purely an xfs issue. We have an f_ops->release method which is getting
called out of nfsd_close, it does some operations more in line with a
close system call than every write to the file. I made changes within the
xfs code to avoid this issue and our performance went way up. So I cannot
say we have a scenario where the dcache entry is getting pruned out from
under us all the time.
Steve
> On Tuesday March 6, ak@xxxxxxx wrote:
> >
> > I think it would be best to fix nfsd to cache file handles instead of dentr
> ies
> > and only do release when the nfsfh leaves the nfsfh cache.
> > This way could also help ext2, which caches readahead context in struct fil
> e.
> >
> > Put Neil Brown into cc. Neil, do you see any obstacles with that?
> >
> >
> > -Andi
>
> Firstly, lets check that I understand the situation.
>
> You find that under high memory pressure dentries for files that are
> currently being used by an NFS client get pruned from the dcache (and
> so the associated inodes leave the icache) resulting in slower
> filehandle-to-dentry lookup times when the next access is made.
>
> You are suggesting that some sort of filehandle cache go into nfsd to
> hold onto these denties a bit longer so that on the next lookup they
> are more likely to still be there.
>
> Is that right?
>
> I had hoped that the normal dcache mechanisms would have held onto
> them long enough, but I can see that under high memory pressure this
> may not be long enough.
> It might be enough just to add:
>
> if (!IS_ERR(result) result->d_flags |= DCACHE_REFERENCED;
>
> after the first call to nfsd_iget in find_fh_dentry.
> This would give the dentry an extra turn around the lru list.
> If that isn't enough, I think that the best fix would be to modify
> prune_dcache to be a bit more generous.
> Something like:
>
> add
>
> #define DCACHE_DELAY 0x0010 /* recently used, don't discard for 20ms */
>
> to dache.h, then change prune_dcache to add
>
> if ((dentry->d_flags & DCACHE_DELAY) &&
> time_before(jiffies, dentry->d_reftime+(HZ-1)/50+1)) {
> list_add(&dentry->d_lru, &dentry_unused);
> goto next;
> }
>
> Then whenever nfsd lets go of a dentry after an operation that would
> likely be followed by some other operations on the same dentry, the
> DCACHE_DELAY bit would get set.
>
> Ofcourse you would have to try to get that change passed Linux and
> AlViro, but they might go for it, especially if you can point to
> performance numbers. The "20ms" is a bit icky but I don't know what
> else to do about that.
>
> You could ofcourse do all this inside knfsd, or even inside the
> individual filesystem, but it makes the code a lot more complex. You
> need to make sure you flush the cache in a timely fashion, and let go
> of denties that have been unlinked and all that stuff.
> If we get the dcache to do the delay, then it is easy as it already
> copes with those other things.
>
> NeilBrown
|