[PATCH] xfs: reduce lock traffic on incore sb lock

[Dave Chinner]

On Wed, Sep 29, 2010 at 12:04:25AM -0400, Christoph Hellwig wrote:
> On Wed, Sep 29, 2010 at 10:51:40AM +1000, Dave Chinner wrote:
> > From: Dave Chinner <dchinner at redhat.com>
> > 
> > Under heavy parallel unlink workloads, the incore superblock lock is
> > heavily trafficed in xfs_mod_incore_sb_batch(). This is despite the
> > fact that the counters being modified are typically the counters
> > that are per-cpu and do not require the lock. IOWs, we are locking
> > and unlocking the superblock lock needlessly, and the result is that
> > it is third most heavily contended lock in the system under these
> > workloads.
> > 
> > Fix this by only locking the superblock lock when we are modifying a
> > counter protected by it. This completely removes the m_sb_lock from
> > lock_stat traces during create/remove workloads.
> 
> God spot of the idiocy there, but I really don't like the patch.

Fair enough - it is a rather quick hack. ;)

> I've started writing a small patches series solving the issue slightly
> better by cleaning up this area a bit.

Oh, cool. That code is quite a tangle.

> After this we will never use xfs_mod_incore_sb/xfs_mod_incore_sb_batch
> for the percpu counters but rather make those always go through
> xfs_icsb_modify_counters.  I'll need to quickly finish it up and will
> send it out soon.

FWIW, I've got a prototype that converts the per-cpu counters to the
generic per-cpu counter infrastructure. It chops out almost all the
xfs_icsb_* stuff (including xfs_icsb_modify_counters()) and has a
diffstat of:

6 files changed, 317 insertions(+), 709 deletions(-)

It needs a significant cleanup of xfs_mod_incore_sb() before/after
the conversion which I haven't done yet because I haven't quite got
my new percpu_counter_test_and_add_delta() function working
correctly yet. I spotted this locking problem when testing the
patch...

That said, there's no reason why my percpu counter code needs to run
through xfs_mod_incore_sb() at all. If we have a separate path for
per-cpu counters then I can rework my code on top of that....

Cheers,

Dave.
-- 
Dave Chinner
david at fromorbit.com