[Top] [All Lists]

Re: [PATCH] xfs: reduce lock traffic on incore sb lock

To: Christoph Hellwig <hch@xxxxxxxxxxxxx>
Subject: Re: [PATCH] xfs: reduce lock traffic on incore sb lock
From: Dave Chinner <david@xxxxxxxxxxxxx>
Date: Wed, 29 Sep 2010 15:57:48 +1000
Cc: xfs@xxxxxxxxxxx
In-reply-to: <20100929040425.GA29691@xxxxxxxxxxxxx>
References: <1285721500-5671-1-git-send-email-david@xxxxxxxxxxxxx> <20100929040425.GA29691@xxxxxxxxxxxxx>
User-agent: Mutt/1.5.20 (2009-06-14)
On Wed, Sep 29, 2010 at 12:04:25AM -0400, Christoph Hellwig wrote:
> On Wed, Sep 29, 2010 at 10:51:40AM +1000, Dave Chinner wrote:
> > From: Dave Chinner <dchinner@xxxxxxxxxx>
> > 
> > Under heavy parallel unlink workloads, the incore superblock lock is
> > heavily trafficed in xfs_mod_incore_sb_batch(). This is despite the
> > fact that the counters being modified are typically the counters
> > that are per-cpu and do not require the lock. IOWs, we are locking
> > and unlocking the superblock lock needlessly, and the result is that
> > it is third most heavily contended lock in the system under these
> > workloads.
> > 
> > Fix this by only locking the superblock lock when we are modifying a
> > counter protected by it. This completely removes the m_sb_lock from
> > lock_stat traces during create/remove workloads.
> God spot of the idiocy there, but I really don't like the patch.

Fair enough - it is a rather quick hack. ;)

> I've started writing a small patches series solving the issue slightly
> better by cleaning up this area a bit.

Oh, cool. That code is quite a tangle.

> After this we will never use xfs_mod_incore_sb/xfs_mod_incore_sb_batch
> for the percpu counters but rather make those always go through
> xfs_icsb_modify_counters.  I'll need to quickly finish it up and will
> send it out soon.

FWIW, I've got a prototype that converts the per-cpu counters to the
generic per-cpu counter infrastructure. It chops out almost all the
xfs_icsb_* stuff (including xfs_icsb_modify_counters()) and has a
diffstat of:

6 files changed, 317 insertions(+), 709 deletions(-)

It needs a significant cleanup of xfs_mod_incore_sb() before/after
the conversion which I haven't done yet because I haven't quite got
my new percpu_counter_test_and_add_delta() function working
correctly yet. I spotted this locking problem when testing the

That said, there's no reason why my percpu counter code needs to run
through xfs_mod_incore_sb() at all. If we have a separate path for
per-cpu counters then I can rework my code on top of that....


Dave Chinner

<Prev in Thread] Current Thread [Next in Thread>