[Top] [All Lists]

Re: PARTIAL TAKE 946630 - Contention on incore superblock lock

To: Christoph Hellwig <hch@xxxxxxxxxxxxx>
Subject: Re: PARTIAL TAKE 946630 - Contention on incore superblock lock
From: David Chinner <dgc@xxxxxxx>
Date: Thu, 9 Feb 2006 18:30:04 +1100
Cc: David Chinner <dgc@xxxxxxx>, linux-xfs@xxxxxxxxxxx, sgi.bugs.xfs@xxxxxxxxxxxx
In-reply-to: <20060208092128.GA9543@xxxxxxxxxxxxx>
References: <20060203061930.8672A494E42E@xxxxxxxxxxxxxxxxxxxxxxx> <20060208092128.GA9543@xxxxxxxxxxxxx>
Sender: linux-xfs-bounce@xxxxxxxxxxx
User-agent: Mutt/
On Wed, Feb 08, 2006 at 09:21:28AM +0000, Christoph Hellwig wrote:
> On Fri, Feb 03, 2006 at 05:19:30PM +1100, David Chinner wrote:
> > When running parallel I/O threads on machines with lots of
> > CPUs, the incore superblock lock becomes the limiting factor
> > for buffered write throughput. Make the contended fields in
> > the incore superblock use per-cpu counters so that there is
> > no global lock to limit scalability.
> This looks like it's wrong in case of hotplug cpus.

Yup, I know. There's even a comment to say that it doesn't yet
support hotplug CPUs.

I need to add a notifier chain function that initialises/destroys
and then rebalances the per-cpu stuff when an event occurs.

> Changing all for_each_online_cpu to
> for_each_possible_cpu should fix all this.

I can't find that function in 2.6.16-rc2. I see:

#define for_each_cpu(cpu)         for_each_cpu_mask((cpu), cpu_possible_map)
#define for_each_online_cpu(cpu)  for_each_cpu_mask((cpu), cpu_online_map)
#define for_each_present_cpu(cpu) for_each_cpu_mask((cpu), cpu_present_map)

Do you mean for_each_cpu()?

If that is the case, then that means we balance over a per-cpu
list of NR_CPUs rather than the number of CPUS actually in the machine.
That is bad because we distribute counts to cpus that don't even
exist in most machines and hence would never get used, leading
to excessive rebalancing due to the existing cpus running the
counters dry far faster....

And cpu_present_map is the same as cpu_possible_map for machines that
do not support hotplug cpus, so that's to be avoided as well.

In light of this, what choice do I have if I only want to balance
across the actual present CPUs? The only thing that I can see
will work is for_each_online_cpu()....


Dave Chinner
R&D Software Enginner
SGI Australian Software Group

<Prev in Thread] Current Thread [Next in Thread>