xfs
[Top] [All Lists]

Re: [PATCH 2/5] repair: per AG locks contend for cachelines

To: Brian Foster <bfoster@xxxxxxxxxx>
Subject: Re: [PATCH 2/5] repair: per AG locks contend for cachelines
From: Dave Chinner <david@xxxxxxxxxxxxx>
Date: Fri, 13 Dec 2013 07:46:23 +1100
Cc: xfs@xxxxxxxxxxx
Delivered-to: xfs@xxxxxxxxxxx
In-reply-to: <52AA0773.1000506@xxxxxxxxxx>
References: <1386832945-19763-1-git-send-email-david@xxxxxxxxxxxxx> <1386832945-19763-3-git-send-email-david@xxxxxxxxxxxxx> <52AA0773.1000506@xxxxxxxxxx>
User-agent: Mutt/1.5.21 (2010-09-15)
On Thu, Dec 12, 2013 at 01:58:59PM -0500, Brian Foster wrote:
> On 12/12/2013 02:22 AM, Dave Chinner wrote:
> > From: Dave Chinner <dchinner@xxxxxxxxxx>
> > 
> > The per-ag locks used to protect per-ag block lists are located in a tightly
> > packed array. That means that they share cachelines, so separate them out 
> > inot
> > separate 64 byte regions in the array.
> > 
> > pahole confirms the padding is correctly applied:
> > 
> > struct aglock {
> >         pthread_mutex_t            lock;                 /*     0    40 */
> > 
> >         /* size: 64, cachelines: 1, members: 1 */
> >         /* padding: 24 */
> > };
> > 
> > Signed-off-by: Dave Chinner <dchinner@xxxxxxxxxx>
> > ---
> 
> Out of curiosity, any data on this one?

There's a small improvement but it's within the margins of error. At
higher levels of concurrency it will make a bigger difference, but
right now we can't get to a thread per AG (where the contention
would really show) because of other scalability limitations

e.g. mmap_sem contention becomes a limiting factor at 20-25 threads
because of page faults occurring on hash cache lookups (read lock)
occurring in parallel with memory allocation (libc mprotect calls
take a write lock). So more scalability will come from a CPU cache
friendlier cache index - I'm tempted to drop in per-AG rb trees and
see where that leads (now where would I have got that idea?)....

Cheers,

Dave.
-- 
Dave Chinner
david@xxxxxxxxxxxxx

<Prev in Thread] Current Thread [Next in Thread>