xfs
[Top] [All Lists]

Re: [PATCH 4/4] mm: numa: Slow PTE scan rate if migration failures occur

To: Dave Chinner <david@xxxxxxxxxxxxx>
Subject: Re: [PATCH 4/4] mm: numa: Slow PTE scan rate if migration failures occur
From: Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx>
Date: Tue, 17 Mar 2015 14:30:57 -0700
Cc: Mel Gorman <mgorman@xxxxxxx>, Ingo Molnar <mingo@xxxxxxxxxx>, Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>, Aneesh Kumar <aneesh.kumar@xxxxxxxxxxxxxxxxxx>, Linux Kernel Mailing List <linux-kernel@xxxxxxxxxxxxxxx>, Linux-MM <linux-mm@xxxxxxxxx>, xfs@xxxxxxxxxxx, ppc-dev <linuxppc-dev@xxxxxxxxxxxxxxxx>
Delivered-to: xfs@xxxxxxxxxxx
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=F5f+ojz4YjxxG55vSI7O6QwAw5XOkvVZqeeE3B1P23Y=; b=lKnozDq7OgrlepE2MYp9PZpxkAyVIei7N1OuSxpcy/wa2BiS0CUvZEncEDbJo5iTqf 30dAGaAKmqKDzLzC7D615X/sZtOFUMWXCaRC2uA/+2sjNlXfL9MKtrka6oh3HNizZ05j w0I7PxwckR3uLVSh7dd2GrqHlKtJ05OBl1stM1nLoii04N0S2t+iSksI8SzU/kPc0DxN cF7x64ll+KUX5PYohMtbzWT00XRy0CvA7jdCRFCdzizUB8hMl2t8pLA6yImQY3SxM477 RPPXns1BZkgQmnGsnJmBUrLio16dtY8iGPwlAj/jBzmSMoNY1hl6QS7pMNLWJzMMjXbF 9vPQ==
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux-foundation.org; s=google; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=F5f+ojz4YjxxG55vSI7O6QwAw5XOkvVZqeeE3B1P23Y=; b=Nwi7qKZ/9aJOdF+h8UKT47x6IPFDs6e5XntpymB/1QHBI0lMs1RTD+dEyRDnEWk29D TkkQLQozxN+ElK9KcLwrQ1CbMF1i454BI4U/cjGQT5ZfzXGKQj/LvC/T4COZPd+/5/5T kHIrBmv9/dmlw6i2rcXPBYA0PhvL/vzDfIGGY=
In-reply-to: <20150317205104.GA28621@dastard>
References: <CA+55aFyQyZXu2fi7X9bWdSX0utk8=sccfBwFaSoToROXoE_PLA@xxxxxxxxxxxxxx> <20150309112936.GD26657@destitution> <CA+55aFywW5JLq=BU_qb2OG5+pJ-b1v9tiS5Ygi-vtEKbEZ_T5Q@xxxxxxxxxxxxxx> <20150309191943.GF26657@destitution> <CA+55aFzFt-vX5Jerci0Ty4Uf7K4_nQ7wyCp8hhU_dB0X4cBpVQ@xxxxxxxxxxxxxx> <20150312131045.GE3406@xxxxxxx> <CA+55aFx=81BGnQFNhnAGu6CetL7yifPsnD-+v7Y6QRqwgH47gQ@xxxxxxxxxxxxxx> <20150312184925.GH3406@xxxxxxx> <20150317070655.GB10105@dastard> <CA+55aFzdLnFdku-gnm3mGbeS=QauYBNkFQKYXJAGkrMd2jKXhw@xxxxxxxxxxxxxx> <20150317205104.GA28621@dastard>
Sender: linus971@xxxxxxxxx
On Tue, Mar 17, 2015 at 1:51 PM, Dave Chinner <david@xxxxxxxxxxxxx> wrote:
>
> On the -o ag_stride=-1 -o bhash=101073 config, the 60s perf stat I
> was using during steady state shows:
>
>      471,752      migrate:mm_migrate_pages ( +-  7.38% )
>
> The migrate pages rate is even higher than in 4.0-rc1 (~360,000)
> and 3.19 (~55,000), so that looks like even more of a problem than
> before.

Hmm. How stable are those numbers boot-to-boot?

That kind of extreme spread makes me suspicious. It's also interesting
that if the numbers really go up even more (and by that big amount),
then why does there seem to be almost no correlation with performance
(which apparently went up since rc1, despite migrate_pages getting
even _worse_).

> And the profile looks like:
>
> -   43.73%     0.05%  [kernel]            [k] native_flush_tlb_others

Ok, that's down from rc1 (67%), but still hugely up from 3.19 (13.7%).
And flush_tlb_page() does seem to be called about ten times more
(flush_tlb_mm_range used to be 1.4% of the callers, now it's invisible
at 0.13%)

Damn. From a performance number standpoint, it looked like we zoomed
in on the right thing. But now it's migrating even more pages than
before. Odd.

> And the vmstats are:
>
> 3.19:
>
> numa_hit 5163221
> numa_local 5153127

> 4.0-rc1:
>
> numa_hit 36952043
> numa_local 36927384
>
> 4.0-rc4:
>
> numa_hit 23447345
> numa_local 23438564
>
> Page migrations are still up by a factor of ~20 on 3.19.

The thing is, those "numa_hit" things come from the zone_statistics()
call in buffered_rmqueue(), which in turn is simple from the memory
allocator. That has *nothing* to do with virtual memory, and
everything to do with actual physical memory allocations.  So the load
is simply allocating a lot more pages, presumably for those stupid
migration events.

But then it doesn't correlate with performance anyway..

Can you do a simple stupid test? Apply that commit 53da3bc2ba9e ("mm:
fix up numa read-only thread grouping logic") to 3.19, so that it uses
the same "pte_dirty()" logic as 4.0-rc4. That *should* make the 3.19
and 4.0-rc4 numbers comparable.

It does make me wonder if your load is "chaotic" wrt scheduling. The
load presumably wants to spread out across all cpu's, but then the
numa code tries to group things together for numa accesses, but
depending on just random allocation patterns and layout in the hash
tables, there either are patters with page access or there aren't.

Which is kind of why I wonder how stable those numbers are boot to
boot. Maybe this is at least partly about lucky allocation patterns.

                              Linus

<Prev in Thread] Current Thread [Next in Thread>