xfs
[Top] [All Lists]

Re: Very aggressive memory reclaim

To: Dave Chinner <david@xxxxxxxxxxxxx>
Subject: Re: Very aggressive memory reclaim
From: Minchan Kim <minchan.kim@xxxxxxxxx>
Date: Tue, 29 Mar 2011 07:52:14 +0900
Cc: John Lepikhin <johnlepikhin@xxxxxxxxx>, linux-kernel@xxxxxxxxxxxxxxx, xfs@xxxxxxxxxxx, linux-mm@xxxxxxxxx
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=R9y1xflKhyFU3MMmzCxoiGfArf5o8bjxND6oFG3DuE4=; b=YEdlp1WBCsBKQ22UGQwVxyiWtRHysY3nr2hGQTZxwySvxS/qIN024T6kEEGcrnsAJ/ FxpCRUOZAk/zQcbAKqW56jGMay7lSMoJagEgqvKYg8WC9jT62VGXWiNF6blogGEU4mcj 4+kJH3azuY+Qt03QNYz8ajn1ozMIKtA/VRowk=
Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=X63h33+1y/LvVo5gnPO2TQU/kHAkwZ0eyOZsAj1CB7cl09zwRYebEqg3dF6UAK6rWF 3Dowvc3iwBBkgwh5LhJLky+AQGy3PdxOFYrm/TVBfmRnokeKdcGirJKhF3kD6ycgyWuW IcR5IO7BgEa9+tw18NRrLECCl/GNUwVW+Vd8w=
In-reply-to: <20110328215344.GC3008@dastard>
References: <AANLkTinFqqmE+fTMTLVU-_CwPE+LQv7CpXSQ5+CdAKLK@xxxxxxxxxxxxxx> <20110328215344.GC3008@dastard>
On Tue, Mar 29, 2011 at 6:53 AM, Dave Chinner <david@xxxxxxxxxxxxx> wrote:
> [cc xfs and mm lists]
>
> On Mon, Mar 28, 2011 at 08:39:29PM +0400, John Lepikhin wrote:
>> Hello,
>>
>> I use high-loaded machine with 10M+ inodes inside XFS, 50+ GB of
>> memory, intensive HDD traffic and 20..50 forks per second. Vanilla
>> kernel 2.6.37.4. The problem is that kernel frees memory very
>> aggressively.
>>
>> For example:
>>
>> 25% of memory is used by processes
>> 50% for page caches
>> 7% for slabs, etc.
>> 18% free.
>>
>> That's bad but works. After few hours:
>>
>> 25% of memory is used by processes
>> 62% for page caches
>> 7% for slabs, etc.
>> 5% free.
>>
>> Most of files are cached, works perfectly. This is the moment when
>> kernel decides to free some memory. After memory reclaim:
>>
>> 25% of memory is used by processes
>> 25% for page caches(!)
>> 7% for slabs, etc.
>> 43% free(!)
>>
>> Page cache is dropped, server becomes too slow. This is the beginning
>> of new cycle.
>>
>> I didn't found any huge mallocs at that moment. Looks like because of
>> large number of small mallocs (forks) kernel have pessimistic forecast
>> about future memory usage and frees too much memory. Is there any
>> options of tuning this? Any other variants?
>
> First it would be useful to determine why the VM is reclaiming so
> much memory. If it is somewhat predictable when the excessive
> reclaim is going to happen, it might be worth capturing an event
> trace from the VM so we can see more precisely what it is doiing
> during this event. In that case, recording the kmem/* and vmscan/*
> events is probably sufficient to tell us what memory allocations
> triggered reclaim and how much reclaim was done on each event.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@xxxxxxxxxxxxx
>

Recently, We had a similar issue.
http://www.spinics.net/lists/linux-mm/msg12243.html
But it seems to not merge. I don't know why since I didn't follow up the thread.
Maybe Cced guys can help you.

Is it a sudden big cache drop at the moment or accumulated small cache
drop for long time?
What's your zones' size?

Please attach the result of cat /proc/zoneinfo for others.

<Prev in Thread] Current Thread [Next in Thread>