On Wed, Sep 08, 2004 at 10:21:17PM +0200, Jakob Oestergaard wrote:
> Would *you* base a production environment on that ? ;)
I might if I used ext3, they didn't look all the bad but I only
glanced over them.
> 2.5G RAM IIRC - but slab is in lowmem so the machine will barf when
> it reaches ~850MB slab.
Ick. This is a 'known problem' that some people hit. Basically you
get bazillions of dentries as the backup runs (or whatever, find /fs
-noleaf, works pretty well too) which consume slab. What's worse is
that they pin other slab objects used by XFS. Checking 'bloatmeter'
on my desktop I see:
xfs_inode: 90327KB 115723KB 78.5
linvfs_icache: 84445KB 106528KB 79.27
dentry_cache: 24118KB 62147KB 38.80
This shows for me the largest slab users are xfs_inode and
linvfs_icache, these can't be freed until the dentry_cache objects
that pin them are also freed (the slab-freeing logic isn't aware of
this and I'm not sure that even if it were what would help much even
if it were but I always meant to test if it would help).
The problem is basically we don't reclaim slab/lowmem hard enough at
times and even when we do slab fragmentation means we can still have
lots of memory tied up until we really aggressively prune the slab.
> There seems to be a common trend that NFS+XFS+SMP is causing this. I
> suppose that could explain some of that.
It it's OOM death due to lack of lowmem then no, it's not XFS to blame
but XFS doesn't help as it uses a bit more slab that other fs' for the
same use (I don't have numbers on this, but I'm pretty sure it's a
fair bit more last I checked). I looked at the in-core structures to
see if they could be shrunk I while ago and nothing jumped out as
being an obvious candidate for assassination.
Ideally the slab blanacing logic needs more work. I really don't know
this personally at all but akpm and nick did some changes in the 2.6.5
time-frame which helped and I wonder if more work can be done here?
It was suggested to me:
"you can try changing include/linux/mm.h DEFAULT_SEEKS from 2 to 1.
that will make slab be reclaimed more aggressively."
does that help at all?
I also wonder if -mm kernels might behave differently?
... not that this helps you, but because slab/lowmem balancing is such
a pain on i386, I'll personally never get another i386 platform again
almost entirely because of this when amd64 platforms are so cheap and
don't have this problem[1] (you don't have this special case were a
large chunk of memory is only usable for certain things).
Sorry I can't think of anything more productive right this second (and
I know saying get an opteron isn't what yoiu want to head).
--cw
[1] Well, arguably slab can still get out of hand but it usually
doesn't consume all of a special class of memory and cause the
machine to die...
|