xfs
[Top] [All Lists]

Re: PROBLEM + POSS FIX: kernel stack overflow, xfs, many disks, heavy wr

To: Chris Mason <chris.mason@xxxxxxxxxx>, Dave Chinner <david@xxxxxxxxxxxxx>, John Berthels <john@xxxxxxxxx>, linux-kernel@xxxxxxxxxxxxxxx, Nick Gregory <nick@xxxxxxxxx>, Rob Sanderson <rob@xxxxxxxxx>, xfs@xxxxxxxxxxx, linux-mm@xxxxxxxxx
Subject: Re: PROBLEM + POSS FIX: kernel stack overflow, xfs, many disks, heavy write load, 8k stack, x86-64
From: John Berthels <john@xxxxxxxxx>
Date: Tue, 13 Apr 2010 10:51:44 +0100
In-reply-to: <20100409113850.GE13327@think>
References: <4BBC6719.7080304@xxxxxxxxx> <20100407140523.GJ11036@dastard> <4BBCAB57.3000106@xxxxxxxxx> <20100407234341.GK11036@dastard> <20100408030347.GM11036@dastard> <4BBDC92D.8060503@xxxxxxxxx> <4BBDEC9A.9070903@xxxxxxxxx> <20100408233837.GP11036@dastard> <20100409113850.GE13327@think>
User-agent: Thunderbird 2.0.0.24 (X11/20100317)
Chris Mason wrote:
shrink_zone on my box isn't 500 bytes, but lets try the easy stuff
first.  This is against .34, if you have any trouble applying to .32,
just add the word noinline after the word static on the function
definitions.

Hi Chris,

Thanks for this, we've been soaking it for a while and get the stack trace below (which is still >8k), which still has shrink_zone at 528 bytes.

I find it odd that the shrink_zone stack usage is different on our systems. This is a stock kernel 2.6.33.2 kernel, x86_64 arch (plus your patch + Dave Chinner's patch) built using ubuntu make-kpkg, with gcc (Ubuntu 4.3.3-5ubuntu4) 4.3.3 (.vmscan.o.cmd with full build options is below, gzipped .config attached).

Can you see any difference between your system and ours which might explain the discrepancy? I note -g and -pg in there. (Does -pg have any stack overhead? It seems to be enabled in ubuntu release kernels).

regards,

jb



mm/.vmscan.o.cmd:

cmd_mm/vmscan.o := gcc -Wp,-MD,mm/.vmscan.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-linux-gnu/4.3.3/include -I/usr/local/src/kern/linux-2.6.33.2/arch/x86/include -Iinclude -include include/generated/autoconf.h -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -fno-delete-null-pointer-checks -O2 -m64 -mtune=generic -mno-red-zone -mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -fstack-protector -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -fno-omit-frame-pointer -fno-optimize-sibling-calls -g -pg -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -D"KBUILD_STR(s)=\#s" -D"KBUILD_BASENAME=KBUILD_STR(vmscan)" -D"KBUILD_MODNAME=KBUILD_STR(vmscan)" -c -o mm/.tmp_vmscan.o mm/vmscan.c



Apr 12 22:06:35 nas17 kernel: [36346.599076] apache2 used greatest stack
depth: 7904 bytes left
        Depth    Size   Location    (56 entries)
        -----    ----   --------
  0)     7904      48   __call_rcu+0x67/0x190
  1)     7856      16   call_rcu_sched+0x15/0x20
  2)     7840      16   call_rcu+0xe/0x10
  3)     7824     272   radix_tree_delete+0x159/0x2e0
  4)     7552      32   __remove_from_page_cache+0x21/0x110
  5)     7520      64   __remove_mapping+0xe8/0x130
  6)     7456     384   shrink_page_list+0x400/0x860
  7)     7072     528   shrink_zone+0x636/0xdc0
  8)     6544     112   do_try_to_free_pages+0xc2/0x3c0
  9)     6432     112   try_to_free_pages+0x64/0x70
 10)     6320     256   __alloc_pages_nodemask+0x3d2/0x710
 11)     6064      48   alloc_pages_current+0x8c/0xe0
 12)     6016      32   __page_cache_alloc+0x67/0x70
 13)     5984      80   find_or_create_page+0x50/0xb0
 14)     5904     160   _xfs_buf_lookup_pages+0x145/0x350 [xfs]
 15)     5744      64   xfs_buf_get+0x74/0x1d0 [xfs]
 16)     5680      48   xfs_buf_read+0x2f/0x110 [xfs]
 17)     5632      80   xfs_trans_read_buf+0x2bf/0x430 [xfs]
 18)     5552      80   xfs_btree_read_buf_block+0x5d/0xb0 [xfs]
 19)     5472     176   xfs_btree_rshift+0xd7/0x530 [xfs]
 20)     5296      96   xfs_btree_make_block_unfull+0x5b/0x190 [xfs]
 21)     5200     224   xfs_btree_insrec+0x39c/0x5b0 [xfs]
 22)     4976     128   xfs_btree_insert+0x86/0x180 [xfs]
 23)     4848      96   xfs_alloc_fixup_trees+0x1fa/0x350 [xfs]
 24)     4752     144   xfs_alloc_ag_vextent_near+0x916/0xb30 [xfs]
 25)     4608      32   xfs_alloc_ag_vextent+0xe5/0x140 [xfs]
 26)     4576      96   xfs_alloc_vextent+0x49f/0x630 [xfs]
 27)     4480     160   xfs_bmbt_alloc_block+0xbe/0x1d0 [xfs]
 28)     4320     208   xfs_btree_split+0xb3/0x6a0 [xfs]
 29)     4112      96   xfs_btree_make_block_unfull+0x151/0x190 [xfs]
 30)     4016     224   xfs_btree_insrec+0x39c/0x5b0 [xfs]
 31)     3792     128   xfs_btree_insert+0x86/0x180 [xfs]
 32)     3664     352   xfs_bmap_add_extent_delay_real+0x41e/0x1670 [xfs]
 33)     3312     208   xfs_bmap_add_extent+0x41c/0x450 [xfs]
 34)     3104     448   xfs_bmapi+0x982/0x1200 [xfs]
 35)     2656     256   xfs_iomap_write_allocate+0x248/0x3c0 [xfs]
 36)     2400     208   xfs_iomap+0x3d8/0x410 [xfs]
 37)     2192      32   xfs_map_blocks+0x2c/0x30 [xfs]
 38)     2160     256   xfs_page_state_convert+0x443/0x730 [xfs]
 39)     1904      64   xfs_vm_writepage+0xab/0x160 [xfs]
 40)     1840      32   __writepage+0x1a/0x60
 41)     1808     288   write_cache_pages+0x1f7/0x400
 42)     1520      16   generic_writepages+0x27/0x30
 43)     1504      48   xfs_vm_writepages+0x5a/0x70 [xfs]
 44)     1456      16   do_writepages+0x24/0x40
 45)     1440      64   writeback_single_inode+0xf1/0x3e0
 46)     1376     128   writeback_inodes_wb+0x31e/0x510
 47)     1248      16   writeback_inodes_wbc+0x1e/0x20
 48)     1232     224   balance_dirty_pages_ratelimited_nr+0x277/0x410
 49)     1008     192   generic_file_buffered_write+0x19b/0x240
 50)      816     288   xfs_write+0x849/0x930 [xfs]
 51)      528      16   xfs_file_aio_write+0x5b/0x70 [xfs]
 52)      512     272   do_sync_write+0xd1/0x120
 53)      240      48   vfs_write+0xcb/0x1a0
 54)      192      64   sys_write+0x55/0x90
 55)      128     128   system_call_fastpath+0x16/0x1b

Attachment: config.gz
Description: GNU Zip compressed data

<Prev in Thread] Current Thread [Next in Thread>