Dave Chinner wrote:
Can you send me a link to a metadump image of the filesystem so I can
look at the broken directories in detail?
----
I'll have to work on that -- right now I'm taking a copy and trying to
delete everything off of it, so the meta copy, hopefully will have just the
stuff that won't delete... (at least that's the theory)...
BTW -- reviewing log files...there some stuff there that corresponds to not
begin able to read files and metadata, though I don't know the exact runtime
messages
I was getting at the time:
Jul 6 08:15:54 Ishtar kernel: [33919.157874] xfsdump: page allocation failure.
order:4, mode:0x40d0
Jul 6 08:17:12 Ishtar kernel: [33997.243321] xfsdump: page allocation failure.
order:4, mode:0x40d0
Jul 6 08:17:51 Ishtar kernel: [34036.405012] xfsdump: page allocation failure.
order:4, mode:0x40d0
....
....
Jul 6 08:17:12 Ishtar kernel: [33997.243326] Pid: 5047, comm: xfsdump Not
tainted 2.6.34T610-Vanilla #1
Jul 6 08:17:12 Ishtar kernel: [33997.243329] Call Trace:
Jul 6 08:17:12 Ishtar kernel: [33997.243338] [<ffffffff810c24d6>]
__alloc_pages_nodemask+0x5b6/0x6a0
Jul 6 08:17:12 Ishtar kernel: [33997.243343] [<ffffffff810c25d2>]
__get_free_pages+0x12/0x50
Jul 6 08:17:12 Ishtar kernel: [33997.243349] [<ffffffff810f26d9>]
__kmalloc+0x189/0x1e0
Jul 6 08:17:12 Ishtar kernel: [33997.243353] [<ffffffff8125b82e>]
xfs_attrlist_by_handle+0xbe/0x130
Jul 6 08:17:12 Ishtar kernel: [33997.243358] [<ffffffff8125c6c9>]
xfs_file_ioctl+0x5f9/0x980
Jul 6 08:17:12 Ishtar kernel: [33997.243364] [<ffffffff8158f4e1>] ?
down_read+0x11/0x30
Jul 6 08:17:12 Ishtar kernel: [33997.243370] [<ffffffff8123276d>] ?
xfs_iunlock+0xdd/0x130
Jul 6 08:17:12 Ishtar kernel: [33997.243374] [<ffffffff81250d05>] ?
xfs_free_eofblocks+0xd5/0x260
Jul 6 08:17:12 Ishtar kernel: [33997.243379] [<ffffffff81107ca8>]
vfs_ioctl+0x38/0xd0
Jul 6 08:17:12 Ishtar kernel: [33997.243382] [<ffffffff81107e7a>]
do_vfs_ioctl+0x8a/0x5a0
Jul 6 08:17:12 Ishtar kernel: [33997.243387] [<ffffffff810fa40c>] ?
__fput+0x19c/0x210
Jul 6 08:17:12 Ishtar kernel: [33997.243391] [<ffffffff81108411>]
sys_ioctl+0x81/0xa0
Jul 6 08:17:12 Ishtar kernel: [33997.243396] [<ffffffff81002deb>]
system_call_fastpath+0x16/0x1b
Jul 6 08:17:12 Ishtar kernel: [33997.243398] Mem-Info:
Jul 6 08:17:12 Ishtar kernel: [33997.243400] DMA per-cpu:
Jul 6 08:17:12 Ishtar kernel: [33997.243402] CPU 0: hi: 0, btch: 1
usd: 0
Jul 6 08:17:12 Ishtar kernel: [33997.243405] CPU 1: hi: 0, btch: 1
usd: 0
Jul 6 08:17:12 Ishtar kernel: [33997.243407] CPU 2: hi: 0, btch: 1
usd: 0
Jul 6 08:17:12 Ishtar kernel: [33997.243409] CPU 3: hi: 0, btch: 1
usd: 0
Jul 6 08:17:12 Ishtar kernel: [33997.243411] DMA32 per-cpu:
Jul 6 08:17:12 Ishtar kernel: [33997.243413] CPU 0: hi: 186, btch: 31
usd: 0
Jul 6 08:17:12 Ishtar kernel: [33997.243416] CPU 1: hi: 186, btch: 31
usd: 10
Jul 6 08:17:12 Ishtar kernel: [33997.243418] CPU 2: hi: 186, btch: 31
usd: 0
Jul 6 08:17:12 Ishtar kernel: [33997.243425] CPU 3: hi: 186, btch: 31
usd: 0
Jul 6 08:17:12 Ishtar kernel: [33997.243427] Normal per-cpu:
Jul 6 08:17:12 Ishtar kernel: [33997.243431] CPU 0: hi: 186, btch: 31
usd: 12
Jul 6 08:17:12 Ishtar kernel: [33997.243435] CPU 1: hi: 186, btch: 31
usd: 36
Jul 6 08:17:12 Ishtar kernel: [33997.243439] CPU 2: hi: 186, btch: 31
usd: 15
Jul 6 08:17:12 Ishtar kernel: [33997.243443] CPU 3: hi: 186, btch: 31
usd: 0
Jul 6 08:17:12 Ishtar kernel: [33997.243451] active_anon:39844
inactive_anon:31818 isolated_anon:0
Jul 6 08:17:12 Ishtar kernel: [33997.243454] active_file:854853
inactive_file:4597578 isolated_file:44
Jul 6 08:17:12 Ishtar kernel: [33997.243457] unevictable:262348 dirty:348876
writeback:2 unstable:0
Jul 6 08:17:12 Ishtar kernel: [33997.243460] free:59942
slab_reclaimable:294751 slab_unreclaimable:26479
Jul 6 08:17:12 Ishtar kernel: [33997.243463] mapped:18561 shmem:6
pagetables:2966 bounce:0
Jul 6 08:17:12 Ishtar kernel: [33997.243473] DMA free:15868kB min:12kB
low:12kB high:16kB active_anon:0kB inactive_anon:0kB active_file:0kB
inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:15772kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jul 6 08:17:12 Ishtar kernel: [33997.243491] lowmem_reserve[]: 0 2990 24200
24200
Jul 6 08:17:12 Ishtar kernel: [33997.243504] DMA32 free:199276kB min:2456kB
low:3068kB high:3684kB active_anon:48kB inactive_anon:2672kB
active_file:195724kB inactive_file:1726428kB unevictable:55768kB
isolated(anon):0kB isolated(file):0kB present:3062596kB mlocked:55768kB
dirty:107612kB writeback:0kB mapped:6352kB shmem:0kB slab_reclaimable:509064kB
slab_unreclaimable:22184kB kernel_stack:64kB pagetables:444kB unstable:0kB
bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jul 6 08:17:12 Ishtar kernel: [33997.243524] lowmem_reserve[]: 0 0 21210 21210
Jul 6 08:17:12 Ishtar kernel: [33997.243537] Normal free:24624kB min:17440kB
low:21800kB high:26160kB active_anon:159328kB inactive_anon:124600kB
active_file:3223688kB inactive_file:16663884kB unevictable:993624kB
isolated(anon):0kB isolated(file):176kB present:21719040kB mlocked:993624kB
dirty:1287892kB writeback:8kB mapped:67892kB shmem:24kB
slab_reclaimable:669940kB slab_unreclaimable:83732kB kernel_stack:3320kB
pagetables:11420kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
Jul 6 08:17:12 Ishtar kernel: [33997.243557] lowmem_reserve[]: 0 0 0 0
Jul 6 08:17:12 Ishtar kernel: [33997.243565] DMA: 1*4kB 1*8kB 1*16kB 1*32kB
1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15868kB
Jul 6 08:17:12 Ishtar kernel: [33997.243583] DMA32: 26095*4kB 10064*8kB
798*16kB 52*32kB 3*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
199516kB
Jul 6 08:17:12 Ishtar kernel: [33997.243602] Normal: 5486*4kB 0*8kB 0*16kB
4*32kB 2*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 24376kB
Jul 6 08:17:12 Ishtar kernel: [33997.243621] 5462983 total pagecache pages
Jul 6 08:17:12 Ishtar kernel: [33997.243625] 10294 pages in swap cache
Jul 6 08:17:12 Ishtar kernel: [33997.243629] Swap cache stats: add 455795,
delete 445501, find 338025/357649
Jul 6 08:17:12 Ishtar kernel: [33997.243633] Free swap = 8151268kB
Jul 6 08:17:12 Ishtar kernel: [33997.243637] Total swap = 8393924kB
Jul 6 08:17:12 Ishtar kernel: [33997.345120] 6291455 pages RAM
Jul 6 08:17:12 Ishtar kernel: [33997.345122] 104293 pages reserved
Jul 6 08:17:12 Ishtar kernel: [33997.345124] 3464575 pages shared
Jul 6 08:17:12 Ishtar kernel: [33997.345125] 2682993 pages non-shared
Jul 6 08:17:51 Ishtar kernel: [34036.405017] Pid: 5047, comm: xfsdump Not
tainted 2.6.34T610-Vanilla #1
Jul 6 08:17:51 Ishtar kernel: [34036.405020] Call Trace:
Jul 6 08:17:51 Ishtar kernel: [34036.405029] [<ffffffff810c24d6>]
__alloc_pages_nodemask+0x5b6/0x6a0
Jul 6 08:17:51 Ishtar kernel: [34036.405034] [<ffffffff810c25d2>]
__get_free_pages+0x12/0x50
Jul 6 08:17:51 Ishtar kernel: [34036.405040] [<ffffffff810f26d9>]
__kmalloc+0x189/0x1e0
Jul 6 08:17:51 Ishtar kernel: [34036.405044] [<ffffffff8125b82e>]
xfs_attrlist_by_handle+0xbe/0x130
Jul 6 08:17:51 Ishtar kernel: [34036.405048] [<ffffffff8125c6c9>]
xfs_file_ioctl+0x5f9/0x980
Jul 6 08:17:51 Ishtar kernel: [34036.405055] [<ffffffff8158f4e1>] ?
down_read+0x11/0x30
Jul 6 08:17:51 Ishtar kernel: [34036.405060] [<ffffffff8123276d>] ?
xfs_iunlock+0xdd/0x130
Jul 6 08:17:51 Ishtar kernel: [34036.405065] [<ffffffff81250d05>] ?
xfs_free_eofblocks+0xd5/0x260
Jul 6 08:17:51 Ishtar kernel: [34036.405070] [<ffffffff81107ca8>]
vfs_ioctl+0x38/0xd0
Jul 6 08:17:51 Ishtar kernel: [34036.405073] [<ffffffff81107e7a>]
do_vfs_ioctl+0x8a/0x5a0
Jul 6 08:17:51 Ishtar kernel: [34036.405084] [<ffffffff810fa40c>] ?
__fput+0x19c/0x210
Jul 6 08:17:51 Ishtar kernel: [34036.405087] [<ffffffff81108411>]
sys_ioctl+0x81/0xa0
Jul 6 08:17:51 Ishtar kernel: [34036.405093] [<ffffffff81002deb>]
system_call_fastpath+0x16/0x1b
Jul 6 08:17:51 Ishtar kernel: [34036.405095] Mem-Info:
Jul 6 08:17:51 Ishtar kernel: [34036.405097] DMA per-cpu:
Jul 6 08:17:51 Ishtar kernel: [34036.405099] CPU 0: hi: 0, btch: 1
usd: 0
Jul 6 08:17:51 Ishtar kernel: [34036.405101] CPU 1: hi: 0, btch: 1
usd: 0
Jul 6 08:17:51 Ishtar kernel: [34036.405104] CPU 2: hi: 0, btch: 1
usd: 0
Jul 6 08:17:51 Ishtar kernel: [34036.405106] CPU 3: hi: 0, btch: 1
usd: 0
Jul 6 08:17:51 Ishtar kernel: [34036.405108] DMA32 per-cpu:
Jul 6 08:17:51 Ishtar kernel: [34036.405110] CPU 0: hi: 186, btch: 31
usd: 0
Jul 6 08:17:51 Ishtar kernel: [34036.405112] CPU 1: hi: 186, btch: 31
usd: 0
Jul 6 08:17:51 Ishtar kernel: [34036.405114] CPU 2: hi: 186, btch: 31
usd: 0
Jul 6 08:17:51 Ishtar kernel: [34036.405117] CPU 3: hi: 186, btch: 31
usd: 0
Jul 6 08:17:51 Ishtar kernel: [34036.405118] Normal per-cpu:
Jul 6 08:17:51 Ishtar kernel: [34036.405120] CPU 0: hi: 186, btch: 31
usd: 0
Jul 6 08:17:51 Ishtar kernel: [34036.405123] CPU 1: hi: 186, btch: 31
usd: 0
Jul 6 08:17:51 Ishtar kernel: [34036.405125] CPU 2: hi: 186, btch: 31
usd: 30
Jul 6 08:17:51 Ishtar kernel: [34036.405127] CPU 3: hi: 186, btch: 31
usd: 15
Jul 6 08:17:51 Ishtar kernel: [34036.405133] active_anon:39844
inactive_anon:31812 isolated_anon:0
Jul 6 08:17:51 Ishtar kernel: [34036.405134] active_file:865726
inactive_file:4534161 isolated_file:35
Jul 6 08:17:51 Ishtar kernel: [34036.405135] unevictable:262348 dirty:195473
writeback:0 unstable:0
Jul 6 08:17:51 Ishtar kernel: [34036.405136] free:57638
slab_reclaimable:344326 slab_unreclaimable:32158
Jul 6 08:17:51 Ishtar kernel: [34036.405137] mapped:26657 shmem:6
pagetables:2971 bounce:0
Jul 6 08:17:51 Ishtar kernel: [34036.405144] DMA free:15868kB min:12kB
low:12kB high:16kB active_anon:0kB inactive_anon:0kB active_file:0kB
inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:15772kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jul 6 08:17:51 Ishtar kernel: [34036.405150] lowmem_reserve[]: 0 2990 24200
24200
Jul 6 08:17:51 Ishtar kernel: [34036.405159] DMA32 free:179088kB min:2456kB
low:3068kB high:3684kB active_anon:48kB inactive_anon:2648kB
active_file:196548kB inactive_file:1548864kB unevictable:55768kB
isolated(anon):0kB isolated(file):0kB present:3062596kB mlocked:55768kB
dirty:52076kB writeback:0kB mapped:10560kB shmem:0kB slab_reclaimable:701508kB
slab_unreclaimable:27664kB kernel_stack:64kB pagetables:444kB unstable:0kB
bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jul 6 08:17:51 Ishtar kernel: [34036.405166] lowmem_reserve[]: 0 0 21210 21210
Jul 6 08:17:51 Ishtar kernel: [34036.405175] Normal free:35596kB min:17440kB
low:21800kB high:26160kB active_anon:159328kB inactive_anon:124600kB
active_file:3266356kB inactive_file:16587780kB unevictable:993624kB
isolated(anon):0kB isolated(file):140kB present:21719040kB mlocked:993624kB
dirty:729816kB writeback:0kB mapped:96068kB shmem:24kB
slab_reclaimable:675796kB slab_unreclaimable:100968kB kernel_stack:3320kB
pagetables:11440kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
Jul 6 08:17:51 Ishtar kernel: [34036.405181] lowmem_reserve[]: 0 0 0 0
Jul 6 08:17:51 Ishtar kernel: [34036.405185] DMA: 1*4kB 1*8kB 1*16kB 1*32kB
1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15868kB
Jul 6 08:17:51 Ishtar kernel: [34036.405194] DMA32: 26498*4kB 8649*8kB
150*16kB 41*32kB 3*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
179088kB
Jul 6 08:17:51 Ishtar kernel: [34036.405204] Normal: 7071*4kB 557*8kB 20*16kB
2*32kB 2*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 35428kB
Jul 6 08:17:51 Ishtar kernel: [34036.405214] 5410107 total pagecache pages
Jul 6 08:17:51 Ishtar kernel: [34036.405216] 10290 pages in swap cache
Jul 6 08:17:51 Ishtar kernel: [34036.405218] Swap cache stats: add 455867,
delete 445577, find 338111/357759
Jul 6 08:17:51 Ishtar kernel: [34036.405221] Free swap = 8151272kB
Jul 6 08:17:51 Ishtar kernel: [34036.405222] Total swap = 8393924kB
Jul 6 08:17:52 Ishtar kernel: [34036.530592] 6291455 pages RAM
Jul 6 08:17:52 Ishtar kernel: [34036.530595] 104293 pages reserved
Jul 6 08:17:52 Ishtar kernel: [34036.530597] 3301623 pages shared
Jul 6 08:17:52 Ishtar kernel: [34036.530598] 2859224 pages non-shared
|