xfs
[Top] [All Lists]

Re: 3.14-rc2 XFS backtrace because irqs_disabled.

To: Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx>
Subject: Re: 3.14-rc2 XFS backtrace because irqs_disabled.
From: Steven Rostedt <rostedt@xxxxxxxxxxx>
Date: Wed, 12 Feb 2014 07:40:36 -0500
Cc: Dave Chinner <david@xxxxxxxxxxxxx>, Jens Axboe <axboe@xxxxxxxxx>, Tejun Heo <tj@xxxxxxxxxx>, Dave Jones <davej@xxxxxxxxxx>, Al Viro <viro@xxxxxxxxxxxxxxxxxx>, Eric Sandeen <sandeen@xxxxxxxxxxx>, Linux Kernel <linux-kernel@xxxxxxxxxxxxxxx>, xfs@xxxxxxxxxxx, Ingo Molnar <mingo@xxxxxxxxxx>, Peter Zijlstra <peterz@xxxxxxxxxxxxx>, Frederic Weisbecker <fweisbec@xxxxxxxxx>
Delivered-to: xfs@xxxxxxxxxxx
In-reply-to: <CA+55aFyp91=1seVT0ZV8T+GjOuafWvvjHNHn+MKGydsG+8eUEQ@xxxxxxxxxxxxxx>
References: <20140211210841.GM13647@dastard> <52FA9ADA.9040803@xxxxxxxxxxx> <20140212004403.GA17129@xxxxxxxxxx> <20140212010941.GM18016@xxxxxxxxxxxxxxxxxx> <CA+55aFwoWT-0A_KTkXMkNqOy8hc=YmouTMBgWUD_z+8qYPphjA@xxxxxxxxxxxxxx> <20140212040358.GA25327@xxxxxxxxxx> <20140212042215.GN18016@xxxxxxxxxxxxxxxxxx> <20140212054043.GB13997@dastard> <20140212055027.GA28502@xxxxxxxxxx> <20140212061038.GC13997@dastard> <20140212063150.GD13997@dastard> <CA+55aFyp91=1seVT0ZV8T+GjOuafWvvjHNHn+MKGydsG+8eUEQ@xxxxxxxxxxxxxx>
[ Added the perf tracepoint maintainers ]

On Tue, 11 Feb 2014 22:59:58 -0800
Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx> wrote:

> On Tue, Feb 11, 2014 at 10:31 PM, Dave Chinner <david@xxxxxxxxxxxxx> wrote:
> >
> > FYI, just creating lots of files with open(O_CREAT):
> >
> > [  348.718357] fs_mark (4828) used greatest stack depth: 2968 bytes left
> > [  348.769846] fs_mark (4814) used greatest stack depth: 2312 bytes left
> > [  349.777717] fs_mark (4826) used greatest stack depth: 2280 bytes left
> > [  418.139415] fs_mark (4928) used greatest stack depth: 1936 bytes left
> > [  460.492282] fs_mark (4993) used greatest stack depth: 1336 bytes left
> > [  544.825418] fs_mark (5104) used greatest stack depth: 1112 bytes left
> > [  689.503970] fs_mark (5265) used greatest stack depth: 1000 bytes left
> >
> > We've got absolutely no spare stack space anymore in the IO path.
> > And the IO path can't get much simpler than filesystem -> virtio
> > block device.
> 
> Ugh, that's bad. A thousand bytes of stack space is much too close to
> any limits.
> 
> Do you have the stack traces for these things so that we can look at
> worst offenders?
> 
> If the new block-mq code is to blame, it needs to be fixed.
> __virtblk_add_req() has a 300-byte stack frame, it seems. Looking
> elsewhere, blkdev_issue_discard() has 350 bytes of stack frame, but is
> hopefully not in any normal path - online discard is moronic, and I'm
> assuming XFS doesn't do that.
> 
> There's a lot of 200+ byte stack frames in block/blk-core.s, and they
> all seem to be of the type perf_trace_block_buffer() - things created
> with DECLARE_EVENT_CLASS(), afaik. Why they all have 200+ bytes of
> frame, I have no idea. That sounds like a potential disaster too,
> although hopefully it's mostly leaf functions - but leaf functions
> *deep* in the callchain. Tejun? Steven, why _do_ they end up with such
> huge frames?

The perf_trace_##event is defined in include/trace/ftrace.h.

There we have this:

perf_trace_##call(void *__data, proto)                                  \
{                                                                       \
        struct ftrace_event_call *event_call = __data;                  \
        struct ftrace_data_offsets_##call __maybe_unused __data_offsets;\
        struct ftrace_raw_##call *entry;                                \
        struct pt_regs __regs;                                          \
        u64 __addr = 0, __count = 1;                                    \
        struct task_struct *__task = NULL;                              \
        struct hlist_head *head;                                        \
        int __entry_size;                                               \
        int __data_size;                                                \
        int rctx;                                                       \
                                                                        \

Mostly pointers except for two structures. The __data_offests, is
dynamically defined, and only consists of values from the tracepoint
entry_structure that defines dynamic arrays. But the other structure on
the stack looks a bit harrier. The pt_regs structure.

That's what? 21 unsigned longs? 21 * 8 = 168. I think that's the
culprit here.

Peter and Frederic, is there a way not to store that on the stack?

-- Steve

> 
> And if the stack use comes from the VFS layer, we can probably work on
> that too. But I don't think that has really changed much lately..
> 
>                 Linus

<Prev in Thread] Current Thread [Next in Thread>