xfs
[Top] [All Lists]

Re: Still seeing hangs in xlog_grant_log_space

To: Dave Chinner <david@xxxxxxxxxxxxx>
Subject: Re: Still seeing hangs in xlog_grant_log_space
From: Juerg Haefliger <juergh@xxxxxxxxx>
Date: Fri, 27 Apr 2012 01:00:08 +0200
Cc: xfs@xxxxxxxxxxx
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=pBAaZzZefp+LOWOntuD8e5jjqVEpTT4mx0wD+H0iMQ0=; b=u0WZ16RgTp9PrUppShOzKnad1cl6HvGQScrFIntZZYSewUZOmWP+uAlCpaPWBrwmtm aWVAmPn3f0FVnbQzmQTg/6K9PYXXoFn6/8efJQzS/TzCPxs8xG7yTxLx+JUu6S3zS+s/ AhV4dEGLL/NUxxuQ3JmRdYIDz0uFHAsRfb2icLUXx3t2G116czqn5FiJU/+T/Xb1m88m GzPCHnG9rcwegl5unkRzGjHA5b4uFdsekWwcnj4e3Bq8UHE/CafJWSR+eUbYB/0q/daq yvnJ+GMvBm7RaZfOq6RwSOWk02kr48yuPQXojtF7Bq2rsh0WMbZ7mJNWbjDb0DhKJtcv L91g==
In-reply-to: <20120426224412.GA9541@dastard>
References: <CADLDEKsP4DsXf_G07ub+a-ODbrJbsiprRJUX1fJdaQ41TB7+Xg@xxxxxxxxxxxxxx> <20120423143843.GN9541@dastard> <CADLDEKvFF3FvEHVtmwdWhbM58_jrCRX+Uk9vLBg1hA8sizh5BQ@xxxxxxxxxxxxxx> <20120423235840.GQ9541@dastard> <CADLDEKsfckBw2oVYFfaaTbpe8Ri+rYJr2e5SB7-pM0BU9nRUeA@xxxxxxxxxxxxxx> <20120424120731.GT9541@dastard> <CADLDEKs01GnxgYh2UTt1waVDUXHbB_RcBcUTBr5REFg5aD5jHA@xxxxxxxxxxxxxx> <20120425223845.GX9541@dastard> <CADLDEKvYkpUnMrqdMyqCmsYrZcUtiJ6ZRhrRu_ERTjn=r7M3Pg@xxxxxxxxxxxxxx> <20120426224412.GA9541@dastard>
On Fri, Apr 27, 2012 at 12:44 AM, Dave Chinner <david@xxxxxxxxxxxxx> wrote:
> On Thu, Apr 26, 2012 at 02:37:50PM +0200, Juerg Haefliger wrote:
>> On Thu, Apr 26, 2012 at 12:38 AM, Dave Chinner <david@xxxxxxxxxxxxx> wrote:
>> > On Tue, Apr 24, 2012 at 08:26:04PM +0200, Juerg Haefliger wrote:
>> >> On Tue, Apr 24, 2012 at 2:07 PM, Dave Chinner <david@xxxxxxxxxxxxx> wrote:
>> >> > On Tue, Apr 24, 2012 at 10:55:22AM +0200, Juerg Haefliger wrote:
>> >> >> > Alright, then I need all the usual information. I suspect an event
>> >> >> > trace is the only way I'm going to see what is happening. I just
>> >> >> > updated the FAQ entry, so all the necessary info for gathering a
>> >> >> > trace should be there now.
>> >> >> >
>> >> >> > http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
>> >> >>
>> >> >> Very good. Will do. What kernel do you want me to run? I would prefer
>> >> >> our current production kernel (2.6.38-8-server) but I understand if
>> >> >> you want something newer.
>> >> >
>> >> > If you can reproduce it on a current kernel - 3.4-rc4 if possible, if
>> >> > not a 3.3.x stable kernel would be best. 2.6.38 is simply too old to
>> >> > be useful for debugging these sorts of problems...
>> >>
>> >> OK, I reproduced a hang running 3.4-rc4. The data is here but it's a
>> >> whopping 2GB (yes it's compressed):
>> >> https://region-a.geo-1.objects.hpcloudsvc.com:443/v1.0/AUTH_9630ead2-6194-40df-afd3-7395448d4536/xfs-hang/report-2012-04-24.tar
>> >
>> > That's a bit big to be useful, and far bigger than I'm willing to
>> > download given that I'm on the end of a wet piece of string, not a
>> > big fat intarwebby pipe.
>>
>> Fair enough.
>>
>>
>> > I'm assuming it is the event trace
>> > that is causing it to blow out? If so, just the 30-60s either side of
>> > the hang first showing up is probaby necessary, and that should cut
>> > the size down greatly....
>>
>> Can I shorten the existing trace.dat?
>
> No idea, but that's likely the problem - I don't want the binary
> trace.dat file. I want the text output of the report command
> generated from the binary trace.dat file...

Well yes. I did RTFM :-) trace.dat is 15GB.


>> I stopped the trace
>> automatically 10 secs after the the xlog_... trace showed up in syslog
>> so effectively some 130+ secs after the hang occured.
>
> Extract the text report from it, and compress that. For example, a
> trace i've just done:
>
> $ ~/trace-cmd/trace-cmd report > trace.out
> $ ls -ltr |tail -4
> -rw-r--r-- 1 root root  21430272 Apr 27 08:36 trace.dat
> -rw-r--r-- 1 root root  10039296 Apr 27 08:36 trace.dat.cpu1
> -rw-r--r-- 1 root root  10035200 Apr 27 08:36 trace.dat.cpu0
> -rw-r--r-- 1 dave dave  48255670 Apr 27 08:37 trace.out
> $ gzip trace.out
> $ ls -ltr |tail -4
> -rw-r--r-- 1 root root  21430272 Apr 27 08:36 trace.dat
> -rw-r--r-- 1 root root  10039296 Apr 27 08:36 trace.dat.cpu1
> -rw-r--r-- 1 root root  10035200 Apr 27 08:36 trace.dat.cpu0
> -rw-r--r-- 1 dave dave   2500733 Apr 27 08:37 trace.out.gz
>
> Has 200MB of binary trace data, which generates a 470MB text output
> file, which compresses really well - down to 2.5MB in this case.

Compressed trace_report.txt is 2GB.
Sorry, haven't had the time today to look into this. I'll cut the size
down somehow.

...Juerg


> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@xxxxxxxxxxxxx

<Prev in Thread] Current Thread [Next in Thread>