xfs
[Top] [All Lists]

Re: 2.6.38.8 kernel bug in XFS or megaraid driver with heavy I/O load

To: Dave Chinner <david@xxxxxxxxxxxxx>
Subject: Re: 2.6.38.8 kernel bug in XFS or megaraid driver with heavy I/O load
From: Anders Ossowicki <aowi@xxxxxxxxxxxxx>
Date: Wed, 12 Oct 2011 14:29:33 +0200
Cc: Christoph Hellwig <hch@xxxxxxxxxxxxx>, "linux-kernel@xxxxxxxxxxxxxxx" <linux-kernel@xxxxxxxxxxxxxxx>, "aradford@xxxxxxxxx" <aradford@xxxxxxxxx>, "xfs@xxxxxxxxxxx" <xfs@xxxxxxxxxxx>
In-reply-to: <20111012003526.GI3159@dastard>
Mail-followup-to: Dave Chinner <david@xxxxxxxxxxxxx>, Christoph Hellwig <hch@xxxxxxxxxxxxx>, "linux-kernel@xxxxxxxxxxxxxxx" <linux-kernel@xxxxxxxxxxxxxxx>, "aradford@xxxxxxxxx" <aradford@xxxxxxxxx>, "xfs@xxxxxxxxxxx" <xfs@xxxxxxxxxxx>
References: <20111011091757.GA32589@xxxxxxxxxxxxxxx> <20111011133448.GA10692@xxxxxxxxxxxxx> <20111011141338.GA11808@xxxxxxxxxxxxxxx> <20111012003526.GI3159@dastard>
Reply-to: <aowi@xxxxxxxxxxxxx>
User-agent: Mutt/1.5.21 (2010-09-15)
On Wed, Oct 12, 2011 at 02:35:26AM +0200, Dave Chinner wrote:
> Memory corruption can be caused by more than just a bad memory
> stick. You've got a brand new driver running your brand new
> controller and it may still have bugs - it might be scribbling over
> memory it doesn't own because of off-by-one index errors, etc. It's
> much more likely that that new hardware or driver code is the cause
> of your problem than an undetected ECC memory error or core VM
> problem.
Ah, now that I agree on. A few more observations from today's experiments:

First of all, there are two MegaRAID controllers in the machine. The 
old'n'reliable 8888ELP and the new'n'wonky 9285-8e. Both are using the megaraid
driver and the 8888ELP card ran with the megaraid driver prior to the 
refactoring that introduced support for 9285-8e without a hitch for about a
year.

We've gotten to a point where we can reliably reproduce this by running certain
queries in postgresql when data from the disk is cached. E.g 
 foo=# select count(*) from foo.sequence;
 ERROR:  invalid page header in block 529134 of relation base/16385/58318945

If we echo 3 >/proc/sys/vm/drop_caches and reload postgres, the same queries
work. This does indeed smell like memory corruption.

The 9285-8e controller has FastPath enabled.

> FWIW, if it's a repeatable problem, you might want to update the
> driver and controller firmware to something more recent and see if
> that solves the problem....
I upgraded the firmware (post-accident) but we're still seeing the corruption.
-- 
Anders

<Prev in Thread] Current Thread [Next in Thread>