[Top] [All Lists]

Re: Interesting possible XFS crash condition

Subject: Re: Interesting possible XFS crash condition
From: Shawn Usry <shawn@xxxxxxxxxxxxxxxx>
Date: Wed, 20 Oct 2010 14:01:00 -0500
Cc: xfs@xxxxxxxxxxx
In-reply-to: <201010201012.39778@xxxxxx>
References: <4CBE887F.6020506@xxxxxxxxxxxxxxxx> <201010201012.39778@xxxxxx>
User-agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv: Gecko/20101013 Lightning/1.0b2 Thunderbird/3.1.5

On 10/20/2010 3:12 AM, Michael Monnerie wrote:
On Mittwoch, 20. Oktober 2010 Shawn Usry wrote:
Limited information shown in what I've been able to capture in the
kernel crash.  Nothing really specific or repeatable (different
message  each time) - some instances to the term "atomic" and "xfs"
- other times "irq" related
I'm not a dev, but I'd say a kernel crash dump would be very helpful.
Can't you at least take pictures of the messages?

I've never read about such XFS errors, maybe you should
1) xfs_metadump
2) xfs_mdrestore (into a file)
3) mount that file
and try to access files there. If this also crashes, it will really be
XFS related.

Also, can you try putting the hard disks onto another system, possibly
with changing the RAID controller? It might be a hardware error.

Thanks for the suggestions / comments guys.

@Emmanuel - I've run a verify on the unit several times, some purposely, some that start automatically after the system reboots after a crash. All have completed without a problem. I even forcibly removed a disk, and re-added it to the array, to force a rebuild. This completed without error,
or any messages other than start/completed in dmesg.

@Michael - I can try to capture some of the kernel dump - but getting this info is often sketchy - most often, no dump is ever produced to even the console screen. Even using netconsole to redirect console output and kernel debugging set, there is often little if any information. What data is sometimes produced is rarely the same (seemingly) information - but I'll try to capture what I can on several repeat offenses.

I gave the xfs_metadata/xfs_mdrestore procedure a run and this produced no problems. I could access the filesystem and files just fine - of course they are all basically empty files so I couldn't really do any real work with them, but I could traverse the filesystem copy/move files just fine. If there are any other
detailed tests I could try there please let me know.

On hardware swapping - I'll have to find an MB with a 64-bit pci slot in it. Otherwise, I sadly don't have a second controller to work with.

A couple of other notes:

1. I thought this might be driver-related (3w-xxxx) but I've tried several versions of the driver, by using different distributions (Centos 5, Fedora 13) with the same results. To note, the array was originally created, and expanded, under Centos 5.5. I reinstalled the OS to Fedora 13, hoping that newer
code might resolve the issue.  Same results.

2. I did upgrade the firmware on the controller to a newer version AFTER the issue appeared, hoping this would resolve it. Same results.

At this point I'm leaning toward faulty hardware somewhere.

<Prev in Thread] Current Thread [Next in Thread>