Interesting possible XFS crash condition

[Shawn Usry]

On 10/20/2010 3:12 AM, Michael Monnerie wrote:
> On Mittwoch, 20. Oktober 2010 Shawn Usry wrote:
>> Limited information shown in what I've been able to capture in the
>> kernel crash.  Nothing really specific or repeatable (different
>> message  each time) - some instances to the term "atomic" and "xfs"
>> - other times "irq" related
> I'm not a dev, but I'd say a kernel crash dump would be very helpful.
> Can't you at least take pictures of the messages?
>
> I've never read about such XFS errors, maybe you should
> 1) xfs_metadump
> 2) xfs_mdrestore (into a file)
> 3) mount that file
> and try to access files there. If this also crashes, it will really be
> XFS related.
>
> Also, can you try putting the hard disks onto another system, possibly
> with changing the RAID controller? It might be a hardware error.
>
Thanks for the suggestions / comments guys.

@Emmanuel - I've run a verify on the unit several times, some purposely, 
some that start automatically after the system reboots after a crash.
All have completed without a problem.   I even forcibly removed a disk, 
and re-added it to the array, to force a rebuild.  This completed 
without error,
or any messages other than start/completed in dmesg.

@Michael -  I can try to capture some of the kernel dump - but getting 
this info is often sketchy - most often, no dump is ever produced to 
even the console
screen.  Even using netconsole to redirect console output and kernel 
debugging set, there is often little if any information.   What data is 
sometimes
produced is rarely the same (seemingly) information - but I'll try to 
capture what I can on several repeat offenses.

I gave the xfs_metadata/xfs_mdrestore procedure a run and this produced 
no problems.  I could access the filesystem and files just fine - of 
course they are
all basically empty files so I couldn't really do any real work with 
them, but I could traverse the filesystem copy/move files just fine.  If 
there are any other
detailed tests I could try there please let me know.

On hardware swapping - I'll have to find an MB with a 64-bit pci slot in 
it.  Otherwise, I sadly don't have a second controller to work with.

A couple of other notes:

1.  I thought this might be driver-related (3w-xxxx) but I've tried 
several versions of the driver, by using different distributions (Centos 
5, Fedora 13)
with the same results.  To note, the array was originally created, and 
expanded, under Centos 5.5.   I reinstalled the OS to Fedora 13, hoping 
that newer
code might resolve the issue.  Same results.

2.  I did upgrade the firmware on the controller to a newer version 
AFTER the issue appeared, hoping this would resolve it.  Same results.

At this point I'm leaning toward faulty hardware somewhere.