xfs_repair of critical volume

[Steve Costaras]

On 2010-10-31 09:10, Emmanuel Florac wrote:
> Did you try to actually freeze the failed drives (it may revive them 
> for a while)?

Do NOT try this.   It's only good for some /very/ specific types of 
issues with older drives.   With an array of your size you are probably 
running relatively current drives (i.e. past 5-7 years) and this has a 
vary large probability of causing more damage.

The other questions are to the point to determine the circumstances 
around the failure and what the  state of the array was at the 
time.      Take your time, do not rush anything; you are already hanging 
over a cliff.

First thing if you are able is to do a bit copy of the physical drives 
to spares that way you can always get back to the same point where you 
are now.    This may not be practical with such a large array but if you 
have the means it's worth it.

You want to start from the lowest component and work your way up.    So 
you want to make sure that your raid array itself is sane before looking 
to fix any volume management functions and that before looking at your 
file systems.    When dealing with degraded or failed arrays be careful 
on what you do if you have write cache enabled on your controllers.   
Talk to the vendor!   Whatever operations you do on the card could cause 
this data to be lost and that can  be substantial with some controllers 
(MiB->GiB ranges).       Normally we  run w/ write cache disabled (both 
on the drive and on the raid controllers) for critical data to avoid 
having too much data in flight if a problem ever did occur.

The points that Emmanuel mentioned are valid;    Though would hold off 
on powering down until you are able to get all the geometry information 
from your raid's (unless you already have them).   Also would hold off 
until you determine if you have any dirty caches on the raid 
controllers.    Most controllers keep a rotating buffer of events 
including failure pointers that if you re-boot the re-scanning of drives 
upon start may push that pointer further down the stack until it gets 
lost and then you won't be able to recover outstanding data.   I've seen 
this set at 128 - 256 entries on various systems, another reason to keep 
drives per controller counts down.

Steve