On 2010-10-31 09:10, Emmanuel Florac wrote:
Did you try to actually freeze the failed drives (it may revive them
for a while)?
Do NOT try this. It's only good for some /very/ specific types of
issues with older drives. With an array of your size you are probably
running relatively current drives (i.e. past 5-7 years) and this has a
vary large probability of causing more damage.
The other questions are to the point to determine the circumstances
around the failure and what the state of the array was at the
time. Take your time, do not rush anything; you are already hanging
over a cliff.
First thing if you are able is to do a bit copy of the physical drives
to spares that way you can always get back to the same point where you
are now. This may not be practical with such a large array but if you
have the means it's worth it.
You want to start from the lowest component and work your way up. So
you want to make sure that your raid array itself is sane before looking
to fix any volume management functions and that before looking at your
file systems. When dealing with degraded or failed arrays be careful
on what you do if you have write cache enabled on your controllers.
Talk to the vendor! Whatever operations you do on the card could cause
this data to be lost and that can be substantial with some controllers
(MiB->GiB ranges). Normally we run w/ write cache disabled (both
on the drive and on the raid controllers) for critical data to avoid
having too much data in flight if a problem ever did occur.
The points that Emmanuel mentioned are valid; Though would hold off
on powering down until you are able to get all the geometry information
from your raid's (unless you already have them). Also would hold off
until you determine if you have any dirty caches on the raid
controllers. Most controllers keep a rotating buffer of events
including failure pointers that if you re-boot the re-scanning of drives
upon start may push that pointer further down the stack until it gets
lost and then you won't be able to recover outstanding data. I've seen
this set at 128 - 256 entries on various systems, another reason to keep
drives per controller counts down.