XFS corruption problem

[Bjoern Metzdorf <bm@xxxxxxxxxxxxxxxxxxxxxxx>]

Hello everybody,

we have some major problems with a 10 TB XFS filesystem mounted as an 
ISCSI device under Linux 2.6. After a firmware upgrade on the underlying 
raid-controller, the device (/dev/sdb) has lost its partition table. 
/dev/sdb1 was the only device in an EVMS cluster volume.

Kevin Corry, one of the fellow EVMS developers who tries to help us, had 
a look at the first 1024k bytes of /dev/sdb1 and could not find rests of 
EVMS metadata, but suspected rests of the XFS filesystem beginning at 
sector 31:

Here are his comments:

> There doesn't appear to be any EVMS metadata at the front of sdb1, but there 
> does seem to be some sort of filesystem data starting at sector 31. There are 
> definitely XFS inode signatures, but unfortunately I don't see anything that 
> looks like an XFS superblock signature.
>
> One thing you could do to see if you can find an alternate superblock is run:
> hexdump -C /dev/evms/.nodes/sdb1 |grep XFSB
> It could take a while, since you've got a 9 TB disk. You might want to just 
> let it run for a bit and then kill it. Let me know if that reveals anything.
>
> To get the EVMS cluster container back, you can simply recreate it on sdb1. 
> Just delete the /dev/evms/sdb1 compatibility volume, and assign the Cluster 
> Segment Manager to sdb1 with the same name/options you used when you 
> originally created it. If you had an EVMS volume on that cluster segment, you 
> can recreate it as well. None of these operations will overwrite any of the 
> filesystem data.
>
> Unfortunately, I don't know how to restore the XFS superblock without running 
> mkfs.xfs (which you probably don't want to do). That's outside the scope of 
> what EVMS handles. At this point I would suspect some sort of data 
> corruption, seemingly related to the firmware update. You might ask on the 
> XFS mailing lists if there are any recovery tools that would help in this 
> situation, since it seems like a lot of your data is still there.

We already tried running xfs_repair -n on /dev/sdb1, with only the 
following results repeating like 10 times:

primary superblock not found, looking for secodaries ...
..........
.. found candidate secondary superblock...
.. unable to verify superblock, continuing...
.. found candidate secondary superblock...
.. unable to verify superblock, continuing...

and so on.

We did not run xfs_repair without -n yet, because we still were not able 
to make a full dump of the whole 10 TB to another device :/

Is there anything we can do to get our data back? Are there any other 
tools or manual debug methods we can use to recover things?

This is an urgent case for us, we appreciate every help we can get, we 
are also willing to pay for consulting and support.

Please reply also off list to me, as I am not subscribed to linux-xfs 
right now, and please excuse my cross post to xfs and linux-xfs.

Thanks in advance

Regards,
Bjoern