[Top] [All Lists]

Re: XFS corruption with failover

To: John Quigley <jquigley@xxxxxxxxxxxx>
Subject: Re: XFS corruption with failover
From: Felix Blyakher <felixb@xxxxxxx>
Date: Thu, 13 Aug 2009 16:44:05 -0500
Cc: XFS Development <xfs@xxxxxxxxxxx>
In-reply-to: <4A8474D2.7050508@xxxxxxxxxxxx>
References: <4A8474D2.7050508@xxxxxxxxxxxx>

On Aug 13, 2009, at 3:17 PM, John Quigley wrote:


We're deploying XFS in a configuration where the file system is being exported with NFS. XFS is being mounted on Linux, with default options; an iSCSI volume is the formatted media. We're working out a failover solution for this deployment utilizing Linux HA. Things appear to work correctly in the general case, but in continuous testing we're getting XFS superblock corruption on a very reproducible basis.
The sequence of events in our test scenario:

1. NFS server #1 online
2. Run IO to NFS server #1 from NFS client
3. NFS server #1 offline, (via passing 'b' to /proc/sysrq-trigger)
4. NFS server #2 online
5. XFS mounted as part of failover mechanism, mount fails

The mount fails with the following:

kernel: XFS mounting filesystem sde
kernel: Starting XFS recovery on filesystem: sde (logdev: internal)
kernel: XFS: xlog_recover_process_data: bad clientid
kernel: XFS: log mount/recovery failed: error 5

This is an IO error. Is the block device (/dev/sde) accessible
from the server #2 OK? Can you dd from that device?

kernel: XFS: log mount failed

When running xfs_repair:

That's not a good time to run xfs_repair. There were no
indication that the filesystem is corrupted.

Let's take for a sec "NFS server #2" out of the picture.
Can you mount the filesystem from the original server
after it reboots?


[root@machine ~]# xfs_repair /dev/sde xfs_repair: warning - cannot set blocksize on block device /dev/sde: Invalid argument
Phase 1 - find and verify superblock...
Phase 2 - using internal log
      - zero log...
ERROR: The filesystem has valuable metadata changes in a log which needs ...

Any advice or insight into what we're doing wrong would be very much appreciated. My apologies in advance for the somewhat off-topic question.

- John Quigley

xfs mailing list

<Prev in Thread] Current Thread [Next in Thread>