xfs
[Top] [All Lists]

Re: xfs_trans_read_buf error / xfs_force_shutdown with LVM snapshot and

To: Wolfram Schlich <lists@xxxxxxxxxxxxxxxxxxx>
Subject: Re: xfs_trans_read_buf error / xfs_force_shutdown with LVM snapshot and Xen kernel 2.6.18
From: Eric Sandeen <sandeen@xxxxxxxxxxx>
Date: Thu, 18 Jun 2009 08:57:00 -0500
Cc: linux-xfs@xxxxxxxxxxx
In-reply-to: <20090618065621.GD16867@xxxxxxxxxxxxx>
References: <20090618065621.GD16867@xxxxxxxxxxxxx>
User-agent: Thunderbird 2.0.0.21 (Macintosh/20090302)
Wolfram Schlich wrote:
> Hi!
> 
> I'm currently using LVM snapshots to create full system backups
> of a bunch of Xen-based virtual machines (so-called domUs).
> Those domUs all run Xen kernel 2.6.18 from the Xen 3.2.0 release
> (32bit domU on 32bit dom0, I can post the .config if needed).
> All domUs are using XFS on their LVM logical volumes.
> The backup of all mounted snapshot volumes is made using
> rsnapshot/rsync. This has been running smoothly for some
> weeks now on 5 domUs.
> 
> Yesterday this happened during the backup on 1 domU:
> --8<--
> kernel: I/O error in filesystem ("dm-21") meta-data dev dm-21 block 0x604d68  
>      ("xfs_trans_read_buf") error 5 buf count 4096
> kernel: I/O error in filesystem ("dm-21") meta-data dev dm-21 block 0x66c5a0  
>      ("xfs_trans_read_buf") error 5 buf count 4096
> kernel: I/O error in filesystem ("dm-21") meta-data dev dm-21 block 0x202f70  
>      ("xfs_trans_read_buf") error 5 buf count 4096
> kernel: I/O error in filesystem ("dm-21") meta-data dev dm-21 block 0x2701f8  
>      ("xfs_trans_read_buf") error 5 buf count 4096
> kernel: I/O error in filesystem ("dm-21") meta-data dev dm-21 block 0x6a78    
>    ("xfs_trans_read_buf") error 5 buf count 4096
> kernel: I/O error in filesystem ("dm-21") meta-data dev dm-21 block 0x600500  
>      ("xfs_trans_read_buf") error 5 buf count 8192
> kernel: I/O error in filesystem ("dm-21") meta-data dev dm-21 block 0x600520  
>      ("xfs_trans_read_buf") error 5 buf count 8192
> kernel: I/O error in filesystem ("dm-21") meta-data dev dm-21 block 0x600520  
>      ("xfs_trans_read_buf") error 5 buf count 8192
> kernel: I/O error in filesystem ("dm-21") meta-data dev dm-21 block 0xdd0     
>   ("xfs_trans_read_buf") error 5 buf count 8192
> kernel: I/O error in filesystem ("dm-21") meta-data dev dm-21 block 0x4055d0  
>      ("xfs_trans_read_buf") error 5 buf count 8192
> [...many more of such messages...]

Well these are all I/O errors happening -to- xfs, so xfs is unlikely to
be at fault here.  Any block layer messages before that?

> kernel: xfs_force_shutdown(dm-21,0x1) called from line 424 of file 
> fs/xfs/xfs_rw.c.  Return address = 0xc02b1cbb
> kernel: Filesystem "dm-21": I/O Error Detected.  Shutting down filesystem: 
> dm-21
> kernel: Please umount the filesystem, and rectify the problem(s)
> kernel: xfs_force_shutdown(dm-21,0x1) called from line 424 of file 
> fs/xfs/xfs_rw.c.  Return address = 0xc02b1cbb

> --8<--
> The rsync process was then terminated with SIGBUS (exit code 135 -> 128+7).
> 
> The device dm-21 was the snapshot of the /var filesystem and
> was mounted using nouuid,norecovery.
> 
> Is it possible that the LVM snapshot (that should be using
> xfs_freeze/xfs_unfreeze) has created an inconsistent/damaged
> snapshot that was kept from being repaired through norecovery?
> Any other ideas?

If it was a proper snapshot norecovery shouldn't matter, as the fs
should be clean already (well, hopefully, 2.6.18 was a long time ago;
this is true today, anyway)

I suppose it's possible that the snapshot was not consistent, and you're
hitting problems there, but things like:

> kernel: I/O error in filesystem ("dm-21") meta-data dev dm-21 block
0xdd0       ("xfs_trans_read_buf") error 5 buf count 8192

looks like a failure to read a perfectly normal block, not out of bounds
or anything, so I'd most likely point to problems outside xfs.

-Eric

<Prev in Thread] Current Thread [Next in Thread>