Input/Output error

[Dave Chinner]

On Tue, Feb 22, 2011 at 09:14:24PM +0530, Srinivasan T wrote:
> Hi,
> 
> We are running an C++ application in AWS EC2 instance (CentOS 5.4)
> mounted with an EBS Volume (say /mymountpoint). We do more
> simultaneous writes to the EBS Volume from our application. But at
> some point we get 'ERROR: Input/output error'. After this, 'ls -l
> /mymountpoint' command itself fails with the i/o error. The
> filesystem which we use for the EBS Volume is xfs.
> 
> I unmounted the drive and done xfs_check and again mounted the
> drive. Now, everything seems to be working fine. But the issue still
> persists everytime when we do simultaneous writes.
> 
> I believe the following details will be useful,
> 
> [root at domU-12-31-39-07-81-36 StoreGrid]# cat /etc/redhat-release
> CentOS release 5.4 (Final)
> 
> [root at domU-12-31-39-07-81-36 StoreGrid]# df -lTi
> Filesystem    Type    Inodes   IUsed   IFree IUse% Mounted on
> /dev/sda1     ext3   1310720  107566 1203154    9% /
> /dev/sdb      ext3   19546112      11 19546101    1% /mnt
> none         tmpfs    186059       1  186058    1% /dev/shm
> /dev/sdh       xfs   1934272  495857 1438415   26% /mymountpoint
> 
> [root at domU-12-31-39-07-81-36 StoreGrid]# uname -a
> Linux domU-12-31-39-07-81-36 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15
> 12:39:36 EST 2008 i686 i686 i386 GNU/Linux
> 
> Output of dmesg :
> 
> SGI XFS with ACLs, security attributes, large block numbers, no
> debug enabled
> SGI XFS Quota Management subsystem
> Filesystem "sdh": Disabling barriers, not supported by the underlying device
> XFS mounting filesystem sdh
> Ending clean XFS mount for filesystem: sdh
> Filesystem "sdh": XFS internal error xfs_trans_cancel at line 1138
> of file fs/xfs/xfs_trans.c.  Caller 0xee201944
>  [<ee2032fe>] xfs_trans_cancel+0x59/0xe3 [xfs]
>  [<ee201944>] xfs_rename+0x8f8/0x954 [xfs]
>  [<ee201944>] xfs_rename+0x8f8/0x954 [xfs]
>  [<ee21458c>] xfs_vn_rename+0x30/0x70 [xfs]
>  [<c10bb5e3>] selinux_inode_rename+0x11f/0x16d
>  [<c1078d88>] vfs_rename+0x2c3/0x441
>  [<c107a77f>] sys_renameat+0x15a/0x1b4
>  [<c1074b7f>] sys_stat64+0xf/0x23
>  [<c1072d3b>] __fput+0x140/0x16a
>  [<c10841ee>] mntput_no_expire+0x11/0x6a
>  [<c107a800>] sys_rename+0x27/0x2b
>  [<c1005688>] syscall_call+0x7/0xb
>  =======================
> xfs_force_shutdown(sdh,0x8) called from line 1139 of file
> fs/xfs/xfs_trans.c.  Return address = 0xee217778
> Filesystem "sdh": Corruption of in-memory data detected.  Shutting
> down filesystem: sdh
> Please umount the filesystem, and rectify the problem(s)
> I/O error in filesystem ("sdh") meta-data dev sdh block 0x3c0001
> ("xfs_trans_read_buf") error 5 buf count 512
> I/O error in filesystem ("sdh") meta-data dev sdh block 0x780001
> ("xfs_trans_read_buf") error 5 buf count 512

Looks like an IO error reading the AGF during a transaction. That
would indicate a problem with your storage, not XFS.

Cheers,

Dave.
-- 
Dave Chinner
david at fromorbit.com