XFS: I/O Error Detected / 2.6.27.39

[Emmanuel Florac]

Le Tue, 16 Nov 2010 14:10:51 +0100 vous écriviez:

> Hi,
> 
> Our environment is following:
> - we have 24GB RAM,
> - we are using 3ware controller (and it does not report any errors),

What model? 9550SX? 9650SE? 9690SA? 9750 ? 

> - we have one big logical volume (20TB) exported via NFS with large 
> amount of small files (about 150k),
> - we are doing periodically backup of this logical volume using rsync
> to another server.
> - we have kernel 2.6.27.39,

What distribution, architecture? What is the version of xfs tools
(try xfs_info -V for instance)? What are the xfs mount options?

> Unfortunately our system is freezing unexpectedly without reason.

What are the symptoms ? does the whole system freeze up? Or does it
crash with kernel panic, or otherwise "Oops" messages?

Older 3Ware cards (9550, early 9650) are prone to overheating and may
fail.

> We 
> started investigating this problem and noticed that cache memory is 
> slowly increasing.

This is completely normal and expected. Linux uses up all available
memory as a disk cache.

> We tried to dump this cache memory using:
> /bin/echo "3" > /proc/sys/vm/drop_caches
> 
> In a result, cache was dumped, but in logs we noticed a lot of errors 
> with XFS:
> 
> [kern.warning] kernel: xfs_iunlink_remove: xfs_inotobp()  returned an 
> error 22 on dm-16.  Returning error.
> [kern.notice] kernel: xfs_inactive:\011xfs_ifree() returned an error
> = 22 on dm-16
> [kern.notice] kernel: xfs_force_shutdown(dm-16,0x1) called from line 
> 1406 of file fs/xfs/xfs_vnodeops.c.  Return address = 0x
> [kern.alert] kernel: Filesystem \"dm-16\": I/O Error Detected.
> Shutting down filesystem: dm-16
> [kern.alert] kernel: Please umount the filesystem, and rectify the 
> problem(s)
> [kern.warning] kernel: xfs_imap_to_bp: xfs_trans_read_buf()returned
> an error 5 on dm-16.  Returning error.
> [kern.warning] kernel: xfs_imap_to_bp: xfs_trans_read_buf()returned
> an error 5 on dm-16.  Returning error.
> [kern.warning] kernel: xfs_imap_to_bp: xfs_trans_read_buf()returned
> an error 5 on dm-16.  Returning error.
> 
> We are wondering if this is problem connected to hardware or rather
> this is XFS problem (if yes, was it fixed?).

This may be an xfs bug but more details would be necessary.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac at intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------