XFS errors on large Infiniband fileserver setup

[Christian Herzog]

Dear all,

we (Physics Dept. at ETH Zurich) are trying to set up a large file 
server combo (two disk backends connected to a frontend by Infiniband, 
all running Ubuntu 10.04) and keep getting XFS internal error 
xfs_da_do_buf(2) messages when copying large amounts of data, resulting 
in 'structure needs cleaning' warnings. We have tried a lot of 
different kernels, iSCSI implementations, LVM configurations, whatnot, 
but these errors persist. The setup right now looks like this:

2 disk backends, each: Quad-Xeon X5550, 12G of RAM, 28T HW SATA-RAID6 
sliced into 2T chunks by LVM2 and exported via tgt 1.0.0-2, Ubuntu 
10.04 LTS, connected via Mellanox MHRH19B-XTR Infiniband + ISER to

1 frontend Octo-Xeon E5520, 12G of RAM, open-iscsi 2.0.871 initiator, 
Ubuntu 10.04 LTS. LMV2 stitches together the 2T-iSCSI-LUNs and provides 
a 10T test XFS filesystem

right now we're performing stress tests and when copying large amounts 
of data to the XFS filesystem, at some point we get

Filesystem "dm-3": XFS internal error xfs_da_do_buf(2) at line 2113 of 
file /home/kernel-ppa/mainline/build/fs/xfs/xfs_da_btree.c.  Caller 
0xffffffffa0299a1a
This can be provoked by running a 'du' or 'find' while writing the 
data.

on the frontend and XFS reports 'structure needs cleaning'. The 
following modifications have been suggested and we're working on them 
right now:

- try w/o ISER (direct IB over TCP)
- try an XFS filesystem < 2T
- try RHEL or SLES (will take more time)

We already had to change the I/O scheduler from Deadline to CFQ in 
order to get it up and running at all and also tried to change the 
kernel from stock LTS to 2.6.34-020634-generic, but we still get the FS 
errors.


root at phd-san-gw1:~# xfs_info /export/astrodata
meta-data=/dev/mapper/vgastro-lvastro isize=256    agcount=10, 
agsize=268435455 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=2684354550, 
imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=0 blks, 
lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0


we're slowly but surely running out of ideas here. Needless to say the 
system should have been deployed quite some time ago. Any help would be 
greatly appreciated. We're also happy to provide any further 
information that might be useful.

thanks a lot and kind regards,
-Christian



--
Dr. Christian Herzog <herzog at phys.ethz.ch>  support: +41 44 633 26 68
IT Services Group, HPT D 17                   voice: +41 44 633 39 50
Department of Physics, ETH Zurich
8093 Zurich, Switzerland                     http://nic.phys.ethz.ch