xfs
[Top] [All Lists]

Re: XFS errors on large Infiniband fileserver setup

To: Christian Herzog <horeizo@xxxxxxxxxxxx>
Subject: Re: XFS errors on large Infiniband fileserver setup
From: Dave Chinner <david@xxxxxxxxxxxxx>
Date: Fri, 24 Sep 2010 09:53:55 +1000
Cc: xfs@xxxxxxxxxxx, isg@xxxxxxxxxxxx
In-reply-to: <29252416bd0d9dc973a909e411dbec6a@xxxxxxxxxxxx>
References: <29252416bd0d9dc973a909e411dbec6a@xxxxxxxxxxxx>
User-agent: Mutt/1.5.20 (2009-06-14)
On Thu, Sep 23, 2010 at 09:22:29AM +0200, Christian Herzog wrote:
> 
> Dear all,
> 
> we (Physics Dept. at ETH Zurich) are trying to set up a large file
> server combo (two disk backends connected to a frontend by
> Infiniband, all running Ubuntu 10.04) and keep getting XFS internal
> error xfs_da_do_buf(2) messages when copying large amounts of data,
> resulting in 'structure needs cleaning' warnings. We have tried a
> lot of different kernels, iSCSI implementations, LVM configurations,
> whatnot, but these errors persist. The setup right now looks like
> this:
> 
> 2 disk backends, each: Quad-Xeon X5550, 12G of RAM, 28T HW
> SATA-RAID6 sliced into 2T chunks by LVM2 and exported via tgt
> 1.0.0-2, Ubuntu 10.04 LTS, connected via Mellanox MHRH19B-XTR
> Infiniband + ISER to
> 
> 1 frontend Octo-Xeon E5520, 12G of RAM, open-iscsi 2.0.871
> initiator, Ubuntu 10.04 LTS. LMV2 stitches together the
> 2T-iSCSI-LUNs and provides a 10T test XFS filesystem

Out of curiousity, why are you using such a complex storage
configuration?

IMO, it is unneccessarily complex - you could easily do this (~30
drives) with a single server with a couple of external SAS JBOD
arrays and SAS RAID controllers. That would give you the same
performance (or better), with many fewer points of failure (both
hardware and software), use less rack space, and probably be
significantly cheaper....

Cheers,

Dave.

-- 
Dave Chinner
david@xxxxxxxxxxxxx

<Prev in Thread] Current Thread [Next in Thread>