xfs
[Top] [All Lists]

XFS with nfs over rdma performance

To: xfs@xxxxxxxxxxx
Subject: XFS with nfs over rdma performance
From: Samuel Kvasnica <samuel.kvasnica@xxxxxxxxx>
Date: Tue, 22 Jan 2013 12:10:28 +0100
Delivered-to: xfs@xxxxxxxxxxx
Organization: IMS Nanofabrication AG
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130105 Thunderbird/17.0.2
Hi folks,

I would like to hear about your experience with the performance of XFS when
used on NFS client mounted using Infininband RDMA connection on 3.4.11
kernel.

What we observe is following:

- we do have local RAID storage with 1.4GB/s read and write performance
(both dd on raw partition
and on xfs filesystem give basically the same performance)

- we do have QDR Infiniband connection (Mellanox), the rdma benchmark
gives 29Gbit/s throughput

Now, both above points look pretty Ok but if we mount an nfs export
using rdma on client we never get the 1.4GB/s throughput.
Sporadically (and especially at the beginning) it comes up to some
1.3GB/s for short period but then it starts oscillating
between 300MB/s and some 1.2GB/s with an average of 500-600MB/s. Even
when using more clients in parallel,
the net throughput behaves the same so it seems to be a server-side
related bottleneck.
We do not see any remarkable CPU load.

The interesting point is, we use btrfs filesystem on server instead of
xfs now (with otherwise same config) and we are getting consistent,
steady throughput
around 1.2-1.3GB/s.

What is wrong with XFS on 3.x kernel ? Any hints what parameters to look
at ?


Samuel

<Prev in Thread] Current Thread [Next in Thread>