xfs
[Top] [All Lists]

Performance degradation over time

To: xfs@xxxxxxxxxxx
Subject: Performance degradation over time
From: Marcin Deranek <marcin.deranek@xxxxxxxxxxx>
Date: Wed, 10 Oct 2012 10:51:42 +0200
Dkim-signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=booking.com; s=bk; h=Content-Transfer-Encoding:Content-Type:Mime-Version:Message-ID:Subject:To:From:Date; bh=v1hFacFq9mWV7pHose68G91PFgW5skmkowIyFH0Yvsw=; b=ywnl8l6bqy6AtOuPG8jnOTKt6G6MZUcNZU48RB79HFdxsf7nMvWX5Xn+8sSPpV8TwVeuZGMLnUVgW1puxSfnT7MJaqobDBwWUwuHDyrQk5H8siZsDTqMHp9vIw4OUZoFg32tyIIzPzcWP6xt79GRm5+9GtWw72dO5fBMIy8ZLo0=;
Organization: Booking
Hi,

We are running XFS filesystem on one of out machines which is a big
store (~3TB) of different data files (mostly images). Quite recently we
experienced some performance problems - machine wasn't able to keep up
with updates. After some investigation it turned out that open()
syscalls (open for writing) were taking significantly more time than
they should eg. 15-20ms vs 100-150us.
Some more info about our workload as I think it's important here:
our XFS filesystem is exclusively used as data store, so we only
read and write our data (we mostly write). When new update comes it's
written to a temporary file eg.

/mountpoint/some/path/.tmp/file

When file is completely stored we move it to final location eg.

/mountpoint/some/path/different/subdir/newname

That means that we create lots of files in /mountpoint/some/path/.tmp
directory, but directory is empty as they are moved (rename() syscall)
shortly after file creation to a different directory on the same
filesystem.
The workaround which I found so far is to remove that directory
(/mountpoint/some/path/.tmp in our case) with its content and re-create
it. After this operation open() syscall goes down to 100-150us again.
Is this a known problem ?
Information regarding our system:
CentOS 5.8 / kernel 2.6.18-308.el5 / kmod-xfs-0.4-2
Let me know if you need to know anything more.
Cheers,

Marcin

<Prev in Thread] Current Thread [Next in Thread>