[Top] [All Lists]

Re: Files not touched in weeks got truncated after a crash

To: Guido Winkelmann <guido@xxxxxxxxxxxxxxxxxxxxxxxx>, xfs@xxxxxxxxxxx
Subject: Re: Files not touched in weeks got truncated after a crash
From: Ric Wheeler <ricwheeler@xxxxxxxxx>
Date: Thu, 14 Nov 2013 05:39:35 +0900
Delivered-to: xfs@xxxxxxxxxxx
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=jEpmQ/YF/WG9GwrMb9GK99LyAx57HYJiRGO9TFzZTDo=; b=U+mktw5TomlDGNGUeizC1PK3HIXQwXe/26Y9OAhsM/lyDa/aIaXR0NJjyEtxagwSQn TBJFM4A3BAQ+SJ1J0a9ZDWf2l/OyHnwsvhFvfBtrZ+vNygWNkX1GRL1oFX2gEFkglX1b ccEdtlALAfg7dhaOKkrVt5RSeYUifaysVWsgKJtdil8xFZVsg3vpVQdsHMlwNhfOuGhp VRncjq/0aQOIa0Y5F2w0mAyqIElcm57gHNHCIHcD4Y48fzepa1XlhfmSskI79G6prgN1 masLfPIL/EjhUivP1FonkkBSvJe4+PAVSuMlS2/F8Ky/4abZ5j4ko90m2CMU8/5H9FLp lWDw==
In-reply-to: <2662179.4mj0dgORXu@r008>
References: <2662179.4mj0dgORXu@r008>
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.1.0
You should update your kernel - this sounds like an issue that Dave fixed quite a few months back (and got shipped in RHEL and other distros, I don't know about when Centos would pick it up)


On 11/14/2013 01:36 AM, Guido Winkelmann wrote:

We are having some trouble with one of our fileservers using XFS (on linux).
Yesterday, one of the external RAIDs on the server failed. Of course, it is
unavoidable that some data would get lost from the fileserver in such an
event, however, we lost a lot more files than would seem reasonable. In
particular, we lost a number of files that had not been written to (but had
been been read from, in some cases) in several weeks.

The data loss manifested itself through files being truncated to length 0 or
to some other size short of what they should be. (We happen to have an
external database that keeps track of that.)

The fileserver is based on CentOS 6.3 with kernel version
2.6.32-279.9.1.el6.x86_64. It has got several external RAIDs in the 100 TB
range, connected via FibreChannel.

In case it matters: The server's primary role is as a samba server servicing a
large number of Windows XP and Windows 7 machines.

We had already been trying to reduce the possible impact of a hardware failure
by setting a few tunables in /etc/sysctl.conf to try and make the kernel not
keep dirty buffers around too long:

vm.dirty_background_bytes = 536870912
vm.dirty_bytes = 134217728
vm.dirty_writeback_centisecs = 500
vm.dirty_expire_centisecs = 3000

and by issuing a sync from cron every 15 minutes:

0,15,30,45 * * * * /bin/sync

Unfortunately, I seem to be unable so far to reproduce the issue on a smaller
system - and I cannot exactly just walk up to the in-production fileserver and
rip out yet another array just to see what happens...

This leaves me with a few questions:

Why did we lose so much data through the crash?

Why did not even a sync every 15 minutes prevent further damage?

What can we do to prevent this from happening again in the future?



xfs mailing list

<Prev in Thread] Current Thread [Next in Thread>