[Top] [All Lists]

Re: Corruption errors with growfs

To: Fredrik Tolf <fredrik@xxxxxxxxxxxxx>
Subject: Re: Corruption errors with growfs
From: Dave Chinner <david@xxxxxxxxxxxxx>
Date: Fri, 4 Oct 2013 21:52:53 +1000
Cc: xfs@xxxxxxxxxxx
Delivered-to: xfs@xxxxxxxxxxx
In-reply-to: <alpine.DEB.2.10.1310040905380.3223@xxxxxxxxxxxxxxxxxxx>
References: <alpine.DEB.2.10.1310040905380.3223@xxxxxxxxxxxxxxxxxxx>
User-agent: Mutt/1.5.21 (2010-09-15)
On Fri, Oct 04, 2013 at 09:19:06AM +0200, Fredrik Tolf wrote:
> Dear list,
> I recently consolidated two filesystems that have previously been
> separate; I'll refer to them below as /home and /home/pub (since
> that's what they're actually called).
> When I first did so, I needed to grow /home quite a bit to
> accomodate the files in /home/pub. This is on LVM, so I extended the
> LV quite a bit (from about 550 GB to about 3.5 TB), and tried to
> grow the filesystem. At this point I encountered an error about
> corruption, prompting me to unmount the filesystem and run
> xfs_repair on it. I did so, it completed successfully and retained
> the filesystem at the size it was supposed to be grown to, so I
> ascribed the errors to some latent corruption by some older kernel
> version or something and went on with my life.
> However, today I tried to grow the filesystem by another 500 GB,
> encountering again a very similar error. Clearly, this couldn't just
> be left-over corruption from some earlier kernel bug since I'm still
> using the exact same kernel. What's worse, however, is that
> xfs_repair restored the filesystem to its size prior to running
> growfs, so it seems I can't grow the filesystem and am stuck at its
> current size.
> Does someone know what is happening, and what I can do to fix it?

Old kernel versions didn't zero the empty part of the secondary
superblocks when growing the filesystem. This commit in 3.8 fixed
the kernel growfs code not to put garbage in the new secondary

commit 1375cb65e87b327a8dd4f920c3e3d837fb40e9c2
Author: Dave Chinner <dchinner@xxxxxxxxxx>
Date:   Tue Oct 9 14:50:52 2012 +1100

    xfs: growfs: don't read garbage for new secondary superblocks

    When updating new secondary superblocks in a growfs operation, the
    superblock buffer is read from the newly grown region of the
    underlying device. This is not guaranteed to be zero, so violates
    the underlying assumption that the unused parts of superblocks are
    zero filled. Get a new buffer for these secondary superblocks to
    ensure that the unused regions are zero filled correctly.

    Signed-off-by: Dave Chinner <dchinner@xxxxxxxxxx>
    Reviewed-by: Carlos Maiolino <cmaiolino@xxxxxxxxxx>
    Signed-off-by: Ben Myers <bpm@xxxxxxx>

The only time the kernel reads secondary superblocks is during a
growfs operation, so that's the only time the kernel will detect
such an error. More extensive validity tests were added during 3.9
and 3.10, and these now throw corruption errors over secondary
superblocks that have not been correctly zeroed.

To fix this, you need to grab xfsprogs from the git repo
(3.2.0-alpha will do) as this commit to xfs_repair detects and fixes
the corrupted superblocks:

commit cbd7508db4c9597889ad98d5f027542002e0e57c
Author: Eric Sandeen <sandeen@xxxxxxxxxx>
Date:   Thu Aug 15 02:26:40 2013 +0000

    xfs_repair: zero out unused parts of superblocks
    Prior to:
    1375cb65 xfs: growfs: don't read garbage for new secondary superblocks
    we ran the risk of allowing garbage in secondary superblocks
    beyond the in-use sb fields.  With kernels 3.10 and beyond, the
    verifiers will kick these out as invalid, but xfs_repair does
    not detect or repair this condition.
    There is superblock stale-data zeroing code, but it is under a
    narrow conditional - the bug addressed in the above commit did not
    meet that conditional.  So change this to check unconditionally.
    Further, the checking code was looking at the in-memory
    superblock buffer, which was zeroed prior to population, and
    would therefore never possibly show any stale data beyond the
    last up-rev superblock field.
    So instead, check the disk buffer for this garbage condition.
    If we detect garbage, we must zero out both the in-memory sb
    and the disk buffer; the former may contain unused data
    in up-rev sb fields which will be written back out; the latter
    may contain garbage beyond all fields, which won't be updated
    when we translate the in-memory sb back to disk.
    The V4 superblock case was zeroing out the sb_bad_features2
    field; we also fix that to leave that field alone.
    Lastly, use offsetof() instead of the tortured (__psint_t)
    casts & pointer math.
    Reported-by: Michael Maier <m1278468@xxxxxxxxxxx>
    Signed-off-by: Eric Sandeen <sandeen@xxxxxxxxxx>
    Reviewed-by: Rich Johnston <rjohnston@xxxxxxx>
    Signed-off-by: Rich Johnston <rjohnston@xxxxxxx>


Dave Chinner

<Prev in Thread] Current Thread [Next in Thread>