Dave Chinner wrote:
> On Tue, Aug 13, 2013 at 05:30:58PM +0200, Michael Maier wrote:
>> Dave Chinner wrote:
>>> [ re-ccing the list, because finding this is in everyone's interest ]
>>> On Mon, Aug 12, 2013 at 06:25:16PM +0200, Michael Maier wrote:
>>>> Eric Sandeen wrote:
>>>>> On 8/11/13 2:11 AM, Michael Maier wrote:
>>>>>> I think I'm facing the same problem as already described here:
>>>>> Maybe you can try the tracing Dave suggested in that thread?
>>>>> It certainly does look similar.
>>>> I attached a trace report while executing xfs_growfs /mnt on linux 3.10.5
>>>> (does not happen with 3.9.8).
>>>> xfs_growfs /mnt
>>>> meta-data=/dev/mapper/backupMy-daten3 isize=256 agcount=42,
>>>> agsize=7700480 blks
>>>> = sectsz=512 attr=2
>>>> data = bsize=4096 blocks=319815680, imaxpct=25
>>>> = sunit=0 swidth=0 blks
>>>> naming =version 2 bsize=4096 ascii-ci=0
>>>> log =internal bsize=4096 blocks=60160, version=2
>>>> = sectsz=512 sunit=0 blks, lazy-count=1
>>>> realtime =none extsz=4096 blocks=0, rtextents=0
>>>> xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning
>>>> data blocks changed from 319815680 to 346030080
>>>> The entry in messages was:
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368030] ffff8801e8dbd400: 58 46 53 42
>>>> 00 00 10 00 00 00 00 00 13 10 00 00 XFSB............
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368037] ffff8801e8dbd410: 00 00 00 00
>>>> 00 00 00 00 00 00 00 00 00 00 00 00 ................
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368042] ffff8801e8dbd420: 46 91 c6 80
>>>> a9 a9 4d 8c 8f e2 18 fd e8 7f 66 e1 F.....M.......f.
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368045] ffff8801e8dbd430: 00 00 00 00
>>>> 04 00 00 04 00 00 00 00 00 00 00 80 ................
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368051] XFS (dm-33): Internal error
>>>> xfs_sb_read_verify at line 730 of file
>>>> Caller 0xffffffffa099a2fd
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368533] XFS (dm-33): Corruption
>>>> detected. Unmount and run xfs_repair
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368611] XFS (dm-33): metadata I/O
>>>> error: block 0x3ac00000 ("xfs_trans_read_buf_map") error 117 numblks 1
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368623] XFS (dm-33): error 117
>>>> reading secondary superblock for ag 16
>>> Ok, so that's reading the secondary superblock for AG 16. You're
>>> growing the filesystem from 42 to 45 AGs, so this problem is not
>>> related to the actual grow operation - it's tripping over a problem
>>> that already exists on disk before the grow operation is started.
>>> i.e. this is likely to be a real corruption being seen, and it
>>> happened some time in the distant past and so we probably won't ever
>>> be able to pinpoint the cause of the problem.
>>> That said, let's have a look at the broken superblock. Can you post
>>> the output of the commands:
>>> # xfs_db -r -c "sb 16" -c p <dev>
>> done after the failed growfs mentioned above:
> Looks fine....
>>> # xfs_db -r -c "sb 16" -c "type data" -c p <dev>
>> 000: 58465342 00001000 00000000 13100000 00000000 00000000 00000000 00000000
>> 020: 4691c680 a9a94d8c 8fe218fd e87f66e1 00000000 04000004 00000000 00000080
>> 040: 00000000 00000081 00000000 00000082 00000001 00758000 0000002a 00000000
>> 060: 0000eb00 b4a40200 01000010 00000000 00000000 00000000 0c090804 17000019
>> 080: 00000000 00001940 00000000 00000277 00000000 001126ba 00000000 00000000
>> 0a0: 00000000 00000000 00000000 00000000 00000000 00000002 00000000 00000000
>> 0c0: 00000000 00000001 0000000a 0000000a 8f980320 73987e9e db829704 ef73fe2e
>> 0e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 100: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 120: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 140: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 160: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 180: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 1a0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 1c0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 1e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
> There's your problem - the empty space in the superblock is supposed
> to be zero. mkfs zeros it and we rely on it being zero for various
> And one of those reasons is that we use the fact it shoul dbe zero
> to determine if we should be checking the CRC of the superblock.
> That is if there's a single bit error in the superblock and we are
> missing the correct bit in the version numbers that say CRCs are
> enabled, we use the fact that the superblock CRC field - which your
> filesystem knowns nothing about - should be zero to validate that
> the CRC feature bit is correctly set. The above superblock will
> indicate that there is a CRC set on the superblock, find the
> necessary version number is not correct, and so therefore we have a
> corruption in that superblock that the kernel code cannot handle
> without a user telling it what is correct.
> So, the fact grwofs is failing is actually the correct behaviour for
> the filesystem to have in this case - the superblock is corrupt,
> just not obviously so.
>>> so we can see the exact contents of that superblock?
>>> FWIW, how many times has this filesystem ben grown?
>> I can't say for sure, about 4 or 5 times?
>>> Did it start
>>> with only 32 AGs (i.e. 10TB in size)?
>> 10TB? No. The device just has 3 TB. You most probably meant 10GB?
>> I'm not sure, but it definitely started with > 100GB.
> I misplaced a digit A block size of 4096 bytes and:
> agcount=42, agsize=7700480 blks
> So the filesystem size is 42 * 7700480 * 4096 = 1.26TB.
> The question I'm asking is how many AGs did the filesystem start
> with, because this:
> commit 1375cb65e87b327a8dd4f920c3e3d837fb40e9c2
> Author: Dave Chinner <dchinner@xxxxxxxxxx>
> Date: Tue Oct 9 14:50:52 2012 +1100
> xfs: growfs: don't read garbage for new secondary superblocks
> When updating new secondary superblocks in a growfs operation, the
> superblock buffer is read from the newly grown region of the
> underlying device. This is not guaranteed to be zero, so violates
> the underlying assumption that the unused parts of superblocks are
> zero filled. Get a new buffer for these secondary superblocks to
> ensure that the unused regions are zero filled correctly.
> Signed-off-by: Dave Chinner <dchinner@xxxxxxxxxx>
> Reviewed-by: Carlos Maiolino <cmaiolino@xxxxxxxxxx>
> Signed-off-by: Ben Myers <bpm@xxxxxxx>
> Is the only possible reason I can think of that would result in
> non-zero empty space in a secondary superblock. And that implies
> that the filesystem started with 16 AGs or less,
> and was grown with
> an older kernel with this bug in it.
> If it makes you feel any better, the bug that caused this had been
> in the code for 15+ years and you are the first person I know of to
> have ever hit it....
Probably the second one :-) See
> xfs_repair doesn't appear to have any checks in it to detect this
> situation or repair it - there are some conditions for zeroing the
> unused parts of a superblock, but they are focussed around detecting
> and correcting damage caused by a buggy Irix 6.5-beta mkfs from 15
> years ago.
The _big problem_ is: xfs_repair not just doesn't repair it, but it
_causes data loss_ in some situations!
Given the following situation I ran in:
- xfs_growfs started running linux 3.10.5.
- Saw the error message on the konsole:
XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning
data blocks changed from 319815680 to 346030080
- Checked with df -> The growing seems to be done. Decision: Analyse the
problem later when there is more time.
- Some days later, entry found in messages:
"Corruption detected. Unmount and run xfs_repair"
- I did it as suggested.
Result: FS has the original size again before growing the FS and
complete loss of all data written since this faulty growing. And: FS
If it is not a problem at all (that's how I understood you here), why is
there a error message and the suggest to run xfs_repair, which obviously
isn't able at all to repair this problem but leads directly to data loss?
Thanks for your clarification. I hope other people read this thread
before they are loosing data :-(.
What to do now?
- Don't use >= 3.10.x kernel. Or:
- Ignore it (how can I distinguish this case from other cases?) Or:
- Recreate the complete FS.
Thanks for clarification,