On Thu, Mar 13, 2014 at 02:47:58PM +0000, Mears, Morgan wrote:
> On Wed, Mar 12, 2014 at 07:03:14PM -0400, Dave Chinner wrote:
> > On Wed, Mar 12, 2014 at 08:14:32PM +0000, Mears, Morgan wrote:
> >> Hi,
> >> Please CC me on any responses; I don't subscribe to this list.
> >> I ran into a possible XFS bug while doing some Oracle benchmarking. My
> >> test
> >> system is running a 3.14.0-rc3+ kernel built from the for-next branch of
> >> git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git
> >> on 2014-02-19 (last commit 1342f11e713792e53e4b7aa21167fe9caca81c4a).
> >> The XFS instance in question is 200 GB and should have all default
> >> parameters (mkfs.xfs /dev/mapper/<my_lun_partition>). It contains Oracle
> >> binaries and trace files. At the time the issue occurred I had been
> >> running Oracle with SQL*NET server tracing enabled. The affected XFS
> >> had filled up 100% with trace files several times; I was periodically
> >> executing rm -f * in the trace file directory, which would reduce the
> >> file system occupancy from 100% to 3%. I had an Oracle load generating
> >> tool running, so new log files were being created with some frequency.
> >> The issue occurred during one of my rm -f * executions; afterwards the
> >> file system would only produce errors. Here is the traceback:
> >> [1552067.297192] XFS: Internal error XFS_WANT_CORRUPTED_GOTO at line 1602
> >> of file fs/xfs/xfs_alloc.c. Caller 0xffffffffa04c4905
> > So, freeing a range that is already partially free. The problem
> > appears to be in AG 15, according to the repair output.
> >> https://dl.dropboxusercontent.com/u/31522929/xfs-double-free-xfs_metadump-before-repair.gz
> > AGF 15 is full:
> > on the unlinked list:
> > agi unlinked bucket 24 is 6477464 in ag 14 (inode=946001560)
> > So, prior to recovery, what did it contain? it's got 287 bytes of
> > date, and a single extent:
> > u.bmx = [startoff,startblock,blockcount,extentflag] 0:[0,59154425,1,0]
> > xfs_db> convert fsb 59154425 agno
> > 0xe (14)
> > xfs_db> convert fsb 59154425 agbno
> > 0x69ff9 (434169)
> > Ok, so the corruption, whatever it was, happened a long time ago,
> > and it's only when removing the file that it was tripped over.
> > There's nothing more I can really get from this - the root cause of
> > the corruption is long gone.
> > Cheers,
> > Dave.
> > --
> > Dave Chinner
> > david@xxxxxxxxxxxxx
> Thanks Dave.
> Upon restarting my testing I immediately hit this error again (or a very
> similar one in any case). I suspect that the corruption you've noted was
> not properly repaired by xfs_repair.
What happens if you run xfs_repair twice in a row?
> I captured all the same data as before, as well as an xfs_metadump from
> after the xfs_repair. If you're interested, it's all in this tarball:
Ok, that's triggered the right side btree checks, not the left side
like the previous one. It's probably AG 14 again.
EFI: cnt:1 total:1 a:0x19c77b0 len:48
EFI: #regs:1 num_extents:2 id:0xffff880f8011c640
(s: 0x3816a3b, l: 112) (s: 0x3817cb1, l: 1920)
So, two extents being freed:
xfs_db> convert fsb 0x3816a3b agno
xfs_db> convert fsb 0x3816a3b agbno
xfs_db> convert fsb 0x3817cb1 agbno
Surrounding free space regions:
66:[92551,180] 67:[92856,2] -> used space range [92731,125]
172:[97415,42] 173:[97622,4] -> used space range [97457,65]
So the first extent is good. The second, however, aligns correctly
to the free space region to the left, but massively overruns the
used space region which is only 165 blocks long. So it's a similar
problem here - both the free space trees are internally consistent,
the inode BMBT is internally consistent, but the space that they
track is not consistent.
63:[92551,292] 64:[92856,2] -> correctly accounted
169:[97415,49] 170:[97468,56] 171:[97528,168]
-> used space [97464,2], [97524,4]
But that's a very different freespace map around the second extent
in the EFI. It's most definitely not a contiguous range of 1920
blocks now that repair has made sure the inode has no extents and
the range is correctly accounted for, so that indicates that the
length in the EFI is suspect. Maybe it should only be 7 blocks?
Did you run the filesystem out of space again before this happened?
If you don't hit enospc, does removing files trigger this