free space of root partition decreases unaccountably by some 1024 blocks on every umount+linux shutdown : additional informations

[Roland Eggner]

On Thursday August 13th 05:16:40 2009 Eric Sandeen wrote:
> Roland Eggner wrote:
> > On July 18th I noticed the first time this unaccountable decrease of free space of my root partition /dev/hda7:
> > For at least several boot-shutdown-cycles it has decreased on every cycle by some 1020 … 1030 blocks from originally above 100 MB to 96 MB.
> > Expected change at most ±1 block.  Neither xfs_check nor xfs_repair -dn could detect any flaws.
> Maybe I missed it in the email, but how have you ruled out the
> possibility that files are simply growing, thereby using the space?
Look at the last but one paragraph in my mail from August 12th
http://oss.sgi.com/archives/xfs/2009-08/msg00113.html
(obviously the find argument “-mount” got lost on copy+paste, sorry).



Facts summarized:
----------------

(a)  Free space decreases unaccountably even if I circumvent all shutdown scripts und shutdown by SysReq R S U B.  Note: no SIGTERM, no SIGKILL.

(b)  Unaccountable free space decrease is triggered EXACTLY, ALWAYS and EXCLUSIVELY by this “remount,ro”.  Note: NOT by any other write activities.

(c)  Binary comparison of images written with dd exhibited NO unexpected writes to me (I do not have deeper knowledge of xfs internals):  in a particular case free space difference reported by df has been 1026 blocks = 1050624 byte, whereas count of differing bytes reported by “cmp -b -l” has been only 135189.  “cmp -b -l” DID exhibit expected writes e.g. /etc/mtab.  (Eric Sandeen got details via private mail and offered kindly to have a look at xfs_metadump extraction from this images — thanks!).

(d)  Until now I could NOT detect this problem at any other partition, it seems that ONLY the root partition is affected.

(e)  I performed xfsdump | mkfs.xfs | xfsrestore and got back some 200 Mbyte free space, but only temporarily — at subsequent linux shutdowns free space CONTINUES to decrease, just starting from a new offset.

(f)  I booted a sidux image and reset the lazy-counter attribute by “xfs_admin -c 0” ➔ Apart from “XFS: correcting sb_features alignment problem” message at next mount, EXACTLY the same result as after measures (e).  (WARNING:  Never use “xfs_admin -c 0” unless you have a current backup of your valuable data!!)

(g)  Kernel 2.6.30.4 shows the problem too.  (And introduces some other flaws, “show stopping” for me — therefore I stay at kernel 2.6.29.6).

(h)  Apart from following single incident, I got never any error messages from this filesystem, neither from xfs_check nor from xfs_repair:
On July 17th a run of xfs_repair yielded following report — beeing busy on that day, I ignored the message apart from saving it for later analysis:
#  xfs_repair -dn /dev/hda7
bad nblocks 1 for free inode 9722
bad nlink 1 for free inode 9722
bad mode 0100644 for free inode 9722
link count mismatch for inode 9722 (name ?), nlink 0, counted 1
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.



If I can provide any additional debugging info, let me know.

-- 
Roland Eggner
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part.
URL: <http://oss.sgi.com/pipermail/xfs/attachments/20090902/fa48d4fb/attachment.sig>