xfs
[Top] [All Lists]

df bigger than ls?

To: xfs@xxxxxxxxxxx
Subject: df bigger than ls?
From: Brian Candler <B.Candler@xxxxxxxxx>
Date: Wed, 7 Mar 2012 15:54:39 +0000
Dkim-signature: v=1; a=rsa-sha1; c=relaxed; d=pobox.com; h=date:from:to :subject:message-id:mime-version:content-type; s=sasl; bh=zpvuBf ALmAqw2XO6oyPxw3JPpoQ=; b=Zv96UR2L/C5h3QalYlvCyvvFGctlzrwObE+Sgz xT0emQvkAIBr9uH/RYfga6wbGWaFYAWkXYaamxpfjA03iqxiRL4y/kw2DOk2XUHd B25irOWccdd3Z+ExvYMJY6EjXstAWgto/NoczsSHSjyZat9RIgoaqammE9hsc6oi gqGWI=
Domainkey-signature: a=rsa-sha1; c=nofws; d=pobox.com; h=date:from:to :subject:message-id:mime-version:content-type; q=dns; s=sasl; b= VrFZPVkAZyUEDgK2LmzxXzoSoVQR2W9HwYJKv/D6pImoiPgtYTfCtTy5z4ZK9Vrk wu3rN98scLg1sCkp5ZQZpKVTSXsDwz1mI/AzaYZP19/yGAkQkbxFBgB9p7cYg5Vh Gl+4W9fe5NJhTKhj3hEvFxBqQpzU2/y6zwGvwdq9Bvo=
User-agent: Mutt/1.5.21 (2010-09-15)
I have created some files spread across 12 XFS filesystems, using a
glusterfs "striped" volume.  This writes files with lots of holes(*) - so I
would expect the space reported by 'du' to be less than the space reported
by 'ls'.

However it's the other way round - du is reporting more space used than ls! 
Here's what I mean:  I'm looking at the files directly on the underlying
disk mount point, not via glusterfs at all.

$ ls -lh /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
-rw-rw-r-- 1 brian brian 1.1G 2012-03-07 15:19 
/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
$ ls -lk /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
-rw-rw-r-- 1 brian brian 1059968 2012-03-07 15:19 
/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
$ du -h /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G    /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
$ du -k /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2096392 /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff

And here's what xfs_bmap reports:

root@storage1:~# xfs_bmap -vp /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff:
 EXT: FILE-OFFSET         BLOCK-RANGE            AG AG-OFFSET              
TOTAL FLAGS
   0: [0..255]:           2933325744..2933325999  2 (3059152..3059407)       
256 00000
   1: [256..3071]:        hole                                              2816
   2: [3072..3327]:       2933326000..2933326255  2 (3059408..3059663)       
256 00000
   3: [3328..6143]:       hole                                              2816
   4: [6144..8191]:       2933326472..2933328519  2 (3059880..3061927)      
2048 00000
   5: [8192..9215]:       hole                                              1024
   6: [9216..13311]:      2933369480..2933373575  2 (3102888..3106983)      
4096 00000
   7: [13312..15359]:     hole                                              2048
   8: [15360..23551]:     2933375624..2933383815  2 (3109032..3117223)      
8192 00000
   9: [23552..24575]:     hole                                              1024
  10: [24576..40959]:     2933587168..2933603551  2 (3320576..3336959)     
16384 00000
  11: [40960..43007]:     hole                                              2048
  12: [43008..75775]:     2933623008..2933655775  2 (3356416..3389183)     
32768 00000
  13: [75776..76799]:     hole                                              1024
  14: [76800..142335]:    2933656800..2933722335  2 (3390208..3455743)     
65536 00000
  15: [142336..144383]:   hole                                              2048
  16: [144384..275455]:   2933724384..2933855455  2 (3457792..3588863)    
131072 00000
  17: [275456..276479]:   hole                                              1024
  18: [276480..538623]:   2935019808..2935281951  2 (4753216..5015359)    
262144 00000
  19: [538624..540671]:   hole                                              2048
  20: [540672..1064959]:  2935284000..2935808287  2 (5017408..5541695)    
524288 00000
  21: [1064960..1065983]: hole                                              1024
  22: [1065984..2114559]: 2935809312..2936857887  2 (5542720..6591295)   
1048576 00000
  23: [2114560..2116607]: hole                                              2048
  24: [2116608..2119935]: 2943037984..2943041311  2 (12771392..12774719)    
3328 00000
root@storage1:~# 

Given that these values are all in 512-byte disk blocks, the total file size
is (2119935 + 1) * 512 which agrees with ls. And some proportion of it
is holes, so du should report less than this, shouldn't it?

(Aside: it starts off being 11/12th holes as expected, but after a while
isn't.  This may be a different bug, possibly in glusterfs itself)

I guess 'du' gets its info from stat(), and stat() also says the file is
using 4192784 blocks which is 2096392 KB:

$ stat /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
  File: `/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff'
  Size: 1085407232      Blocks: 4192784    IO Block: 4096   regular file
Device: 810h/2064d      Inode: 8595657903  Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/   brian)   Gid: ( 1000/   brian)
Access: 2012-03-07 15:20:36.044365215 +0000
Modify: 2012-03-07 15:19:33.640364277 +0000
Change: 2012-03-07 15:19:33.640364277 +0000

Finally, here is xfs_db dump of the inode:

# ls -i /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
8595657903 /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
...
xfs_db> inode 8595657903
xfs_db> p
core.magic = 0x494e
core.mode = 0100664
core.version = 2
core.format = 3 (btree)
core.nlinkv2 = 1
core.onlink = 0
core.projid_lo = 0
core.projid_hi = 0
core.uid = 1000
core.gid = 1000
core.flushiter = 231
core.atime.sec = Wed Mar  7 15:20:36 2012
core.atime.nsec = 044365215
core.mtime.sec = Wed Mar  7 15:19:33 2012
core.mtime.nsec = 640364277
core.ctime.sec = Wed Mar  7 15:19:33 2012
core.ctime.nsec = 640364277
core.size = 1085407232
core.nblocks = 262370
core.extsize = 0
core.nextents = 13
core.naextents = 1
core.forkoff = 15
core.aformat = 2 (extents)
core.dmevmask = 0
core.dmstate = 0
core.newrtbm = 0
core.prealloc = 0
core.realtime = 0
core.immutable = 0
core.append = 0
core.sync = 0
core.noatime = 0
core.nodump = 0
core.rtinherit = 0
core.projinherit = 0
core.nosymlinks = 0
core.extsz = 0
core.extszinherit = 0
core.nodefrag = 0
core.filestream = 0
core.gen = 1875303423
next_unlinked = null
u.bmbt.level = 1
u.bmbt.numrecs = 1
u.bmbt.keys[1] = [startoff] 1:[0]
u.bmbt.ptrs[1] = 1:537294687
a.bmx[0] = [startoff,startblock,blockcount,extentflag] 0:[0,537253238,1,0]
xfs_db> 

Platform: ubuntu 11.10
Linux 3.0.0-15-server #26-Ubuntu SMP Fri Jan 20 19:07:39 UTC 2012 x86_64 x86_64 
x86_64 GNU/Linux

Have I missed something obvious, or is there a bug of some sort here?

Regards,

Brian.

(*) 
http://gluster.org/community/documentation//index.php/GlusterFS_Technical_FAQ#Stripe_behavior_not_working_as_expected

<Prev in Thread] Current Thread [Next in Thread>