<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content="text/html; charset=iso-8859-1" http-equiv=Content-Type>
<META name=GENERATOR content="MSHTML 8.00.6001.23588">
<STYLE></STYLE>
</HEAD>
<BODY bgColor=#ffffff>
<DIV><FONT size=2 face=Arial>
<DIV><FONT size=2 face=Arial>Hello xfs experts,</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>I am crawling in the dark from few days and I have
no idea how to fix the following problem. On a centos 7 system:</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial># uname -a<BR>Linux 1a 3.10.0-327.el7.x86_64 #1 SMP
Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>df is reporting 100% full of / and du is reporting
only 1.7G usage from 50GB available (less than 4%). I want to mention that / is
xfs. See below:</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial># df -a|grep ^/<BR>/dev/mapper/centos-root
52403200 52400396 2804 100%
/<BR>
^^^^^^^^^^
^^^^^^^^^^<BR>/dev/sda1
503040 131876 371164 27%
/boot<BR>/dev/mapper/centos-home 210529792 35204
210494588 1% /home</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>du is estimating just 1.7G usage of /<BR># du -sch
/* --exclude=home --exclude=boot<BR>0
/bin<BR>0
/dev<BR>25M
/etc<BR>0
/lib<BR>0 /lib64<BR>744K
/luarocks-2.3.0<BR>0
/media<BR>0 /mnt<BR>125M
/openresty-1.9.7.4<BR>0
/opt<BR>420K /root<BR>49M
/run<BR>0
/sbin<BR>0
/srv<BR>0
/sys<BR>0 /tmp<BR>1.3G
/usr<BR>227M /var<BR>1.7G
total<BR>[root@localhost ~]#</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>df is also reporting 80% of inode
usage:</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial># df
-i<BR>Filesystem
Inodes IUsed IFree IUse% Mounted
on<BR>/dev/mapper/centos-root 78160
66218 11942 85%
/<BR>
^^^^^^^^<BR>devtmpfs
8218272 519 8217753 1%
/dev<BR>tmpfs
8221010 1 8221009 1%
/dev/shm<BR>tmpfs
8221010 648 8220362 1%
/run<BR>tmpfs
8221010 13 8220997 1%
/sys/fs/cgroup<BR>/dev/sda1
509952 330 509622 1%
/boot<BR>/dev/mapper/centos-home 210632704 99
210632605 1%
/home<BR>tmpfs
8221010 1 8221009 1%
/run/user/0<BR>#</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>/ partition is created on top of a LVM having also
50GB size.</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial># lvdisplay /dev/centos/root<BR> --- Logical
volume ---<BR> LV
Path
/dev/centos/root<BR> LV
Name
root<BR> VG
Name
centos</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial> LV
Status
available<BR> #
open
1<BR> LV
Size
50.00 GiB<BR> Current
LE
12800<BR>
Segments
1<BR>
Allocation
inherit<BR> Read ahead sectors auto<BR> -
currently set to 256<BR> Block
device
253:0</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>I've already checked against rootkit without
finding anything wrong!</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>I have another system, identical with this one
which is healthy. The only difference I found between those systems is regarding
max number of inodes available on / (which has the same size, 50GB on booth
servers). On the second one (healthy), max number of inodes are ~52 milions and
not only just ~85.000 as are reported on "seek" server.</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial># df -i|grep ^/<BR>/dev/mapper/centos-root
52424704 66137 52358567 1%
/<BR>
^^^^^^^^^^^^^<BR>/dev/sda1
509952 330 509622 1%
/boot<BR>/dev/mapper/centos-home 210632704 26
210632678 1% /home<BR>[root@localhost ~]#</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>Suspected also large number of files on /. Counted
total number of files and or booth servers are the same: ~180K. So no difference
here.</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>Look to find also files larger than 100M and on
booth servers and found just 1 (104M size):</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>find / -type f -size +100000k -exec ls -lh {}
\;<BR>#<BR>/usr/lib/locale/locale-archive<BR>#</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>Looking to find files larger than 10M, I found just
~20 on booth servers.</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial># find / -type f -size +10000k -exec ls -lh {} \;
|wc -l<BR>16<BR>#</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>So for sure, there are NO files exhausting free
space.</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>On booth servers, number of used inodes are
identical: ~66K. Also xfs_info report is identical for booth. What is different
is number of AVAILABLE inodes: 85K (on seek node) vs 52 milion (on healthy
node)!!! How is possible that!!! Booth servers has the same size (50GB) for
/!</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>#lsof -nP |grep -i delete|wc -l<BR>0<BR>#find
/proc/*/fd -ls | grep -i dele|wc -l<BR>0</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>so lsof and find does not report anything wrong
(any file deleted and still open)!</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>reboot does not fix the problem, / remain 100%
full</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>After reboot, on 25th July:</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial># df -ah|grep
centos-root<BR>/dev/mapper/centos-root 50G 50G
4.0M 100% /<BR>#</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>Also max number of inodes = 67k:<BR># df
-i<BR>Filesystem
Inodes IUsed IFree IUse% Mounted
on<BR>/dev/mapper/centos-root 66960
66165 795 99%
/<BR>^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<BR>devtmpfs
8218272 519 8217753 1%
/dev<BR>tmpfs
8221010 1 8221009 1%
/dev/shm<BR>tmpfs
8221010 630 8220380 1%
/run<BR>tmpfs
8221010 13 8220997 1%
/sys/fs/cgroup<BR>/dev/sda1
509952 330 509622 1%
/boot<BR>/dev/mapper/centos-home 210632704 28
210632676 1%
/home<BR>tmpfs
8221010 1 8221009 1%
/run/user/0<BR>#</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>Lets try to run intentionally xfs_grow (which
normally should not produce any change)</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial># xfs_growfs
/dev/mapper/centos-root<BR>meta-data=/dev/mapper/centos-root
isize=256 agcount=16, agsize=819136
blks<BR>
=
sectsz=512 attr=2,
projid32bit=1<BR>
=
crc=0
finobt=0<BR>data
=
bsize=4096 blocks=13106176,
imaxpct=25<BR>
=
sunit=64 swidth=64 blks<BR>naming =version
2
bsize=4096 ascii-ci=0 ftype=0<BR>log
=internal
bsize=4096 blocks=6400,
version=2<BR>
=
sectsz=512 sunit=64 blks, lazy-count=1<BR>realtime
=none
extsz=4096 blocks=0, rtextents=0<BR>data blocks changed from
13106176 to
13107200<BR>^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<BR>#</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>Partition remain the same, 50GB
size:<BR>[root@nl-hvs-ov001a ~]# df -ah|grep
centos-root<BR>/dev/mapper/centos-root 50G 50G
4.0M 100% /</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>But number of inodes INCREASED with more tha
20%!!!<BR># df
-i<BR>Filesystem
Inodes IUsed IFree IUse% Mounted
on<BR>/dev/mapper/centos-root 83200
66165 17035 80%
/<BR>^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<BR>devtmpfs
8218272 519 8217753 1%
/dev<BR>tmpfs
8221010 1 8221009 1%
/dev/shm<BR>tmpfs
8221010 630 8220380 1%
/run<BR>tmpfs
8221010 13 8220997 1%
/sys/fs/cgroup<BR>/dev/sda1
509952 330 509622 1%
/boot<BR>/dev/mapper/centos-home 210632704 28
210632676 1%
/home<BR>tmpfs
8221010 1 8221009 1%
/run/user/0<BR>#</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>On 27July without changing anything there, max
number inodes available for / decreased to ~67k (the same size like 2 days ago,
before xfs_grow)!</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial># df
-i<BR>Filesystem
Inodes IUsed IFree IUse% Mounted
on<BR>/dev/mapper/centos-root 67024
66225 799 99%
/<BR>^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<BR>devtmpfs
8218272 519 8217753 1%
/dev<BR>tmpfs
8221010 1 8221009 1%
/dev/shm<BR>tmpfs
8221010 632 8220378 1%
/run<BR>tmpfs
8221010 13 8220997 1%
/sys/fs/cgroup<BR>/dev/mapper/centos-home 210632704 99
210632605 1%
/home<BR>/dev/sda1
509952 330 509622 1%
/boot<BR>tmpfs
8221010 1 8221009 1%
/run/user/0<BR>#</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>Please note that all that time, number of files
remain unchanged ~180K, the same for inodes used, the number remain constant
~66K. Just max number of inodes available decreased which is an abnormal
behavior.</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=2 face=Arial>How can be fixed? Looks like xfs is crrupted
or like a bug.</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>Thanks in advance for help.<BR>Alex</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV></FONT></DIV></BODY></HTML>