Enabling quota on XFS filesystem with many files hangs

Sun Oct 28 07:20:50 CDT 2012

On Fri, 26 Oct 2012 14:01:48 -0500
Eric Sandeen <sandeen at sandeen.net> wrote:

> On 10/26/12 12:15 PM, Milan Holzäpfel wrote:
> > Hello all, 
> > 
> > I have an XFS filesystem of size 1.2 TiB with 101 GiB free space and 14
> > million inodes in use. 

Meanwhile, I deleted 200 GiB of data on that filesystem, with 9.9
million inodes still in use. Now, quotacheck just works. 

> > With 3.5.7 and 3.6.3, the OOM does not occur. 

Correction: I couldn't boot 3.6.3 because of a regression [1]. I don't
know whether the problem exists with 3.6.3. 

> > In dmesg, I find
> > 
> > INFO: task mount:8806 blocked for more than 120 seconds.
> 
> And then what?  Probably a backtrace, right?

Yes, of course. Sorry. Here it is:

Oct 24 15:23:39 bombax kernel: [  221.122875] XFS (dm-3): Mounting Filesystem
Oct 24 15:23:39 bombax kernel: [  221.431585] XFS (dm-3): Ending clean mount
Oct 24 15:23:39 bombax kernel: [  221.445026] XFS (dm-3): Quotacheck needed: Please wait.
Oct 24 15:28:01 bombax kernel: [  482.960045] INFO: task mount:8806 blocked for more than 120 seconds.
Oct 24 15:28:01 bombax kernel: [  482.966422] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 24 15:28:01 bombax kernel: [  482.974281] mount           D ffffffff8180cba0     0  8806   8703 0x00000000
Oct 24 15:28:01 bombax kernel: [  482.974286]  ffff880036be38a8 0000000000000086 ffff880036be3878 ffffffffa042b4e9
Oct 24 15:28:01 bombax kernel: [  482.974290]  ffff880036be3fd8 ffff880036be3fd8 ffff880036be3fd8 0000000000013980
Oct 24 15:28:01 bombax kernel: [  482.974293]  ffffffff81c13440 ffff88007908dc00 ffff880036be3898 7fffffffffffffff
Oct 24 15:28:01 bombax kernel: [  482.974297] Call Trace:
Oct 24 15:28:01 bombax kernel: [  482.974338]  [<ffffffffa042b4e9>] ? xfs_buf_iowait+0xa9/0x100 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974344]  [<ffffffff81698f59>] schedule+0x29/0x70
Oct 24 15:28:01 bombax kernel: [  482.974347]  [<ffffffff81697675>] schedule_timeout+0x2a5/0x320
Oct 24 15:28:01 bombax kernel: [  482.974373]  [<ffffffffa0486c75>] ? xfs_trans_read_buf+0x265/0x480 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974395]  [<ffffffffa0459ae7>] ? xfs_btree_check_sblock+0xc7/0x130 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974398]  [<ffffffff81698daf>] wait_for_common+0xdf/0x180
Oct 24 15:28:01 bombax kernel: [  482.974403]  [<ffffffff8108a280>] ? try_to_wake_up+0x200/0x200
Oct 24 15:28:01 bombax kernel: [  482.974406]  [<ffffffff81698f2d>] wait_for_completion+0x1d/0x20
Oct 24 15:28:01 bombax kernel: [  482.974430]  [<ffffffffa048c8a4>] xfs_qm_flush_one+0x74/0xb0 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974455]  [<ffffffffa048c830>] ? xfs_qm_dqattach_grouphint+0x90/0x90 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974479]  [<ffffffffa048c3ae>] xfs_qm_dquot_walk.isra.5+0xde/0x160 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974505]  [<ffffffffa048e21c>] xfs_qm_quotacheck+0x2bc/0x2e0 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974529]  [<ffffffffa048e3f4>] xfs_qm_mount_quotas+0x124/0x1b0 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974554]  [<ffffffffa047b8e5>] xfs_mountfs+0x615/0x6b0 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974573]  [<ffffffffa043af7d>] xfs_fs_fill_super+0x21d/0x2b0 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974577]  [<ffffffff81189996>] mount_bdev+0x1c6/0x210
Oct 24 15:28:01 bombax kernel: [  482.974597]  [<ffffffffa043ad60>] ? xfs_parseargs+0xb80/0xb80 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974616]  [<ffffffffa0439025>] xfs_fs_mount+0x15/0x20 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974620]  [<ffffffff8118a7d3>] mount_fs+0x43/0x1b0
Oct 24 15:28:01 bombax kernel: [  482.974624]  [<ffffffff811a4ab6>] vfs_kern_mount+0x76/0x120
Oct 24 15:28:01 bombax kernel: [  482.974628]  [<ffffffff811a5424>] do_kern_mount+0x54/0x110
Oct 24 15:28:01 bombax kernel: [  482.974631]  [<ffffffff811a7114>] do_mount+0x1a4/0x260
Oct 24 15:28:01 bombax kernel: [  482.974634]  [<ffffffff811a75f0>] sys_mount+0x90/0xe0
Oct 24 15:28:01 bombax kernel: [  482.974637]  [<ffffffff816a26e9>] system_call_fastpath+0x16/0x1b

> sysrq-w to get hung tasks or sysrq-t to get all task traces might
> help.
> 
> The sysrqs are one of the things suggested in:
> http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

I prepared all the information mentioned here, including the trace-cmd
report, but then I noticed that the problem had disappeared (see
above). 

If you are really interested in the information, I could try to move
the 200 GiB back on the filesystem and see whether the problem
reappears. 

Here is some more information on the system:

Linux bombax 3.5.7-030507-generic #201210130556 SMP Sat Oct 13 09:57:36 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
xfs_repair version 3.1.7
2 CPUs

Storage layers are:
mdadm RAID-5 256 KiB chunk size on sd[abcd]8
Block-device encryption with cryptsetup-luks 
XFS file system with the quotacheck problem

(no LVM below the XFS file system. / is ext4 on LVM on mdadm RAID-1)

disks: 4x SATA, 3.0 Gbps, NCQ enabled
hdparm -W says: "write-caching =  1 (on)" on all drives
no battery-backed write cache

mount options: logbsize=256k
xfs_info: 

meta-data=/dev/mapper/r5a-decrypt isize=256    agcount=32, agsize=9827264 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=314472448, imaxpct=5
         =                       sunit=64     swidth=192 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=153600, version=2
         =                       sectsz=512   sunit=64 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Regards,
Milan Holzäpfel

[1]: https://lkml.org/lkml/2012/10/11/155

-- 
Milan Holzäpfel <listen at mjh.name>