Hi,
I've hit an XFS internal error then filesystem shutdown with 4.8-rc3
kernel but not with 4.8-rc2
[ 8841.923617] XFS (sda6): Internal error xfs_trans_cancel at line 984 of file
fs/xfs/xfs_trans.c. Caller xfs_iomap_write_allocate+0x2d7/0x380 [xfs]
[ 8841.938286] CPU: 3 PID: 56 Comm: kswapd0 Not tainted 4.8.0-rc3 #1
[ 8841.945073] Hardware name: IBM IBM System x3550 M4 Server
-[7914I21]-/00J6242, BIOS -[D7E120CUS-1.20]- 08/23/2012
[ 8841.956526] 0000000000000286 00000000c8d39410 ffff88046890b7a8
ffffffff8135c53c
[ 8841.964818] ffff8804144e4cb0 0000000000000001 ffff88046890b7c0
ffffffffa02d99cb
[ 8841.973116] ffffffffa02e5537 ffff88046890b7e8 ffffffffa02f53e6
ffff8801ad37e580
[ 8841.981402] Call Trace:
[ 8841.984134] [<ffffffff8135c53c>] dump_stack+0x63/0x87
[ 8841.989900] [<ffffffffa02d99cb>] xfs_error_report+0x3b/0x40 [xfs]
[ 8841.996813] [<ffffffffa02e5537>] ? xfs_iomap_write_allocate+0x2d7/0x380
[xfs]
[ 8842.004891] [<ffffffffa02f53e6>] xfs_trans_cancel+0xb6/0xe0 [xfs]
[ 8842.011803] [<ffffffffa02e5537>] xfs_iomap_write_allocate+0x2d7/0x380 [xfs]
[ 8842.019684] [<ffffffffa02cf949>] xfs_map_blocks+0x1a9/0x220 [xfs]
[ 8842.026593] [<ffffffffa02d0c5b>] xfs_do_writepage+0x16b/0x560 [xfs]
[ 8842.033695] [<ffffffffa02d108b>] xfs_vm_writepage+0x3b/0x70 [xfs]
[ 8842.040584] [<ffffffff811b00dd>] pageout.isra.41+0x18d/0x2d0
[ 8842.046993] [<ffffffff811b1f3a>] shrink_page_list+0x78a/0x9b0
[ 8842.053501] [<ffffffff811b293d>] shrink_inactive_list+0x21d/0x570
[ 8842.060396] [<ffffffff811b350e>] shrink_node_memcg+0x51e/0x7d0
[ 8842.067000] [<ffffffff810a3900>] ? workqueue_congested+0x70/0x90
[ 8842.073799] [<ffffffff810a4862>] ? __queue_work+0x142/0x420
[ 8842.080112] [<ffffffff810a4862>] ? __queue_work+0x142/0x420
[ 8842.086425] [<ffffffff811b38a1>] shrink_node+0xe1/0x310
[ 8842.092351] [<ffffffff811b48d1>] kswapd+0x301/0x6f0
[ 8842.097889] [<ffffffff811b45d0>] ? mem_cgroup_shrink_node+0x180/0x180
[ 8842.105172] [<ffffffff810aca28>] kthread+0xd8/0xf0
[ 8842.110614] [<ffffffff816f8dbf>] ret_from_fork+0x1f/0x40
[ 8842.116636] [<ffffffff810ac950>] ? kthread_park+0x60/0x60
[ 8842.122784] XFS (sda6): xfs_do_force_shutdown(0x8) called from line 985 of
file fs/xfs/xfs_trans.c. Return address = 0xffffffffa02f53ff
[ 8842.522306] XFS (sda6): Corruption of in-memory data detected. Shutting
down filesystem
[ 8842.531358] XFS (sda6): Please umount the filesystem and rectify the
problem(s)
[ 8842.540470] Buffer I/O error on dev sda6, logical block 56162821, lost async
page write
[ 8842.549431] audit: netlink_unicast sending to audit_pid=1123 returned error:
-111
[ 8842.549433] audit: audit_lost=1 audit_rate_limit=0 audit_backlog_limit=64
[ 8842.549434] audit: audit_pid=1123 reset
[ 8842.552890] audit: type=1701 audit(1472234261.632:184): auid=4294967295
uid=0 gid=0 ses=4294967295 subj=system_u:system_r:syslogd_t:s0 pid=2064
comm="in:imjournal" exe="/usr/sbin/rsyslogd" sig=7
[ 8842.552909] XFS (sda6): xfs_do_force_shutdown(0x1) called from line 203 of
file fs/xfs/libxfs/xfs_defer.c. Return address = 0xffffffffa02b5459
[ 8842.554230] audit: type=1701 audit(1472234261.633:185): auid=4294967295
uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=1
comm="systemd" exe="/usr/lib/systemd/systemd" sig=11
[ 8842.555324] Kernel panic - not syncing: Attempted to kill init!
exitcode=0x0000000b
So it's likely a regression introduced in 4.8-rc3, and my bisect test
pointed to commit 0af32fb468b4 ("xfs: fix bogus space reservation in
xfs_iomap_write_allocate").
The test I ran is "bash-shared-mapping", it's available in
autotest(bash-shared-mapping.c from ext3-tools.tar.gz).
https://github.com/autotest/autotest-client-tests/raw/master/bash_shared_mapping/ext3-tools.tar.gz
You may have to do some modifications to make it compile. I attached an
updated version of bash-shared-mapping.c, you can downloand and compile
it directly.
I attached a script too to reproduce it. Please note that the XFS
partition needs about 40G frees space, and it may take hours to finish
based on your memory setup on your host.
I reproduced it on multiple hosts e.g. host with 64G memory & 16 cpus
and host with 16G memory & 16 cpus, but I haven't seen it on my test vm
which has 8G memory & 4 vcpus.
Detailed information of the host with 64G memory is:
[root@hp-dl360g9-15 ~]# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 1
Core(s) per socket: 8
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Model name: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
Stepping: 2
CPU MHz: 2400.000
BogoMIPS: 4802.86
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 20480K
NUMA node0 CPU(s): 0-3,8-11
NUMA node1 CPU(s): 4-7,12-15
[root@hp-dl360g9-15 ~]# free -m
total used free shared buff/cache available
Mem: 64305 38757 224 5 25323 61944
Swap: 16379 65 16314
[root@hp-dl360g9-15 ~]# xfs_info /
meta-data=/dev/mapper/systemvg-root isize=256 agcount=16, agsize=2927744 blks
= sectsz=512 attr=2, projid32bit=1
= crc=0 finobt=0 spinodes=0
data = bsize=4096 blocks=46843904, imaxpct=25
= sunit=64 swidth=192 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
log =internal bsize=4096 blocks=22912, version=2
= sectsz=512 sunit=64 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@hp-dl360g9-15 ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync
Convert
home systemvg -wi-ao---- 2.54t
root systemvg -wi-ao---- 178.70g
swap systemvg -wi-ao---- 16.00g
If more information is needed please let me know.
Thanks,
Eryu
bashmemory.sh
Description: Bourne shell script
console.log
Description: Text document
bash-shared-mapping.c
Description: Text document
|