xfs
[Top] [All Lists]

XFS kernal crash

To: xfs@xxxxxxxxxxx
Subject: XFS kernal crash
From: Thomas Stagge <thomas.stagge@xxxxxxxxxxx>
Date: Tue, 23 Sep 2014 14:53:59 +0200
Delivered-to: xfs@xxxxxxxxxxx
User-agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0

Hello,

we have in the last few weeks always problems with the XFS file system.

XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)


After a certain time there is a kernel crash which looks as follows:
September 23 10:17:57 hostname kernel: [1282081.010814] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:17:59 hostname kernel: [1282082.913437] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:01 hostname kernel: [1282084.908582] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:03 hostname kernel: [1282086.691791] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:05 hostname kernel: [1282088.691319] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:07 hostname kernel: [1282090.690182] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:09 hostname kernel: [1282092.673355] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:11 hostname kernel: [1282094.596504] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:13 hostname kernel: [1282096.595971] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:15 hostname kernel: [1282098.595301] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:17 hostname kernel: [1282100.594232] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:19 hostname kernel: [1282102.433262] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:21 hostname kernel: [1282104.412432] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:23 hostname kernel: [1282106.411598] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:25 hostname kernel: [1282108.310740] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:27 hostname kernel: [1282110.289892] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:29 hostname kernel: [1282112.289106] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:31 hostname kernel: [1282114.288247] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:33 hostname kernel: [1282116.215953] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:35 hostname kernel: [1282118.198488] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:37 hostname kernel: [1282120.193686] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:39 hostname kernel: [1282122.288716] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:41 hostname kernel: [1282124.215970] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:43 hostname kernel: [1282126.307031] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:45 hostname kernel: [1282128.306190] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:47 hostname kernel: [1282130.405310] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:49 hostname kernel: [1282132.076611] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:51 hostname kernel: [1282134.027790] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:53 hostname kernel: [1282136.122911] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:55 hostname kernel: [1282138.218032] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:57 hostname kernel: [1282140.145365] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:59 hostname kernel: [1282142.140528] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:19:01 hostname kernel: [1282144.091708] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:19:01 hostname kernel: [1282144.223485] INFO: task kswapd0: 87 blocked for more than 120 seconds.
September 23 10:19:01 hostname kernel: [1282144.224156] Not tainted 3.13.0-34-generic # 60-Ubuntu ~ precise1
September 23 10:19:01 hostname kernel: [1282144.224837] "echo 0> / proc / sys / kernel / hung_task_timeout_secs" disables this message.
September 23 10:19:01 hostname kernel: [1282144.225597] kswapd0 D ffff88031662a020 0 87 2 0x00000000
September 23 10:19:01 hostname kernel: [1282144.225609] ffff88019500d8f8 0000000000000002 ffff880199d94440 ffff88019500dfd8
September 23 10:19:01 hostname kernel: [1282144.225621] 0000000000014440 0000000000014440 ffff880316cd2fe0 ffff880316f3afe0
September 23 10:19:01 hostname kernel: [1282144.225629] ffff88019500d970 ffff880314119128 ffff88031411912c 00000000ffffffff
September 23 10:19:01 hostname kernel: [1282144.225638] Call Trace:
September 23 10:19:01 hostname kernel: [1282144.225658] [<ffffffff8175b8b9>] schedule + 0x29 / 0x70
September 23 10:19:01 hostname kernel: [1282144.225669] [<ffffffff8175bbde>] schedule_preempt_disabled + 0xe / 0x10
September 23 10:19:01 hostname kernel: [1282144.225680] [<ffffffff8175da14>] __mutex_lock_slowpath + 0x114 / 0x1b0
September 23 10:19:01 hostname kernel: [1282144.225771] [<ffffffffa0218291>]? xfs_perag_get_tag + 0x41 / 0xf0 [xfs]
September 23 10:19:01 hostname kernel: [1282144.225781] [<ffffffff8175dad3>] mutex_lock + 0x23 / 0x37
September 23 10:19:01 hostname kernel: [1282144.225832] [<ffffffffa01c7540>] xfs_reclaim_inodes_ag + 0x2e0 / 0x3a0 [xfs]
September 23 10:19:01 hostname kernel: [1282144.225843] [<ffffffff8101d0b9>]? sched_clock + 0x9 / 0x10
September 23 10:19:01 hostname kernel: [1282144.225852] [<ffffffff81042c9b>]? native_smp_send_reschedule + 0x4b / 0x60
September 23 10:19:01 hostname kernel: [1282144.225862] [<ffffffff8109cf7d>]? ttwu_queue + 0x8D / 0xD0
September 23 10:19:01 hostname kernel: [1282144.225871] [<ffffffff8109f860>]? try_to_wake_up + 0x190 / 0x210
September 23 10:19:01 hostname kernel: [1282144.225879] [<ffffffff8109f947>]? wake_up_process + 0x27 / 0x50

uname -a
hostname 3.13.0-34-generic # 60-Ubuntu SMP Wed precise1 ~ August 13 15:55:33 UTC 2014 x86_64 x86_64 x86_64 GNU / Linux


xfs_repair -V
xfs_repair Version 3.1.7


7 CPUs  -> cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 16
model           : 4
model name      : Quad-Core AMD Opteron(tm) Processor 2376
stepping        : 2
microcode       : 0x1000086
cpu MHz         : 800.000
cache size      : 512 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt hw_pstate npt lbrv svm_lock nrip_save
bogomips        : 4600.18
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate


cat /proc/meminfo
MemTotal:       12304884 kB
MemFree:          151816 kB
Buffers:            1076 kB
Cached:         10328496 kB
SwapCached:            0 kB
Active:          2658232 kB
Inactive:        7766540 kB
Active(anon):      42644 kB
Inactive(anon):    53636 kB
Active(file):    2615588 kB
Inactive(file):  7712904 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       4878268 kB
SwapFree:        4878268 kB
Dirty:              1220 kB
Writeback:             0 kB
AnonPages:         95728 kB
Mapped:            32208 kB
Shmem:               352 kB
Slab:            1412904 kB
SReclaimable:    1188188 kB
SUnreclaim:       224716 kB
KernelStack:        3512 kB
PageTables:        28816 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    11030708 kB
Committed_AS:     949792 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      310312 kB
VmallocChunk:   34353030140 kB
HardwareCorrupted:     0 kB
AnonHugePages:     12288 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       75520 kB
DirectMap2M:     3069952 kB
DirectMap1G:     9437184 kB


 cat /proc/mounts
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=6108796k,nr_inodes=1527199,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=1230492k,mode=755 0 0
/dev/disk/by-uuid/d6821e50-e82b-448d-bc03-9c286ef59c8f / xfs rw,relatime,attr2,inode64,noquota 0 0
none /sys/fs/fuse/connections fusectl rw,relatime 0 0
none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
/dev/sdc1 /extern/data0 xfs rw,relatime,attr2,inode64,noquota 0 0
rpc_pipefs /run/rpc_pipefs rpc_pipefs rw,relatime 0 0
nfsd /proc/fs/nfsd nfsd rw,relatime 0 0


cat /proc/partitions
major minor  #blocks  name

  11        0    1048575 sr0
   8        0  245175336 sda
   8        1   97654784 sda1
   8        2    4882432 sda2
   8       16  245175336 sdb
   8       17   97654784 sdb1
   8       18    4882432 sdb2
   9        2   97589120 md2
   9        1    4878272 md1
   8       32 27345461248 sdc
   8       33 27345461214 sdc1


xfs_info /dev/md2
Metadaten =/dev/disk/by-uuid/d6821e50-e82b-448d-bc03-9c286ef59c8f isize=256    agcount=4, agsize=6099320 blks
          =                       sectsz=512   attr=2
Daten     =                       bsize=4096   Blöcke=24397280, imaxpct=25
          =                       sunit=0      swidth=0 blks
Benennung =Version 2              bsize=4096   ascii-ci=0
Protokoll =Intern                 bsize=4096   Blöcke=11912, Version=2
          =                       sectsz=512   sunit=0 blks, lazy-count=1
Echtzeit  =keine                  extsz=4096   Blöcke=0, rtextents=0


 xfs_info /dev/md2
Metadaten =/dev/disk/by-uuid/d6821e50-e82b-448d-bc03-9c286ef59c8f isize=256    agcount=4, agsize=6099320 blks
          =                       sectsz=512   attr=2
Daten     =                       bsize=4096   Blöcke=24397280, imaxpct=25
          =                       sunit=0      swidth=0 blks
Benennung =Version 2              bsize=4096   ascii-ci=0
Protokoll =Intern                 bsize=4096   Blöcke=11912, Version=2
          =                       sectsz=512   sunit=0 blks, lazy-count=1
Echtzeit  =keine                  extsz=4096   Blöcke=0, rtextents=0


xfs_info /dev/sdc1
Metadaten =/dev/sdc1              isize=256    agcount=26, agsize=268435455 blks
          =                       sectsz=512   attr=2
Daten     =                       bsize=4096   Blöcke=6836365303, imaxpct=5
          =                       sunit=0      swidth=0 blks
Benennung =Version 2              bsize=4096   ascii-ci=0
Protokoll =Intern                 bsize=4096   Blöcke=32768, Version=2
          =                       sectsz=512   sunit=0 blks, lazy-count=0
Echtzeit  =keine                  extsz=4096   Blöcke=0, rtextents=0


Can you help us?


Best regards,

-- 
Mit freundlichen Grüssen,

Thomas Stagge
--------------------------------------

Thomas Stagge, Dipl.Ing.(FH)
System Administrator

OPTIMARE Systems GmbH
Am Luneort 15a
27572 Bremerhaven, Germany

T: +49 471 48361-27
F: +49 471 48361-11
M: +49 172 8698575

 

thomas.stagge@xxxxxxxxxxx
Internet: http://www.optimare.de

CEO: Dipl.-Ing. Hans J. Stahl; Sitz der Gesellschaft: Bremerhaven; Registergericht: Amtsgericht Bremen HRB 28618 HB

<Prev in Thread] Current Thread [Next in Thread>
  • XFS kernal crash, Thomas Stagge <=