xf kernel trace , what is the cause ?

[Gregory Machin <gdm@xxxxxxxxxxxxxx>]

Hi.

Â

I have a file server that has had 2 xfs Kernel trace dumps about 2 weeks apart.ÂÂ

Â

The RAID 60 data store is mounted as /home with xfs as the file system. The /home space isÂshares viaÂNFS exportÂandÂCIFS share. At the time of theÂTrace dumpsÂNFS access hangs and crashed users sessions on the client side.ÂÂTheÂoutputÂof the last traces is below.Â

Â

Can you advise as to the causeÂof this dump and possible resolution ?ÂÂÂ

Â

System information:

CentOS release 6.5 (Final)

Â

Physical

24 GB Ram

Intel(R) Xeon(R) CPUÂÂX5560Â @ 2.80GHz

2 x RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 1078 (rev 04)

RAID configured in RAID 6 with software RAID 0 providing final RAID 60

Â

Kernel 2.6.32-431.5.1.el6.x86_64
xfsdump-3.0.4-3.el6.x86_64
xfsprogs-3.1.1-14.el6.x86_64

nfs-utils-1.2.3-39.el6.x86_64
nfs4-acl-tools-0.3.3-6.el6.x86_64
nfs-utils-lib-1.1.5-6.el6.x86_64
samba-common-3.6.9-168.el6_5.x86_64
samba4-libs-4.0.0-61.el6_5.rc4.x86_64
samba-3.6.9-168.el6_5.x86_64
samba-winbind-clients-3.6.9-168.el6_5.x86_64
samba-client-3.6.9-168.el6_5.x86_64
samba-winbind-3.6.9-168.el6_5.x86_64

Â

[root@nzhmlfpr010 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sdc2ÂÂÂÂÂÂÂÂÂÂÂÂÂ 97GÂ 3.2GÂÂ 89GÂÂ 4% /
tmpfsÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 12GÂÂÂÂ 0ÂÂ 12GÂÂ 0% /dev/shm
/dev/sdc1ÂÂÂÂÂÂÂÂÂÂÂÂ 504MÂ 100MÂ 379MÂ 21% /boot
/dev/md0ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 11TÂ 1.7TÂ 9.3TÂ 15% /home

Â

[root@nzhmlfpr010 ~]# cat /proc/mdstat
Personalities : [raid0]
md0 : active raid0 sdb1[1] sda1[0]
ÂÂÂÂÂ 11707018240 blocks super 1.1 512k chunks

unused devices: <none>

Â

Kernel traces:

Â

May 2 17:49:39 nzhmlfpr010 kernel: INFO: task nfsd:2529 blocked for more than 120 seconds.
May 2 17:49:39 nzhmlfpr010 kernel: Not tainted 2.6.32-431.5.1.el6.x86_64 #1
May 2 17:49:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 2 17:49:39 nzhmlfpr010 kernel: nfsd D 0000000000000008 0 2529 2 0x00000080
May 2 17:49:39 nzhmlfpr010 kernel: ffff8806384c95e0 0000000000000046 0000000000001000 0000000000000000
May 2 17:49:39 nzhmlfpr010 kernel: 000000000026b000 000000000026c000 0000000000000100 ffffffffffffffff
May 2 17:49:39 nzhmlfpr010 kernel: ffff880623df3ab8 ffff8806384c9fd8 000000000000fbc8 ffff880623df3ab8
May 2 17:49:39 nzhmlfpr010 kernel: Call Trace:
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81086036>] ? mod_timer_pending+0x126/0x200
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa03cea95>] ? __nf_ct_refresh_acct+0xe5/0x100 [nf_conntrack]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027ac87>] xfs_trans_read_buf+0x1f7/0x410 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa025d164>] xfs_read_agi+0x74/0x100 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa025d21e>] xfs_ialloc_read_agi+0x2e/0x90 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa025d503>] xfs_ialloc_ag_select+0x133/0x270 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa025ebc7>] xfs_dialloc+0x3d7/0x850 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8144b4d4>] ? release_sock+0xe4/0xf0
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff814a4aa1>] ? tcp_recvmsg+0x821/0xe80
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0262980>] xfs_ialloc+0x60/0x6e0 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028006a>] ? kmem_zone_zalloc+0x3a/0x50 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027b814>] xfs_dir_ialloc+0x74/0x2b0 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027d8f0>] xfs_create+0x440/0x640 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028aa3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028ab80>] xfs_vn_create+0x10/0x20 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff811983b6>] vfs_create+0xe6/0x110
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa055f3d5>] nfsd_create_v3+0x3d5/0x530 [nfsd]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0566c13>] nfsd3_proc_create+0x123/0x1b0 [nfsd]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
May 2 17:49:39 nzhmlfpr010 kernel: INFO: task nfsd:2532 blocked for more than 120 seconds.
May 2 17:49:39 nzhmlfpr010 kernel: Not tainted 2.6.32-431.5.1.el6.x86_64 #1
May 2 17:49:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 2 17:49:39 nzhmlfpr010 kernel: nfsd D 0000000000000008 0 2532 2 0x00000080
May 2 17:49:39 nzhmlfpr010 kernel: ffff88063a073430 0000000000000046 0000000000000000 ffff88063a0735d8
May 2 17:49:39 nzhmlfpr010 kernel: 000000043a0735e4 ffff880638681000 ffff88015a9ed838 00ff88063a073580
May 2 17:49:39 nzhmlfpr010 kernel: ffff8806386fbaf8 ffff88063a073fd8 000000000000fbc8 ffff8806386fbaf8
May 2 17:49:39 nzhmlfpr010 kernel: Call Trace:
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027ac87>] xfs_trans_read_buf+0x1f7/0x410 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa022faa6>] xfs_read_agf+0x66/0x1a0 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa022fc19>] xfs_alloc_read_agf+0x39/0xd0 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0232642>] xfs_alloc_fix_freelist+0x432/0x4a0 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0278187>] ? xfs_trans_add_item+0x57/0x70 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0274938>] ? xfs_perag_get+0x38/0xb0 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0232bb6>] xfs_alloc_vextent+0x3d6/0x610 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa025e453>] xfs_ialloc_ag_alloc+0x143/0x4e0 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa025e8cf>] xfs_dialloc+0xdf/0x850 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8144b4d4>] ? release_sock+0xe4/0xf0
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff814a4aa1>] ? tcp_recvmsg+0x821/0xe80
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0262980>] xfs_ialloc+0x60/0x6e0 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028006a>] ? kmem_zone_zalloc+0x3a/0x50 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027b814>] xfs_dir_ialloc+0x74/0x2b0 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027d8f0>] xfs_create+0x440/0x640 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028aa3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028ab80>] xfs_vn_create+0x10/0x20 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff811983b6>] vfs_create+0xe6/0x110
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa055f3d5>] nfsd_create_v3+0x3d5/0x530 [nfsd]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0566c13>] nfsd3_proc_create+0x123/0x1b0 [nfsd]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
May 2 17:49:39 nzhmlfpr010 kernel: INFO: task nfsd:2533 blocked for more than 120 seconds.
May 2 17:49:39 nzhmlfpr010 kernel: Not tainted 2.6.32-431.5.1.el6.x86_64 #1
May 2 17:49:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 2 17:49:39 nzhmlfpr010 kernel: nfsd D 0000000000000008 0 2533 2 0x00000080
May 2 17:49:39 nzhmlfpr010 kernel: ffff880623f7d8d0 0000000000000046 0000000000000000 ffff880623f7d894
May 2 17:49:39 nzhmlfpr010 kernel: ffff880200000000 ffff88033fc25300 ffff880028276840 0000000000000400
May 2 17:49:39 nzhmlfpr010 kernel: ffff88063875bab8 ffff880623f7dfd8 000000000000fbc8 ffff88063875bab8
May 2 17:49:39 nzhmlfpr010 kernel: Call Trace:
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0245208>] ? xfs_bmbt_get_all+0x18/0x20 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa023b42d>] ? xfs_bmap_search_multi_extents+0xad/0x120 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027ac87>] xfs_trans_read_buf+0x1f7/0x410 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa025d164>] xfs_read_agi+0x74/0x100 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa026342b>] xfs_iunlink+0x4b/0x170 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff810792c7>] ? current_fs_time+0x27/0x30
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027b427>] ? xfs_trans_ichgtime+0x27/0xb0 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027b78b>] xfs_droplink+0x5b/0x70 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027d1ce>] xfs_remove+0x27e/0x390 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff811a3767>] ? __d_lookup+0xa7/0x150
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81196203>] ? generic_permission+0x23/0xb0
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028aca8>] xfs_vn_unlink+0x48/0x90 [xfs]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff811979a2>] vfs_unlink+0xa2/0xf0
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa055ef5f>] nfsd_unlink+0x1af/0x250 [nfsd]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa056369f>] ? nfsd_cache_lookup+0x59f/0x700 [nfsd]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa05665d3>] nfsd3_proc_remove+0x83/0x120 [nfsd]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
May 2 17:51:39 nzhmlfpr010 kernel: INFO: task nfsd:2527 blocked for more than 120 seconds.
May 2 17:51:39 nzhmlfpr010 kernel: Not tainted 2.6.32-431.5.1.el6.x86_64 #1
May 2 17:51:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 2 17:51:39 nzhmlfpr010 kernel: nfsd D 0000000000000000 0 2527 2 0x00000080
May 2 17:51:39 nzhmlfpr010 kernel: ffff880623ea1600 0000000000000046 0000000000000000 ffffffff8105afee
May 2 17:51:39 nzhmlfpr010 kernel: ffff880623ea1670 ffff880623ea15e0 ffff880623ea1748 0000000200001000
May 2 17:51:39 nzhmlfpr010 kernel: ffff880623e43098 ffff880623ea1fd8 000000000000fbc8 ffff880623e43098
May 2 17:51:39 nzhmlfpr010 kernel: Call Trace:
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8105afee>] ? find_busiest_group+0x90e/0x9f0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa03cea95>] ? __nf_ct_refresh_acct+0xe5/0x100 [nf_conntrack]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027ac27>] xfs_trans_read_buf+0x197/0x410 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d164>] xfs_read_agi+0x74/0x100 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d21e>] xfs_ialloc_read_agi+0x2e/0x90 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d8fe>] xfs_imap_lookup+0x3e/0x170 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8116ecf0>] ? cache_alloc_refill+0x1c0/0x240
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025dbfe>] xfs_imap+0x1ce/0x280 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa02650f0>] xfs_iread+0x40/0x1b0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff811a59de>] ? inode_init_always+0x11e/0x1c0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025fc8e>] xfs_iget+0x27e/0x6e0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff814b121f>] ? tcp_event_new_data_sent+0x4f/0x110
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa02860a6>] xfs_nfs_get_inode+0x46/0xe0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0444860>] ? cache_check+0x60/0x360 [sunrpc]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028621b>] xfs_fs_fh_to_dentry+0x8b/0x90 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa015271c>] exportfs_decode_fh+0x5c/0x2bc [exportfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055a800>] ? nfsd_acceptable+0x0/0x120 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0444860>] ? cache_check+0x60/0x360 [sunrpc]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0560712>] ? exp_find_key+0x62/0xb0 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810ec17e>] ? rcu_start_gp+0x1be/0x230
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055acfa>] fh_verify+0x32a/0x640 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa056a7c0>] nfsd4_putfh+0x50/0x60 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa056a458>] nfsd4_proc_compound+0x3d8/0x490 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
May 2 17:51:39 nzhmlfpr010 kernel: INFO: task nfsd:2528 blocked for more than 120 seconds.
May 2 17:51:39 nzhmlfpr010 kernel: Not tainted 2.6.32-431.5.1.el6.x86_64 #1
May 2 17:51:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 2 17:51:39 nzhmlfpr010 kernel: nfsd D 0000000000000008 0 2528 2 0x00000080
May 2 17:51:39 nzhmlfpr010 kernel: ffff880623ed75e0 0000000000000046 ffff88033745c0f0 0000000019bfcc00
May 2 17:51:39 nzhmlfpr010 kernel: ffff880623ed7690 ffffffffa03d4b5b 0007fffffffff07f ffffffff00000000
May 2 17:51:39 nzhmlfpr010 kernel: ffff880623d6daf8 ffff880623ed7fd8 000000000000fbc8 ffff880623d6daf8
May 2 17:51:39 nzhmlfpr010 kernel: Call Trace:
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa03d4b5b>] ? tcp_packet+0x68b/0x1750 [nf_conntrack]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81086036>] ? mod_timer_pending+0x126/0x200
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81297e28>] ? swiotlb_dma_mapping_error+0x18/0x30
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027ac87>] xfs_trans_read_buf+0x1f7/0x410 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d164>] xfs_read_agi+0x74/0x100 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d21e>] xfs_ialloc_read_agi+0x2e/0x90 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d503>] xfs_ialloc_ag_select+0x133/0x270 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025ebc7>] xfs_dialloc+0x3d7/0x850 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8116ec8b>] ? cache_alloc_refill+0x15b/0x240
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0262980>] xfs_ialloc+0x60/0x6e0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028006a>] ? kmem_zone_zalloc+0x3a/0x50 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027b814>] xfs_dir_ialloc+0x74/0x2b0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027d8f0>] xfs_create+0x440/0x640 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028aa3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028ab80>] xfs_vn_create+0x10/0x20 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff811983b6>] vfs_create+0xe6/0x110
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055f3d5>] nfsd_create_v3+0x3d5/0x530 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0566c13>] nfsd3_proc_create+0x123/0x1b0 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
May 2 17:51:39 nzhmlfpr010 kernel: INFO: task nfsd:2529 blocked for more than 120 seconds.
May 2 17:51:39 nzhmlfpr010 kernel: Not tainted 2.6.32-431.5.1.el6.x86_64 #1
May 2 17:51:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 2 17:51:39 nzhmlfpr010 kernel: nfsd D 0000000000000008 0 2529 2 0x00000080
May 2 17:51:39 nzhmlfpr010 kernel: ffff8806384c95e0 0000000000000046 0000000000001000 0000000000000000
May 2 17:51:39 nzhmlfpr010 kernel: 000000000026b000 000000000026c000 0000000000000100 ffffffffffffffff
May 2 17:51:39 nzhmlfpr010 kernel: ffff880623df3ab8 ffff8806384c9fd8 000000000000fbc8 ffff880623df3ab8
May 2 17:51:39 nzhmlfpr010 kernel: Call Trace:
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81086036>] ? mod_timer_pending+0x126/0x200
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa03cea95>] ? __nf_ct_refresh_acct+0xe5/0x100 [nf_conntrack]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027ac87>] xfs_trans_read_buf+0x1f7/0x410 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d164>] xfs_read_agi+0x74/0x100 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d21e>] xfs_ialloc_read_agi+0x2e/0x90 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d503>] xfs_ialloc_ag_select+0x133/0x270 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025ebc7>] xfs_dialloc+0x3d7/0x850 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8144b4d4>] ? release_sock+0xe4/0xf0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff814a4aa1>] ? tcp_recvmsg+0x821/0xe80
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0262980>] xfs_ialloc+0x60/0x6e0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028006a>] ? kmem_zone_zalloc+0x3a/0x50 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027b814>] xfs_dir_ialloc+0x74/0x2b0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027d8f0>] xfs_create+0x440/0x640 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028aa3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028ab80>] xfs_vn_create+0x10/0x20 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff811983b6>] vfs_create+0xe6/0x110
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055f3d5>] nfsd_create_v3+0x3d5/0x530 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0566c13>] nfsd3_proc_create+0x123/0x1b0 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
May 2 17:51:39 nzhmlfpr010 kernel: INFO: task nfsd:2530 blocked for more than 120 seconds.
May 2 17:51:39 nzhmlfpr010 kernel: Not tainted 2.6.32-431.5.1.el6.x86_64 #1
May 2 17:51:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 2 17:51:39 nzhmlfpr010 kernel: nfsd D 0000000000000000 0 2530 2 0x00000080
May 2 17:51:39 nzhmlfpr010 kernel: ffff880623fbd600 0000000000000046 0000000000000000 ffffffffa0260e6c
May 2 17:51:39 nzhmlfpr010 kernel: 0000000000800000 0002005a00000004 0000000000000003 0000000000000286
May 2 17:51:39 nzhmlfpr010 kernel: ffff880626601058 ffff880623fbdfd8 000000000000fbc8 ffff880626601058
May 2 17:51:39 nzhmlfpr010 kernel: Call Trace:
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0260e6c>] ? xfs_iext_bno_to_ext+0x8c/0x170 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027ac27>] xfs_trans_read_buf+0x197/0x410 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d164>] xfs_read_agi+0x74/0x100 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d21e>] xfs_ialloc_read_agi+0x2e/0x90 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d8fe>] xfs_imap_lookup+0x3e/0x170 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8149a3e8>] ? ip_finish_output+0x148/0x310
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025dbfe>] xfs_imap+0x1ce/0x280 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa02650f0>] xfs_iread+0x40/0x1b0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff811a59de>] ? inode_init_always+0x11e/0x1c0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025fc8e>] xfs_iget+0x27e/0x6e0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff814b121f>] ? tcp_event_new_data_sent+0x4f/0x110
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa02860a6>] xfs_nfs_get_inode+0x46/0xe0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0444860>] ? cache_check+0x60/0x360 [sunrpc]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028621b>] xfs_fs_fh_to_dentry+0x8b/0x90 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa015271c>] exportfs_decode_fh+0x5c/0x2bc [exportfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055a800>] ? nfsd_acceptable+0x0/0x120 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0444860>] ? cache_check+0x60/0x360 [sunrpc]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0560712>] ? exp_find_key+0x62/0xb0 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81068a53>] ? dequeue_entity+0x113/0x2e0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055acfa>] fh_verify+0x32a/0x640 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa044392d>] ? cache_revisit_request+0x2d/0x120 [sunrpc]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa056a7c0>] nfsd4_putfh+0x50/0x60 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa056a458>] nfsd4_proc_compound+0x3d8/0x490 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
May 2 17:51:39 nzhmlfpr010 kernel: INFO: task nfsd:2531 blocked for more than 120 seconds.
May 2 17:51:39 nzhmlfpr010 kernel: Not tainted 2.6.32-431.5.1.el6.x86_64 #1
May 2 17:51:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 2 17:51:39 nzhmlfpr010 kernel: nfsd D 0000000000000008 0 2531 2 0x00000080
May 2 17:51:39 nzhmlfpr010 kernel: ffff880623131c80 0000000000000046 0000000000000000 00000000329ac000
May 2 17:51:39 nzhmlfpr010 kernel: ffff880338a05200 0000000000000001 ffff8803330c3d88 ffff8803330c3d88
May 2 17:51:39 nzhmlfpr010 kernel: ffff880623d91ab8 ffff880623131fd8 000000000000fbc8 ffff880623d91ab8
May 2 17:51:39 nzhmlfpr010 kernel: Call Trace:
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8152938e>] __mutex_lock_slowpath+0x13e/0x180
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8152922b>] mutex_lock+0x2b/0x50
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055f102>] nfsd_create_v3+0x102/0x530 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0566c13>] nfsd3_proc_create+0x123/0x1b0 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
May 2 17:51:39 nzhmlfpr010 kernel: INFO: task nfsd:2532 blocked for more than 120 seconds.
May 2 17:51:39 nzhmlfpr010 kernel: Not tainted 2.6.32-431.5.1.el6.x86_64 #1
May 2 17:51:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 2 17:51:39 nzhmlfpr010 kernel: nfsd D 0000000000000008 0 2532 2 0x00000080
May 2 17:51:39 nzhmlfpr010 kernel: ffff88063a073430 0000000000000046 0000000000000000 ffff88063a0735d8
May 2 17:51:39 nzhmlfpr010 kernel: 000000043a0735e4 ffff880638681000 ffff88015a9ed838 00ff88063a073580
May 2 17:51:39 nzhmlfpr010 kernel: ffff8806386fbaf8 ffff88063a073fd8 000000000000fbc8 ffff8806386fbaf8
May 2 17:51:39 nzhmlfpr010 kernel: Call Trace:
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027ac87>] xfs_trans_read_buf+0x1f7/0x410 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa022faa6>] xfs_read_agf+0x66/0x1a0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa022fc19>] xfs_alloc_read_agf+0x39/0xd0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0232642>] xfs_alloc_fix_freelist+0x432/0x4a0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0278187>] ? xfs_trans_add_item+0x57/0x70 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0274938>] ? xfs_perag_get+0x38/0xb0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0232bb6>] xfs_alloc_vextent+0x3d6/0x610 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025e453>] xfs_ialloc_ag_alloc+0x143/0x4e0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025e8cf>] xfs_dialloc+0xdf/0x850 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8144b4d4>] ? release_sock+0xe4/0xf0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff814a4aa1>] ? tcp_recvmsg+0x821/0xe80
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0262980>] xfs_ialloc+0x60/0x6e0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028006a>] ? kmem_zone_zalloc+0x3a/0x50 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027b814>] xfs_dir_ialloc+0x74/0x2b0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027d8f0>] xfs_create+0x440/0x640 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028aa3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028ab80>] xfs_vn_create+0x10/0x20 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff811983b6>] vfs_create+0xe6/0x110
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055f3d5>] nfsd_create_v3+0x3d5/0x530 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0566c13>] nfsd3_proc_create+0x123/0x1b0 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
May 2 17:51:39 nzhmlfpr010 kernel: INFO: task nfsd:2533 blocked for more than 120 seconds.
May 2 17:51:39 nzhmlfpr010 kernel: Not tainted 2.6.32-431.5.1.el6.x86_64 #1
May 2 17:51:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 2 17:51:39 nzhmlfpr010 kernel: nfsd D 0000000000000008 0 2533 2 0x00000080
May 2 17:51:39 nzhmlfpr010 kernel: ffff880623f7d8d0 0000000000000046 0000000000000000 ffff880623f7d894
May 2 17:51:39 nzhmlfpr010 kernel: ffff880200000000 ffff88033fc25300 ffff880028276840 0000000000000400
May 2 17:51:39 nzhmlfpr010 kernel: ffff88063875bab8 ffff880623f7dfd8 000000000000fbc8 ffff88063875bab8
May 2 17:51:39 nzhmlfpr010 kernel: Call Trace:
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0245208>] ? xfs_bmbt_get_all+0x18/0x20 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa023b42d>] ? xfs_bmap_search_multi_extents+0xad/0x120 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027ac87>] xfs_trans_read_buf+0x1f7/0x410 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d164>] xfs_read_agi+0x74/0x100 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa026342b>] xfs_iunlink+0x4b/0x170 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810792c7>] ? current_fs_time+0x27/0x30
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027b427>] ? xfs_trans_ichgtime+0x27/0xb0 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027b78b>] xfs_droplink+0x5b/0x70 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027d1ce>] xfs_remove+0x27e/0x390 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff811a3767>] ? __d_lookup+0xa7/0x150
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81196203>] ? generic_permission+0x23/0xb0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028aca8>] xfs_vn_unlink+0x48/0x90 [xfs]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff811979a2>] vfs_unlink+0xa2/0xf0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055ef5f>] nfsd_unlink+0x1af/0x250 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa056369f>] ? nfsd_cache_lookup+0x59f/0x700 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa05665d3>] nfsd3_proc_remove+0x83/0x120 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20

Â

Â

Thank you