<div dir="ltr"><div>Hi. </div><div> </div><div>I have a file server that has had 2 xfs Kernel trace dumps about 2 weeks apart. </div><div> </div><div>The RAID 60 data store is mounted as /home with xfs as the file system. The /home space is shares via NFS export and CIFS share. At the time of the Trace dumps NFS access hangs and crashed users sessions on the client side. The output of the last traces is below. </div>
<div> </div><div>Can you advise as to the cause of this dump and possible resolution ? </div><div> </div><div>System information: </div><div><br>CentOS release 6.5 (Final)</div><div> </div><div>Physical </div><div>24 GB Ram</div>
<div>Intel(R) Xeon(R) CPU X5560 @ 2.80GHz</div><div>2 x RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 1078 (rev 04) </div><div>RAID configured in RAID 6 with software RAID 0 providing final RAID 60</div><div>
</div><div>Kernel 2.6.32-431.5.1.el6.x86_64<br>xfsdump-3.0.4-3.el6.x86_64<br>xfsprogs-3.1.1-14.el6.x86_64</div><div>nfs-utils-1.2.3-39.el6.x86_64<br>nfs4-acl-tools-0.3.3-6.el6.x86_64<br>nfs-utils-lib-1.1.5-6.el6.x86_64<br>
samba-common-3.6.9-168.el6_5.x86_64<br>samba4-libs-4.0.0-61.el6_5.rc4.x86_64<br>samba-3.6.9-168.el6_5.x86_64<br>samba-winbind-clients-3.6.9-168.el6_5.x86_64<br>samba-client-3.6.9-168.el6_5.x86_64<br>samba-winbind-3.6.9-168.el6_5.x86_64<br>
</div><div> </div><div>[root@nzhmlfpr010 ~]# df -h<br>Filesystem Size Used Avail Use% Mounted on<br>/dev/sdc2 97G 3.2G 89G 4% /<br>tmpfs 12G 0 12G 0% /dev/shm<br>/dev/sdc1 504M 100M 379M 21% /boot<br>
/dev/md0 11T 1.7T 9.3T 15% /home</div><div> </div><div>[root@nzhmlfpr010 ~]# cat /proc/mdstat<br>Personalities : [raid0]<br>md0 : active raid0 sdb1[1] sda1[0]<br> 11707018240 blocks super 1.1 512k chunks</div>
<div>unused devices: <none><br></div><div> </div><div>Kernel traces:</div><div> </div><div>May 2 17:49:39 nzhmlfpr010 kernel: INFO: task nfsd:2529 blocked for more than 120 seconds.<br>May 2 17:49:39 nzhmlfpr010 kernel: Not tainted 2.6.32-431.5.1.el6.x86_64 #1<br>
May 2 17:49:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>May 2 17:49:39 nzhmlfpr010 kernel: nfsd D 0000000000000008 0 2529 2 0x00000080<br>
May 2 17:49:39 nzhmlfpr010 kernel: ffff8806384c95e0 0000000000000046 0000000000001000 0000000000000000<br>May 2 17:49:39 nzhmlfpr010 kernel: 000000000026b000 000000000026c000 0000000000000100 ffffffffffffffff<br>May 2 17:49:39 nzhmlfpr010 kernel: ffff880623df3ab8 ffff8806384c9fd8 000000000000fbc8 ffff880623df3ab8<br>
May 2 17:49:39 nzhmlfpr010 kernel: Call Trace:<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81086036>] ? mod_timer_pending+0x126/0x200<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa03cea95>] ? __nf_ct_refresh_acct+0xe5/0x100 [nf_conntrack]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027ac87>] xfs_trans_read_buf+0x1f7/0x410 [xfs]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa025d164>] xfs_read_agi+0x74/0x100 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa025d21e>] xfs_ialloc_read_agi+0x2e/0x90 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa025d503>] xfs_ialloc_ag_select+0x133/0x270 [xfs]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa025ebc7>] xfs_dialloc+0x3d7/0x850 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8144b4d4>] ? release_sock+0xe4/0xf0<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff814a4aa1>] ? tcp_recvmsg+0x821/0xe80<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0262980>] xfs_ialloc+0x60/0x6e0 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028006a>] ? kmem_zone_zalloc+0x3a/0x50 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027b814>] xfs_dir_ialloc+0x74/0x2b0 [xfs]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027d8f0>] xfs_create+0x440/0x640 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028aa3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028ab80>] xfs_vn_create+0x10/0x20 [xfs]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff811983b6>] vfs_create+0xe6/0x110<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa055f3d5>] nfsd_create_v3+0x3d5/0x530 [nfsd]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0566c13>] nfsd3_proc_create+0x123/0x1b0 [nfsd]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20<br>May 2 17:49:39 nzhmlfpr010 kernel: INFO: task nfsd:2532 blocked for more than 120 seconds.<br>May 2 17:49:39 nzhmlfpr010 kernel: Not tainted 2.6.32-431.5.1.el6.x86_64 #1<br>
May 2 17:49:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>May 2 17:49:39 nzhmlfpr010 kernel: nfsd D 0000000000000008 0 2532 2 0x00000080<br>
May 2 17:49:39 nzhmlfpr010 kernel: ffff88063a073430 0000000000000046 0000000000000000 ffff88063a0735d8<br>May 2 17:49:39 nzhmlfpr010 kernel: 000000043a0735e4 ffff880638681000 ffff88015a9ed838 00ff88063a073580<br>May 2 17:49:39 nzhmlfpr010 kernel: ffff8806386fbaf8 ffff88063a073fd8 000000000000fbc8 ffff8806386fbaf8<br>
May 2 17:49:39 nzhmlfpr010 kernel: Call Trace:<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027ac87>] xfs_trans_read_buf+0x1f7/0x410 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa022faa6>] xfs_read_agf+0x66/0x1a0 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa022fc19>] xfs_alloc_read_agf+0x39/0xd0 [xfs]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0232642>] xfs_alloc_fix_freelist+0x432/0x4a0 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0278187>] ? xfs_trans_add_item+0x57/0x70 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0274938>] ? xfs_perag_get+0x38/0xb0 [xfs]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0232bb6>] xfs_alloc_vextent+0x3d6/0x610 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa025e453>] xfs_ialloc_ag_alloc+0x143/0x4e0 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa025e8cf>] xfs_dialloc+0xdf/0x850 [xfs]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8144b4d4>] ? release_sock+0xe4/0xf0<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff814a4aa1>] ? tcp_recvmsg+0x821/0xe80<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0262980>] xfs_ialloc+0x60/0x6e0 [xfs]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028006a>] ? kmem_zone_zalloc+0x3a/0x50 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027b814>] xfs_dir_ialloc+0x74/0x2b0 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027d8f0>] xfs_create+0x440/0x640 [xfs]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028aa3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028ab80>] xfs_vn_create+0x10/0x20 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff811983b6>] vfs_create+0xe6/0x110<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa055f3d5>] nfsd_create_v3+0x3d5/0x530 [nfsd]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0566c13>] nfsd3_proc_create+0x123/0x1b0 [nfsd]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20<br>
May 2 17:49:39 nzhmlfpr010 kernel: INFO: task nfsd:2533 blocked for more than 120 seconds.<br>May 2 17:49:39 nzhmlfpr010 kernel: Not tainted 2.6.32-431.5.1.el6.x86_64 #1<br>May 2 17:49:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>
May 2 17:49:39 nzhmlfpr010 kernel: nfsd D 0000000000000008 0 2533 2 0x00000080<br>May 2 17:49:39 nzhmlfpr010 kernel: ffff880623f7d8d0 0000000000000046 0000000000000000 ffff880623f7d894<br>May 2 17:49:39 nzhmlfpr010 kernel: ffff880200000000 ffff88033fc25300 ffff880028276840 0000000000000400<br>
May 2 17:49:39 nzhmlfpr010 kernel: ffff88063875bab8 ffff880623f7dfd8 000000000000fbc8 ffff88063875bab8<br>May 2 17:49:39 nzhmlfpr010 kernel: Call Trace:<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0245208>] ? xfs_bmbt_get_all+0x18/0x20 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa023b42d>] ? xfs_bmap_search_multi_extents+0xad/0x120 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027ac87>] xfs_trans_read_buf+0x1f7/0x410 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa025d164>] xfs_read_agi+0x74/0x100 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa026342b>] xfs_iunlink+0x4b/0x170 [xfs]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff810792c7>] ? current_fs_time+0x27/0x30<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027b427>] ? xfs_trans_ichgtime+0x27/0xb0 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027b78b>] xfs_droplink+0x5b/0x70 [xfs]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027d1ce>] xfs_remove+0x27e/0x390 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff811a3767>] ? __d_lookup+0xa7/0x150<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81196203>] ? generic_permission+0x23/0xb0<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028aca8>] xfs_vn_unlink+0x48/0x90 [xfs]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff811979a2>] vfs_unlink+0xa2/0xf0<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa055ef5f>] nfsd_unlink+0x1af/0x250 [nfsd]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa056369f>] ? nfsd_cache_lookup+0x59f/0x700 [nfsd]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa05665d3>] nfsd3_proc_remove+0x83/0x120 [nfsd]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0<br>
May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0<br>May 2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20<br>
May 2 17:51:39 nzhmlfpr010 kernel: INFO: task nfsd:2527 blocked for more than 120 seconds.<br>May 2 17:51:39 nzhmlfpr010 kernel: Not tainted 2.6.32-431.5.1.el6.x86_64 #1<br>May 2 17:51:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>
May 2 17:51:39 nzhmlfpr010 kernel: nfsd D 0000000000000000 0 2527 2 0x00000080<br>May 2 17:51:39 nzhmlfpr010 kernel: ffff880623ea1600 0000000000000046 0000000000000000 ffffffff8105afee<br>May 2 17:51:39 nzhmlfpr010 kernel: ffff880623ea1670 ffff880623ea15e0 ffff880623ea1748 0000000200001000<br>
May 2 17:51:39 nzhmlfpr010 kernel: ffff880623e43098 ffff880623ea1fd8 000000000000fbc8 ffff880623e43098<br>May 2 17:51:39 nzhmlfpr010 kernel: Call Trace:<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8105afee>] ? find_busiest_group+0x90e/0x9f0<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa03cea95>] ? __nf_ct_refresh_acct+0xe5/0x100 [nf_conntrack]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027ac27>] xfs_trans_read_buf+0x197/0x410 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d164>] xfs_read_agi+0x74/0x100 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d21e>] xfs_ialloc_read_agi+0x2e/0x90 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d8fe>] xfs_imap_lookup+0x3e/0x170 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8116ecf0>] ? cache_alloc_refill+0x1c0/0x240<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025dbfe>] xfs_imap+0x1ce/0x280 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa02650f0>] xfs_iread+0x40/0x1b0 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff811a59de>] ? inode_init_always+0x11e/0x1c0<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025fc8e>] xfs_iget+0x27e/0x6e0 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff814b121f>] ? tcp_event_new_data_sent+0x4f/0x110<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa02860a6>] xfs_nfs_get_inode+0x46/0xe0 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0444860>] ? cache_check+0x60/0x360 [sunrpc]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028621b>] xfs_fs_fh_to_dentry+0x8b/0x90 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa015271c>] exportfs_decode_fh+0x5c/0x2bc [exportfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055a800>] ? nfsd_acceptable+0x0/0x120 [nfsd]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0444860>] ? cache_check+0x60/0x360 [sunrpc]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0560712>] ? exp_find_key+0x62/0xb0 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810ec17e>] ? rcu_start_gp+0x1be/0x230<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055acfa>] fh_verify+0x32a/0x640 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa056a7c0>] nfsd4_putfh+0x50/0x60 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa056a458>] nfsd4_proc_compound+0x3d8/0x490 [nfsd]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20<br>May 2 17:51:39 nzhmlfpr010 kernel: INFO: task nfsd:2528 blocked for more than 120 seconds.<br>May 2 17:51:39 nzhmlfpr010 kernel: Not tainted 2.6.32-431.5.1.el6.x86_64 #1<br>
May 2 17:51:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>May 2 17:51:39 nzhmlfpr010 kernel: nfsd D 0000000000000008 0 2528 2 0x00000080<br>
May 2 17:51:39 nzhmlfpr010 kernel: ffff880623ed75e0 0000000000000046 ffff88033745c0f0 0000000019bfcc00<br>May 2 17:51:39 nzhmlfpr010 kernel: ffff880623ed7690 ffffffffa03d4b5b 0007fffffffff07f ffffffff00000000<br>May 2 17:51:39 nzhmlfpr010 kernel: ffff880623d6daf8 ffff880623ed7fd8 000000000000fbc8 ffff880623d6daf8<br>
May 2 17:51:39 nzhmlfpr010 kernel: Call Trace:<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa03d4b5b>] ? tcp_packet+0x68b/0x1750 [nf_conntrack]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81086036>] ? mod_timer_pending+0x126/0x200<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81297e28>] ? swiotlb_dma_mapping_error+0x18/0x30<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027ac87>] xfs_trans_read_buf+0x1f7/0x410 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d164>] xfs_read_agi+0x74/0x100 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d21e>] xfs_ialloc_read_agi+0x2e/0x90 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d503>] xfs_ialloc_ag_select+0x133/0x270 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025ebc7>] xfs_dialloc+0x3d7/0x850 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8116ec8b>] ? cache_alloc_refill+0x15b/0x240<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0262980>] xfs_ialloc+0x60/0x6e0 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028006a>] ? kmem_zone_zalloc+0x3a/0x50 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027b814>] xfs_dir_ialloc+0x74/0x2b0 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027d8f0>] xfs_create+0x440/0x640 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028aa3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028ab80>] xfs_vn_create+0x10/0x20 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff811983b6>] vfs_create+0xe6/0x110<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055f3d5>] nfsd_create_v3+0x3d5/0x530 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0566c13>] nfsd3_proc_create+0x123/0x1b0 [nfsd]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20<br>May 2 17:51:39 nzhmlfpr010 kernel: INFO: task nfsd:2529 blocked for more than 120 seconds.<br>May 2 17:51:39 nzhmlfpr010 kernel: Not tainted 2.6.32-431.5.1.el6.x86_64 #1<br>
May 2 17:51:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>May 2 17:51:39 nzhmlfpr010 kernel: nfsd D 0000000000000008 0 2529 2 0x00000080<br>
May 2 17:51:39 nzhmlfpr010 kernel: ffff8806384c95e0 0000000000000046 0000000000001000 0000000000000000<br>May 2 17:51:39 nzhmlfpr010 kernel: 000000000026b000 000000000026c000 0000000000000100 ffffffffffffffff<br>May 2 17:51:39 nzhmlfpr010 kernel: ffff880623df3ab8 ffff8806384c9fd8 000000000000fbc8 ffff880623df3ab8<br>
May 2 17:51:39 nzhmlfpr010 kernel: Call Trace:<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81086036>] ? mod_timer_pending+0x126/0x200<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa03cea95>] ? __nf_ct_refresh_acct+0xe5/0x100 [nf_conntrack]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027ac87>] xfs_trans_read_buf+0x1f7/0x410 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d164>] xfs_read_agi+0x74/0x100 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d21e>] xfs_ialloc_read_agi+0x2e/0x90 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d503>] xfs_ialloc_ag_select+0x133/0x270 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025ebc7>] xfs_dialloc+0x3d7/0x850 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8144b4d4>] ? release_sock+0xe4/0xf0<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff814a4aa1>] ? tcp_recvmsg+0x821/0xe80<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0262980>] xfs_ialloc+0x60/0x6e0 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028006a>] ? kmem_zone_zalloc+0x3a/0x50 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027b814>] xfs_dir_ialloc+0x74/0x2b0 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027d8f0>] xfs_create+0x440/0x640 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028aa3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028ab80>] xfs_vn_create+0x10/0x20 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff811983b6>] vfs_create+0xe6/0x110<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055f3d5>] nfsd_create_v3+0x3d5/0x530 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0566c13>] nfsd3_proc_create+0x123/0x1b0 [nfsd]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20<br>May 2 17:51:39 nzhmlfpr010 kernel: INFO: task nfsd:2530 blocked for more than 120 seconds.<br>May 2 17:51:39 nzhmlfpr010 kernel: Not tainted 2.6.32-431.5.1.el6.x86_64 #1<br>
May 2 17:51:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>May 2 17:51:39 nzhmlfpr010 kernel: nfsd D 0000000000000000 0 2530 2 0x00000080<br>
May 2 17:51:39 nzhmlfpr010 kernel: ffff880623fbd600 0000000000000046 0000000000000000 ffffffffa0260e6c<br>May 2 17:51:39 nzhmlfpr010 kernel: 0000000000800000 0002005a00000004 0000000000000003 0000000000000286<br>May 2 17:51:39 nzhmlfpr010 kernel: ffff880626601058 ffff880623fbdfd8 000000000000fbc8 ffff880626601058<br>
May 2 17:51:39 nzhmlfpr010 kernel: Call Trace:<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0260e6c>] ? xfs_iext_bno_to_ext+0x8c/0x170 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027ac27>] xfs_trans_read_buf+0x197/0x410 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d164>] xfs_read_agi+0x74/0x100 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d21e>] xfs_ialloc_read_agi+0x2e/0x90 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d8fe>] xfs_imap_lookup+0x3e/0x170 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8149a3e8>] ? ip_finish_output+0x148/0x310<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025dbfe>] xfs_imap+0x1ce/0x280 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa02650f0>] xfs_iread+0x40/0x1b0 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff811a59de>] ? inode_init_always+0x11e/0x1c0<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025fc8e>] xfs_iget+0x27e/0x6e0 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff814b121f>] ? tcp_event_new_data_sent+0x4f/0x110<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa02860a6>] xfs_nfs_get_inode+0x46/0xe0 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0444860>] ? cache_check+0x60/0x360 [sunrpc]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028621b>] xfs_fs_fh_to_dentry+0x8b/0x90 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa015271c>] exportfs_decode_fh+0x5c/0x2bc [exportfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055a800>] ? nfsd_acceptable+0x0/0x120 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0444860>] ? cache_check+0x60/0x360 [sunrpc]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0560712>] ? exp_find_key+0x62/0xb0 [nfsd]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81068a53>] ? dequeue_entity+0x113/0x2e0<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055acfa>] fh_verify+0x32a/0x640 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa044392d>] ? cache_revisit_request+0x2d/0x120 [sunrpc]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa056a7c0>] nfsd4_putfh+0x50/0x60 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa056a458>] nfsd4_proc_compound+0x3d8/0x490 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20<br>
May 2 17:51:39 nzhmlfpr010 kernel: INFO: task nfsd:2531 blocked for more than 120 seconds.<br>May 2 17:51:39 nzhmlfpr010 kernel: Not tainted 2.6.32-431.5.1.el6.x86_64 #1<br>May 2 17:51:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>
May 2 17:51:39 nzhmlfpr010 kernel: nfsd D 0000000000000008 0 2531 2 0x00000080<br>May 2 17:51:39 nzhmlfpr010 kernel: ffff880623131c80 0000000000000046 0000000000000000 00000000329ac000<br>May 2 17:51:39 nzhmlfpr010 kernel: ffff880338a05200 0000000000000001 ffff8803330c3d88 ffff8803330c3d88<br>
May 2 17:51:39 nzhmlfpr010 kernel: ffff880623d91ab8 ffff880623131fd8 000000000000fbc8 ffff880623d91ab8<br>May 2 17:51:39 nzhmlfpr010 kernel: Call Trace:<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8152938e>] __mutex_lock_slowpath+0x13e/0x180<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8152922b>] mutex_lock+0x2b/0x50<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055f102>] nfsd_create_v3+0x102/0x530 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0566c13>] nfsd3_proc_create+0x123/0x1b0 [nfsd]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20<br>May 2 17:51:39 nzhmlfpr010 kernel: INFO: task nfsd:2532 blocked for more than 120 seconds.<br>May 2 17:51:39 nzhmlfpr010 kernel: Not tainted 2.6.32-431.5.1.el6.x86_64 #1<br>
May 2 17:51:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>May 2 17:51:39 nzhmlfpr010 kernel: nfsd D 0000000000000008 0 2532 2 0x00000080<br>
May 2 17:51:39 nzhmlfpr010 kernel: ffff88063a073430 0000000000000046 0000000000000000 ffff88063a0735d8<br>May 2 17:51:39 nzhmlfpr010 kernel: 000000043a0735e4 ffff880638681000 ffff88015a9ed838 00ff88063a073580<br>May 2 17:51:39 nzhmlfpr010 kernel: ffff8806386fbaf8 ffff88063a073fd8 000000000000fbc8 ffff8806386fbaf8<br>
May 2 17:51:39 nzhmlfpr010 kernel: Call Trace:<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027ac87>] xfs_trans_read_buf+0x1f7/0x410 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa022faa6>] xfs_read_agf+0x66/0x1a0 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa022fc19>] xfs_alloc_read_agf+0x39/0xd0 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0232642>] xfs_alloc_fix_freelist+0x432/0x4a0 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0278187>] ? xfs_trans_add_item+0x57/0x70 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0274938>] ? xfs_perag_get+0x38/0xb0 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0232bb6>] xfs_alloc_vextent+0x3d6/0x610 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025e453>] xfs_ialloc_ag_alloc+0x143/0x4e0 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025e8cf>] xfs_dialloc+0xdf/0x850 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8144b4d4>] ? release_sock+0xe4/0xf0<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff814a4aa1>] ? tcp_recvmsg+0x821/0xe80<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0262980>] xfs_ialloc+0x60/0x6e0 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028006a>] ? kmem_zone_zalloc+0x3a/0x50 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027b814>] xfs_dir_ialloc+0x74/0x2b0 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027d8f0>] xfs_create+0x440/0x640 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028aa3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028ab80>] xfs_vn_create+0x10/0x20 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff811983b6>] vfs_create+0xe6/0x110<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055f3d5>] nfsd_create_v3+0x3d5/0x530 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0566c13>] nfsd3_proc_create+0x123/0x1b0 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20<br>
May 2 17:51:39 nzhmlfpr010 kernel: INFO: task nfsd:2533 blocked for more than 120 seconds.<br>May 2 17:51:39 nzhmlfpr010 kernel: Not tainted 2.6.32-431.5.1.el6.x86_64 #1<br>May 2 17:51:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>
May 2 17:51:39 nzhmlfpr010 kernel: nfsd D 0000000000000008 0 2533 2 0x00000080<br>May 2 17:51:39 nzhmlfpr010 kernel: ffff880623f7d8d0 0000000000000046 0000000000000000 ffff880623f7d894<br>May 2 17:51:39 nzhmlfpr010 kernel: ffff880200000000 ffff88033fc25300 ffff880028276840 0000000000000400<br>
May 2 17:51:39 nzhmlfpr010 kernel: ffff88063875bab8 ffff880623f7dfd8 000000000000fbc8 ffff88063875bab8<br>May 2 17:51:39 nzhmlfpr010 kernel: Call Trace:<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0245208>] ? xfs_bmbt_get_all+0x18/0x20 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa023b42d>] ? xfs_bmap_search_multi_extents+0xad/0x120 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027ac87>] xfs_trans_read_buf+0x1f7/0x410 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d164>] xfs_read_agi+0x74/0x100 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa026342b>] xfs_iunlink+0x4b/0x170 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810792c7>] ? current_fs_time+0x27/0x30<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027b427>] ? xfs_trans_ichgtime+0x27/0xb0 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027b78b>] xfs_droplink+0x5b/0x70 [xfs]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027d1ce>] xfs_remove+0x27e/0x390 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff811a3767>] ? __d_lookup+0xa7/0x150<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81196203>] ? generic_permission+0x23/0xb0<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028aca8>] xfs_vn_unlink+0x48/0x90 [xfs]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff811979a2>] vfs_unlink+0xa2/0xf0<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055ef5f>] nfsd_unlink+0x1af/0x250 [nfsd]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa056369f>] ? nfsd_cache_lookup+0x59f/0x700 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa05665d3>] nfsd3_proc_remove+0x83/0x120 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0<br>
May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0<br>May 2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20</div>
<div> </div><div> </div><div>Thank you</div></div>