<div dir="ltr">Hello,<div><br></div><div>I am running a server with heavy workload on a XFS mount:</div><div><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">/dev/mapper/v2-d1 on /d/d1 type xfs (rw,nodev,noatime,inode64,</span><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">allocsize=16m)</span><br>
</div><div><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">2.6.32-424.el6.x86_64 #1 SMP Mon Oct 14 20:11:50 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux</span><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"><br>
</span></div><div><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">We get errors in log indicating processes are </span><font face="arial, sans-serif">blocked for more than 120 seconds.</font></div>
<div><br></div><div>1) Is this expected during heavy workload?</div><div>2) What would be the impact on the processes?  Are they basically hung in userspace waiting for IO?</div><div>3) Is there anything we tune here?</div>
<div><br></div><div>Below is the output.</div><div><br></div><div>Cheers,</div><div>Ivan Novick</div><div><br></div><div><div><div>INFO: task flush-253:1:6882 blocked for more than 120 seconds.</div><div>      Tainted: P           ---------------    2.6.32-424.el6.x86_64 #1</div>
<div>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.</div><div>flush-253:1   D 0000000000000000     0  6882      2 0x00000000</div><div> ffff881011fa5690 0000000000000046 0000000000000000 ffff881013cba040</div>
<div> 0000000000000001 ffff88012a447d40 ffff881013cba040 ffff881016eb4830</div><div> ffff881013cba5f8 ffff881011fa5fd8 000000000000fbc8 ffff881013cba5f8</div><div>Call Trace:</div><div> [<ffffffff815289d3>] io_schedule+0x73/0xc0</div>
<div> [<ffffffff81267d18>] get_request_wait+0x108/0x1d0</div><div> [<ffffffff8109b4a0>] ? autoremove_wake_function+0x0/0x40</div><div> [<ffffffff812618ce>] ? elv_merge+0x17e/0x1c0</div><div> [<ffffffff81267e79>] blk_queue_bio+0x99/0x620</div>
<div> [<ffffffff81266f00>] generic_make_request+0x240/0x5a0</div><div> [<ffffffffa01c3f97>] ? dm_merge_bvec+0xc7/0x100 [dm_mod]</div><div> [<ffffffff812672d0>] submit_bio+0x70/0x120</div><div> [<ffffffffa03b9c53>] xfs_submit_ioend_bio+0x33/0x40 [xfs]</div>
<div> [<ffffffffa03b9d2c>] xfs_submit_ioend+0xcc/0x140 [xfs]</div><div> [<ffffffffa03ba678>] xfs_vm_writepage+0x308/0x590 [xfs]</div><div> [<ffffffff81133967>] __writepage+0x17/0x40</div><div> [<ffffffff81134c2d>] write_cache_pages+0x1fd/0x4c0</div>
<div> [<ffffffff8100bdae>] ? call_function_single_interrupt+0xe/0x20</div><div> [<ffffffff81133950>] ? __writepage+0x0/0x40</div><div> [<ffffffff81134f14>] generic_writepages+0x24/0x30</div><div> [<ffffffffa03b98ad>] xfs_vm_writepages+0x5d/0x80 [xfs]</div>
<div> [<ffffffff81134f41>] do_writepages+0x21/0x40</div><div> [<ffffffff811b52ad>] writeback_single_inode+0xdd/0x290</div><div> [<ffffffff811b56ad>] writeback_sb_inodes+0xbd/0x170</div><div> [<ffffffff811b580b>] writeback_inodes_wb+0xab/0x1b0</div>
<div> [<ffffffff811b5c03>] wb_writeback+0x2f3/0x410</div><div> [<ffffffff81528240>] ? thread_return+0x4e/0x76e</div><div> [<ffffffff81084fb2>] ? del_timer_sync+0x22/0x30</div><div> [<ffffffff811b5ec5>] wb_do_writeback+0x1a5/0x240</div>
<div> [<ffffffff811b5fc3>] bdi_writeback_task+0x63/0x1b0</div><div> [<ffffffff8109b327>] ? bit_waitqueue+0x17/0xd0</div><div> [<ffffffff81143c40>] ? bdi_start_fn+0x0/0x100</div><div> [<ffffffff81143cc6>] bdi_start_fn+0x86/0x100</div>
<div> [<ffffffff81143c40>] ? bdi_start_fn+0x0/0x100</div><div> [<ffffffff8109b0f6>] kthread+0x96/0xa0</div><div> [<ffffffff8100c20a>] child_rip+0xa/0x20</div><div> [<ffffffff8109b060>] ? kthread+0x0/0xa0</div>
<div> [<ffffffff8100c200>] ? child_rip+0x0/0x20</div><div>INFO: task ivanserver:6969 blocked for more than 120 seconds.</div><div>      Tainted: P           ---------------    2.6.32-424.el6.x86_64 #1</div><div>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.</div>
<div>ivanserver      D 0000000000000000     0  6969   6947 0x00000000</div><div> ffff881f4ef179d8 0000000000000082 0000000000000000 000000000000000f</div><div> 0000000000000001 ffff8800622143c0 0000000000000001 ffff880062216840</div>
<div> ffff881f4eeae638 ffff881f4ef17fd8 000000000000fbc8 ffff881f4eeae638</div><div>Call Trace:</div><div> [<ffffffff815290e5>] schedule_timeout+0x215/0x2e0</div><div> [<ffffffff8152a0e2>] __down+0x72/0xb0</div>
<div> [<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]</div><div> [<ffffffff810a1591>] down+0x41/0x50</div><div> [<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]</div><div> [<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]</div>
<div> [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]</div><div> [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]</div><div> [<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]</div><div> [<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]</div>
<div> [<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]</div><div> [<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]</div><div> [<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]</div><div>
 [<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]</div><div> [<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]</div><div> [<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]</div><div>
 [<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]</div><div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div><div> [<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]</div><div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div>
<div> [<ffffffff8100bdae>] ? call_function_single_interrupt+0xe/0x20</div><div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div><div> [<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]</div><div> [<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0</div>
<div> [<ffffffff8118a5b0>] ? fget+0x30/0x80</div><div> [<ffffffff8119f429>] sys_getdents+0x89/0xf0</div><div> [<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20</div><div> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b</div>
<div>INFO: task ivanserver:6972 blocked for more than 120 seconds.</div><div>      Tainted: P           ---------------    2.6.32-424.el6.x86_64 #1</div><div>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.</div>
<div>ivanserver      D 0000000000000008     0  6972   6948 0x00000000</div><div> ffff88100f1219d8 0000000000000086 0000000000000000 000000000000000f</div><div> 0000000000000001 ffff8810788143c0 0000000000000001 ffff881078816840</div>
<div> ffff88100fd62638 ffff88100f121fd8 000000000000fbc8 ffff88100fd62638</div><div>Call Trace:</div><div> [<ffffffff815290e5>] schedule_timeout+0x215/0x2e0</div><div> [<ffffffff8152a0e2>] __down+0x72/0xb0</div>
<div> [<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]</div><div> [<ffffffff810a1591>] down+0x41/0x50</div><div> [<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]</div><div> [<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]</div>
<div> [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]</div><div> [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]</div><div> [<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]</div><div> [<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]</div>
<div> [<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]</div><div> [<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]</div><div> [<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]</div><div>
 [<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]</div><div> [<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]</div><div> [<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]</div><div>
 [<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]</div><div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div><div> [<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]</div><div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div>
<div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div><div> [<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]</div><div> [<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0</div><div> [<ffffffff8119f429>] sys_getdents+0x89/0xf0</div>
<div> [<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20</div><div> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b</div><div>INFO: task ivanserver:6985 blocked for more than 120 seconds.</div><div>      Tainted: P           ---------------    2.6.32-424.el6.x86_64 #1</div>
<div>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.</div><div>ivanserver      D 0000000000000000     0  6985   6963 0x00000000</div><div> ffff8810103ad9d8 0000000000000082 0000000000000000 000000000000000f</div>
<div> 0000000000000001 ffff8800622143c0 0000000000000001 ffff880062216840</div><div> ffff88100e01e638 ffff8810103adfd8 000000000000fbc8 ffff88100e01e638</div><div>Call Trace:</div><div> [<ffffffff815290e5>] schedule_timeout+0x215/0x2e0</div>
<div> [<ffffffff8152a0e2>] __down+0x72/0xb0</div><div> [<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]</div><div> [<ffffffff810a1591>] down+0x41/0x50</div><div> [<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]</div>
<div> [<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]</div><div> [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]</div><div> [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]</div><div> [<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]</div>
<div> [<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]</div><div> [<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]</div><div> [<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]</div><div> [<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]</div>
<div> [<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]</div><div> [<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]</div><div> [<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]</div>
<div> [<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]</div><div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div><div> [<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]</div><div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div>
<div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div><div> [<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]</div><div> [<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0</div><div> [<ffffffff8119f429>] sys_getdents+0x89/0xf0</div>
<div> [<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20</div><div> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b</div><div>INFO: task ivanserver:6989 blocked for more than 120 seconds.</div><div>      Tainted: P           ---------------    2.6.32-424.el6.x86_64 #1</div>
<div>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.</div><div>ivanserver      D 0000000000000008     0  6989   6981 0x00000000</div><div> ffff8810101279d8 0000000000000086 0000000000000000 000000000000000f</div>
<div> 0000000000000001 ffff8810788143c0 0000000000000001 ffff881078816840</div><div> ffff8810252a4638 ffff881010127fd8 000000000000fbc8 ffff8810252a4638</div><div>Call Trace:</div><div> [<ffffffff815290e5>] schedule_timeout+0x215/0x2e0</div>
<div> [<ffffffff8152a0e2>] __down+0x72/0xb0</div><div> [<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]</div><div> [<ffffffff810a1591>] down+0x41/0x50</div><div> [<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]</div>
<div> [<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]</div><div> [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]</div><div> [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]</div><div> [<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]</div>
<div> [<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]</div><div> [<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]</div><div> [<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]</div><div> [<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]</div>
<div> [<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]</div><div> [<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]</div><div> [<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]</div>
<div> [<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]</div><div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div><div> [<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]</div><div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div>
<div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div><div> [<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]</div><div> [<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0</div><div> [<ffffffff8119f429>] sys_getdents+0x89/0xf0</div>
<div> [<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20</div><div> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b</div><div>usxxnovicim1:Downloads novici$ cat dmesg.txt </div><div>INFO: task flush-253:1:6882 blocked for more than 120 seconds.</div>
<div>      Tainted: P           ---------------    2.6.32-424.el6.x86_64 #1</div><div>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.</div><div>flush-253:1   D 0000000000000000     0  6882      2 0x00000000</div>
<div> ffff881011fa5690 0000000000000046 0000000000000000 ffff881013cba040</div><div> 0000000000000001 ffff88012a447d40 ffff881013cba040 ffff881016eb4830</div><div> ffff881013cba5f8 ffff881011fa5fd8 000000000000fbc8 ffff881013cba5f8</div>
<div>Call Trace:</div><div> [<ffffffff815289d3>] io_schedule+0x73/0xc0</div><div> [<ffffffff81267d18>] get_request_wait+0x108/0x1d0</div><div> [<ffffffff8109b4a0>] ? autoremove_wake_function+0x0/0x40</div>
<div> [<ffffffff812618ce>] ? elv_merge+0x17e/0x1c0</div><div> [<ffffffff81267e79>] blk_queue_bio+0x99/0x620</div><div> [<ffffffff81266f00>] generic_make_request+0x240/0x5a0</div><div> [<ffffffffa01c3f97>] ? dm_merge_bvec+0xc7/0x100 [dm_mod]</div>
<div> [<ffffffff812672d0>] submit_bio+0x70/0x120</div><div> [<ffffffffa03b9c53>] xfs_submit_ioend_bio+0x33/0x40 [xfs]</div><div> [<ffffffffa03b9d2c>] xfs_submit_ioend+0xcc/0x140 [xfs]</div><div> [<ffffffffa03ba678>] xfs_vm_writepage+0x308/0x590 [xfs]</div>
<div> [<ffffffff81133967>] __writepage+0x17/0x40</div><div> [<ffffffff81134c2d>] write_cache_pages+0x1fd/0x4c0</div><div> [<ffffffff8100bdae>] ? call_function_single_interrupt+0xe/0x20</div><div> [<ffffffff81133950>] ? __writepage+0x0/0x40</div>
<div> [<ffffffff81134f14>] generic_writepages+0x24/0x30</div><div> [<ffffffffa03b98ad>] xfs_vm_writepages+0x5d/0x80 [xfs]</div><div> [<ffffffff81134f41>] do_writepages+0x21/0x40</div><div> [<ffffffff811b52ad>] writeback_single_inode+0xdd/0x290</div>
<div> [<ffffffff811b56ad>] writeback_sb_inodes+0xbd/0x170</div><div> [<ffffffff811b580b>] writeback_inodes_wb+0xab/0x1b0</div><div> [<ffffffff811b5c03>] wb_writeback+0x2f3/0x410</div><div> [<ffffffff81528240>] ? thread_return+0x4e/0x76e</div>
<div> [<ffffffff81084fb2>] ? del_timer_sync+0x22/0x30</div><div> [<ffffffff811b5ec5>] wb_do_writeback+0x1a5/0x240</div><div> [<ffffffff811b5fc3>] bdi_writeback_task+0x63/0x1b0</div><div> [<ffffffff8109b327>] ? bit_waitqueue+0x17/0xd0</div>
<div> [<ffffffff81143c40>] ? bdi_start_fn+0x0/0x100</div><div> [<ffffffff81143cc6>] bdi_start_fn+0x86/0x100</div><div> [<ffffffff81143c40>] ? bdi_start_fn+0x0/0x100</div><div> [<ffffffff8109b0f6>] kthread+0x96/0xa0</div>
<div> [<ffffffff8100c20a>] child_rip+0xa/0x20</div><div> [<ffffffff8109b060>] ? kthread+0x0/0xa0</div><div> [<ffffffff8100c200>] ? child_rip+0x0/0x20</div><div>INFO: task ivanserver:6969 blocked for more than 120 seconds.</div>
<div>      Tainted: P           ---------------    2.6.32-424.el6.x86_64 #1</div><div>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.</div><div>ivanserver      D 0000000000000000     0  6969   6947 0x00000000</div>
<div> ffff881f4ef179d8 0000000000000082 0000000000000000 000000000000000f</div><div> 0000000000000001 ffff8800622143c0 0000000000000001 ffff880062216840</div><div> ffff881f4eeae638 ffff881f4ef17fd8 000000000000fbc8 ffff881f4eeae638</div>
<div>Call Trace:</div><div> [<ffffffff815290e5>] schedule_timeout+0x215/0x2e0</div><div> [<ffffffff8152a0e2>] __down+0x72/0xb0</div><div> [<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]</div><div> [<ffffffff810a1591>] down+0x41/0x50</div>
<div> [<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]</div><div> [<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]</div><div> [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]</div><div> [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]</div>
<div> [<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]</div><div> [<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]</div><div> [<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]</div><div> [<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]</div>
<div> [<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]</div><div> [<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]</div><div> [<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]</div><div>
 [<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]</div><div> [<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]</div><div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div><div> [<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]</div>
<div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div><div> [<ffffffff8100bdae>] ? call_function_single_interrupt+0xe/0x20</div><div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div><div> [<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]</div>
<div> [<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0</div><div> [<ffffffff8118a5b0>] ? fget+0x30/0x80</div><div> [<ffffffff8119f429>] sys_getdents+0x89/0xf0</div><div> [<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20</div>
<div> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b</div><div>INFO: task ivanserver:6972 blocked for more than 120 seconds.</div><div>      Tainted: P           ---------------    2.6.32-424.el6.x86_64 #1</div>
<div>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.</div><div>ivanserver      D 0000000000000008     0  6972   6948 0x00000000</div><div> ffff88100f1219d8 0000000000000086 0000000000000000 000000000000000f</div>
<div> 0000000000000001 ffff8810788143c0 0000000000000001 ffff881078816840</div><div> ffff88100fd62638 ffff88100f121fd8 000000000000fbc8 ffff88100fd62638</div><div>Call Trace:</div><div> [<ffffffff815290e5>] schedule_timeout+0x215/0x2e0</div>
<div> [<ffffffff8152a0e2>] __down+0x72/0xb0</div><div> [<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]</div><div> [<ffffffff810a1591>] down+0x41/0x50</div><div> [<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]</div>
<div> [<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]</div><div> [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]</div><div> [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]</div><div> [<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]</div>
<div> [<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]</div><div> [<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]</div><div> [<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]</div><div> [<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]</div>
<div> [<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]</div><div> [<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]</div><div> [<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]</div>
<div> [<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]</div><div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div><div> [<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]</div><div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div>
<div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div><div> [<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]</div><div> [<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0</div><div> [<ffffffff8119f429>] sys_getdents+0x89/0xf0</div>
<div> [<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20</div><div> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b</div><div>INFO: task ivanserver:6985 blocked for more than 120 seconds.</div><div>      Tainted: P           ---------------    2.6.32-424.el6.x86_64 #1</div>
<div>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.</div><div>ivanserver      D 0000000000000000     0  6985   6963 0x00000000</div><div> ffff8810103ad9d8 0000000000000082 0000000000000000 000000000000000f</div>
<div> 0000000000000001 ffff8800622143c0 0000000000000001 ffff880062216840</div><div> ffff88100e01e638 ffff8810103adfd8 000000000000fbc8 ffff88100e01e638</div><div>Call Trace:</div><div> [<ffffffff815290e5>] schedule_timeout+0x215/0x2e0</div>
<div> [<ffffffff8152a0e2>] __down+0x72/0xb0</div><div> [<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]</div><div> [<ffffffff810a1591>] down+0x41/0x50</div><div> [<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]</div>
<div> [<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]</div><div> [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]</div><div> [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]</div><div> [<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]</div>
<div> [<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]</div><div> [<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]</div><div> [<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]</div><div> [<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]</div>
<div> [<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]</div><div> [<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]</div><div> [<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]</div>
<div> [<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]</div><div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div><div> [<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]</div><div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div>
<div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div><div> [<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]</div><div> [<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0</div><div> [<ffffffff8119f429>] sys_getdents+0x89/0xf0</div>
<div> [<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20</div><div> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b</div><div>INFO: task ivanserver:6989 blocked for more than 120 seconds.</div><div>      Tainted: P           ---------------    2.6.32-424.el6.x86_64 #1</div>
<div>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.</div><div>ivanserver      D 0000000000000008     0  6989   6981 0x00000000</div><div> ffff8810101279d8 0000000000000086 0000000000000000 000000000000000f</div>
<div> 0000000000000001 ffff8810788143c0 0000000000000001 ffff881078816840</div><div> ffff8810252a4638 ffff881010127fd8 000000000000fbc8 ffff8810252a4638</div><div>Call Trace:</div><div> [<ffffffff815290e5>] schedule_timeout+0x215/0x2e0</div>
<div> [<ffffffff8152a0e2>] __down+0x72/0xb0</div><div> [<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]</div><div> [<ffffffff810a1591>] down+0x41/0x50</div><div> [<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]</div>
<div> [<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]</div><div> [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]</div><div> [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]</div><div> [<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]</div>
<div> [<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]</div><div> [<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]</div><div> [<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]</div><div> [<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]</div>
<div> [<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]</div><div> [<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]</div><div> [<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]</div>
<div> [<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]</div><div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div><div> [<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]</div><div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div>
<div> [<ffffffff8119f020>] ? filldir+0x0/0xe0</div><div> [<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]</div><div> [<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0</div><div> [<ffffffff8119f429>] sys_getdents+0x89/0xf0</div>
<div> [<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20</div><div> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b</div></div></div></div>