xfs
[Top] [All Lists]

xfs: blocked for more than 120 seconds

To: xfs@xxxxxxxxxxx
Subject: xfs: blocked for more than 120 seconds
From: Ivan Novick <inovick@xxxxxxxxxxxxx>
Date: Thu, 16 Jan 2014 11:51:52 -0800
Delivered-to: xfs@xxxxxxxxxxx
Hello,

I am running a server with heavy workload on a XFS mount:
/dev/mapper/v2-d1 on /d/d1 type xfs (rw,nodev,noatime,inode64,allocsize=16m)
2.6.32-424.el6.x86_64 #1 SMP Mon Oct 14 20:11:50 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux

We get errors in log indicating processes are blocked for more than 120 seconds.

1) Is this expected during heavy workload?
2) What would be the impact on the processes?  Are they basically hung in userspace waiting for IO?
3) Is there anything we tune here?

Below is the output.

Cheers,
Ivan Novick

INFO: task flush-253:1:6882 blocked for more than 120 seconds.
      Tainted: P           ---------------    2.6.32-424.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
flush-253:1   D 0000000000000000     0  6882      2 0x00000000
 ffff881011fa5690 0000000000000046 0000000000000000 ffff881013cba040
 0000000000000001 ffff88012a447d40 ffff881013cba040 ffff881016eb4830
 ffff881013cba5f8 ffff881011fa5fd8 000000000000fbc8 ffff881013cba5f8
Call Trace:
 [<ffffffff815289d3>] io_schedule+0x73/0xc0
 [<ffffffff81267d18>] get_request_wait+0x108/0x1d0
 [<ffffffff8109b4a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff812618ce>] ? elv_merge+0x17e/0x1c0
 [<ffffffff81267e79>] blk_queue_bio+0x99/0x620
 [<ffffffff81266f00>] generic_make_request+0x240/0x5a0
 [<ffffffffa01c3f97>] ? dm_merge_bvec+0xc7/0x100 [dm_mod]
 [<ffffffff812672d0>] submit_bio+0x70/0x120
 [<ffffffffa03b9c53>] xfs_submit_ioend_bio+0x33/0x40 [xfs]
 [<ffffffffa03b9d2c>] xfs_submit_ioend+0xcc/0x140 [xfs]
 [<ffffffffa03ba678>] xfs_vm_writepage+0x308/0x590 [xfs]
 [<ffffffff81133967>] __writepage+0x17/0x40
 [<ffffffff81134c2d>] write_cache_pages+0x1fd/0x4c0
 [<ffffffff8100bdae>] ? call_function_single_interrupt+0xe/0x20
 [<ffffffff81133950>] ? __writepage+0x0/0x40
 [<ffffffff81134f14>] generic_writepages+0x24/0x30
 [<ffffffffa03b98ad>] xfs_vm_writepages+0x5d/0x80 [xfs]
 [<ffffffff81134f41>] do_writepages+0x21/0x40
 [<ffffffff811b52ad>] writeback_single_inode+0xdd/0x290
 [<ffffffff811b56ad>] writeback_sb_inodes+0xbd/0x170
 [<ffffffff811b580b>] writeback_inodes_wb+0xab/0x1b0
 [<ffffffff811b5c03>] wb_writeback+0x2f3/0x410
 [<ffffffff81528240>] ? thread_return+0x4e/0x76e
 [<ffffffff81084fb2>] ? del_timer_sync+0x22/0x30
 [<ffffffff811b5ec5>] wb_do_writeback+0x1a5/0x240
 [<ffffffff811b5fc3>] bdi_writeback_task+0x63/0x1b0
 [<ffffffff8109b327>] ? bit_waitqueue+0x17/0xd0
 [<ffffffff81143c40>] ? bdi_start_fn+0x0/0x100
 [<ffffffff81143cc6>] bdi_start_fn+0x86/0x100
 [<ffffffff81143c40>] ? bdi_start_fn+0x0/0x100
 [<ffffffff8109b0f6>] kthread+0x96/0xa0
 [<ffffffff8100c20a>] child_rip+0xa/0x20
 [<ffffffff8109b060>] ? kthread+0x0/0xa0
 [<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task ivanserver:6969 blocked for more than 120 seconds.
      Tainted: P           ---------------    2.6.32-424.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ivanserver      D 0000000000000000     0  6969   6947 0x00000000
 ffff881f4ef179d8 0000000000000082 0000000000000000 000000000000000f
 0000000000000001 ffff8800622143c0 0000000000000001 ffff880062216840
 ffff881f4eeae638 ffff881f4ef17fd8 000000000000fbc8 ffff881f4eeae638
Call Trace:
 [<ffffffff815290e5>] schedule_timeout+0x215/0x2e0
 [<ffffffff8152a0e2>] __down+0x72/0xb0
 [<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]
 [<ffffffff810a1591>] down+0x41/0x50
 [<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]
 [<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]
 [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
 [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
 [<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]
 [<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]
 [<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]
 [<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]
 [<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]
 [<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]
 [<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]
 [<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
 [<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffff8100bdae>] ? call_function_single_interrupt+0xe/0x20
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]
 [<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0
 [<ffffffff8118a5b0>] ? fget+0x30/0x80
 [<ffffffff8119f429>] sys_getdents+0x89/0xf0
 [<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ivanserver:6972 blocked for more than 120 seconds.
      Tainted: P           ---------------    2.6.32-424.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ivanserver      D 0000000000000008     0  6972   6948 0x00000000
 ffff88100f1219d8 0000000000000086 0000000000000000 000000000000000f
 0000000000000001 ffff8810788143c0 0000000000000001 ffff881078816840
 ffff88100fd62638 ffff88100f121fd8 000000000000fbc8 ffff88100fd62638
Call Trace:
 [<ffffffff815290e5>] schedule_timeout+0x215/0x2e0
 [<ffffffff8152a0e2>] __down+0x72/0xb0
 [<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]
 [<ffffffff810a1591>] down+0x41/0x50
 [<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]
 [<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]
 [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
 [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
 [<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]
 [<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]
 [<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]
 [<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]
 [<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]
 [<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]
 [<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]
 [<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
 [<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]
 [<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0
 [<ffffffff8119f429>] sys_getdents+0x89/0xf0
 [<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ivanserver:6985 blocked for more than 120 seconds.
      Tainted: P           ---------------    2.6.32-424.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ivanserver      D 0000000000000000     0  6985   6963 0x00000000
 ffff8810103ad9d8 0000000000000082 0000000000000000 000000000000000f
 0000000000000001 ffff8800622143c0 0000000000000001 ffff880062216840
 ffff88100e01e638 ffff8810103adfd8 000000000000fbc8 ffff88100e01e638
Call Trace:
 [<ffffffff815290e5>] schedule_timeout+0x215/0x2e0
 [<ffffffff8152a0e2>] __down+0x72/0xb0
 [<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]
 [<ffffffff810a1591>] down+0x41/0x50
 [<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]
 [<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]
 [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
 [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
 [<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]
 [<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]
 [<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]
 [<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]
 [<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]
 [<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]
 [<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]
 [<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
 [<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]
 [<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0
 [<ffffffff8119f429>] sys_getdents+0x89/0xf0
 [<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ivanserver:6989 blocked for more than 120 seconds.
      Tainted: P           ---------------    2.6.32-424.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ivanserver      D 0000000000000008     0  6989   6981 0x00000000
 ffff8810101279d8 0000000000000086 0000000000000000 000000000000000f
 0000000000000001 ffff8810788143c0 0000000000000001 ffff881078816840
 ffff8810252a4638 ffff881010127fd8 000000000000fbc8 ffff8810252a4638
Call Trace:
 [<ffffffff815290e5>] schedule_timeout+0x215/0x2e0
 [<ffffffff8152a0e2>] __down+0x72/0xb0
 [<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]
 [<ffffffff810a1591>] down+0x41/0x50
 [<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]
 [<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]
 [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
 [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
 [<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]
 [<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]
 [<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]
 [<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]
 [<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]
 [<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]
 [<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]
 [<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
 [<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]
 [<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0
 [<ffffffff8119f429>] sys_getdents+0x89/0xf0
 [<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
usxxnovicim1:Downloads novici$ cat dmesg.txt 
INFO: task flush-253:1:6882 blocked for more than 120 seconds.
      Tainted: P           ---------------    2.6.32-424.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
flush-253:1   D 0000000000000000     0  6882      2 0x00000000
 ffff881011fa5690 0000000000000046 0000000000000000 ffff881013cba040
 0000000000000001 ffff88012a447d40 ffff881013cba040 ffff881016eb4830
 ffff881013cba5f8 ffff881011fa5fd8 000000000000fbc8 ffff881013cba5f8
Call Trace:
 [<ffffffff815289d3>] io_schedule+0x73/0xc0
 [<ffffffff81267d18>] get_request_wait+0x108/0x1d0
 [<ffffffff8109b4a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff812618ce>] ? elv_merge+0x17e/0x1c0
 [<ffffffff81267e79>] blk_queue_bio+0x99/0x620
 [<ffffffff81266f00>] generic_make_request+0x240/0x5a0
 [<ffffffffa01c3f97>] ? dm_merge_bvec+0xc7/0x100 [dm_mod]
 [<ffffffff812672d0>] submit_bio+0x70/0x120
 [<ffffffffa03b9c53>] xfs_submit_ioend_bio+0x33/0x40 [xfs]
 [<ffffffffa03b9d2c>] xfs_submit_ioend+0xcc/0x140 [xfs]
 [<ffffffffa03ba678>] xfs_vm_writepage+0x308/0x590 [xfs]
 [<ffffffff81133967>] __writepage+0x17/0x40
 [<ffffffff81134c2d>] write_cache_pages+0x1fd/0x4c0
 [<ffffffff8100bdae>] ? call_function_single_interrupt+0xe/0x20
 [<ffffffff81133950>] ? __writepage+0x0/0x40
 [<ffffffff81134f14>] generic_writepages+0x24/0x30
 [<ffffffffa03b98ad>] xfs_vm_writepages+0x5d/0x80 [xfs]
 [<ffffffff81134f41>] do_writepages+0x21/0x40
 [<ffffffff811b52ad>] writeback_single_inode+0xdd/0x290
 [<ffffffff811b56ad>] writeback_sb_inodes+0xbd/0x170
 [<ffffffff811b580b>] writeback_inodes_wb+0xab/0x1b0
 [<ffffffff811b5c03>] wb_writeback+0x2f3/0x410
 [<ffffffff81528240>] ? thread_return+0x4e/0x76e
 [<ffffffff81084fb2>] ? del_timer_sync+0x22/0x30
 [<ffffffff811b5ec5>] wb_do_writeback+0x1a5/0x240
 [<ffffffff811b5fc3>] bdi_writeback_task+0x63/0x1b0
 [<ffffffff8109b327>] ? bit_waitqueue+0x17/0xd0
 [<ffffffff81143c40>] ? bdi_start_fn+0x0/0x100
 [<ffffffff81143cc6>] bdi_start_fn+0x86/0x100
 [<ffffffff81143c40>] ? bdi_start_fn+0x0/0x100
 [<ffffffff8109b0f6>] kthread+0x96/0xa0
 [<ffffffff8100c20a>] child_rip+0xa/0x20
 [<ffffffff8109b060>] ? kthread+0x0/0xa0
 [<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task ivanserver:6969 blocked for more than 120 seconds.
      Tainted: P           ---------------    2.6.32-424.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ivanserver      D 0000000000000000     0  6969   6947 0x00000000
 ffff881f4ef179d8 0000000000000082 0000000000000000 000000000000000f
 0000000000000001 ffff8800622143c0 0000000000000001 ffff880062216840
 ffff881f4eeae638 ffff881f4ef17fd8 000000000000fbc8 ffff881f4eeae638
Call Trace:
 [<ffffffff815290e5>] schedule_timeout+0x215/0x2e0
 [<ffffffff8152a0e2>] __down+0x72/0xb0
 [<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]
 [<ffffffff810a1591>] down+0x41/0x50
 [<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]
 [<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]
 [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
 [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
 [<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]
 [<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]
 [<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]
 [<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]
 [<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]
 [<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]
 [<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]
 [<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
 [<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffff8100bdae>] ? call_function_single_interrupt+0xe/0x20
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]
 [<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0
 [<ffffffff8118a5b0>] ? fget+0x30/0x80
 [<ffffffff8119f429>] sys_getdents+0x89/0xf0
 [<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ivanserver:6972 blocked for more than 120 seconds.
      Tainted: P           ---------------    2.6.32-424.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ivanserver      D 0000000000000008     0  6972   6948 0x00000000
 ffff88100f1219d8 0000000000000086 0000000000000000 000000000000000f
 0000000000000001 ffff8810788143c0 0000000000000001 ffff881078816840
 ffff88100fd62638 ffff88100f121fd8 000000000000fbc8 ffff88100fd62638
Call Trace:
 [<ffffffff815290e5>] schedule_timeout+0x215/0x2e0
 [<ffffffff8152a0e2>] __down+0x72/0xb0
 [<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]
 [<ffffffff810a1591>] down+0x41/0x50
 [<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]
 [<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]
 [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
 [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
 [<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]
 [<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]
 [<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]
 [<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]
 [<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]
 [<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]
 [<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]
 [<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
 [<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]
 [<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0
 [<ffffffff8119f429>] sys_getdents+0x89/0xf0
 [<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ivanserver:6985 blocked for more than 120 seconds.
      Tainted: P           ---------------    2.6.32-424.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ivanserver      D 0000000000000000     0  6985   6963 0x00000000
 ffff8810103ad9d8 0000000000000082 0000000000000000 000000000000000f
 0000000000000001 ffff8800622143c0 0000000000000001 ffff880062216840
 ffff88100e01e638 ffff8810103adfd8 000000000000fbc8 ffff88100e01e638
Call Trace:
 [<ffffffff815290e5>] schedule_timeout+0x215/0x2e0
 [<ffffffff8152a0e2>] __down+0x72/0xb0
 [<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]
 [<ffffffff810a1591>] down+0x41/0x50
 [<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]
 [<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]
 [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
 [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
 [<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]
 [<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]
 [<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]
 [<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]
 [<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]
 [<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]
 [<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]
 [<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
 [<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]
 [<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0
 [<ffffffff8119f429>] sys_getdents+0x89/0xf0
 [<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ivanserver:6989 blocked for more than 120 seconds.
      Tainted: P           ---------------    2.6.32-424.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ivanserver      D 0000000000000008     0  6989   6981 0x00000000
 ffff8810101279d8 0000000000000086 0000000000000000 000000000000000f
 0000000000000001 ffff8810788143c0 0000000000000001 ffff881078816840
 ffff8810252a4638 ffff881010127fd8 000000000000fbc8 ffff8810252a4638
Call Trace:
 [<ffffffff815290e5>] schedule_timeout+0x215/0x2e0
 [<ffffffff8152a0e2>] __down+0x72/0xb0
 [<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]
 [<ffffffff810a1591>] down+0x41/0x50
 [<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]
 [<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]
 [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
 [<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
 [<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]
 [<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]
 [<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]
 [<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]
 [<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]
 [<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]
 [<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]
 [<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
 [<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffff8119f020>] ? filldir+0x0/0xe0
 [<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]
 [<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0
 [<ffffffff8119f429>] sys_getdents+0x89/0xf0
 [<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
<Prev in Thread] Current Thread [Next in Thread>