xfs
[Top] [All Lists]

xfs: two deadlock problems occur when kswapd writebacks XFS pages.

To: Dave Chinner <david@xxxxxxxxxxxxx>
Subject: xfs: two deadlock problems occur when kswapd writebacks XFS pages.
From: Masayoshi Mizuma <m.mizuma@xxxxxxxxxxxxxx>
Date: Tue, 17 Jun 2014 17:50:02 +0900
Cc: <xfs@xxxxxxxxxxx>, <linux-mm@xxxxxxxxx>
Delivered-to: xfs@xxxxxxxxxxx
User-agent: Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Thunderbird/24.2.0
I found two deadlock problems occur when kswapd writebacks XFS pages.
I detected these problems on RHEL kernel actually, and I suppose these
also happen on upstream kernel (3.16-rc1).

1.

A process (processA) has acquired read semaphore "xfs_cil.xc_ctx_lock"
at xfs_log_commit_cil() and it is waiting for the kswapd. Then, a
kworker has issued xlog_cil_push_work() and it is waiting for acquiring
the write semaphore. kswapd is waiting for acquiring the read semaphore
at xfs_log_commit_cil() because the kworker has been waiting before for
acquiring the write semaphore at xlog_cil_push(). Therefore, a deadlock
happens.

The deadlock flow is as follows.

  processA              | kworker                  | kswapd              
  ----------------------+--------------------------+----------------------
| xfs_trans_commit      |                          |
| xfs_log_commit_cil    |                          |
| down_read(xc_ctx_lock)|                          |
| xlog_cil_insert_items |                          |
| xlog_cil_insert_format_items                     |
| kmem_alloc            |                          |
| :                     |                          |
| shrink_inactive_list  |                          |
| congestion_wait       |                          |
| # waiting for kswapd..|                          |
|                       | xlog_cil_push_work       |
|                       | xlog_cil_push            |
|                       | xfs_trans_commit         |
|                       | down_write(xc_ctx_lock)  |
|                       | # waiting for processA...|
|                       |                          | shrink_page_list
|                       |                          | xfs_vm_writepage
|                       |                          | xfs_map_blocks
|                       |                          | xfs_iomap_write_allocate
|                       |                          | xfs_trans_commit
|                       |                          | xfs_log_commit_cil
|                       |                          | down_read(xc_ctx_lock)
V(time)                 |                          | # waiting for kworker...
  ----------------------+--------------------------+-----------------------

To fix this, should we up the read semaphore before calling kmem_alloc()
at xlog_cil_insert_format_items() to avoid blocking the kworker? Or,
should we the second argument of kmem_alloc() from KM_SLEEP|KM_NOFS
to KM_NOSLEEP to avoid waiting for the kswapd. Or...

2. 

A kworker (kworkerA), whish is a writeback thread, is waiting for
the XFS allocation thread (kworkerB) while it writebacks XFS pages.
kworkerB has started the allocation and it is waiting for kswapd to
allocate free pages. kswapd has started writeback XFS pages and
it is waiting for more log space. The reason why exhaustion of the
log space is both the writeback thread and kswapd are stuck, so
some processes, who have allocated the log space and are requesting
free pages, are also stuck.

The deadlock flow is as follows.

  kworkerA              | kworkerB                 | kswapd            
  ----------------------+--------------------------+-----------------------
| wb_writeback          |                          |
| :                     |                          |
| xfs_vm_writepage      |                          |
| xfs_map_blocks        |                          |
| xfs_iomap_write_allocate                         |
| xfs_bmapi_write       |                          |
| xfs_bmapi_allocate    |                          |
| wait_for_completion   |                          |
| # waiting for kworkerB...                        |
|                       | xfs_bmapi_allocate_worker|
|                       | :                        |
|                       | xfs_buf_get_map          |
|                       | xfs_buf_allocate_memory  |
|                       | alloc_pages_current      |
|                       | :                        |
|                       | shrink_inactive_list     |
|                       | congestion_wait          |
|                       | # waiting for kswapd...  |
|                       |                          | shrink_page_list
|                       |                          | xfs_vm_writepage
|                       |                          | :
|                       |                          | xfs_log_reserve
|                       |                          | :
|                       |                          | xlog_grant_head_check
|                       |                          | xlog_grant_head_wait
|                       |                          | # waiting for more
|                       |                          | # space...
V(time)                 |                          |
  ----------------------+--------------------------+-----------------------

I don't have any ideas to fix this...

Thanks,
Masayoshi Mizuma

<Prev in Thread] Current Thread [Next in Thread>