xfs
[Top] [All Lists]

Re: On-stack work item completion race? (was Re: XFS crash?)

To: Dave Chinner <david@xxxxxxxxxxxxx>
Subject: Re: On-stack work item completion race? (was Re: XFS crash?)
From: Tejun Heo <tj@xxxxxxxxxx>
Date: Wed, 25 Jun 2014 10:18:36 -0400
Cc: Austin Schuh <austin@xxxxxxxxxxxxxxxx>, xfs <xfs@xxxxxxxxxxx>, linux-kernel@xxxxxxxxxxxxxxx
Delivered-to: xfs@xxxxxxxxxxx
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=q6bZajDqpH/OQoCHZ1RFpWe5oIJOql8kruXDUL0eHQ0=; b=f3pHlywDXaqvFV0ZxDvcOwrWvrj8shYuhoqjcV2UYnshXRlPLzwPD381Xhtc1iNK7L ohF5sYvrsmNmUoLAleCvXc7qml213UgYBYkTa8q0B5lFYPYWND07YzZWFPS2EwhrOZ08 KbPne1LFIUFko9uY6lIloxZUewYIpO5jgWw3zy+GI1RtzRplqjw5xRycugwwaPRI6tQX XrFRhEscKQus0XhuCekrpBdDm7kuUdVLiK/v7srdECmU0aBCdlwcCkRQnbs1xQbIVEp2 Drnb1dckK72epaOLtyNe/j/xrbzT2Ni+fuAxrZxuTxb1J3MKTa2V6VLetaT8nx1L3bRi rOgQ==
In-reply-to: <20140625055641.GL9508@dastard>
References: <20140513034647.GA5421@dastard> <CANGgnMZ0q9uE3NHj2i0SBK1d0vdKLx7QBJeFNb+YwP-5EAmejQ@xxxxxxxxxxxxxx> <20140513063943.GQ26353@dastard> <CANGgnMYn++1++UyX+D2d9GxPxtytpQJv0ThFwdxM-yX7xDWqiA@xxxxxxxxxxxxxx> <20140513090321.GR26353@dastard> <CANGgnMZqQc_NeaDpO_aX+bndmHrQ9VWo9mkfxhPBkRD-J=N6sQ@xxxxxxxxxxxxxx> <CANGgnMZ8OwzfBj5m9H7c6q2yahGhU7oFZLsJfVxnWoqZExkZmQ@xxxxxxxxxxxxxx> <20140624030240.GB9508@dastard> <20140624032521.GA12164@xxxxxxxxxxxxxx> <20140625055641.GL9508@dastard>
Sender: Tejun Heo <htejun@xxxxxxxxx>
User-agent: Mutt/1.5.23 (2014-03-12)
Hello, Dave.

On Wed, Jun 25, 2014 at 03:56:41PM +1000, Dave Chinner wrote:
> Hmmm - that's different from my understanding of what the original
> behaviour WQ_MEM_RECLAIM gave us. i.e. that WQ_MEM_RECLAIM
> workqueues had a rescuer thread created to guarantee that the
> *workqueue* could make forward progress executing work in a
> reclaim context.

>From Documentation/workqueue.txt

  WQ_MEM_RECLAIM

        All wq which might be used in the memory reclaim paths _MUST_
        have this flag set.  The wq is guaranteed to have at least one
        execution context regardless of memory pressure.

So, all that's guaranteed is that the workqueue has at least one
worker executing its work items.  If that one worker is serving a work
item which can't make forward progress, the workqueue is not
guaranteed to make forward progress.

> The concept that the *work being executed* needs to guarantee
> forwards progress is something I've never heard stated before.
> That worries me a lot, especially with all the memory reclaim
> problems that have surfaced in the past couple of months....

I'd love to provide that but guaranteeing that at least one work is
always being executed requires unlimited task allocation (the ones
which get blocked gotta store their context somewhere).

> > As long as a WQ_RECLAIM workqueue dosen't depend upon itself,
> > forward-progress is guaranteed.
> 
> I can't find any documentation that actually defines what
> WQ_MEM_RECLAIM means, so I can't tell when or how this requirement
> came about. If it's true, then I suspect most of the WQ_MEM_RECLAIM
> workqueues in filesystems violate it. Can you point me at
> documentation/commits/code describing the constraints of
> WQ_MEM_RECLAIM and the reasons for it?

Documentation/workqueue.txt should be it but maybe we should be more
explicit.  The behavior is maintaining what the
pre-concurrency-management workqueue provided with static
per-workqueue workers.  Each workqueue reserved its workers (either
one per cpu or one globally) and it only supported single level of
concurrency on each CPU.  WQ_MEM_RECLAIM is providing equivalent
amount of forward progress guarantee and all the existing users
shouldn't have issues on this front.  If we have grown incorrect
usages from then on, we need to fix them.

Thanks.

-- 
tejun

<Prev in Thread] Current Thread [Next in Thread>