[Top] [All Lists]

Re: XFS Freezing occasionally hangs

To: xfs@xxxxxxxxxxx
Subject: Re: XFS Freezing occasionally hangs
From: Ryan Campbell <ryan.campbell@xxxxxxxxx>
Date: Fri, 25 Jan 2013 12:04:28 -0600
Delivered-to: xfs@xxxxxxxxxxx
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=yK2rmJjvfxfOJ8Bryr7n/IGJUylpXkA/FMAC4feXvqk=; b=twPex3sw51VR1E/SBL0+IxVCpsSjsDoDvSXqgkEa5R8OUQzLFWjb+kQfk3YtnCS3Q1 vdGQJIHj8s0A1vMfrDeZJ7cSdKvhxBYCQG/NEM8GDzQYg/lLnAzAFRdFMd6EODel5aFC rh4llBCvT/nAUEKFw3XlUdbxazu3yATfwANOmseV1XCSapThNXCcuIK8Tm7kWGb6gGCh PjMhaimkp3kJOi038L4Oz6jI+e2kzAKgBJCXgfoiHIIzwp2kbVAAS6S+U/F6J/Iuw8UK 7bziDqtRtm6z0nf9wC8mSgKGyGBhos6WtOiuEG8lbdjRlXwr/5+kzqd3KOu8ygBKiYfi nUQg==
In-reply-to: <CANHK=J2ZvA_aafnEG3CqWi5O3iynXeNnjjCGPJiOWO=Hu_K7Lg@xxxxxxxxxxxxxx>
References: <CANHK=J0ZZ+Wx=GYPGh+m-=Dofn_FUnY3H=tdEM5JTrwj=8yq+Q@xxxxxxxxxxxxxx> <CANHK=J2ZvA_aafnEG3CqWi5O3iynXeNnjjCGPJiOWO=Hu_K7Lg@xxxxxxxxxxxxxx>
In #xfs, bpm recommended I run echo t > /proc/sysrq-trigger. Here's the output of that command on the host with the stuck xfs_freeze. : https://gist.github.com/4636507

On Fri, Jan 25, 2013 at 11:08 AM, Ryan Campbell <ryan.campbell@xxxxxxxxx> wrote:
Is there any additional information I can provide to debug this issue? We are seeing it quite often.

On Thu, Jan 24, 2013 at 9:38 AM, Ryan Campbell <ryan.campbell@xxxxxxxxx> wrote:
We use XFS on EC2 EBS running on Arch Linux (kernel 3.6.7-1-ARCH). Before we take an EC2 snapshot, we call xfs_freeze on the mount so that we can have a consistent snapshot.

On instances running for around 30 days, we see the XFS freeze hang, resulting in a skyrocketing load (but not CPU). We can't kill the xfs_freeze process. The only route available is usually to force a reboot of the server without cleaning unmounting.

There are backtraces in dmesg complaining both of hung xfs processes, and hung java processes. https://gist.github.com/3d46830eac52df44d30f

Is this a known issue? We call xfs_freeze quite often on thousands of XFS volumes, so I wouldn't be surprised if we have encountered an edge case considering how often we do this.

Any help appreciated. I'm recampbell on #xfs/freenode.


<Prev in Thread] Current Thread [Next in Thread>