[Top] [All Lists]

Re: mdadm/raid5 hung/d-state

To: "Justin Piszcz" <jpiszcz@xxxxxxxxxxxxxxx>
Subject: Re: mdadm/raid5 hung/d-state
From: "Fabiano Silva" <fabiano@xxxxxxxxxxxx>
Date: Fri, 9 Nov 2007 12:09:24 -0200
Cc: "Carlos Carvalho" <carlos@xxxxxxxxxxxxxx>, "Jeff Lessem" <Jeff@xxxxxxxxxx>, root@xxxxxxxxxxxx, "Dan Williams" <dan.j.williams@xxxxxxxxx>, "BERTRAND Joël" <joel.bertrand@xxxxxxxxxxx>, "Neil Brown" <neilb@xxxxxxx>, linux-kernel@xxxxxxxxxxxxxxx, linux-raid@xxxxxxxxxxxxxxx, xfs@xxxxxxxxxxx
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:sender:to:subject:cc:in-reply-to:mime-version:content-type:references:x-google-sender-auth; bh=0Y6W1bg1EB8oyZ+kPaaOFGaRejveHLlNPWtaLM3r6MU=; b=UGskOKJpj62NjBlAuTwMORWsOakzg5mJh0+6xusb9x4hzl9U4kHS2WyK9JQZgK1IjCWJweZ3jI7bTC274ylU430cybp/t4XNS77BiMJnEUyy1z4uE0SN//ai52Sax/rH0OXCmEmfxa/9Rc062JWJxMmFUCgocoi70jTvogSeEaM=
Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:sender:to:subject:cc:in-reply-to:mime-version:content-type:references:x-google-sender-auth; b=DvvIO4nZO2Wz23PtQv1f0Guzhc7AELqJBbHArF+5LrL2fWTGBn/QC36RmeXux3pWHGay/IQRuP8cC/veXxPJe0Km+f4YgyRXFoBPeAE3V5fiK014V9turJWqQeLLN99W6V6rxsE65N29q7PQ6Q0dm44o1gr2ETKGnFZmV5rQHFM=
In-reply-to: <Pine.LNX.4.64.0711090414090.26103@p34.internal.lan>
References: <Pine.LNX.4.64.0711040658180.30831@p34.internal.lan> <Pine.LNX.4.64.0711041651250.23496@p34.internal.lan> <e9c3a7c20711051035m78ba90ck68f4fbc10480462a@mail.gmail.com> <Pine.LNX.4.64.0711051335450.11422@p34.internal.lan> <e9c3a7c20711051619u7054aab9l208b604b9e58fb61@mail.gmail.com> <47303FB8.7000801@systella.fr> <1194398700.2970.18.camel@dwillia2-linux.ch.intel.com> <47314653.80905@Lessem.org> <18227.33346.994456.270194@fisica.ufpr.br> <Pine.LNX.4.64.0711090414090.26103@p34.internal.lan>
Reply-to: fabiano@xxxxxxxxxxxx
Sender: xfs-bounce@xxxxxxxxxxx
On Nov 9, 2007 7:14 AM, Justin Piszcz <jpiszcz@xxxxxxxxxxxxxxx> wrote:
> On Thu, 8 Nov 2007, Carlos Carvalho wrote:
> > Jeff Lessem (Jeff@xxxxxxxxxx) wrote on 6 November 2007 22:00:
> > >Dan Williams wrote:
> > > > The following patch, also attached, cleans up cases where the code looks
> > > > at sh->ops.pending when it should be looking at the consistent
> > > > stack-based snapshot of the operations flags.
> > >
> > >I tried this patch (against a stock 2.6.23), and it did not work for
> > >me.  Not only did I/O to the effected RAID5 & XFS partition stop, but
> > >also I/O to all other disks.  I was not able to capture any debugging
> > >information, but I should be able to do that tomorrow when I can hook
> > >a serial console to the machine.
> > >
> > >I'm not sure if my problem is identical to these others, as mine only
> > >seems to manifest with RAID5+XFS.  The RAID rebuilds with no problem,
> > >and I've not had any problems with RAID5+ext3.
> >
> > Us too! We're stuck trying to build a disk server with several disks
> > in a raid5 array, and the rsync from the old machine stops writing to
> > the new filesystem. It only happens under heavy IO. We can make it
> > lock without rsync, using 8 simultaneous dd's to the array. All IO
> > stops, including the resync after a newly created raid or after an
> > unclean reboot.
> >
> > We could not trigger the problem with ext3 or reiser3; it only happens
> > with xfs.

In our case all process using md4, including md4_resync, stay in D state.
Call Trace:
  [<ffffffff803615ac>] __generic_unplug_device+0x13/0x24
  [<ffffffff803622cf>] generic_unplug_device+0x18/0x28
  [<ffffffff803f2cf7>] get_active_stripe+0x22b/0x472
see dmesg (sysrq t) attached.

We can reproduce this problem in two machines with the same configuration:
  - 2 x Dual-Core Opteron 2.8GHz
  - 8GB memory
  - 3ware 9000 with 10 x 750GB sata disks
  - Debian Etch x86_64
  - raid5 + xfs (/dev/md4)
in all these stock kernel's:
  -,,, 2.6.24-rc2
  - for i in f{0..7}; do (dd bs=1M count=100000 if=/dev/zero of=$i &); done

If we increase /sys/block/md4/md/stripe_cache_size the device and process
back to work.

Attachment: dmesg_sysrq_t.txt.gz
Description: GNU Zip compressed data

<Prev in Thread] Current Thread [Next in Thread>