xfs
[Top] [All Lists]

Re: long hangs when deleting large directories (3.0-rc3)

To: Dave Chinner <david@xxxxxxxxxxxxx>
Subject: Re: long hangs when deleting large directories (3.0-rc3)
From: Markus Trippelsdorf <markus@xxxxxxxxxxxxxxx>
Date: Mon, 20 Jun 2011 08:03:51 +0200
Cc: xfs@xxxxxxxxxxx
Dkim-signature: v=1; a=rsa-sha256; c=simple; d=mail.ud10.udmedia.de; h= date:from:to:cc:subject:message-id:references:mime-version: content-type:content-transfer-encoding:in-reply-to; q=dns/txt; s= beta; bh=UUDKIlLCNgaoYcc0MYMT37xwj8b99DmQHeuzIMLE4cU=; b=DXKIY+d cgI5R7j8Vx0ZoV+f33STiT09vssMq6BQEZ8lBhuygoSUxYnx2YOFAFbooSvTKe4z KDOz83ZhtiW5pY0CXXBuOcWbcSoErY9G+EqZSTR+s/Sh5fvAWi6k5loAFgpXzCEr G0cP6W+KQGZbMr6XRwaWXC46b7gVjojCfO7g=
In-reply-to: <20110620023625.GP561@dastard>
References: <20110618141950.GA1685@xxxxxxxxxxxxxx> <20110619222447.GI561@dastard> <20110620005415.GA1730@xxxxxxxxxxxxxx> <20110620013449.GO561@dastard> <20110620020236.GB1730@xxxxxxxxxxxxxx> <20110620023625.GP561@dastard>
On 2011.06.20 at 12:36 +1000, Dave Chinner wrote:
> On Mon, Jun 20, 2011 at 04:02:36AM +0200, Markus Trippelsdorf wrote:
> > On 2011.06.20 at 11:34 +1000, Dave Chinner wrote:
> > > On Mon, Jun 20, 2011 at 02:54:15AM +0200, Markus Trippelsdorf wrote:
> > > > On 2011.06.20 at 08:24 +1000, Dave Chinner wrote:
> > > > > On Sat, Jun 18, 2011 at 04:19:50PM +0200, Markus Trippelsdorf wrote:
> > > > > > Running the latest git kernel (3.0-rc3) my machine hangs for long
> > > > > > periods (1-2 sec) whenever I delete a large directory recursively 
> > > > > > on my
> > > > > > xfs partition. During the hang I cannot move the mouse pointer or 
> > > > > > use
> > > > > > the keyboard (but the music keeps playing without stuttering). A 
> > > > > > quick
> > > > > > way to reproduce is to "rm -fr" a kernel tree. 
> > > > > 
> > > > > So what is the system doing when it "hangs"? Is it CPU bound (e.g.
> > > > > cpu scheduler issue)? Is the system running out of memory and
> > > > > stalling everything in memory reclaim? What IO is occurring?
> > > > 
> > > > It's totally idle otherwise; just a desktop with a single xterm. The
> > > > machine has four cores (and also runs with "CONFIG_PREEMPT=y"), so I
> > > > don't think it is CPU bound at all. It has 8GB of memory (and the
> > > > "hangs" even occur after reboot when most of it is free). No other IO
> > > > activity is occurring.
> > > 
> > > Sure, the system might be otherwise idle, but what I was asking is
> > > what load does the "rm -rf" cause. What IO does it cause? is it cpu
> > > bound? etc.
> > 
> > I have not measured this, so I cannot tell.
> 
> And so you are speculating as to the cause of the problem. What I'm
> trying to do is work from the bottom up to ensure that the layers
> below the fs are not the cause of the problem.
> 
> > > > > Is your partition correctly sector aligned for however your drive
> > > > > maps it's 4k sectors?
> > > > 
> > > > Yes, it's a GPT partition that is aligned to 1MB.
> > > 
> > > Ok, that is fine, but the big question now is how does the drive
> > > align sector 0? Is that 4k aligned, or is it one of those drives
> > > that aligns an odd 512 byte logical sector to the physical 4k sector
> > > boundary (i.e. sector 63 is 4k aligned to work with msdos
> > > partitions). FYI, some drives have jumpers on them to change this
> > > odd/even sector alignment configuration.....
> > 
> > No, it's none of those (it's a Seagate Barracuda Green ST1500). Sector 0
> > is 4k aligned for sure. The odd 512 byte offset was present only on some
> > first generation drives. 
> > But I think the whole alignment issue is a red herring, because I cannot
> > reproduce the "hangs" on the next partition on the same drive. This
> > partition is larger and contains my music and film collection (so mostly
> > static content and no traffic).
> 
> Which also means you might have one unaligned and one aligned
> partition.  i.e. the test results you have presented does not
> necessarily point at a filesystem problem. We always ask for exact
> details of your storage subsystem for these reasons - so we can
> understand if there's something that you missed or didn't think was
> important enough to tell us. You may have already checked those
> things, but we don't know that if you don't tell us....

Understood.

> So, is the sector alignment of the second partition the same as the
> first partition?

Yes.

> > And as I wrote in my other reply to this
> > thread: »it appears that the observed "hangs" are the result of a
> > strongly aged file-system.«
> 
> There is no evidence that points to any cause. Hell, I don't even
> know what you consider a "strongly aged filesystem" looks like....
> 
> If the alignment is the cause of the problem, you should be able to
> see a difference in performance when doing random 4k synchronous
> writes to a large file on differently aligned partitions. Can you
> run the same random 4k sync write test on both partitions (make sure
> barriers are enabled) and determine if they perform the same?
> 
> If the filesystem layout is the cause of the problem, you should be
> able to take a metadump of the problematic filesystem, restore it to
> a normal 512 sector drive and reproduce the "rm -rf" problem. Can
> you try this as well?

OK. I was able to reproduce the same hang on a conventional 512 sector drive.
The partition that I've used was the predecessor to the one on the 4k drive. So
it saw roughly the same usage pattern.
This is the output of "dstat -cdyl -C 0,1,2,3 -D sdc --disk-tps" during the
hang:

-------cpu0-usage--------------cpu1-usage--------------cpu2-usage--------------cpu3-usage------
 --dsk/sdc-- ---system-- ---load-avg--- --dsk/sdc--
usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys 
idl wai hiq siq| read  writ| int   csw | 1m   5m  15m |reads writs

  0   0 100   0   0   0:  1   0  99   0   0   0:  0   1  99   0   0   0:  1   0 
 99   0   0   0|   0     0 | 249   354 |0.33 0.58 0.38|   0     0 
  0   0 100   0   0   0:  0   0 100   0   0   0:  0   0 100   0   0   0:  0   0 
100   0   0   0|   0     0 | 244   228 |0.33 0.58 0.38|   0     0 
  1   2  97   0   0   0:  0   1  99   0   0   0:  0   1  99   0   0   0:  0   1 
 99   0   0   0|   0     0 | 559   614 |0.33 0.58 0.38|   0     0 
  0   0 100   0   0   0:  1   0  99   0   0   0:  1   0  99   0   0   0:  1   0 
 99   0   0   0|   0     0 | 341   426 |0.33 0.58 0.38|   0     0 
  1   0  99   0   0   0:  1   4  95   0   0   0:  0   1  99   0   0   0:  1  16 
 83   0   0   0|   0     0 | 874   796 |0.33 0.58 0.38|   0     0 
  2  50  49   0   0   0:  1   9  90   0   0   0:  1   9  90   0   0   0:  1  23 
 76   0   0   0|   0  6400k|2803  2073 |0.46 0.60 0.39|   0    25 
  1  29  70   0   0   0:  1   1  98   0   0   0:  1   9  90   0   0   0:  1  53 
 46   0   0   0|   0  6400k|2047  1414 |0.46 0.60 0.39|   0    25 
  0   4  96   0   0   0:  0   0 100   0   0   0:  1  19  80   0   0   0:  0  80 
 20   0   0   0|   0  2048k|1425   685 |0.46 0.60 0.39|   0     8 
  2   1  97   0   0   0:  1   6  93   0   0   0:  0   5  95   0   0   0:  0  83 
 17   0   0   0|   0  4608k|1624   849 |0.46 0.60 0.39|   0    18 
  2  45  53   0   0   0:  1  16  83   0   0   0:  3  20  77   0   0   0:  1  15 
 84   0   0   0|   0  6400k|2420  1984 |0.46 0.60 0.39|   0    26 
  1  19  80   0   0   0:  2   8  90   0   0   0:  0  33  67   0   0   0:  0  33 
 67   0   0   0|   0  6400k|2694  2134 |0.59 0.63 0.40|   0    25 
  2   7  91   0   0   0:  2   1  97   0   0   0:  1   0  99   0   0   0:  0  49 
 10  41   0   0|   0  8269k|1865  1571 |0.59 0.63 0.40|   0   363 
  1   1  98   0   0   0:  1   1  98   0   0   0:  1   1  98   0   0   0:  0   1 
  0  99   0   0|   0  4778k|1509  1639 |0.59 0.63 0.40|   0   410 
  2   0  98   0   0   0:  2   1  97   0   0   0:  1   1  98   0   0   0:  2   0 
  0  98   0   0|   0  5318k|1663  1809 |0.59 0.63 0.40|   0   426 
  1   1  98   0   0   0:  2   7  91   0   0   0:  1   0  99   0   0   0:  1   0 
  0  99   0   0|   0  5446k|1659  1806 |0.59 0.63 0.40|   0   432 
  0   1  99   0   0   0:  1   0  99   0   0   0:  2   0  98   0   0   0:  0   1 
 17  82   0   0|   0  5472k|1572  1837 |0.62 0.63 0.40|   0   439 
  2   0  98   0   0   0:  2   2  96   0   0   0:  0   1  99   0   0   0:  0   1 
 99   0   0   0|   0   397k|1058  1049 |0.62 0.63 0.40|   0    36 
  1   1  98   0   0   0:  1   1  98   0   0   0:  1   1  98   0   0   0:  0   0 
100   0   0   0|   0     0 | 617   689 |0.62 0.63 0.40|   0     0 
  9   4  87   0   0   0:  4   0  96   0   0   0:  1   1  98   0   0   0:  8   6 
 87   0   0   0|   0     0 |1234  1961 |0.62 0.63 0.40|   0     0 
  0   1  99   0   0   0:  1   1  98   0   0   0:  0   1  99   0   0   0:  0   1 
 99   0   0   0|   0     0 | 391   403 |0.62 0.63 0.40|   0     0 
  1   0  99   0   0   0:  1   1  98   0   0   0:  0   0 100   0   0   0:  0   0 
100   0   0   0|   0     0 | 366   375 |0.57 0.62 0.40|   0     0 



-- 
Markus

<Prev in Thread] Current Thread [Next in Thread>