xfs
[Top] [All Lists]

Re: very slow file deletion on an SSD

To: Dave Chinner <david@xxxxxxxxxxxxx>
Subject: Re: very slow file deletion on an SSD
From: Joe Landman <joe.landman@xxxxxxxxx>
Date: Sat, 26 May 2012 21:49:57 -0400
Cc: xfs@xxxxxxxxxxx, linux-raid <linux-raid@xxxxxxxxxxxxxxx>
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=CRwPxrixWy/OZAkRe7Gw9geij4I9QCBqEFykEc6pqmI=; b=yGFeLA/+fGWhlRMUlV2DCm4mvRuoR6ImvrbpDrzzzitqjJyOMdPakXRYb1Vyo0irla oxVoc10m57pQTaZ/M0xtv4dWTpZXWdXZBvYmPu0zu3JABwpIEns1HISJe5bH6jtRR/me VA7i+YA5POCNO7KQMTBAuj0G9g4mxqs1mv+bq1UNG2evEnp+umPD7SDH7himI5w63LVW l0S7mg7TF2voRduQhV9IS5xUkTvm5VCmcl16drjWXpMWToS2mZRr3/zdDZDmjCUUrY6A jMO99xjZ4ekH2RSoO//qjmPm95h41o9/VEIgVcNMQZeM3TWs5Bho0xcgFayZzeyt2x+D J+XQ==
In-reply-to: <20120527000701.GS25351@dastard>
References: <4FBF60D1.80104@xxxxxxxxx> <20120526231838.GR25351@dastard> <4FC16683.9060800@xxxxxxxxx> <20120527000701.GS25351@dastard>
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:12.0) Gecko/20120430 Thunderbird/12.0.1
On 05/26/2012 08:07 PM, Dave Chinner wrote:
On Sat, May 26, 2012 at 07:25:55PM -0400, Joe Landman wrote:
[root@siFlash test]# ls -alF  | wc -l
59
[root@siFlash test]# /usr/bin/time rm -f *
^C0.00user 8.46system 0:09.55elapsed 88%CPU (0avgtext+0avgdata
2384maxresident)k
25352inputs+0outputs (0major+179minor)pagefaults 0swaps

It's burning an awful lot of CPU time during this remove.

[root@siFlash test]# ls -alF  | wc -l
48

So, 48 files were removed, it was basically CPU bound and one took
2.6 seconds.

So, how big are the files, and does the one that took 2.6s have tens
of thousands of extents ('xfs_bmap -vp *' will dump the extent maps
for all the files)?

Getting some sort of out of memory error with bmap

[root@siFlash test]# ls -alF
total 50466476
drwxr-xr-x 2 root root       4096 May 26 21:40 ./
drwxr-xr-x 3 root root         17 May 26 19:32 ../
-rw------- 1 root root 1073741824 May 26 19:36 2.r.49.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.50.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.51.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.52.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.53.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.54.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.55.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.56.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.57.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.58.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.59.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.60.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.61.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.62.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.63.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.64.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.65.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.66.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.67.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.68.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.69.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.70.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.71.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.72.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.73.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.74.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.75.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.76.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.77.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.78.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.79.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.80.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.81.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.82.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.83.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.84.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.85.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.86.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.87.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.88.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.89.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.90.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.91.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.92.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.93.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.94.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.95.0
-rw------- 1 root root 1073741824 May 26 19:36 2.r.96.0
-rw-r--r-- 1 root root          0 May 26 21:40 x

[root@siFlash test]# ls -alF > x

[root@siFlash test]# xfs_bmap -vp x
x:
EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS 0: [0..7]: 212681896..212681903 2 (7555752..7555759) 8 01111

[root@siFlash test]# xfs_bmap -vp 2.r.96.0
xfs_bmap: xfsctl(XFS_IOC_GETBMAPX) iflags=0x4 ["2.r.96.0"]: Cannot allocate memory

These are 3.1.8 from git (had same error with earlier version).





if not, can you use perf top to get an ida of the CPU usage profile
duing the rm by doing:

# perf record rm -f *
.....

and capturing the profile via:

# perf report>  profile.txt

And attaching te profile.txt file so we can see where all the CPU
time is being spent? You can find perf in your kernel source tree
under the tools subdir....

Cheers,

Dave.

<Prev in Thread] Current Thread [Next in Thread>