[Top] [All Lists]

Re: rm -f * on large files very slow on XFS + MD RAID 6 volume of 15x 4T

To: Dave Chinner <david@xxxxxxxxxxxxx>
Subject: Re: rm -f * on large files very slow on XFS + MD RAID 6 volume of 15x 4TB of HDDs (52TB)
From: Ivan Pantovic <gyro.ivan@xxxxxxxxx>
Date: Wed, 23 Apr 2014 09:23:41 +0200
Cc: Speedy Milan <speedy.milan@xxxxxxxxx>, linux-kernel@xxxxxxxxxxxxxxx, xfs@xxxxxxxxxxx
Delivered-to: xfs@xxxxxxxxxxx
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=OZaLEPx4RF454N3jMOd7Un3e9Uonu7voW0vIEgg8jxM=; b=Jese46ZANXycqSAW3s/KvyxX5dx3YC+Enw3aVtbnlPzGMHgzS6Sh8o9nO0qGNOE3CO XNpec+grD6AX6cr/X+dtQ6OSoYGpuCogABRId57rUf7KSWZM5YT9bN2WMVMslmNf1df0 zjQmLMAXNlwYAkIBK/m/EKHaMxgnQuVJbDWutI1tacgMUm4gKP5Qw870b/cJljeVgT1t 7EtFJRc8ZMADtFretC5sObuTXMGG+OS6XOx3Q0meud3KXeJtx/YvEImpAEF97lNAgplA N9wDMOQS9d7/Ne/IQL8UMIFhOn8m2PPl06B72slnuXqJaliV+39BccyNI551OPNdZpD7 /z/w==
In-reply-to: <20140423021835.GI15995@dastard>
References: <CAHuzUScfp19c_th_pfsZs05+yDz34MuEH-P1f+FF1dcivfH=5Q@xxxxxxxxxxxxxx> <20140423021835.GI15995@dastard>
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130215 Thunderbird/17.0.3

[root@drive-b ~]# xfs_db -r /dev/md0
xfs_db> frag
actual 11157932, ideal 11015175, fragmentation factor 1.28%

this is current level of fragmentation ... is it bad?

some say over 1% is candidate for defrag? ...

we can leave it like this and wait for a next full backup and then check on the fragmentation of that file.

On 04/23/2014 04:18 AM, Dave Chinner wrote:
[cc xfs@xxxxxxxxxxx]

On Mon, Apr 21, 2014 at 10:58:53PM +0200, Speedy Milan wrote:
I want to report very slow deletion of 24 50GB files (in total 12 TB),
all present in the same folder.
total = 1.2TB?

OS is CentOS 6.4, with upgraded kernel 3.13.1.

The hardware is a Supermicro server with 15x 4TB WD Se drives in MD
RAID 6, totalling 52TB of free space.

XFS is formated directly on the RAID volume, without LVM layers.

Deletion was done with rm -f * command, and it took upwards of 1 hour
to delete the files.

File system was filled completely prior to deletion.
Oh, that's bad. it's likely you fragmented the files into
millions of extents?

rm was mostly waiting (D state), probably for kworker threads, and
No, waiting for IO.

iostat was showing big HDD utilization numbers and very low throughput
so it looked like a random HDD workload was in effect.
Yup, smells like file fragmentation. Non-fragmented 50GB files
should be removed in a few milliseconds. but if you've badly
fragmented the files, there could be 10 million extents in a 50GB
file. A few milliseconds per extent removal gives you....



<Prev in Thread] Current Thread [Next in Thread>