Performance problem with multiple parallel rm -rf's

[Justin Piszcz]

On Wed, 2 Dec 2009, Jens Rosenboom wrote:

> On a large 13TB XFS volume that is being used for backups, I am seeing bad 
> performance if multiple "rm -rf" processes are running in parallel. The 
> backups are being done with rsnapshot and the first operation it does is 
> removing the oldest snapshot. A single rsnapshot does this in reasonable 
> time, but if four jobs are started at the same time, all their rm processes 
> run for hours without making much progress.
>
> This seems to be related to the planned optimizations in
>
> http://xfs.org/index.php/Improving_Metadata_Performance_By_Reducing_Journal_Overhead
>
> Are there any other tuning options I might try? I'm already using 
> "noatime,nodiratime,nobarrier,logbufs=8,logbsize=256k" as mount options and 
> did enable lazy_counters for the fs.
>
> _______________________________________________
> xfs mailing list
> xfs at oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
>

For faster rm performance you need a bigger log (128m-256mb), that is 
shown to increase delete performance.  But I am not sure there is a way to 
change the size of it once the array has been created.  An alternative may 
be creating another volume for a log.

Also: nodiratime is not needed, as it is implied by noatime.

Justin.