xfs
[Top] [All Lists]

Re: Deleting files with extended attributes is dead slow

To: Christoph Hellwig <hch@xxxxxxxxxxxxx>
Subject: Re: Deleting files with extended attributes is dead slow
From: Bernd Schubert <bernd.schubert@xxxxxxxxxxxxxxxxxx>
Date: Wed, 17 Aug 2011 15:05:28 +0200
Cc: linux-xfs@xxxxxxxxxxx
In-reply-to: <20110816161357.GA18201@xxxxxxxxxxxxx>
References: <j23qs9$1c3$1@xxxxxxxxxxxxxxx> <20110812204746.GB30615@xxxxxxxxxxxxx> <20110816161357.GA18201@xxxxxxxxxxxxx>
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.18) Gecko/20110626 Iceowl/1.0b2 Icedove/3.1.11
On 08/16/2011 06:13 PM, Christoph Hellwig wrote:
The patch below should help with the performance of deleting files
that have external attributes.  But I still wonder why you see
attributes outside the inode if you are using 512 byte inodes and around
256k of attrs.  Can you check tat you really have external blocks for
attributes using xfs_bmap -a on the files in your workload.  Can you
also check if this goes away using 1k or 2k inode sizes?

Thanks! Deletes behave much better now!

Compared to ext4, it is still rather slow. Below are xfs and ext4 benchmarks using our patched bonnie. Here is getfattr and xfs_bmap output of one of those bonnie files:

(squeeze-x86_64)fslab2:~# getfattr -d 
/mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n
getfattr: Removing leading '/' from absolute path names
# file: mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n
user.bonnie=0sMDAwMDkAAACotThkkX8AAFAvbQAAAAAAUC9tAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIAA7usEhkXAAAAAAAAAAAEAAAA0C9tAAAAAADRRwAAAAAAAGAQbQAAAAAAmK44ZJF/AAAAAAAAAAAAAAAAAAAAAAAAqLU4ZJF/AACotThkkX8AAOAvbQAAAAAA4C9tAAAAAAAFAAAAAAAAAAAAAAAAAAAA/////////////////////wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAYGg4ZJF/AAAAAAAAAAAAAA==

(squeeze-x86_64)fslab2:~# xfs_bmap -a 
/mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n
/mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n:
        0: [0..7]: 92304..92311

(Sorry, I have no idea what "0: [0..7]: 92304..9231" is supposed to tell me).

Looking at 'top' and 'iostat -x' outout, I noticed we are actually not limited by io to disk, but CPU bound. If you should be interested, I have attached 'perf record -g' and 'perf report -g' outout, of the bonnie file create (create + fsetfattr() ) phase.



ext4:
mkfs.ext4 -J size=400 -I512 -i2048 /dev/sdd

1) ext4 and bonnie without xattr
(squeeze-x86_64)fslab2:~# ./bonnie++ -d /mnt/ext4/ -s0 -n 100:256:256:10 -r 0 
-u 500
Using uid:500, gid:100.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Create------ --------Random Create--------
fslab2              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
     100:256:256/10 13191  41 96158  97 29075  94 11812  35 98398  97 27271  94
Latency              3457us    1460us    2793us     529us    1106us    1018us
1.96,1.96,fslab2,1,1313594531,,,,,,,,,,,,,,100,256,256,,10,13191,41,96158,97,29075,94,11812,35,98398,97,27271,94,,,,,,,3457us,1460us,2793us,529us,1106us,1018us

2) ext4 and bonnie with xattr


(squeeze-x86_64)fslab2:~# ./bonnie++ -d /mnt/ext4/ -s0 -n 100:256:256:10 -r 0 
-u 500 -X
Using uid:500, gid:100.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Create------ --------Random Create--------
fslab2              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
     100:256:256/10 37026  91 +++++ +++ 43691  93 35960  92 +++++ +++ 40708  92
Latency              4328us     765us    2960us     527us     440us    1075us
1.96,1.96,fslab2,1,1313594619,,,,,,,,,,,,,,100,256,256,,10,37026,91,+++++,+++,43691,93,35960,92,+++++,+++,40708,92,,,,,,,4328us,765us,2960us,527us,440us,1075us


xfs:
mkfs.xfs -f -i size=512 -i maxpct=90  -l lazy-count=1 -n size=64k /dev/sdd

3) xfs and bonnie without xattr

(squeeze-x86_64)fslab2:~# ./bonnie++ -d /mnt/xfs/ -s0 -n 100:256:256:10 -r 0 -u 
500
Using uid:500, gid:100.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Create------ --------Random Create--------
fslab2              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
     100:256:256/10  6028  52 148478  98  9085  96  6484  56 136627  96  4429  
98
Latency             88887us    1501us    4202us   89499us    1144us   97805us
1.96,1.96,fslab2,1,1313595178,,,,,,,,,,,,,,100,256,256,,10,6028,52,148478,98,9085,96,6484,56,136627,96,4429,98,,,,,,,88887us,1501us,4202us,89499us,1144us,97805us

4) xfs and bonnie with xattr

(squeeze-x86_64)fslab2:~# ./bonnie++ -d /mnt/xfs/ -s0 -n 100:256:256:10 -r 0 -u 
500 -X
Using uid:500, gid:100.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Create------ --------Random Create--------
fslab2              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
     100:256:256/10  6860  85 +++++ +++  8254  96  5877  72 +++++ +++  4132  96
Latency               137ms     126us   14419us     461ms      16us     123ms
1.96,1.96,fslab2,1,1313594712,,,,,,,,,,,,,,100,256,256,,10,6860,85,+++++,+++,8254,96,5877,72,+++++,+++,4132,96,,,,,,,137ms,126us,14419us,461ms,16us,123ms



No idea why bonnie reports only 85% CPU here, 'top' shows 99%...

Cheers,
Bernd

Attachment: xfs_perf_create.txt
Description: Text document

<Prev in Thread] Current Thread [Next in Thread>