On 08/16/2011 06:13 PM, Christoph Hellwig wrote:
The patch below should help with the performance of deleting files
that have external attributes. But I still wonder why you see
attributes outside the inode if you are using 512 byte inodes and around
256k of attrs. Can you check tat you really have external blocks for
attributes using xfs_bmap -a on the files in your workload. Can you
also check if this goes away using 1k or 2k inode sizes?
Thanks! Deletes behave much better now!
Compared to ext4, it is still rather slow. Below are xfs and ext4
benchmarks using our patched bonnie. Here is getfattr and xfs_bmap
output of one of those bonnie files:
(squeeze-x86_64)fslab2:~# getfattr -d
/mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n
getfattr: Removing leading '/' from absolute path names
# file: mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n
user.bonnie=0sMDAwMDkAAACotThkkX8AAFAvbQAAAAAAUC9tAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIAA7usEhkXAAAAAAAAAAAEAAAA0C9tAAAAAADRRwAAAAAAAGAQbQAAAAAAmK44ZJF/AAAAAAAAAAAAAAAAAAAAAAAAqLU4ZJF/AACotThkkX8AAOAvbQAAAAAA4C9tAAAAAAAFAAAAAAAAAAAAAAAAAAAA/////////////////////wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAYGg4ZJF/AAAAAAAAAAAAAA==
(squeeze-x86_64)fslab2:~# xfs_bmap -a
/mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n
/mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n:
0: [0..7]: 92304..92311
(Sorry, I have no idea what "0: [0..7]: 92304..9231" is supposed to tell
me).
Looking at 'top' and 'iostat -x' outout, I noticed we are actually not
limited by io to disk, but CPU bound. If you should be interested, I
have attached 'perf record -g' and 'perf report -g' outout, of the
bonnie file create (create + fsetfattr() ) phase.
ext4:
mkfs.ext4 -J size=400 -I512 -i2048 /dev/sdd
1) ext4 and bonnie without xattr
(squeeze-x86_64)fslab2:~# ./bonnie++ -d /mnt/ext4/ -s0 -n 100:256:256:10 -r 0
-u 500
Using uid:500, gid:100.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Create------ --------Random Create--------
fslab2 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
100:256:256/10 13191 41 96158 97 29075 94 11812 35 98398 97 27271 94
Latency 3457us 1460us 2793us 529us 1106us 1018us
1.96,1.96,fslab2,1,1313594531,,,,,,,,,,,,,,100,256,256,,10,13191,41,96158,97,29075,94,11812,35,98398,97,27271,94,,,,,,,3457us,1460us,2793us,529us,1106us,1018us
2) ext4 and bonnie with xattr
(squeeze-x86_64)fslab2:~# ./bonnie++ -d /mnt/ext4/ -s0 -n 100:256:256:10 -r 0
-u 500 -X
Using uid:500, gid:100.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Create------ --------Random Create--------
fslab2 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
100:256:256/10 37026 91 +++++ +++ 43691 93 35960 92 +++++ +++ 40708 92
Latency 4328us 765us 2960us 527us 440us 1075us
1.96,1.96,fslab2,1,1313594619,,,,,,,,,,,,,,100,256,256,,10,37026,91,+++++,+++,43691,93,35960,92,+++++,+++,40708,92,,,,,,,4328us,765us,2960us,527us,440us,1075us
xfs:
mkfs.xfs -f -i size=512 -i maxpct=90 -l lazy-count=1 -n size=64k /dev/sdd
3) xfs and bonnie without xattr
(squeeze-x86_64)fslab2:~# ./bonnie++ -d /mnt/xfs/ -s0 -n 100:256:256:10 -r 0 -u
500
Using uid:500, gid:100.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Create------ --------Random Create--------
fslab2 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
100:256:256/10 6028 52 148478 98 9085 96 6484 56 136627 96 4429
98
Latency 88887us 1501us 4202us 89499us 1144us 97805us
1.96,1.96,fslab2,1,1313595178,,,,,,,,,,,,,,100,256,256,,10,6028,52,148478,98,9085,96,6484,56,136627,96,4429,98,,,,,,,88887us,1501us,4202us,89499us,1144us,97805us
4) xfs and bonnie with xattr
(squeeze-x86_64)fslab2:~# ./bonnie++ -d /mnt/xfs/ -s0 -n 100:256:256:10 -r 0 -u
500 -X
Using uid:500, gid:100.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Create------ --------Random Create--------
fslab2 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
100:256:256/10 6860 85 +++++ +++ 8254 96 5877 72 +++++ +++ 4132 96
Latency 137ms 126us 14419us 461ms 16us 123ms
1.96,1.96,fslab2,1,1313594712,,,,,,,,,,,,,,100,256,256,,10,6860,85,+++++,+++,8254,96,5877,72,+++++,+++,4132,96,,,,,,,137ms,126us,14419us,461ms,16us,123ms
No idea why bonnie reports only 85% CPU here, 'top' shows 99%...
Cheers,
Bernd
xfs_perf_create.txt
Description: Text document
|