Deleting files with extended attributes is dead slow

[Bernd Schubert]

Hi there,

with FhGFS we may use extended attributes to store file meta data and 
with ext3/ext4 that also works very well and the rate to create files 
and to write those EAs (create() + fsetxattr()  is about 2.5 to 3 times 
faster than with a create() + write(). Size of those EA data is about 
256 bytes depending on the number of storage stripes.
However, with XFS using extended attributes is *extremely* slow. Here 
are some numbers with ext4 and xfs using a patched [1] bonnie++

> schubert at fsdevel3 bonnie++-1.96>./bonnie++ -d /mnt/fstestXFS/ -s0 -n 1:256:256:1 -r 0 -X
> Create files in sequential order...done.
> Stat files in sequential order...done.
> Delete files in sequential order...done.
> Create files in random order...done.
> Stat files in random order...done.
> Delete files in random order...done.
> Version  1.96       ------Sequential Create------ --------Random Create--------
> fsdevel3            -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
> files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>           1:256:256   330   6 +++++ +++    47   1   469   9 +++++ +++    31   1
> Latency               962ms      36us     140ms     866ms      28us     311ms
> 1.96,1.96,fsdevel3,1,1314378941,,,,,,,,,,,,,,1,256,256,,,330,6,+++++,+++,47,1,469,9,+++++,+++,31,1,,,,,,,962ms,36us,140ms,866ms,28us,311ms



> schubert at fsdevel3 bonnie++-1.96>./bonnie++ -d /mnt/fstestEXT4/ -s0 -n 100:256:256:10 -r 0 -X
> Create files in sequential order...done.
> Stat files in sequential order...done.
> Delete files in sequential order...done.
> Create files in random order...done.
> Stat files in random order...done.
> Delete files in random order...done.
> Version  1.96       ------Sequential Create------ --------Random Create--------
> fsdevel3            -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
> files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>      100:256:256/10 21348  52 +++++ +++ 32878  61 25286  60 +++++ +++ 25873  53
> Latency               746us    1926us    2553us     653us     118us   32250us
> 1.96,1.96,fsdevel3,1,1314379136,,,,,,,,,,,,,,100,256,256,,10,21348,52,+++++,+++,32878,61,25286,60,+++++,+++,25873,53,,,,,,,746us,1926us,2553us,653us,118us,32250us


NOTE: For ext4 I had to increase the number of files by factor 100 to 
get any sane number (it would only print '+++++' otherwise). Running the 
benchmark with the same numbers on xfs deleted in so slow delete 
numbers, that it probably still would be busy to delete files by tomorrow.


The xfs file system was formated with these parameters:
> mkfs.xfs -f -i size=512 -i maxpct=90  -l lazy-count=1 -n size=64k /dev/mapper/vg0fsdev3-XFStest


ext4 was formated to have an inode size of 512B and to have a joural 
size of 400MB.

Both file systems are mounted with "nobarrier" (real FhGFS production 
systems usually have raid controllers with battery backups).


Any ideas why xfs is so slow with extended attributes? Without extended 
attributes, so skipping the "-X" option in our patched bonnie also 
results in slower Create numbers (about 4500 on xfs vs. 7500 on ext4) 
compared to ext4, but that is 'only' factor 1.67 and not 50 or more as 
with EAs.


Thanks,
Bernd


[1] Updated bonnie to support extended attributes:
https://bitbucket.org/aakef/bonnie/