Re: retrieving exteneded attributes speed issues

[Nathan Scott <nathans@sgi.com>]

hi,

On Fri, May 31, 2002 at 01:20:23PM -0700, Anmar Oueja wrote:
> What's the best way of efficiently retrieving all extended attributes of
> a file on an XFS partition?
> 
> Now I'm doing what the utility getattr does:

IIRC, getfattr doesn't do a particularly spectacular job here, it's
been on my list of things to fix up for awhile, never got round to
it though.  Perhaps you could invest some time tuning it, as follows:

> 	call listxattr() to get size of all attr names
> 	reallocate memory if necessary
> 	call listxattr() again to get all attr name strings
> 	for each attr name
> 		call getxattr() to get size of attr value
> 		reallocate memory if necessary
> 		call getxattr() again to get the attr value

You can pass in a large buffer to listxattr and getxattr each time,
and chances are it will be big enough.  If you structure your code
intelligently (not like above), you can halve the number of syscalls
you make -- IOW, one can skip all of the "get size of" calls above -
the "get value/names" calls will fail if the buffer is too small,
returning ERANGE in this case, I think, & you can then just enlarge
the buffer and try again.

> 
> The problem is that the approach is not good at handling large number of
> files (in the order of 10000 or more.) Same application, without
> retrieving extended attributes, took 7 minutes finishing its job; 20
> minutes if retrieving extended attributes were included.
> 
> Profiling shows that system calls introduced by the retrieval (such as
> listxattr and getxattr) take most extra time. From the pseudo code above
> we can see this, too. At least 4 system calls are required to retrieve
> all extended attributes of one file.
> 
> Is retrieving extended attributes inherently slow? Or is there better
> way to do the job?
> 

There is another solution as well, as Steve has pointed out - xfsdump
uses some XFS-specific ioctl's to get back multiple attribute values
at once.  Such a program would only work on XFS filesystems though,
where as getfattr will work for all filesystems supporting EAs.

cheers.

-- 
Nathan