1B files, slow file creation, only AG0 used

[Michael Spiegle <mike@xxxxxxxxxxxxxxxx>]

We're seeing some very strange behavior with XFS on the default kernel
for CentOS 5.6 (note, I have also 3.2.9 and witnessed the same issue).
 The dataset on this server is about 1B small files (anywhere from 1KB
to 50KB).  We first noticed it when creating files in a directory.  A
simple 'touch' would take over 300ms on a completely idle system.  If
I simply create a different directory, touching files is 1ms or
faster.  Example:

# time touch 0
real    0m0.323s
user    0m0.000s
sys     0m0.323s

# mkdir tmp2
# time touch tmp2/0
real    0m0.001s
user    0m0.000s
sys     0m0.000s

We've done quite a bit of testing and debugging, and while we don't
have an answer yet, we've noticed that our filesystem was created with
the default of 32 AGs.  When using xfs_db, we notice that all
allocations appear to be in AG0 only.  We've also noticed during
testing that if we create 512 AGs, the distribution appears to be
better.  It seems that the AG is actually encoded into the inode, and
the XFS_INO_TO_AGNO(mp,i) macro is used to determine the AG by
performing a bitshift.  In our case, the bitshift appears to be
32bits, and since the inode is 32bits, we always end up with AG0.
Does anyone know if our slow file creation issue is related to our use
of AG0, and if so, what's the best way to utilize additional AGs?

Per-AG counts:
# for x in {0..31}; do echo -n "${x}: "; xfs_db -c "agi ${x}" -c
"print" -r /dev/sda1 | grep "^count"; done
0: count = 1098927744
1: count = 0
2: count = 0
3: count = 0
4: count = 0
5: count = 0
6: count = 0
7: count = 0
8: count = 0
9: count = 0
10: count = 0
11: count = 0
12: count = 0
13: count = 0
14: count = 0
15: count = 0
16: count = 0
17: count = 0
18: count = 0
19: count = 0
20: count = 0
21: count = 0
22: count = 0
23: count = 0
24: count = 0
25: count = 0
26: count = 0
27: count = 0
28: count = 0
29: count = 0
30: count = 0
31: count = 0

Some general stats on the server:
24x Xeon
24GB RAM
CentOS 5.6
20TB of storage
1B files
RAID6, 14 drives, SATA

Output of "xfs_info /dev/sda1":
meta-data=/dev/sda1              isize=256    agcount=32, agsize=152575999 blks
         =                       sectsz=512   attr=0
data     =                       bsize=4096   blocks=4882431968, imaxpct=25
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=32768, version=1
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0