Weird performance on a FusionIO Octal (Random writes faster than Seq.)

[Settlemyer, Bradley W.]

Hello

  So I'm getting weird performance using XFS on a 5TB FusionIO octal (a
solid state device plugged into my pcie bus).  It seems to be a newish
problem, but I can't go back to an old version of everything to prove
that, because I've only got one working Octal right now (they are a little
pricy).

  At any rate, when doing random 16MB requests to a file with 16 threads,
I get about 4.5GB/s.  When writing sequentially with 16 threads doing 16MB
requests, I get about 3.5GB/s -- the first time.  Once the file is written
the first time, a second pass results in 4.5GB/s.

  The thing is, I'm using preallocate on both types of I/O (that is, I
always preallocate the entire file whether its random or sequential).  I
allocate the exact same size file in both cases, its just faster the first
time with random writes rather than sequential writes.

  So if you had xdd 7.0 and an octal plugged into slot 6 of an HP DL585 G7
(running CentOS 6.3), you could replicate these test results with the
following commands (note xdd's default block size is 1024, so you'll see
everything is accounting for 1024 byte size blocks):

# Generate a set of random seek offsets within a file
data_size=$((256*8*1024*1024*1024))
rand_range=$((data_size / 16384 / 1024 - 1))
    shuf -i 0-$rand_range | awk "{i += 1}; {print i-1, \$0*16384,  \"16384
w 0 0\" };" > wseek_file


# Perform the random writes
numactl --cpunodebind=4 xdd -op write -target
/data/xfs-numa4/baseline-rand1 -reqsize 16384 -qd 16 -dio -verbose -bytes
2199023255552 -preallocate 2199023255552 -seek load wseek file

# Perform sequential writes
numactl --cpunodebind=4 xdd -op write -target
/data/xfs-numa4/baseline-seq1 -reqsize 16384 -qd 16 -dio -verbose -bytes
2199023255552 -preallocate 2199023255552

And so for the results I get the following (if you know xdd results):
COMBINED             1       1      16    2199023255552      131072
483.311   4549.910    271.196     0.230    67.644    write     16777216
 COMBINED             1       1      16    2199023255552      131072
605.366   3632.550    216.517     0.289    53.971    write     16777216


The bandwidths here are 4549.910 MB/s for the random, and 3632.550 MB/s
for the sequential.  If I run it again to the already existing file, I get:

COMBINED 1 1 16 2199023255552 131072 482.851 4554.248 271.454 0.230 66.016
write 16777216

Which is a write bandwidth of 4554.248MB/s.


Weird, right?

Cheers,
Brad