We have been doing some performance testing on a handful of kernels and are
seeing a significant performance regression with lower number of outstanding
I/Os somewhere between 2.6.32 and 2.6.38. The test case shows a significant
drop in random read IOPS (45k -> 8k) and a significantly dirtier latency
We also tested against the raw block device and against ext4. The performance
profiles of those tests were fairly consistent between the .32 and 3.0 based
kernels where most of the testing was done.
Also worth noting, the test case below has 24 thread with one I/O each (~24
outstanding total). We did do a small number of tests that used 4 threads with
libaio and 64 I/Os each (~256 total outstanding) which showed performance
across the various kernel versions to be fairly stable.
lat (usec): min=149 , max=2465 , avg=523.58, stdev=106.68
lat (usec): 250=0.01%, 500=48.93%, 750=48.30%, 1000=2.70%
lat (msec): 2=0.07%, 4=0.01%
18.104.22.168-0.fc15.x86_64 (aka 3.0)
lat (usec): min=77 , max=147441 , avg=452.33, stdev=2773.88
lat (usec): 100=0.01%, 250=61.30%, 500=37.59%, 750=0.01%, 1000=0.01%
lat (msec): 2=0.05%, 4=0.04%, 10=0.30%, 20=0.33%, 50=0.30%
lat (msec): 100=0.07%, 250=0.01%
-- Testing Configuration
Most testing was performed on various 2 socket intel x5600 class server systems
using various models of ioDrive. The results above are from a 160GB ioDrive
with a 2.3.1 driver.
The fio benchmark tool was used for most of the testing, but another benchmark
showed similar results.
-- Testing Process
# load the ioDrive driver
# Reset the ioDrive back to a known state
fio-format -y /dev/fct0
# Setup XFS for testing and create the sample file
mkfs.xfs -i size=2048 /dev/fioa
mkdir -p /mnt/tmp
mount -t xfs /dev/fioa /mnt/tmp
dd if=/dev/zero of=/mnt/tmp/bigfile bs=1M oflag=direct count=$((10*1024))
# Run fio test
fio --direct=1 --rw=randread --bs=16k --numjobs=24 --runtime=60
--group_reporting --norandommap --time_based --ioengine=sync --name=file1
Are there any mount options or other tests that can be run in the failing
configuration that would be helpful to isolate this further?
Please cc Paul and I, we are not subscribed to the list.
Confidentiality Notice: This e-mail message, its contents and any attachments
to it are confidential to the intended recipient, and may contain information
that is privileged and/or exempt from disclosure under applicable law. If you
are not the intended recipient, please immediately notify the sender and
destroy the original e-mail message and any attachments (and any copies that
may have been made) from your system or otherwise. Any unauthorized use,
copying, disclosure or distribution of this information is strictly prohibited.