xfs
[Top] [All Lists]

3ware + RAID5 + xfs performance

To: linux-raid@xxxxxxxxxxxxxxx, linux-xfs@xxxxxxxxxxx
Subject: 3ware + RAID5 + xfs performance
From: Gaspar Bakos <gbakos@xxxxxxxxxxxxxxx>
Date: Mon, 25 Jul 2005 21:11:19 -0400 (EDT)
Reply-to: gbakos@xxxxxxxxxxxxxxx
Sender: linux-xfs-bounce@xxxxxxxxxxx
Dear all,

The purpose of this email is twofold:
- to share the results of the many tests I performed with
  a 3ware RAID card + RAID-5 + XFS, pushing for better file I/O,
- and to initiate some brainstorming on what parameters can be tuned for
  getting a good performance out of this hardware under 2.6.* kernels.

I started all these tests because the performance was quite poor, meaning
that the write speed was slow, the read speed was barely acceptable, and the
system load went very high (10.0) during bonnie++ tests.
My questions are marked below with "Q".

1.
There are many useful links related to the 3ware card and related anomalies.
The bugzilla page:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=121434
contains some 260 comments. It is mostly 2.4 kernel and RHEL specific.

2.
A newer description of the problem can be found in the thread:
http://lkml.org/lkml/2005/4/20/110
http://openlab-debugging.web.cern.ch/openlab-debugging/raid/
by Andreas Hirstius.
There was a nasty fls() bug, which was eliminated recently, and improved
performance and stability.

3.
There are recommendations by 3ware, which can be
summarized in one line: "blockdev --setra 16384".
http://www.3ware.com/reference/techlibrary.asp
"Maximum Performance for Linux Kernel 2.6 Combined with XFS File System",
which actually leads to a PDF that has a different title:
"Benchmarking the 9000 controller with linux 2.6".


Q: Any other useful links?

Briefly, the hardware setup I use
=================================
- Tyan S2882 Thunder K8S Pro motherboard
- Dual AMD opteron CPUs
- 4Gb RAM
- 3ware 9500-8S 8 port serial ATA controller
- 8 x 300GB ST3300831AS SATA Seagate disks in hardware RAID-5
More details at the end of this email.

OS/setup
=======
- Redhat FC3, first with 2.6.9-1.667smp kernel, then with all the upgrades,
  and finally a self-compiled 2.6.12.3 x86_64 kernel
- XFS filesystem
- Raid strip size = 64k, write-cache enabled
Kernel config attached.

==========================================================================
Tuneable parameters
====================
1. Kernel itself. I tried 2.6.9-1.667smp, 2.6.11-1.14_FC3smp, and 2.6.12.3
(self-compiled)

        1.a Kernel config (NUMA system, etc.)

2. Raid setup on the card.
        - Write-cache enabled? (I use "YES")
        - Raid strip size
        - firmware, bios, etc. on the card
        - staggered spinup (I use "YES", but the drives may not support it.
          I always "warm up" the unit before the tests, )

3. 3ware driver version
- 3w-9xxx_2.26.02.002 the older version in the kernels
- 3w-9xxx_2.26.03.015fw from the 3ware website, containing the firmware as
  well.

4. Run-time kernel parameters (my device is /dev/sde):

        4.a
                /sys/class/scsi_host/host6/
                cmd_per_lun
                can_queue

        4.b
                /sys/block/sde/queue/, e.g.
                iosched            max_sectors_kb  read_ahead_kb
                max_hw_sectors_kb  nr_requests     scheduler

        4.c
                /sys/block/sde/device/ e.g.
                queue_depth

        4.d Other params from the 2.4 kernel, if they have an alternative in
                2.6:

                /proc/sys/vm/max-readahead

        Q: Anything else?

5. blockdev --setra
        This is possibly belongs to those points mentioned under 4.)

6. For not raw IO (dd), the XFS filesystem parameters.

7. Q: Anything crucial parameter i am missing?

==========================================================================
Tests
=====
I changed the following during the tests. It is not an orthogonal set of
parameters, and I did not try everything with every combination.

- kernel
- raid strip size: 64K and 256K
- 3ware driver and firmware
- /sys/block/sde/queue/nr_requests
- blockdev --setra xxx /dev/sde
- XFS filesystem parameters

I used 5 bonnie++ commands to do not only simple IO, but also combined
filesystem performance:

MOUNT=/mnt/3w1/un0
SIZE=20480
echo "Bonnie test for IO performance"
sync; time bonnie++ -m cfhat5 -n 0   -u 0 -r 4092 -s $SIZE -f -b -d $MOUNT
echo "Testing with zero size files"
sync; time bonnie++ -m cfhat5 -n 50:0:0:50 -u 0 -r 4092 -s 0 -b -d $MOUNT
echo "Testing with tiny files"
sync; time bonnie++ -m cfhat5 -n 20:10:1:20 -u 0 -r 4092 -s 0 -b -d $MOUNT
echo "Testing with 100Kb to 1Mb files"
sync; time bonnie++ -m cfhat5 -n 10:1000000:100000:10 -u 0 -r 4092 -s 0 -b -d 
$MOUNT
echo "Testing with 16Mb size files"
sync; time bonnie++ -m cfhat5 -n 1:17000000:17000000:10 -u 0 -r 4092 -s 0 -b -d 
$MOUNT

==========================================================================
System information during the tests
===================================
This is just to make sure the system is behaving OK, and to catch some
errors. Done only outside the recorded tests, so as not to affect the
results.

1. top, or cat /proc/loadavg
to see the load

2. iostat, iostat -x

3. vmstat

4. ps -eaf
If the system behaves strange, as if locked.

Q: Anything else recommended that can be useful to check healthy system
behaviour?

==========================================================================
Other testing tools?
====================

1.  iozone
mentioning an Excel table in the man page made me uncertain whether
to try it...

2. dd
for raw IO.

Q: What else?

==========================================================================
Conclusions in a nutshell
=========================
1. With any of the kernels below 2.6.12.3, on the ___ x86_64 ___
architecture, the performance is poor. Load becomes huge, system
unresponsive, kswapd0, kswapd1 running on top of the "top".

2. The blockdev --setra 16384 does almost nothing else than increases the
read speed from the disks by also consuming much more CPU time. The write
and re-write speed do not change considerably. It is not really a
solution, when a system is run in hw raid based on an expensive card so as
to save CPU cycles for other tasks. (Then we can use sw RAID-5 on JBOD,
which is just much faster with more CPU usage)

3. The best I got during normal operation (no kswapd anomaly and
unresponsive system) was about 80Mb/s write, 40Mb/s rewrite and 350Mb/s
read. However, this was with "blockdev --setra 4092" and 43% CPU usage.
I would rather quote a more conservative 180Mb/s at setra 256 and 20% CPU.

4. I made tests Migration from 64kb to 256kb stripe size on a 2Tb array would 
take
forever. The performance during this migration is really bad, indifferent
from what the IO priority is set up in the 3ware interface:
50Mb/s write, 8Mb/s rewrite (!) and 12Mb/s read.

As I had no data yet to loose, it was much faster to reboot, and delete
unit, create one with 256Kb stripe size, and initialize it.

5. The performance of the 3ware card seemed worse with the 256k strip size.
Write: 68 Rewrite: 21, read: 60Mb/s

6. Changing /sys/block/sde/queue/nr_requests from 128 to 512 does a moderate
improvement. Going to higher numbers, such as 1024 does not make it better
any more.

==========================================================================
QUESTIONS:
=========

Q: Where is useful information on how to tune the various /sys/*
   parameters.?
   What are recommended values for a 2Tb array running on 3ware card?
   What are the relation between these parameters?

   Notably: nr_requests, can_queue, command_per_lun, max-readahead, etc.

Q: Are there any benchmarks showing better (re)write performance on an eight
   disk SATA RAID-5 with similar capacity (2Tb)?

Q: (mostly to 3ware/amcc inc.) Why is the 256K strip size so inefficient
    compared to the 64k?

==========================================================================
TEST RESULTS
============

        
---------------------------------------------------------------------------
        TEST2.1
        -------
        raid strip size = 64k
        blockdev --setra 256 /dev/sde
        /sys/block/sde/queue/nr_requests = 128
        mkfs.xfs -f -b size=4k -d su=64k,sw=7 -i size=1k -l version=2

        xfs_info /mnt/3w1/un0/
        meta-data=/mnt/3w1/un0           isize=1024   agcount=32, 
agsize=16021136 blks
                 =                       sectsz=512
        data     =                       bsize=4096   blocks=512676288, 
imaxpct=25
                 =                       sunit=16     swidth=112 blks, 
unwritten=1
        naming   =version 2              bsize=4096
        log      =internal               bsize=4096   blocks=32768, version=2
                 =                       sectsz=512   sunit=16 blks
        realtime =none                   extsz=65536  blocks=0, rtextents=0

        Testing with zero size files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
                      files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                    100/100   577   5 +++++ +++   914   5   763   6 +++++ +++   
 97   0
        real    24m32.187s
        user    0m0.365s
        sys     0m32.705s

        Testing with tiny files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max            /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
               100:10:0/100   125   2 103182 100   824   7   127   2 84106  99  
  82   1
        real    49m47.104s
        user    0m0.494s
        sys     1m5.833s

        Testing with 100Kb to 1Mb files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
        10:1000000:100000/10    42   5    75   5   685  11    41   5    24   1  
 212   4
        real    18m29.176s
        user    0m0.240s
        sys     0m45.138s

        16Mb files:
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
        1:17000000:17000000     4  14     7  14   461  39     4  15     5  10   
562  43

        Testing with 16Mb size files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
        1:17000000:17000000     3  14     7  14   522  40     4  14     6  11   
493  39
        real    13m43.331s
        user    0m0.455s
        sys     1m53.656s

        
-----------------------------------------------------------------------------
        TEST 2.2
        --------
        -> change inode size

        Strip size 64Kb
        blockdev --setra 256 /dev/sde
        /sys/block/sde/queue/nr_requests = 128
        mkfs.xfs -f -b size=4k -d su=64k,sw=7 -i size=2k -l version=2 /dev/sde1

        meta-data=/dev/sde1              isize=2048   agcount=32, 
agsize=16021136 blks
                 =                       sectsz=512
        data     =                       bsize=4096   blocks=512676288, 
imaxpct=25
                 =                       sunit=16     swidth=112 blks, 
unwritten=1
        naming   =version 2              bsize=4096
        log      =internal log           bsize=4096   blocks=32768, version=2
                 =                       sectsz=512   sunit=16 blks
        realtime =none                   extsz=65536  blocks=0, rtextents=0

        Disk IO
        Version  1.03       ------Sequential Output------ --Sequential Input- 
--Random-
                            -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
        Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  
/sec %CP
        cfhat5          20G 57019  97 75887  16 47033  10 35907  61 192411  22 
311.6   0

        Testing with zero size files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
                      files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                      50/50   655   6 +++++ +++   944   5   717   6 +++++ +++   
112   0
        real    10m58.033s
        user    0m0.182s
        sys     0m16.954s

        Testing with tiny files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                 20:10:1/20   111   2 +++++ +++   805   7   107   2 +++++ +++   
126   1
        real    9m23.056s
        user    0m0.105s
        sys     0m12.835s

        Testing with 100Kb to 1Mb files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
        10:1000000:100000/10    44   5   221  13   504   7    43   5    22   1  
 164   2
        real    17m25.308s
        user    0m0.207s
        sys     0m42.914s

        ==> Seq. read speed increased to 3x, seq. delete decreased

        Testing with 16Mb size files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
        1:17000000:17000000/10  4  14    10  20   450  34     4  14     5   9   
419  34
        real    13m24.856s
        user    0m0.483s
        sys     1m53.478s

        ==> Delete speed decreased. Seq. read speed somewhat increased.
        ==> No significant difference compared to smaller inode size.

        
-----------------------------------------------------------------------------
        TEST2.3
        --------
        Tests done while migrating from Stripe 64kB to Stripe 256kB.
        /sys/block/sde/queue/nr_requests = 128
        blockdev --setra 256 /dev/sde
        Extremely slow.

        Bonnie test for IO performance
        Version  1.03       ------Sequential Output------ --Sequential Input- 
--Random-
                            -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
        Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  
/sec %CP
        cfhat5          20G           53072  11  8848   1           12039   1 
139.3   0

        Testing with zero size files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
                      files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                      50/50  289   3 +++++ +++   603   3   444   4 +++++ +++    
77   0
        real    17m19.235s
        user    0m0.186s
        sys     0m17.566s

        Testing with tiny files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                 20:10:1/20   86   1 +++++ +++   564   5    86   1 +++++ +++    
90   0
        real    12m16.227s
        user    0m0.099s
        sys     0m12.125s

        Testing with 100Kb to 1Mb files
        Delete files in random order...done.
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
        10:1000000:100000/10  29   3    13   0   466   6    25   3    11   0   
125   2
        real    41m4.151s
        user    0m0.255s
        sys     0m42.095s

        Testing with 16Mb size files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
        1:17000000:17000000/10 2   9     2   5   273  20     2   8     1   3   
258  19
        real    29m20.672s
        user    0m0.469s
        sys     1m49.345s

        ===> Disk IO becomes extreme slow when array is migrating strip size

        
-----------------------------------------------------------------------------
        TEST 2.4
        --------
        Tests done with 256Kb RAID array size
        blockdev --setra 256 /dev/sde
        /sys/block/sde/queue/nr_requests = 128
        mkfs.xfs -f -b size=4k -d su=256k,sw=7 -i size=1k -l version=2 -L 
cfhat5_1_un0 /dev/sde1

        meta-data=/dev/sde1              isize=1024   agcount=32, 
agsize=16021184 blks
                 =                       sectsz=512
        data     =                       bsize=4096   blocks=512676288, 
imaxpct=25
                 =                       sunit=64     swidth=448 blks, 
unwritten=1
        naming   =version 2              bsize=4096
        log      =internal log           bsize=4096   blocks=32768, version=2
                 =                       sectsz=512   sunit=64 blks
        realtime =none                   extsz=65536  blocks=0, rtextents=0

        top - 11:54:04 up 11:31,  2 users,  load average: 8.52, 7.56, 5.07
        Tasks: 104 total,   1 running, 102 sleeping,   1 stopped,   0 zombie
        Cpu(s):  0.3% us,  4.0% sy,  0.0% ni,  0.7% id, 94.5% wa,  0.0% hi,  
0.5% si
        Mem:   4010956k total,  3988284k used,    22672k free,        0k buffers
        Swap:  7823576k total,      224k used,  7823352k free,  3789640k cached
          PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
        30821 root      18   0  8312  916  776 D  5.3  0.0   1:21.60 bonnie++
          175 root      15   0     0    0    0 D  1.3  0.0   0:16.35 kswapd1
          176 root      15   0     0    0    0 S  1.0  0.0   0:18.38 kswapd0

        Bonnie test for IO performance
        Version  1.03       ------Sequential Output------ --Sequential Input- 
--Random-
                            -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
        Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  
/sec %CP
        cfhat5          20G           68990  14 21157   5           60837   7 
250.2   0
        real    27m58.805s
        user    0m1.118s
        sys     1m58.749s

        Testing with zero size files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
                      files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                      50/50   255   3 +++++ +++   247   2   252   3 +++++ +++   
 61   0
        real    23m59.997s
        user    0m0.186s
        sys     0m26.721s

        ==> Much slower than 64kb size with setra=256

        Testing with tiny files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                 20:10:1/20   110   3 +++++ +++   243   3   112   3 +++++ +++   
 77   1
        real    11m57.399s
        user    0m0.100s
        sys     0m17.356s

        ==> Much slower than 64kb size with setra=256

        Testing with 100Kb to 1Mb files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
        10:1000000:100000/10    36   5    77   5   232   4    40   5    35   2  
  92   2
        real    18m25.701s
        user    0m0.238s
        sys     0m45.724s

        ==> Somewhat slower than 64kb size with setra=256

        Testing with 16Mb size files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
        1:17000000:17000000/10     4  15     3   6   227  18     3  14     2   
4   155  13
        real    20m11.168s
        user    0m0.508s
        sys     1m55.892s

        ==> Somewhat slower than 64kb size with setra=256

        ==> Definitely inferior to the 64kb raid strip size

        
------------------------------------------------------------------------------
        TEST2.5
        -------
        raid strip size = 256K
        Change su to 64k
        blockdev --setra 256 /dev/sde
        /sys/block/sde/queue/nr_requests = 128
        mkfs.xfs -f -b size=4k -d su=64k,sw=7 -i size=1k -l version=2 -L 
cfhat5_1_un0 /dev/sde1

        Bonnie test for IO performance
        Version  1.03       ------Sequential Output------ --Sequential Input- 
--Random-
                            -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
        Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  
/sec %CP
        cfhat5          20G           72627  15 23325   5           63101   7 
272.0   0
        real    25m56.324s
        user    0m1.097s
        sys     1m57.267s

        ===> General IO was slightly faster with su=64k than su=256k

        Testing with zero size files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
                      files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                      50/50   788   7 +++++ +++   989   6   781   7 +++++ +++   
 93   0
        real    12m8.633s
        user    0m0.158s
        sys     0m16.578s


        ===> Filesystem is much faster with su=64k

        Testing with tiny files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                 20:10:1/20   135   2 +++++ +++   818   7   133   2 +++++ +++   
145   1
        real    7m51.365s
        user    0m0.091s
        sys     0m12.182s

        ===> Filesystem is somewhat faster with su=64k

        Testing with 100Kb to 1Mb files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
        10:1000000:100000/10    41   5    91   5   787  12    41   5    24   1  
 224   4
        real    18m6.138s
        user    0m0.243s
        sys     0m42.042s

        ===> For larger files, it becomes almost indifferent if we use su=64k 
or su=256k

        Testing with 16Mb size files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
        1:17000000:17000000/10     4  14     3   6   476  34     3  11     2   
5   546  40
        real    19m37.665s
        user    0m0.548s
        sys     1m49.408s

        ===> For larger files, it becomes almost indifferent if we use su=64k 
or su=256k

        
------------------------------------------------------------------------------
        TEST 2.6
        ---------
        Tests done with 256Kb RAID array size
        blockdev --setra 1024 /dev/sde
        /sys/block/sde/queue/nr_requests = 128
        blockdev --setra 1024 /dev/sde
        mkfs.xfs -f -b size=4k -d su=256k,sw=7 -i size=1k -l version=2 -L 
cfhat5_1_un0 /dev/sde1

        meta-data=/dev/sde1              isize=1024   agcount=32, 
agsize=16021184 blks
                 =                       sectsz=512
        data     =                       bsize=4096   blocks=512676288, 
imaxpct=25
                 =                       sunit=64     swidth=448 blks, 
unwritten=1
        naming   =version 2              bsize=4096
        log      =internal log           bsize=4096   blocks=32768, version=2
                 =                       sectsz=512   sunit=64 blks
        realtime =none                   extsz=65536  blocks=0, rtextents=0

        Bonnie test for IO performance
        Version  1.03       ------Sequential Output------ --Sequential Input- 
--Random-
                            -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
        Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  
/sec %CP
        cfhat5          20G           68794  14 26139   6           118452  14 
255.5   0
        real    22m2.101s
        user    0m1.268s
        sys     1m58.232s

        => Speed increased compared to TEST 2.4 (setra 256). CPU % didn't 
increase.

        Testing with zero size files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
                      files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                      50/50   253   3 +++++ +++   247   2   251   3 +++++ +++   
 60   0
        real    24m14.398s
        user    0m0.178s
        sys     0m27.186s

        => No change compared to 2.4

        Testing with tiny files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                 20:10:1/20   112   3 +++++ +++   241   3   109   3 +++++ +++   
 71   1
        real    12m21.663s
        user    0m0.089s
        sys     0m17.502s

        => No change.

        Testing with 100Kb to 1Mb files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
        10:1000000:100000/10    39   5    90   5   237   4    37   5    32   1  
  82   1
        real    18m47.223s
        user    0m0.260s
        sys     0m45.430s

        => No change.

        Testing with 16Mb size files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
        1:17000000:17000000/10 4  13     6  12   215  16     4  14     5   9   
171  13
        real    14m21.865s
        user    0m0.474s
        sys     1m49.301s

        ==> Improved.

        
------------------------------------------------------------------------------
        TEST 2.6
        --------
        Back to raid-strip = 64k
        /sys/block/sde/queue/nr_requests = 128
        mkfs.xfs -f -b size=4k -d su=64k,sw=7 -i size=1k -l version=2 -L 
cfhat5_1_un0 /dev/sde1
        blockdev --setra 256 /dev/sde

        top - 10:51:03 up  8:06,  3 users,  load average: 9.69, 4.18, 1.63
        Tasks: 128 total,   1 running, 127 sleeping,   0 stopped,   0 zombie
        Cpu(s):  0.2% us,  5.0% sy,  0.0% ni,  5.2% id, 88.5% wa,  0.0% hi,  
1.2% si
        Mem:   4010956k total,  3987456k used,    23500k free,       52k buffers
        Swap:  7823576k total,      224k used,  7823352k free,  3677224k cached

        System stays responsive despite the giant load.

          PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
         5757 root      18   0  8308  916  776 D  6.3  0.0   0:35.69 bonnie++
          176 root      15   0     0    0    0 D  1.3  0.0   0:05.27 kswapd0
          175 root      15   0     0    0    0 S  1.0  0.0   0:05.64 kswapd1

        Bonnie test for IO performance
        Version  1.03       ------Sequential Output------ --Sequential Input- 
--Random-
                            -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
        Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  
/sec %CP
        cfhat5          20G           65322  14 46177  10           183637  21 
293.2   0
        real    15m23.264s
        user    0m1.118s
        sys     1m58.544s

        Testing with zero size files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
                      files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                      50/50   701   6 +++++ +++   983   5   733   6 +++++ +++   
111   0
        real    10m56.735s
        user    0m0.171s
        sys     0m15.877s

        Testing with tiny files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                 20:10:1/20   109   2 +++++ +++   824   7   108   2 +++++ +++   
147   1
        real    8m58.359s
        user    0m0.107s
        sys     0m12.546s

        Testing with 100Kb to 1Mb files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
        10:1000000:100000/10    45   5   214  13   642   9    45   5    22   1  
 211   3
        real    16m59.573s
        user    0m0.230s
        sys     0m42.618s

        Testing with 16Mb size files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
        1:17000000:17000000/10     4  13    11  20   467  32     4  13     5   
9   416  30
        real    13m15.243s
        user    0m0.534s
        sys     1m47.777s

        
------------------------------------------------------------------------------
        TEST 2.7
        ---------
        Change setra:
        blockdev --setra 4092 /dev/sde
        raid-strip = 64k
        /sys/block/sde/queue/nr_requests = 128
        mkfs.xfs -f -b size=4k -d su=64k,sw=7 -i size=1k -l version=2 -L 
cfhat5_1_un0 /dev/sde1

        [root@cfhat5 diskio]# iostat -x /dev/sde
        Linux 2.6.12.3-GB2 (cfhat5)     07/25/2005

        avg-cpu:  %user   %nice    %sys %iowait   %idle
                   0.29    0.04    1.00    4.88   93.80
        Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s 
avgrq-sz avgqu-sz   await  svctm  %util
        sde          0.04 903.28 19.74 44.03 4757.48 8632.40  2378.74  4316.20  
 209.94     7.73  121.17   1.96  12.51

        Bonnie test for IO performance
        Version  1.03       ------Sequential Output------ --Sequential Input- 
--Random-
                            -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
        Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  
/sec %CP
        cfhat5          20G           66303  13 41254   9           345730  41 
274.7   0
        real    15m21.055s
        user    0m1.114s
        sys     1m57.199s

        ==> Write does not change. Rewrite decreases. Read increases.

        Testing with zero size files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
                      files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                      50/50   624   6 +++++ +++   904   5   727   6 +++++ +++   
113   0
        real    10m59.528s
        user    0m0.189s
        sys     0m16.520s

        Testing with tiny files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                 20:10:1/20   111   2 +++++ +++   798   7   102   2 +++++ +++   
143   1
        real    9m12.536s
        user    0m0.120s
        sys     0m12.467s

        Testing with 100Kb to 1Mb files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
        10:1000000:100000/10  46   6   323  20   686  10    43   5    30   1   
207   3
        real    14m42.960s
        user    0m0.262s
        sys     0m42.090s

        Testing with 16Mb size files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
        1:17000000:17000000/10     4  14    20  40   524  38     4  13    11  
21   492  35
        real    10m42.784s
        user    0m0.453s
        sys     1m51.078s

        
------------------------------------------------------------------------------
        TEST 2.8
        ---------
        echo 512 > /sys/block/sde/queue/nr_requests
        raid-strip = 64k
        mkfs.xfs -f -b size=4k -d su=64k,sw=7 -i size=1k -l version=2 -L 
cfhat5_1_un0 /dev/sde1
        blockdev --setra 4092 /dev/sde

        Bonnie test for IO performance
        Version  1.03       ------Sequential Output------ --Sequential Input- 
--Random-
                            -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
        Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  
/sec %CP
        cfhat5          20G           78573  16 42444   9           353894  42 
284.6   0
        real    14m14.938s
        user    0m1.213s
        sys     1m55.382s

        Testing with zero size files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
                      files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                      50/50   623   6 +++++ +++   894   5   739   6 +++++ +++   
123   0
        real    10m25.379s
        user    0m0.186s
        sys     0m16.846s

        Testing with tiny files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                 20:10:1/20   107   2 +++++ +++   835   7   100   1 +++++ +++   
159   1
        real    9m7.268s
        user    0m0.104s
        sys     0m12.589s

        Testing with 100Kb to 1Mb files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
        10:1000000:100000/10    47   6   324  19   697  10    44   5    35   2  
 232   4
        real    13m41.706s
        user    0m0.234s
        sys     0m42.614s

        Testing with 16Mb size files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
        1:17000000:17000000/10     4  14    19  38   448  32     4  13    11  
21   506  36
        real    10m40.404s
        user    0m0.469s
        sys     1m51.098s

        
------------------------------------------------------------------------------
        TEST 2.9
        ---------
        echo 1024 > /sys/block/sde/queue/nr_requests
        raid-strip = 64k
        mkfs.xfs -f -b size=4k -d su=64k,sw=7 -i size=1k -l version=2 -L 
cfhat5_1_un0 /dev/sde1
        blockdev --setra 4092 /dev/sde

        Bonnie test for IO performance
        Version  1.03       ------Sequential Output------ --Sequential Input- 
--Random-
                            -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
        Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  
/sec %CP
        cfhat5          20G           79546  16 41227   9           351637  43 
285.0   0
        real    14m26.609s
        user    0m1.136s
        sys     1m57.398s

        ==> No improvement

        Testing with zero size files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
                      files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                      50/50   616   5 +++++ +++   880   5   748   6 +++++ +++   
123   0
        real    10m25.469s
        user    0m0.186s
        sys     0m16.723s

        Testing with tiny files
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                 20:10:1/20    99   2 +++++ +++   779   7   104   2 +++++ +++   
165   1
        real    9m12.385s
        user    0m0.111s
        sys     0m12.947s

        Testing with 100Kb to 1Mb files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
        10:1000000:100000/10  47   6   316  20   616   9    47   6    36   2   
248   4
        real    13m22.360s
        user    0m0.231s
        sys     0m43.679s

        Testing with 16Mb size files
        Version  1.03       ------Sequential Create------ --------Random 
Create--------
        cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
        files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
        1:17000000:17000000/10 3  13    16  31   386  27     4  13    11  22   
558  40
        real    11m1.018s
        user    0m0.464s
        sys     1m49.534s



============================================================================
Hardware info
=============

[root@cfhat5 diskio]# cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 5
model name      : AMD Opteron(tm) Processor 246
stepping        : 10
cpu MHz         : 1991.008
cache size      : 1024 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext lm 3dnowext 3dnow
bogomips        : 3915.77
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

processor       : 1
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 5
model name      : AMD Opteron(tm) Processor 246
stepping        : 10
cpu MHz         : 1991.008
cache size      : 1024 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext lm 3dnowext 3dnow
bogomips        : 3973.12
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

-----------------------------------------------------------
[root@cfhat5 diskio]# cat /sys/class/scsi_host/host6/stats
3w-9xxx Driver version: 2.26.03.015fw
Current commands posted:      0
Max commands posted:         79
Current pending commands:     0
Max pending commands:         1
Last sgl length:              2
Max sgl length:              32
Last sector count:            0
Max sector count:           256
SCSI Host Resets:             0
AEN's:                        0

--------------------------
3ware card info

Model   9500S-8
Serial #      L19403A5100293
Firmware      FE9X 2.06.00.009
Driver        2.26.03.015fw
BIOS  BE9X 2.03.01.051
Boot Loader   BL9X 2.02.00.001
Memory Installed      112 MB
# of Ports    8
# of Units    1
# of Drives   8

Write cache enabled
Auto-spin up enabled, 2 sec between spin-up
Drives, however, probably do not support spinup.

-------------------------------
Disks:
Drive Information (Controller ID 6)
Port    Model   Capacity        Serial #        Firmware        Unit    Status
0       ST3300831AS     279.46 GB       3NF0BZYJ        3.02    0       OK
1       ST3300831AS     279.46 GB       3NF0AC04        3.01    0       OK
2       ST3300831AS     279.46 GB       3NF0A7JE        3.01    0       OK
3       ST3300831AS     279.46 GB       3NF0ABT1        3.01    0       OK
4       ST3300831AS     279.46 GB       3NF0A63J        3.01    0       OK
5       ST3300831AS     279.46 GB       3NF0ACC5        3.01    0       OK
6       ST3300831AS     279.46 GB       3NF09FLP        3.01    0       OK
7       ST3300831AS     279.46 GB       3NF046WY        3.01    0       OK

----------------------------------
[root@cfhat5 diskio]# vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0    380 3781540      0  58004    0    0  2712  3781  243   216  0  2 91  7

[root@cfhat5 diskio]# free
             total       used       free     shared    buffers     cached
Mem:       4010956     229532    3781424          0          0      58004
-/+ buffers/cache:     171528    3839428
Swap:      7823576        380    7823196


============================================================================
Kernel config

See http://www.cfa.harvard.edu/~gbakos/diskio/


<Prev in Thread] Current Thread [Next in Thread>