xfs
[Top] [All Lists]

Linux RAID & XFS Question - Multiple levels of concurrency = faster I/O

To: linux-raid@xxxxxxxxxxxxxxx
Subject: Linux RAID & XFS Question - Multiple levels of concurrency = faster I/O on md/RAID 5?
From: Justin Piszcz <jpiszcz@xxxxxxxxxxxxxxx>
Date: Sat, 1 Nov 2008 04:29:18 -0400 (EDT)
Cc: xfs@xxxxxxxxxxx
User-agent: Alpine 1.10 (DEB 962 2008-03-14)

# echo 1 > /proc/sys/vm/drop_caches ; sync

* Single operation on RAID5 (read/write, like untar for example)
  - It starts when the jump of bi/bo occurs.
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0    176 6423384     12 151356    0    0   173   212   34   48  1  1 98  0
 0  0    176 6421688     12 153004    0    0    16   127 5151 1634  2  1 97  0
 0  0    176 6419952     12 154724    0    0     0    89 5205 1691  2  0 98  0
 0  0    176 6418216     12 156452    0    0     0    32 5346 1768  2  1 97  0
 1  0    176 6323928     12 249380    0    0 50456     0 5350 1854 10  2 82  5
 1  0    176 6072840     12 497488    0    0 127696     0 5462 1565 21  5 73  1
 1  0    176 5829388     12 737636    0    0 116528   108 5576 1830 22  5 73  0
 1  0    176 5639876     12 924496    0    0 97896 98525 6761 2095 15 13 68  4
 1  0    176 5439212     12 1122796    0    0 97676 102516 7403 2697 17 14 67  2
 1  0    176 5241408     12 1318032    0    0 97668 94740 6460 2059 16 12 71  1
 2  0    176 5044000     12 1512528    0    0 97704 98848 8209 2430 17 13 70  1
 1  0    176 4845076     12 1708524    0    0 97668 98761 6879 2490 16 13 70  1

* Two of these operations run on two different sets of data.
 2  0    176 4631564     12 1917264    0    0 111416 78104 7713 3118 17 12 68  2
2 0 176 4260696 12 2283256 0 0 181736 205732 7670 3028 31 23 45 1 2 0 176 3882464 12 2656564 0 0 195392 177052 7324 3271 31 23 39 8 1 1 176 3535608 12 2997788 0 0 160152 185052 9408 3724 32 25 36 7 0 2 176 3184864 12 3342392 0 0 163220 181252 8392 3582 32 24 37 7 3 1 176 2837420 12 3685484 0 0 170424 169939 9071 3242 30 21 43 5 2 1 176 2449528 12 4066656 0 0 190208 196776 7408 3178 33 25 38 4 3 0 176 2058540 12 4452064 0 0 194992 190408 8630 3230 33 24 39 5 7 0 176 1692204 12 4812832 0 0 176528 185336 8583 3838 32 21 40 6
 8  0    176 1302428     12 5195400    0    0 195332 186460 9345 3663 33 25 37

* Three of these operations run on three different sets of data.
 2  1    176 910448     12 5580544    0    0 184484 204909 8533 3109 35 25 34  6
 2  1    176 487040     12 5997284    0    0 211716 205592 9795 3263 36 24 22 19
 2  0    176  40456     12 6437196    0    0 222324 229712 7932 2952 40 29 20 12
 6  0    176  45348     12 6433304    0    0 279344 230608 7553 4077 38 29 25  7
 3  1    176  44784     12 6434256    0    0 197052 247164 9109 4454 42 30 19 10
 3  0    176  44856     12 6433404    0    0 256128 250500 8505 3924 42 31 17 11
 7  0    176  43832     12 6435116    0    0 279352 250440 7998 4171 41 31 21  6
 2  1    176  43888     12 6434544    0    0 214440 234088 9106 4181 41 29 19 11
 5  0    176  45676     12 6433164    0    0 230512 263132 8720 4289 45 30 16  9
 5  0    176  45040     12 6433164    0    0 287536 229856 7886 4669 40 30 19 12
 8  0    176  46012     12 6432800    0    0 257844 147884 9291 4833 46 24 18 12
 9  1    176  46072     12 6432492    0    0 187156 361096 8643 3738 38 35 21  6

Overall the raw speed according to vmstat seems to increase as you add more
load to the server. So I decided to time running three jobs on two parts of data and compare it with a single job that proceses them all.

Three jobs run con-currently: (2 parts/each):

1- 59.99user 18.25system 2:02.07elapsed 64%CPU (0avgtext+0avgdata 0maxresident)k
   0inputs+0outputs (0major+21000minor)pagefaults 0swaps

2- 59.86user 17.78system 1:59.96elapsed 64%CPU (0avgtext+0avgdata 0maxresident)k
   0inputs+0outputs (21major+20958minor)pagefaults 0swaps

3- 74.77user 22.83system 2:13.30elapsed 73%CPU (0avgtext+0avgdata 0maxresident)k
   0inputs+0outputs (36major+21827minor)pagefaults 0swaps

One job with (6 parts):

1 188.66user 56.84system 4:38.52elapsed 88%CPU (0avgtext+0avgdata 0maxresident)k
  0inputs+0outputs (71major+43245minor)pagefaults 0swaps

Why is running 3 jobs con-currently that take care of two parts each more than
twice as fast than running one job for six parts?

I am using XFS and md/RAID-5, the CFQ scheduler and kernel 2.6.27.4.
Is this more of an md/raid issue ( I am guessing ) than XFS? I remember reading of some RAID acceleration patches awhile back that were supposed to boost performance quite a bit, what happened to them?

Justin.

<Prev in Thread] Current Thread [Next in Thread>