Bad performance with XFS + 2.6.38 / 2.6.39
Yann Dupont
Yann.Dupont at univ-nantes.fr
Thu Dec 22 05:02:53 CST 2011
Le 22/12/2011 10:23, Yann Dupont a écrit :
>
>> Can you run a block trace on both kernels (for say five minutes)
>> when the load differential is showing up and provide that to us so
>> we can see how the IO patterns are differing?
here we go.
1st server : Birnie, is running 2.6.26. This is normally the more loaded
server (more active users)
2nd server : Penderyn, is runing a freshly compiled 3.1.6.
blktrace of relevent volumes during 10 minutes. The 2 machines are
identical (poweredge M1610) : same mem & proc, disks, fibre channel
cards, SAN disks ...
birnie:~/TRACE# uptime
11:48:34 up 17:18, 3 users, load average: 0.04, 0.18, 0.23
penderyn:~/TRACE# uptime
11:48:30 up 23 min, 3 users, load average: 4.03, 3.82, 3.21
As you can see, very sensible load difference. keep in mind my
university is on holiday right now, so the load is really _very much
lower_ than usual. In normal times, with 2.6.26 kernels, birnie has a
load in 2 .. 6 range.
here are the results :
birnie:~/TRACE# blktrace /dev/gromelac/gromelac
/dev/POMEROL-R0-P0/gromeldi -w 600
=== dm-18 ===
CPU 0: 26787 events, 1256 KiB data
CPU 1: 530 events, 25 KiB data
CPU 2: 1811 events, 85 KiB data
CPU 3: 104 events, 5 KiB data
CPU 4: 5824 events, 274 KiB data
CPU 5: 146 events, 7 KiB data
CPU 6: 1958 events, 92 KiB data
CPU 7: 176 events, 9 KiB data
CPU 8: 5456 events, 256 KiB data
CPU 9: 175 events, 9 KiB data
CPU 10: 1161 events, 55 KiB data
CPU 11: 216 events, 11 KiB data
CPU 12: 118 events, 6 KiB data
CPU 13: 25 events, 2 KiB data
CPU 14: 287 events, 14 KiB data
CPU 15: 425 events, 20 KiB data
Total: 45199 events (dropped 0), 2119 KiB data
=== dm-16 ===
CPU 0: 27966 events, 1311 KiB data
CPU 1: 311 events, 15 KiB data
CPU 2: 1403 events, 66 KiB data
CPU 3: 1699 events, 80 KiB data
CPU 4: 1706 events, 80 KiB data
CPU 5: 1515 events, 72 KiB data
CPU 6: 30 events, 2 KiB data
CPU 7: 428 events, 21 KiB data
CPU 8: 6774 events, 318 KiB data
CPU 9: 252 events, 12 KiB data
CPU 10: 1299 events, 61 KiB data
CPU 11: 1391 events, 66 KiB data
CPU 12: 111 events, 6 KiB data
CPU 13: 2317 events, 109 KiB data
CPU 14: 130 events, 7 KiB data
CPU 15: 504 events, 24 KiB data
Total: 47836 events (dropped 0), 2243 KiB data
and
penderyn:~/TRACE# blktrace /dev/gromeljo/gromeljo /dev/gromelpz/gromelpz
/dev/POMEROL-R1-P0/gromelpz -w 600
=== dm-14 ===
CPU 0: 12672 events, 595 KiB data
CPU 1: 13248 events, 621 KiB data
CPU 2: 545 events, 26 KiB data
CPU 3: 285 events, 14 KiB data
CPU 4: 574 events, 27 KiB data
CPU 5: 94 events, 5 KiB data
CPU 6: 569 events, 27 KiB data
CPU 7: 172 events, 9 KiB data
CPU 8: 666 events, 32 KiB data
CPU 9: 3231 events, 152 KiB data
CPU 10: 610 events, 29 KiB data
CPU 11: 221 events, 11 KiB data
CPU 12: 11 events, 1 KiB data
CPU 13: 20 events, 1 KiB data
CPU 14: 6 events, 1 KiB data
CPU 15: 30 events, 2 KiB data
Total: 32954 events (dropped 0), 1545 KiB data
=== dm-13 ===
CPU 0: 0 events, 0 KiB data
CPU 1: 0 events, 0 KiB data
CPU 2: 1 events, 1 KiB data
CPU 3: 0 events, 0 KiB data
CPU 4: 0 events, 0 KiB data
CPU 5: 0 events, 0 KiB data
CPU 6: 0 events, 0 KiB data
CPU 7: 0 events, 0 KiB data
CPU 8: 0 events, 0 KiB data
CPU 9: 0 events, 0 KiB data
CPU 10: 0 events, 0 KiB data
CPU 11: 0 events, 0 KiB data
CPU 12: 0 events, 0 KiB data
CPU 13: 0 events, 0 KiB data
CPU 14: 0 events, 0 KiB data
CPU 15: 0 events, 0 KiB data
Total: 1 events (dropped 0), 1 KiB data
=== dm-16 ===
CPU 0: 17499 events, 821 KiB data
CPU 1: 15320 events, 719 KiB data
CPU 2: 1037 events, 49 KiB data
CPU 3: 667 events, 32 KiB data
CPU 4: 278 events, 14 KiB data
CPU 5: 91 events, 5 KiB data
CPU 6: 888 events, 42 KiB data
CPU 7: 67 events, 4 KiB data
CPU 8: 2317 events, 109 KiB data
CPU 9: 3662 events, 172 KiB data
CPU 10: 1756 events, 83 KiB data
CPU 11: 801 events, 38 KiB data
CPU 12: 20 events, 1 KiB data
CPU 13: 618 events, 29 KiB data
CPU 14: 3 events, 1 KiB data
CPU 15: 18 events, 1 KiB data
Total: 45042 events (dropped 0), 2112 KiB data
And The blktrace files are there (for five days) :
http://filex.univ-nantes.fr/get?k=RDxGitXYOf4HKHd7Tan
Hope it can be helpfull,
Thanks,
--
Yann Dupont - Service IRTS, DSI Université de Nantes
Tel : 02.53.48.49.20 - Mail/Jabber : Yann.Dupont at univ-nantes.fr
More information about the xfs
mailing list