In my tests, I have found no mesaurable difference in performance between
stock 2.6.14-rc2, pnotify with the subscriber list protected by the
rwsem lock like it used to be, and pnotify with read protections
implemented with RCU.
Problem: Job allocates memory and uses rw semaphores quite frequently.
This seems to be illegal from within rcu_read_lock() / rcu_read_unlock().
Running with CONFIG_DEBUG_SPINLOCK_SLEEP confirms this. I'm not sure some of
the things we want to do in Job are possible (or at least not easily possible)
if this kernel module subscriber can never sleep. Even the keyring
infrastructure needs to allocate memory sometimes.
My opinion is: Let's not use the RCU version of pnotify and stick with the
rwsem version.
- Since you aren't supposed to sleep, many (most) users can't make use of
pnotify.
- The performance data shows no difference between the rwsem version
with two subscribers (keyring and job) vs a stock kernel with the
non-modified keyring support enabled.
- Using RCU for protection adds comlexity
Here are the details on the test runs. The RCU version "illegally" can
sleep but I didn't hit any problems in my tests.
IMPORTANT: The version of AIM7 used here is not currently tracking
the community version - ours is closer to the original AIM7.
General Test info:
- For the two tests with pnotify (old rwsem protected subscriber list and
new wsem/rcu protected subscriber list), the tests were always fired off
with the shell process being in a job container. This means the tests
don't only compare RCU performance, but also some aspects of Linux job
performance. All child processes will have Job and keyring as subscribers.
Of course, the stock kernel doesn't have pnotify or job and uses the
non-modified keyring.
- forkexit is simple and just forks and exits the number of times supplied on
the command line. It reports the number of times fork returned -1.
- The 2.6.14-rc2 kernel was used for each test. The only variations in
patches and configuration related to which version of keyring, pnotify,
and Job were used (if any).
- jobtest is a mini job test suite. One of the tests forks/exits enough
processes for PIDs to wrap. A job_create is done for each forked
process, which means job is always a subscribed kernenl module to the
forked processes. I snipped out the exessive output in the results.
- keyring support is enabled in all kernels. For both pnotify kernel versions,
keyrings have been changed to support pnotify for task struct entries and
fork/exit calls.
- All kernels include the kdb patch. Otherwise, all kerenels used the
sn2_defconfig as a base.
- The test system was a 2-processor 1400 Mhz SGI Altix 350 (ia64)
pnotify & linux kernel job with RCU, pnotify-aware keyring
------------------------------------------------------------------------------
forkexit test runs (supplied number is the number of forks fired).
minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92468 times
real 0m0.916s
user 0m0.036s
sys 0m0.408s
minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92469 times
real 0m0.914s
user 0m0.024s
sys 0m0.412s
minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92468 times
real 0m0.909s
user 0m0.060s
sys 0m0.332s
minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92467 times
real 0m0.911s
user 0m0.052s
sys 0m0.380s
minime1:~ # time jobtest
[snip]
Great. All tests completed, no errors
real 0m11.339s
user 0m0.524s
sys 0m4.536s
minime1:~ # time jobtest
[snip]
Great. All tests completed, no errors
real 0m11.385s
user 0m0.500s
sys 0m4.592s
Linux version 2.6.14-rc2 (erikj@attica) (gcc version 3.3.3 (SuSE Linux)) #1 SMP
PREEMPT Wed Sep 28 12:11:05 CDT 2005
HOST = minime1
CPUS = 2
DIRS = 1
DISKS= 0
FS = xfs
SCSI = non-xscsi
ID = RCU pnotify+job
Run 1 of 1
Benchmark Version Machine Run Date
AIM Multiuser Benchmark - Suite VII "1.1" AIM7 Run Sep 28 16:00:45
2005
Tasks Jobs/Min JTI Real CPU Jobs/sec/task
1 1962.2 100 3.0 0.5 32.7040
2 4041.7 99 2.9 0.8 33.6806
3 5734.0 98 3.0 1.2 31.8555
4 7232.1 99 3.2 1.5 30.1336
5 8591.7 96 3.4 1.8 28.6389
10 13655.6 93 4.3 3.4 22.7593
20 19812.8 92 5.9 6.7 16.5106
50 26626.4 90 10.9 16.5 8.8755
100 30021.7 87 19.4 32.8 5.0036
150 31080.9 86 28.1 49.1 3.4534
200 31727.0 84 36.7 65.2 2.6439
500 32047.0 84 90.8 163.1 1.0682
1000 31954.7 83 182.1 325.8 0.5326
2000 32633.4 83 356.7 653.9 0.2719
pnotify & linux kernel job WITHOUT RCU - all rwsem, pnotify-aware keyring
------------------------------------------------------------------------------
forkexit test runs (supplied number is the number of forks fired).
minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92468 times
real 0m0.913s
user 0m0.028s
sys 0m0.428s
minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92470 times
real 0m1.032s
user 0m0.044s
sys 0m0.416s
minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92469 times
real 0m1.048s
user 0m0.048s
sys 0m0.368s
minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92469 times
real 0m0.948s
user 0m0.028s
sys 0m0.444s
minime1:/tmp # time jobtest
[snip]
Great. All tests completed, no errors
real 0m11.620s
user 0m0.612s
sys 0m4.540s
minime1:/tmp # time jobtest
[snip]
Great. All tests completed, no errors
real 0m11.268s
user 0m0.548s
sys 0m4.488s
Linux version 2.6.14-rc2 (erikj@attica) (gcc version 3.3.3 (SuSE Linux)) #2 SMP
PREEMPT Wed Sep 28 15:01:33 CDT 2005
HOST = minime1
CPUS = 2
DIRS = 1
DISKS= 0
FS = xfs
SCSI = non-xscsi
ID = non-RCU pnotify+job
Run 1 of 1
Benchmark Version Machine Run Date
AIM Multiuser Benchmark - Suite VII "1.1" AIM7 Run Sep 28 17:17:56
2005
Tasks Jobs/Min JTI Real CPU Jobs/sec/task
1 1964.9 100 3.0 0.5 32.7481
2 4020.7 99 2.9 0.8 33.5060
3 5630.4 98 3.1 1.2 31.2802
4 7218.6 97 3.2 1.5 30.0775
5 8637.6 97 3.4 1.8 28.7919
10 13857.1 94 4.2 3.5 23.0952
20 19429.1 93 6.0 6.7 16.1910
50 26126.8 88 11.1 16.5 8.7089
100 29950.6 86 19.4 32.7 4.9918
150 31052.1 86 28.1 49.1 3.4502
200 31817.2 85 36.6 65.4 2.6514
500 32101.1 84 90.7 163.1 1.0700
1000 31908.8 83 182.4 325.8 0.5318
2000 32718.3 82 355.8 653.7 0.2727
linux kernel without pnotify and without job, non-modified keyring enabled
------------------------------------------------------------------------------
forkexit test runs (supplied number is the number of forks fired).
minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92467 times
real 0m0.910s
user 0m0.024s
sys 0m0.356s
minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92467 times
real 0m0.912s
user 0m0.032s
sys 0m0.472s
minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92467 times
real 0m0.915s
user 0m0.044s
sys 0m0.400s
minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92469 times
real 0m0.923s
user 0m0.036s
sys 0m0.388s
(no jobtest tests since kernel doesn't have pnotify or job)
---------------------------------------------------------------------------
Linux version 2.6.14-rc2 (erikj@attica) (gcc version 3.3.3 (SuSE Linux)) #2 SMP
PREEMPT Wed Sep 28 14:53:02 CDT 2005
HOST = minime1
CPUS = 2
DIRS = 1
DISKS= 0
FS = xfs
SCSI = non-xscsi
ID = NO pnotify+job
Run 1 of 1
Benchmark Version Machine Run Date
AIM Multiuser Benchmark - Suite VII "1.1" AIM7 Run Sep 28 18:36:09
2005
Tasks Jobs/Min JTI Real CPU Jobs/sec/task
1 1964.2 100 3.0 0.5 32.7371
2 4024.9 99 2.9 0.8 33.5408
3 5539.3 99 3.2 1.2 30.7741
4 7123.6 96 3.3 1.5 29.6818
5 8617.1 95 3.4 1.8 28.7237
10 13713.5 96 4.2 3.4 22.8558
20 19732.2 92 5.9 6.7 16.4435
50 26411.3 88 11.0 16.4 8.8038
100 30001.5 86 19.4 32.7 5.0003
150 31184.1 86 28.0 49.0 3.4649
200 31714.9 84 36.7 65.4 2.6429
500 32160.7 84 90.5 163.0 1.0720
1000 31876.8 83 182.6 325.7 0.5313
2000 32708.1 83 355.9 653.0 0.2726
|