Short: TCP puts the system into the idle state even though there are data in
transit.
During coding a distributed application I discovered a TCP latency issue. The
application does a lot of request forwarding like P2P protocols.
I have tried to track down the problem and have written a small program
(random-tcp.c) that shows the long latencies. In this program one message is
passed round between a number om processes. Each time a process receives the
message it randomly chooses a process to forward to next.
This I have compared to a program that doesn't give long latencies
(ring-tcp.c). In this program each process always forwards to the same
process (ring topology).
I have also made the same programs using SCTP and this protocol has no issue
in the random case.
The following is a test with 16 processes forwarding the message 100000 times.
The avg. forwarding time from process to process is messured.
$ ./random-tcp 16 100000
avg forwarding time: 0.000326
$ ./ring-tcp 16 100000
avg forwarding time: 0.000044
$ ./random-sctp 16 100000
avg forwarding time: 0.000068
$ ./ring-sctp 16 100000
avg forwarding time: 0.000067
Using 'top' i have observed that the system spends time in the idle state when
running 'random-tcp'. This I have observed with just 3 processes. With 16
processes the CPU is only 20% loaded on my Mobile Intel(R) Celeron(R) CPU
1.60GHz.
I have also tried with socketpair()'s which didn't have the problem. Therefore
my conclusion is that it must be a TCP issue.
Now this local use of TCP is not that usefull. Therefore, I tried a MPI
version and tested this in a 16 node cluster. Here the random case is 5 times
slower than the ring.
I have tested on many kernel versions from 2.4.25 up until 2.6.12-rc5 and all
had this issue.
A few people on lkml also confirmed it, but I have not got any reply from
someone with a greater knowledge of the inner working of Linux TCP (at least
they didn't tell me that they had this knowledge :-).
I hope this is helpfull.
Regards
Hans Henrik Happe
random-sctp.c
Description: Text Data
random-tcp.c
Description: Text Data
ring-sctp.c
Description: Text Data
ring-tcp.c
Description: Text Data
|