non-growing tcp window size

[Håkon Bugge <Hakon.Bugge@xxxxxxxxx>]

To whom it might concern,

We have observed bandwidth problems of TCP when running benchmarks which
gradually increase its payload and having (most of its) data flowing
unidirectional. The degradation is observed between two dual XEON/E7500
machines, using Gbe and a cross-over cable. The Gbe is eth1 and has an
MTU of 9000. eth0 is a FE with an MTU of 1500. The machines is running
linux 2.4.20, and the NIC in question is an Intel Corp. 82544GC Gigabit
Ethernet Controller (rev 02).

During an attempt to change TCP parameter settings, and also
systematically changing socket rx/tx buffer sizes, I discovered that once
in a while, the benchmark ran well. The OK run was not deterministic, and
had nothing to do with change of the parameter settings. Hence, I let the
machine work during the week-end, and took tcpdumps which I saved for the
successful run. An analysis of the "OK" run vs. the
"BAD" run, discovers a couple of interesting things. The
problem is that the advertised window of the receiver does not increase.
The second problem, which also affects the "OK" run, is that
the ratio of packets sent (from src to dst) and the number of packets
received (#advertisements) is 1 for the "BAD" scenario
(actually this is probably a consequence of the window being only ~2xMTU
size). Surprisingly, it is as large as 0.5 for the "OK"
scenario. IMHO, this violates RFC813, which states:

   There are two reasons  for  prompt  acknowledgement.    One  is  to

prevent  retransmission.  We will discuss later how to determine whether

unnecessary  retransmission  is  occurring.    The  other   reason   one

acknowledges  promptly  is  to permit further data to be sent.  However,

the previous section makes quite clear that it is not  always  desirable

to send a little bit of data, even though the receiver may have room for

it.    Therefore,  one  can  state  a  general  rule  that  under normal

operation, the receiver of data need not,  and  for  efficiency  reasons

should  not,  acknowledge  the data unless either the acknowledgement is

intended to produce an increased useable window, is necessary  in  order

to  prevent  retransmission  or  is  being  sent  as  part  of a reverse

direction segment being sent for some other reason.  We will consider an

algorithm to achieve these goals.

The two tcpdumps has been analyzed by a simple awk scripts, which
gives information for every 1/10 of a second of the runtime:

time: elapsed time relative to the first packet
MB/s: sum of TCP payload (based on TCP sequence numbers) *1e-6 / delta
time
avg_len: average packet length (TCP payload) sent
nsent: no of packets sent from src to dst
avg_win: average window size advertised by the src
adv/sent: ratio between #advertisements (sum of prompt acks and
piggybacked acks) and #packets sent

The extract of the analysis is enclosed as "ok.txt" and
"bad.txt". Also, the information is presented as graphs in
"bw_winsiz.png" and "bw_adv-sent.png". In the graphs,
the "OK" data uses the bottom x-axis, the "BAD" data
uses the upper x-axis (the "BAD" run-time is longer due to the
lower bandwidth). Bandwidth uses the left y-axes, whereas the average
advertised window size and the ratio of the #advertisements and #packets
sent uses the rightmost x-axis.

I do hope this information is useful 4u and that the problem can be
fixed. Since I do not read the mailing lists, I would appreciate a note
back by email of someone triggers on this.

Cheers, Håkon

--
Håkon Bugge; VP Product Development; Scali AS;
mailto:hob@xxxxxxxx; http://www.scali.com; fax: +47 22 62 89 51;
Voice: +47 22 62 89 50; Cellular (Europe+US): +47 924 84 514;
Visiting Addr: Olaf Helsets vei 6, Bogerud, N-0621 Oslo, Norway;
Mail Addr:  Scali AS, Postboks 150, Oppsal, N-0619  Oslo, Norway;

bw_adv-sent.png
Description: PNG image

bad.txt
Description: Text document

bw_winsiz.png
Description: PNG image

ok.txt
Description: Text document