Re: netif_rx packet dumping

[Baruch Even <baruch@xxxxxxxxx>]

jamal wrote:
> On Fri, 2005-03-04 at 03:47, Baruch Even wrote:
>
> > jamal wrote:
>
>
> > > Can you explain a little more? Why does the the throttling cause any
> > > bad behavior thats any different from the queue being full? In both
> > > cases, packets arriving during that transient will be dropped.
> >
> > If you have 300 packets in the queue and the throttling kicks in you now 
> > drop ALL packets until the queue is empty, this will normally take some 
> > time, during all of this time you are dropping all the ACKs that are 
> > coming in, you lose SACK information and potentially you leave no packet 
> > in flight so that the next packet will be sent only due to retransmit 
> > timer waking up, at which point your congestion control algorithm starts
> > from cwnd=1.
> >
> > You can look at the report http://hamilton.ie/net/LinuxHighSpeed.pdf for 
> > some graphs of the effects.
>
> Were the processors tied to NICs? 

No. These are single CPU machines (with HT).

> Your experiment is more than likely a single flow, correct?

Yes.

> In other words the whole queue was infact dedicated just for your one
> flow - thats why you can call this queue a transient burst queue. 

Indeed, For a router or a web server handling several thousand flows it 
might be different, but I don't expect it handles a single packet in one 
 ms (or more) as it happens for the current end-system ack handling code.

> Do you still have the data that shows how many packets were dropped
> during this period. Do you still have the experimental data? I am
> particulary interested in seeing the softnet stats as well as tcp
> netstats.

No, These tests were not run by me, I'll probably rerun similar tests as 
well to base my work on, send me in private how do I get the stats from 
the kernel and I'll add it to my test scripts.

> I think your main problem was the huge amounts of SACK on the writequeue
> and the resultant processing i.e section 1.1 and how you resolved that.

That is my main guess as well, the original work was done rather 
quickly, we are now reorganizing thoughts and redoing the tests in a 
more orderly fashion.

> I dont see any issue in dropping ACKs, many of them even for such large
> windows as you have - TCPs ACKs are cummulative. It is true if you drop
> "large" enough amounts of ACKS, you will end up in timeouts - but large
> enough in your case must be in the minimal 1000 packets. And to say you
> dropped a 1000 packets while processing 300 means you were taking too
> long processing the 300.

With the current code SACK processing takes a long time, so it is 
possible that it happened to drop more than a thousand packets while 
handling 300. I think that after the fixing of the SACK code, the rest 
might work without getting to much into the ingress queue. But that 
might still change when we go to even higher speeds.

> Then what would be really interesting is to see the perfomance you get
> from multiple flows with and without congestion.

We'd need to get a very high speed link for multiple high speed flows.

> I am not against a the benchmarky nature of the single flow and tuning
> for that, but we should also look at a wider scope at the effect before
> you handwave based on the result of one testcase.

I can't say I didn't handwave, but then, there is little experimentation 
done to see if the other claims are correct and that AFQ is really 
needed so early in the packet receive stage. There are also voices that 
say AFQ sucks and causes more damage than good, I don't remember details 
currently.

> So if i was you i would repeat 1.2 with the fix from 1.1 as well as
> tying the NIC to one CPU. And it would be a good idea to present more
> detailed results - not just tcp windows fluctuating (you may not need
> them for the paper, but would be useful to see for debugging purposes
> other parameters).

I'd be happy to hear what other benchmarks you would like to see, I 
currently intend to add some ack processing time analysis and oprofile 
information. With possibly showing the size of the ingress queue as a 
measure as well.

Making it as thorough as possible is one of my goals. Input is always 
welcome.

Baruch