Re: [RFC] netif_rx: receive path optimization

[Rick Jones <rick.jones2@xxxxxx>]

> > > Note Linux is quiet resilient to reordering compared to other OSes (as
> > > you may know) but avoiding this is a better approach - hence my
> > > suggestion to use NAPI when you want to do serious TCP.
> >
> > Would the same apply to NIC->CPU interrupt assignments? That is, bind the NIC to 
> > a single CPU.
>
>
> No reordering there.

Ah, I wasn't clear - would someone doing serious TCP want to have the interrupts 
of a NIC go to a specific CPU.

> > > Dont think we can do that unfortunately: We are screwed by the APIC
> > > architecture on x86.
> >
> > The IPS and TOPS stuff was/is post-NIC-interrupt. Low-level driver processing 
> > still happened/s on a specific CPU, it is the higher-level processing which is 
> > done on another CPU.  The idea - with TOPS at least, is to try to access the ULP 
> > (TCP, UDP etc) structures on the same CPU as last accessed by the app to 
> > minimize that cache to cache migration.
>
>
> But if interupt happens on "wrong" cpu - and you decide higher level
> processing is to be done on the "right" cpu (i assume queueing on some
> per CPU queue); then isnt that expensive? Perhaps IPIs involved even?

More expensive than if one were lucky enough to have the interrupt on the 
"right" CPU in the first place, but as the CPU count goes-up, the chances of 
that go down.  The main idea behind TOPS and prior to that IPS was to spread-out 
the processing of packets across as many CPUs as we could, as "correctly" as we 
could.  Lots of small packets meant/means that a NIC could saturate its 
interrupt CPU before the NIC was saturated.  You don't necessarily see that on 
say single-instance netperf TCP_STREAM (or basic FTP) testing, but certainly can 
on aggregate netperf TCP_RR testing.

IPS, being driven by the packet header info, was good enough for simple 
benchmarking, but once you had more than one connection per process/thread that 
wasn't going to cut it, and even with one connection per process telling the 
process where it should run wasn't terribly easy :)   It wasn't _that_ much more 
expensive than the queueing already happening - IPS was when HP-UX networking 
was BSDish and it was done when things were being queued to the netisr queue(s).

TOPS lets the process (I suppose the scheduler really) decide where some of the 
processing for the packet will happen - the part after the handoff.

rick