Note Linux is quiet resilient to reordering compared to other OSes (as
you may know) but avoiding this is a better approach - hence my
suggestion to use NAPI when you want to do serious TCP.
Would the same apply to NIC->CPU interrupt assignments? That is, bind the NIC to
a single CPU.
No reordering there.
Ah, I wasn't clear - would someone doing serious TCP want to have the interrupts
of a NIC go to a specific CPU.
Dont think we can do that unfortunately: We are screwed by the APIC
architecture on x86.
The IPS and TOPS stuff was/is post-NIC-interrupt. Low-level driver processing
still happened/s on a specific CPU, it is the higher-level processing which is
done on another CPU. The idea - with TOPS at least, is to try to access the ULP
(TCP, UDP etc) structures on the same CPU as last accessed by the app to
minimize that cache to cache migration.
But if interupt happens on "wrong" cpu - and you decide higher level
processing is to be done on the "right" cpu (i assume queueing on some
per CPU queue); then isnt that expensive? Perhaps IPIs involved even?
More expensive than if one were lucky enough to have the interrupt on the
"right" CPU in the first place, but as the CPU count goes-up, the chances of
that go down. The main idea behind TOPS and prior to that IPS was to spread-out
the processing of packets across as many CPUs as we could, as "correctly" as we
could. Lots of small packets meant/means that a NIC could saturate its
interrupt CPU before the NIC was saturated. You don't necessarily see that on
say single-instance netperf TCP_STREAM (or basic FTP) testing, but certainly can
on aggregate netperf TCP_RR testing.
IPS, being driven by the packet header info, was good enough for simple
benchmarking, but once you had more than one connection per process/thread that
wasn't going to cut it, and even with one connection per process telling the
process where it should run wasn't terribly easy :) It wasn't _that_ much more
expensive than the queueing already happening - IPS was when HP-UX networking
was BSDish and it was done when things were being queued to the netisr queue(s).
TOPS lets the process (I suppose the scheduler really) decide where some of the
processing for the packet will happen - the part after the handoff.