netdev
[Top] [All Lists]

RE: Intel and TOE in the news

To: "'Andi Kleen'" <ak@xxxxxx>, "'Leonid Grossman'" <leonid.grossman@xxxxxxxxxxxx>
Subject: RE: Intel and TOE in the news
From: "Alex Aizman" <alex@xxxxxxxxxxxx>
Date: Sun, 20 Feb 2005 17:57:56 -0800
Cc: "'rick jones'" <rick.jones2@xxxxxx>, <netdev@xxxxxxxxxxx>
In-reply-to: <20050220230713.GA62354@xxxxxx>
Organization: s2io
Reply-to: <alex@xxxxxxxxxxxx>
Sender: netdev-bounce@xxxxxxxxxxx
Thread-index: AcUXoOq3jpiGyWGCQqKFfhRTwUUM9AAElz0g
> You can always call netif_rx() multiple times from the 
> interrupt. 

That's what we do, simply because interrupts are coalesced. 

> The function doesn't do the full packet 
> processing,  but just stuffs the packet into a CPU queue that 
> is processed at a lower priority interrupt (softirq). 

There's the netif_rx's own per-packet overhead (including the call itself)
that arguably could be optimized..

> Most interesting would be to use per CPU TX completion 
> interrupts using MSI-X and avoid bouncing packets around between CPUs.

Our experimental Linux driver already supports MSI and MSI-X (the latter not
tested). Once/if multi-MSI support in Linux becomes available it'd be
practically possible to scale TCP connections with a number of CPUs.
Alternative: wait until Xframe II adapter w/MSI-X.. 


Alex

> -----Original Message-----
> From: Andi Kleen [mailto:ak@xxxxxx] 
> Sent: Sunday, February 20, 2005 3:07 PM
> To: Leonid Grossman
> Cc: 'rick jones'; netdev@xxxxxxxxxxx; 'Alex Aizman'
> Subject: Re: Intel and TOE in the news
> 
> On Sun, Feb 20, 2005 at 02:43:59PM -0800, Leonid Grossman wrote:
> > This is an interesting idea, we'll play around...
> 
> What exactly? The software only shadow table?
> 
> > 
> > BTW - does anybody know if it's possible to indicate 
> multiple receive 
> > packets?
> > 
> > In other OS drivers we have an option to indicate a "packet train" 
> > that got received during an interrupt, but I'm not sure 
> if/how it's doable in Linux.
> 
> You can always call netif_rx() multiple times from the 
> interrupt. The function doesn't do the full packet 
> processing,  but just stuffs the packet into a CPU queue that 
> is processed at a lower priority interrupt (softirq). 
> Doesn't work for NAPI unfortunately though; netif_receive_skb 
> always does the protocol stack.
> 
> > We are adding Linux driver support for message-signaled 
> interrupts and 
> > header separation (just recently figured out how to 
> indicate chained 
> > skb for
> 
> I had an experimental patch to enable MSI (without -X) for 
> your cards, but didn't push it out because i wasn't too happy with it.
> 
> Most interesting would be to use per CPU TX completion 
> interrupts using MSI-X and avoid bouncing packets around between CPUs.
> 
> > a packet that has IP and TCP headers separated by the ASIC); If a 
> > packet train indication works then the driver could prefetch the 
> > descriptor ring segment and also a rx buffer segment that holds 
> > headers stored back-to-back, before indicating the train.
> 
> Me and Jamal tried that some time ago, but it did not help too much.
> Probably because the protocol process overhead was not big enough.
> However that was with NAPI, might be perhaps worth trying without it. 
> 
> Problem is that need to fill in skb->protocol/pkt_type before 
> you can pass the packet up; you can perhaps derive it from 
> the RX descriptor (card has a bit that says "this is IP" and 
> "its unicast for my MAC"). 
> But the RX descriptor access is already a cache miss that 
> stalls you.  
> 
> To make the prefetching work well for this would probably 
> require a callback to the driver so that you can do this 
> later after your prefetch succeeded.
> 
> -Andi
> 
> 


<Prev in Thread] Current Thread [Next in Thread>