Rick Jones wrote:
Have you done any tests with 2 tg3 NICs in a single machine to see if
they
can run at or near line speed (full duplex)?
It isn't just a question of two tg3 NICs in the same box is it? You are
running two NICs on the same bus right? And unless my dimm memory is
mistaken, four ports on a card with 5704s means two 5704's a bridge chip
right? So, it would be two tg3 NICs going through the same bridge chip,
not just the same bus or same system. I'd be worrying about DMA
latencies on the system and the bridge chip, and perhaps the efficiency
of the PCI-X bus usage (not sure - is there anything in your system's
chipset to extract that sort of information?)
There will be a bridge chip, and indeed I see better performance when I just
use a 2-port Intel NIC as opposed to a 4 port, even if I am only actively
using 2 of the 4 ports on the 4-port NIC. For the tg3 hardware I only
have a 4-port NIC. I do assume that a 2-port tg3 NIC w/out a bridge chip
would be faster..but probably not too much.
What happens when you turn pktgen around/insideout and source packets
from the bridging system to each of the (two other?) systems?
I looped two ports on the same NIC together for the pktgen tests, so there
is only a single machine in question. With Intel I can source/sink about
960Mbps on two ports simultaneously in this configuration. With the tg3
NIC I can only do about 750Mbps.
And, the tg3 is in the faster PCI-X slot (133Mhz v/s 100Mhz). So, to me
it appears that the tg3 hardware and/or driver can only handle about 80%
of the performance that the intel e1000 can produce. It's possible I have
a particularly sub-optimal configuration for tg3, or maybe a poorly designed
NIC, which is why I'd like to know what others see...
Since you are bridging, does having CKO enabled really matter? Mightn't
that allow the firmware on the 5704(s) to run a triffle faster? Or does
bridging already not request CKO (I suppose it might).
CKO == IP checksum offload?
Since Dave doesn't want to debug my bridge setup (and I don't blame him),
I am going to try to focus my testing/debug reports on the pktgen tests.
If/when pktgen shows better performance with tg3, I can verify that
I see the same speedups with my proprietary bridging module. I've no idea
if CKO would help or hinder pktgen, nor have I tried to enable or disable it.
Are your interface interrupts distributed across the CPUs?
I'm using FC2, basically a default install. It does seem to have
an irq balance daemon running. But, I'm not specifically binding IRQs
or anything like that. pktgen tx is running as a single thread, so the rx
code could run mostly on the other CPU if locking allows...
Thanks,
Ben
--
Ben Greear <greearb@xxxxxxxxxxxxxxx>
Candela Technologies Inc http://www.candelatech.com
|