Andi Kleen wrote:
>
> On Tue, Jan 30, 2001 at 10:08:02AM +0100, Andrew Morton wrote:
> > Lots of interesting things here.
> >
> > - eepro100 generates more interrupts doing TCP Tx, but not
> > TCP Rx. I assume it doesn't do Tx mitigation?
>
> The Intel driver (e100.c) uploads special firmware and does it for RX and TX.
> eepro100 doesn't. Perhaps you could measure that driver too?
Sure. Anyone have a URL for intel's driver?
> >
> > - Changing eepro100 to use IO operations instead of MMIO slows
> > down this dual 500MHz machine by less than one percent at
> > 100 mbps. At 12,000 interrupts per second. Why all the fuss
> > about MMIO?
>
> iirc Ingo at some point found at some monster machine that the IO operations
> in the eepro100 interrupt handler dominated some Tux profile.
mm... My cycle soaker is not aggressive enough in its
generation of memory traffic. I think I need to tune it
so when all CPUs are 'soaking', memory traffic _just_
reaches 100%. This doesn't make much difference at
all at the low utilisation levels of a 100bT NIC.
However I doubt if that's relevant to the PIO vs MMIO
issue. IIRC, an I/O op only flushes the write buffers,
and blocks the CPU on writes.
> ...
> >
> > - Without the zc patch, there is a significant increase (25%) in
> > the number of Rx packets (acks, persumably) when data is sent
> > using sendfile() as opposed to when the same data is sent
> > with send().
>
> RX on the sender?
Amazing but true.
If the sender is sending with sendfile(), he gets 25% more
packets sent back at him than if he's sending the same data
using send().
With the zerocopy patch this discrepancy disappears - the
machine doing the sendfile() receives less packets with zc.
** Yup, I just retested everything. With vanilla 2.4.1,
when data is sent with sendfile() we get 25% more packets
coming back than when sending with 64 kbyte send()s. And
25% more bytes.
With 2.4.1+zc, the number of packets coming back to the
sendfile()r is the same as the number of packets coming back
to the send()er, for both an SG_xsum NIC and a non-SG+xsum
one. Average packets size om the receive path is 13.1 bytes.
So by intent or by accident, something got fixed!
> >
> > Workload: 62 files, average size 350k.
> > sendfile() tries to send the entire file in one hit
> > send() breaks it up into 64kbyte chunks.
> >
> > When the zerocopy patch is applied, the Rx packet rate during
> > sendfile() is the same as the rate during send().
> >
> > Why is this?
>
> Does the send() variant use TCP_CORK ?
`zcc' supports that, and it makes no difference. You see,
all the send()s are really big, and there's always data queued
in the transmit direction so everything gets coalesced into
full-sized frames.
Even when sending 64 byte chunks with send() and no TCP_CORK,
the frames on the wire are 500-1000 bytes long because of this.
> > - I see a consistent 12-13% slowdown on send() with the zerocopy
> > patch. Can this be fixed?
>
> Ugh.
Well, although the ratio is a bit ugh, it's only 2% of the capacity
of a dual 500 box at 100 mbps. It's hard to think of an app which has
high enough bandwidth-over-compute requirements for this to matter.
GigE may be a totally different story. With GigE the ratio
between memory use and interrupt load is vastly different,
and the 12-13% ratio could be a lot larger. But without
knowing the cause, I'm guessing.
I'll swap my postal address for a couple of Acenics :)
-
|