Andi Kleen wrote:
> On Tue, Jan 30, 2001 at 10:08:02AM +0100, Andrew Morton wrote:
> > Lots of interesting things here.
> > - eepro100 generates more interrupts doing TCP Tx, but not
> > TCP Rx. I assume it doesn't do Tx mitigation?
> The Intel driver (e100.c) uploads special firmware and does it for RX and TX.
> eepro100 doesn't. Perhaps you could measure that driver too?
Sure. Anyone have a URL for intel's driver?
> > - Changing eepro100 to use IO operations instead of MMIO slows
> > down this dual 500MHz machine by less than one percent at
> > 100 mbps. At 12,000 interrupts per second. Why all the fuss
> > about MMIO?
> iirc Ingo at some point found at some monster machine that the IO operations
> in the eepro100 interrupt handler dominated some Tux profile.
mm... My cycle soaker is not aggressive enough in its
generation of memory traffic. I think I need to tune it
so when all CPUs are 'soaking', memory traffic _just_
reaches 100%. This doesn't make much difference at
all at the low utilisation levels of a 100bT NIC.
However I doubt if that's relevant to the PIO vs MMIO
issue. IIRC, an I/O op only flushes the write buffers,
and blocks the CPU on writes.
> > - Without the zc patch, there is a significant increase (25%) in
> > the number of Rx packets (acks, persumably) when data is sent
> > using sendfile() as opposed to when the same data is sent
> > with send().
> RX on the sender?
Amazing but true.
If the sender is sending with sendfile(), he gets 25% more
packets sent back at him than if he's sending the same data
With the zerocopy patch this discrepancy disappears - the
machine doing the sendfile() receives less packets with zc.
** Yup, I just retested everything. With vanilla 2.4.1,
when data is sent with sendfile() we get 25% more packets
coming back than when sending with 64 kbyte send()s. And
25% more bytes.
With 2.4.1+zc, the number of packets coming back to the
sendfile()r is the same as the number of packets coming back
to the send()er, for both an SG_xsum NIC and a non-SG+xsum
one. Average packets size om the receive path is 13.1 bytes.
So by intent or by accident, something got fixed!
> > Workload: 62 files, average size 350k.
> > sendfile() tries to send the entire file in one hit
> > send() breaks it up into 64kbyte chunks.
> > When the zerocopy patch is applied, the Rx packet rate during
> > sendfile() is the same as the rate during send().
> > Why is this?
> Does the send() variant use TCP_CORK ?
`zcc' supports that, and it makes no difference. You see,
all the send()s are really big, and there's always data queued
in the transmit direction so everything gets coalesced into
Even when sending 64 byte chunks with send() and no TCP_CORK,
the frames on the wire are 500-1000 bytes long because of this.
> > - I see a consistent 12-13% slowdown on send() with the zerocopy
> > patch. Can this be fixed?
Well, although the ratio is a bit ugh, it's only 2% of the capacity
of a dual 500 box at 100 mbps. It's hard to think of an app which has
high enough bandwidth-over-compute requirements for this to matter.
GigE may be a totally different story. With GigE the ratio
between memory use and interrupt load is vastly different,
and the 12-13% ratio could be a lot larger. But without
knowing the cause, I'm guessing.
I'll swap my postal address for a couple of Acenics :)