netdev
[Top] [All Lists]

Re: More measurements

To: Andi Kleen <ak@xxxxxx>
Subject: Re: More measurements
From: Andrew Morton <andrewm@xxxxxxxxxx>
Date: Wed, 31 Jan 2001 00:47:10 +1100
Cc: netdev@xxxxxxxxxxx
References: <3A75785A.42B9E7CE@xxxxxxxxxx>, <3A75785A.42B9E7CE@xxxxxxxxxx>; from Andrew Morton on Tue, Jan 30, 2001 at 10:08:02AM +0100 <20010130124839.24151@xxxxxxxxxxxx>
Sender: owner-netdev@xxxxxxxxxxx
Andi Kleen wrote:
> 
> On Tue, Jan 30, 2001 at 10:08:02AM +0100, Andrew Morton wrote:
> > Lots of interesting things here.
> >
> > - eepro100 generates more interrupts doing TCP Tx, but not
> >   TCP Rx.  I assume it doesn't do Tx mitigation?
> 
> The Intel driver (e100.c) uploads special firmware and does it for RX and TX.
> eepro100 doesn't. Perhaps you could measure that driver too?

Sure.  Anyone have a URL for intel's driver?

> >
> > - Changing eepro100 to use IO operations instead of MMIO slows
> >   down this dual 500MHz machine by less than one percent at
> >   100 mbps.  At 12,000 interrupts per second. Why all the fuss
> >   about MMIO?
> 
> iirc Ingo at some point found at some monster machine that the IO operations
> in the eepro100 interrupt handler dominated some Tux profile.

mm...  My cycle soaker is not aggressive enough in its
generation of memory traffic.  I think I need to tune it
so when all CPUs are 'soaking', memory traffic _just_
reaches 100%.  This doesn't make much difference at
all at the low utilisation levels of a 100bT NIC.

However I doubt if that's relevant to the PIO vs MMIO
issue.  IIRC, an I/O op only flushes the write buffers,
and blocks the CPU on writes.

> ...
> >
> > - Without the zc patch, there is a significant increase (25%) in
> >   the number of Rx packets (acks, persumably) when data is sent
> >   using sendfile() as opposed to when the same data is sent
> >   with send().
> 
> RX on the sender?

Amazing but true.

If the sender is sending with sendfile(), he gets 25% more
packets sent back at him than if he's sending the same data
using send().

With the zerocopy patch this discrepancy disappears - the
machine doing the sendfile() receives less packets with zc.

** Yup, I just retested everything.  With vanilla 2.4.1,
when data is sent with sendfile() we get 25% more packets
coming back than when sending with 64 kbyte send()s.  And
25% more bytes.

With 2.4.1+zc, the number of packets coming back to the
sendfile()r is the same as the number of packets coming back
to the send()er, for both an SG_xsum NIC and a non-SG+xsum
one.  Average packets size om the receive path is 13.1 bytes.

So by intent or by accident, something got fixed!

> >
> >   Workload: 62 files, average size 350k.
> >             sendfile() tries to send the entire file in one hit
> >             send() breaks it up into 64kbyte chunks.
> >
> >   When the zerocopy patch is applied, the Rx packet rate during
> >   sendfile() is the same as the rate during send().
> >
> >   Why is this?
> 
> Does the send() variant use TCP_CORK ?

`zcc' supports that, and it makes no difference.  You see,
all the send()s are really big, and there's always data queued
in the transmit direction so everything gets coalesced into
full-sized frames.

Even when sending 64 byte chunks with send() and no TCP_CORK,
the frames on the wire are 500-1000 bytes long because of this.

> > - I see a consistent 12-13% slowdown on send() with the zerocopy
> >   patch.  Can this be fixed?
> 
> Ugh.

Well, although the ratio is a bit ugh, it's only 2% of the capacity
of a dual 500 box at 100 mbps.  It's hard to think of an app which has
high enough bandwidth-over-compute requirements for this to matter.

GigE may be a totally different story.  With GigE the ratio
between memory use and interrupt load is vastly different,
and the 12-13% ratio could be a lot larger.   But without
knowing the cause, I'm guessing.

I'll swap my postal address for a couple of Acenics :)

-

<Prev in Thread] Current Thread [Next in Thread>