From owner-netdev@oss.sgi.com Thu Mar 1 02:26:32 2001 Received: by oss.sgi.com id ; Thu, 1 Mar 2001 02:26:12 -0800 Received: from hugin.diku.dk ([130.225.96.144]:30468 "HELO hugin.diku.dk") by oss.sgi.com with SMTP id ; Thu, 1 Mar 2001 02:26:04 -0800 Received: (qmail 15016 invoked from network); 1 Mar 2001 10:25:52 -0000 Received: from ask.diku.dk (firefly@130.225.96.225) by hugin.diku.dk with QMQP; 1 Mar 2001 10:25:52 -0000 Date: Thu, 1 Mar 2001 11:25:51 +0100 (MET) From: Peter Finderup Lund To: netdev@oss.sgi.com Subject: Docs on the routing part of the network stack? Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing Does anybody know where I can find some docs on how the routing part of the network stack works (only interested in IPv4)? In particular, something about the destination cache and the skbuff.dst field... -Peter A room without books is like a body without a soul. -- Marcus Tullius Cicero (106-43 B.C.) From owner-netdev@oss.sgi.com Thu Mar 1 05:56:45 2001 Received: by oss.sgi.com id ; Thu, 1 Mar 2001 05:56:35 -0800 Received: from nero.doit.wisc.edu ([128.104.17.130]:23564 "EHLO nero.doit.wisc.edu") by oss.sgi.com with ESMTP id ; Thu, 1 Mar 2001 05:56:14 -0800 Received: (from jleu@localhost) by nero.doit.wisc.edu (8.8.7/8.8.7) id HAA11611; Thu, 1 Mar 2001 07:55:43 -0600 Date: Thu, 1 Mar 2001 07:55:43 -0600 From: "James R. Leu" To: Rusty Russell Cc: Richard Guy Briggs , netfilter-devel@us5.samba.org, linux-ipsec@freeswan.org, netdev@oss.sgi.com Subject: Re: On Extending NFMark... Message-ID: <20010301075543.B11586@doit.wisc.edu> Reply-To: jleu@mindspring.com References: <20010220110113.C3910@grendel.conscoop.ottawa.on.ca> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2i In-Reply-To: ; from rusty@linuxcare.com.au on Thu, Mar 01, 2001 at 03:44:28PM +1100 Organization: none Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing Hello, I haven't been following this thread long from the beginning, but I have a patch that might be related. I wrote and MPLS implementation which needs to mark skbs and store an piece of info with the skb until it reaches the mpls_output() function. I created an array of void* on each skb. Each protocol that wants to use an entry in this array claims a index in that array (via a header file). There is also an array of function pointers, these are called when we want to clean up the array. The major downfall of this technique is the overhead of always having the array even though it may not be used. I like the idea of having a linked list, except for the case when we cannot allocate memory to add to the list, and because we would have to search the list to find a particular protocols info. (these reasons are why I choose an array) > The classic nfmark field problem; that there is only one. See also, > nfct. > > For a generic linked-list of blobs approach, there are several > problems: > 1) How do I tell which one is mine? > 2) What happens when packet is copied? > 3) What happens when packet is cloned? > 4) What happens when packet is destroyed? > For the array technique: 1. by the protocol index 2. Copy it. 3. Copy it. 4. Call the protocols functions pointer to clean up Jim -- James R. Leu From owner-netdev@oss.sgi.com Thu Mar 1 06:54:34 2001 Received: by oss.sgi.com id ; Thu, 1 Mar 2001 06:54:25 -0800 Received: from adsl-151-196-233-8.baltmd.adsl.bellatlantic.net ([151.196.233.8]:50475 "EHLO vaio.greennet") by oss.sgi.com with ESMTP id ; Thu, 1 Mar 2001 06:54:08 -0800 Received: from localhost (becker@localhost) by vaio.greennet (8.9.3/8.8.7) with ESMTP id JAA08400; Thu, 1 Mar 2001 09:54:19 -0500 Date: Thu, 1 Mar 2001 09:54:19 -0500 (EST) From: Donald Becker X-Sender: becker@vaio.greennet To: Matti Aarnio cc: Jeff Garzik , linux-net@vger.kernel.org, netdev@oss.sgi.com Subject: Re: rx_copybreak... In-Reply-To: <20010301082335.Q15688@mea-ext.zmailer.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing On Thu, 1 Mar 2001, Matti Aarnio wrote: > On Wed, Feb 28, 2001 at 09:21:24PM -0500, Jeff Garzik wrote: > > Instead of unconditionally copying packet sizes over rx_copybreak, in > > many ethernet drivers... is it worth it for the driver to check and see > > if the packet is already aligned? Or is that such a rare case that it > > shouldn't be worth it? > > To know if packet is aligned or not, one must look into > the packet to analyze protocols. ..which breaks an abstraction boundary, and may not be worth the overhead. It's very rare to see non-DIX packets, but the kernel should not crash or be easily D-O-Sed from the unaligned accesses when it does happen. > IMO it is far better to setup the card to do RX DMA into > DIX aligned IP frame (the most common case), but hardware > which is inherently unable to do that has drivers knowing it. I always assume a DIX frame. For every one of my PCI drivers I receive into a + 2 offset if the hardware supports it. This is documented in the operational description at the top of the driver file. It's usually easy to tell the chip capability even without reading the documentation: for bus master drivers look for skb_reserve(skb, 2). Arbitrary Tx and Rx alignment (best) 3c59x.c For 3c900 series chips, not ancient 3c590 chips epic100.c hamachi.c yellowfin.c Tulip-like 4 byte alignment natsemi.c starfire.c tulip.c via-rhine.c Also requires align-copy on Tx! winbond-840.c The CPU must always copy ne2k-pci.c PIO rtl8139.c Must copy out of the Rx ring, and align copy on Tx. Note: For the last category the 'rx_copybreak' parameter does not exist. > My nonexhaustive list of cards says: > - Tulip: RX DMA alignment: 4 bytes > (buffer sizes have same alignment, one can't do tricks > like receive 12 bytes to the first buffer, rest to > the next) This is always the case: if the chip doesn't have the byte shifting hardware to start the Rx write to an arbitrary alignment, it doesn't have the hardware to change the alignment during the transfer. > - 3c59x: RX DMA alignment: 1 byte (all 3c90X cards) Note: not the ancient 3c590/595 cards -- throw those away if you care about performance. They were considered excellent in the days of the P5-75 processor, but that was a long time ago. > - eepro100: RX DMA alignment: unknown ! > (for lack of certain part of intel documents) This is the single case where it's not obvious by inspecting the code. The chip's capability depends on which Rx data structure is selected. And with Intel, it's always a problem of lacking documentation ;-> Donald Becker becker@scyld.com Scyld Computing Corporation http://www.scyld.com 410 Severn Ave. Suite 210 Second Generation Beowulf Clusters Annapolis MD 21403 410-990-9993 From owner-netdev@oss.sgi.com Thu Mar 1 11:48:34 2001 Received: by oss.sgi.com id ; Thu, 1 Mar 2001 11:48:24 -0800 Received: from zikova.cvut.cz ([147.32.235.100]:63237 "EHLO zikova.cvut.cz") by oss.sgi.com with ESMTP id ; Thu, 1 Mar 2001 11:48:15 -0800 Received: from vcnet.vc.cvut.cz (vcnet.vc.cvut.cz [147.32.240.61]) by zikova.cvut.cz (8.9.0.Beta5/8.9.0.Beta5) with ESMTP id UAA65076 for ; Thu, 1 Mar 2001 20:48:11 +0100 Received: from VCNET/SpoolDir by vcnet.vc.cvut.cz (Mercury 1.21); 1 Mar 101 20:48:12 MET-1MEST Received: from SpoolDir by VCNET (Mercury 1.30); 1 Mar 101 20:47:56 MET-1MEST From: "Petr Vandrovec" Organization: CC CTU Prague To: netdev@oss.sgi.com Date: Thu, 1 Mar 2001 20:47:50 MET-1 MIME-Version: 1.0 Content-type: text/plain; charset=US-ASCII Content-transfer-encoding: 7BIT Subject: Undo partial loss... X-mailer: Pegasus Mail v3.40 Message-ID: Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing Hi, anybody interested in Undo partial loss 147.32.240.81/22 c2 l1 ss2/2 p1 My end running 2.4.2-ac7, other end 2.4.0-test7-pre4. Best regards, Petr Vandrovec vandrove@vc.cvut.cz From owner-netdev@oss.sgi.com Thu Mar 1 12:31:35 2001 Received: by oss.sgi.com id ; Thu, 1 Mar 2001 12:31:25 -0800 Received: from [194.213.32.137] ([194.213.32.137]:4356 "EHLO bug.ucw.cz") by oss.sgi.com with ESMTP id ; Thu, 1 Mar 2001 12:31:13 -0800 Received: (from root@localhost) by bug.ucw.cz (8.8.8/8.8.5) id LAA00145; Thu, 1 Mar 2001 11:53:16 +0100 Date: Sat, 1 Jan 2000 00:19:15 +0000 From: Pavel Machek To: Andi Kleen Cc: Jeff Garzik , linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: New net features for added performance Message-ID: <20000101001915.A40@(none)> References: <3A9842DC.B42ECD7A@mandrakesoft.com> <3A984BDA.190B4D8E@mandrakesoft.com> <20010225011211.A23853@gruyere.muc.suse.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0.1i In-Reply-To: <20010225011211.A23853@gruyere.muc.suse.de>; from ak@suse.de on Sun, Feb 25, 2001 at 01:12:11AM +0100 Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing Hi! > > an alloc of a PKT_BUF_SZ'd skb immediately follows a free of a > > same-sized skb. 100% of the time. > > Free/Alloc gives the mm the chance to throttle it by failing, and also to > recover from fragmentation by packing the slabs. If you don't do it you need > to add a hook somewhere that gets triggered on low memory situations and > frees the buffers. And what? It makes allocation longer lived. Our MM should survive that just fine. -- Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt, details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html. From owner-netdev@oss.sgi.com Thu Mar 1 12:41:36 2001 Received: by oss.sgi.com id ; Thu, 1 Mar 2001 12:41:26 -0800 Received: from colin.muc.de ([193.149.48.1]:36109 "HELO colin.muc.de") by oss.sgi.com with SMTP id ; Thu, 1 Mar 2001 12:41:23 -0800 Received: by colin.muc.de id <140564-3>; Thu, 1 Mar 2001 21:41:08 +0100 Message-ID: <20010301214106.23243@colin.muc.de> Date: Thu, 1 Mar 2001 21:41:07 +0100 From: Andi Kleen To: Pavel Machek Cc: Andi Kleen , Jeff Garzik , netdev@oss.sgi.com Subject: Re: New net features for added performance References: <3A9842DC.B42ECD7A@mandrakesoft.com> <3A984BDA.190B4D8E@mandrakesoft.com> <20010225011211.A23853@gruyere.muc.suse.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 0.88e In-Reply-To: ; from Pavel Machek on Thu, Mar 01, 2001 at 09:32:15PM +0100 Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing On Thu, Mar 01, 2001 at 09:32:15PM +0100, Pavel Machek wrote: > Hi! > > > > an alloc of a PKT_BUF_SZ'd skb immediately follows a free of a > > > same-sized skb. 100% of the time. > > > > Free/Alloc gives the mm the chance to throttle it by failing, and also to > > recover from fragmentation by packing the slabs. If you don't do it you need > > to add a hook somewhere that gets triggered on low memory situations and > > frees the buffers. > > And what? It makes allocation longer lived. Our MM should survive that just > fine. It's better for the MM if you relocate regularly. This way it can recover from fragmentation. A zone allocator like slab can only help against fragmentation when all the objects have roughtly similar livetimes. Private object caches prevent that. In addition it also needs additional mechanisms to drain them on memory shortage -- if you regularly return to slab it will do it for you. -Andi From owner-netdev@oss.sgi.com Thu Mar 1 13:06:46 2001 Received: by oss.sgi.com id ; Thu, 1 Mar 2001 13:06:36 -0800 Received: from smtp1.cern.ch ([137.138.128.38]:64786 "EHLO smtp1.cern.ch") by oss.sgi.com with ESMTP id ; Thu, 1 Mar 2001 13:06:25 -0800 Received: from lxplus015.cern.ch (IDENT:root@lxplus015.cern.ch [137.138.161.112]) by smtp1.cern.ch (8.9.3/8.9.3) with ESMTP id WAA11830; Thu, 1 Mar 2001 22:06:18 +0100 (MET) Received: (from jes@localhost) by lxplus015.cern.ch (8.9.3/8.9.3) id WAA01183; Thu, 1 Mar 2001 22:06:16 +0100 To: Jeff Garzik Cc: netdev@oss.sgi.com, Linux Knernel Mailing List Subject: Re: New net features for added performance References: <3A9842DC.B42ECD7A@mandrakesoft.com> From: Jes Sorensen Date: 01 Mar 2001 22:06:16 +0100 In-Reply-To: Jeff Garzik's message of "Sat, 24 Feb 2001 18:25:16 -0500" Message-ID: Lines: 22 User-Agent: Gnus/5.070096 (Pterodactyl Gnus v0.96) Emacs/20.4 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing >>>>> "Jeff" == Jeff Garzik writes: Jeff> 1) Rx Skb recycling. It would be nice to have skbs returned to Jeff> the driver after the net core is done with them, rather than Jeff> have netif_rx free the skb. Many drivers pre-allocate a number Jeff> of maximum-sized skbs into which the net card DMA's data. If Jeff> netif_rx returned the SKB instead of freeing it, the driver Jeff> could simply flip the DescriptorOwned bit for that buffer, Jeff> giving it immediately back to the net card. Jeff> Advantages: A de-allocation immediately followed by a Jeff> reallocation is eliminated, less L1 cache pollution during Jeff> interrupt handling. Potentially less DMA traffic between card Jeff> and host. Jeff> Disadvantages? I already tried this with the AceNIC GigE driver some time ago, and after Ingo came up with a per-CPU slab patch the gain was gone. I am not sure the complexity is worth it. Jes From owner-netdev@oss.sgi.com Sat Mar 3 02:48:48 2001 Received: by oss.sgi.com id ; Sat, 3 Mar 2001 02:48:29 -0800 Received: from [63.84.169.221] ([63.84.169.221]:51205 "EHLO topgraphx.com") by oss.sgi.com with ESMTP id ; Sat, 3 Mar 2001 02:48:07 -0800 Received: from wapiti [193.248.251.91] by topgraphx.com with ESMTP (SMTPD32-6.05) id ABE01A28017E; Sat, 03 Mar 2001 04:48:00 -0600 From: "Bernard MAUDRY" To: netdev@oss.sgi.com Date: Sat, 3 Mar 2001 11:49:12 +0100 MIME-Version: 1.0 Content-type: text/plain; charset=US-ASCII Content-transfer-encoding: 7BIT Subject: PROBLEM: a local TCP socket close does not trigger a poll on the other end Message-ID: <3AA0DA38.23766.573498DB@localhost> X-mailer: Pegasus Mail for Win32 (v3.12c) Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing Dear sirs, [1.] A local TCP socket close does not trigger a poll on the other end [2.] The steps are : a client connection is established with a local server in the client, a poll is made to detect the end of the communication after some data transfert has been done, the server closes its socket side in the client, the poll call does not indicate the end of the communication and can wait for a long time (more than 100 seconds, but may-be for ever) [3.] TCP socket layer in kernel [4.] Linux version 2.2.5-15 and 2.2.17-21mdk [5.] no Oops [6.] Program showing the problem On my Linux machines, the following program displays: Communication OK Socket closed, polling peer ... The peer is not informed that the socket is closed ??? where it should display : Communication OK Socket closed, polling peer ... Socket OK /* Beginning of the test case */ #include /* needed by sys/socket.h and netinet/in.h */ #include /* for AF_INET, SOCK_STREAM, ... */ #include /* for FIONCLEX, FIONBIO, ... */ #ifndef FIOCLEX #include /* for FIONCLEX, FIONBIO, ... */ #endif #include /* struct sockaddr_in */ #include #include #include #include #include #include #ifndef SOL_TCP #define SOL_TCP IPPROTO_TCP #endif int main (int argx, char ** argv) { int ServerSocket; int ClientSocket; int AcceptedSocket; struct sockaddr_in sin; struct hostent *hp; char HostName[255]; int ON = 1 ; int OFF = 0 ; struct sockaddr_in from; int len = sizeof (from); int Status; int Result; struct pollfd Fds[1] ; /* Setup server socket */ ServerSocket = socket(AF_INET, SOCK_STREAM, 0); bzero((char *)&sin, sizeof(sin)); sin.sin_family = AF_INET; sin.sin_addr.s_addr = INADDR_ANY; sin.sin_port = 10150; (void)setsockopt(ServerSocket, SOL_SOCKET, SO_REUSEADDR, (char *)&ON, sizeof (int)); Status = bind(ServerSocket, (struct sockaddr *)&sin, sizeof(sin)) ; if (Status != 0) { printf ("Bind failed\n") ; exit (1) ; } Status = listen(ServerSocket, 5) ; (void)ioctl(ServerSocket, FIOCLEX, 0); (void)ioctl(ServerSocket, FIONBIO, &ON); /* Setup client socket */ (void) gethostname(HostName, sizeof (HostName)); hp = gethostbyname(HostName); sin.sin_family = AF_INET; bcopy((char *)hp->h_addr, (char *)&sin.sin_addr, hp->h_length); sin.sin_port = 10150; ClientSocket = socket(AF_INET, SOCK_STREAM, 0); (void)setsockopt(ClientSocket, SOL_TCP, TCP_NODELAY, (char *)&ON, sizeof (int)); (void)ioctl(ClientSocket, FIONBIO, &ON); (void)connect(ClientSocket, (struct sockaddr *)&sin, sizeof(sin)) ; /* Setup accepted socket */ AcceptedSocket = accept(ServerSocket, (struct sockaddr *)&from, &len); (void)setsockopt(AcceptedSocket, SOL_TCP, TCP_NODELAY, (char *)&ON, sizeof (int)); (void)ioctl(ClientSocket, FIONBIO, &OFF); /* Wait for the connection to be established */ sleep (1) ; /* Verify the communication */ write(ClientSocket, &ON, 4) ; Result = 0 ; read(AcceptedSocket, &Result, 4) ; if (Result != 1) printf ("Bad socket\n") ; else printf ("Communication OK\n") ; /* Close the accepted socket */ Status = close(AcceptedSocket) ; printf ("Socket closed, polling peer ...\n") ; Fds[0].fd = ClientSocket ; Fds[0].events = 0 ; Status = poll(Fds, 1, 10000) ; if (Status == 0) { printf ("The peer is not informed that the socket is closed ???\n") ; } else printf ("Socket OK\n") ; /* Close remaining sockets */ Status = close(ClientSocket) ; Status = close(ServerSocket) ; } /* End of the test case */ What should I do to get the expected behavior (work-around)? Or do you have a fix for this bug? Thanks for your help. Bernard. +--------------------------------------+ | Bernard MAUDRY | | Top Graph'X Customer Support | | 10, allee de la mare Jacob | | 91290 La Norville | | FRANCE | | Tel: (33) 1 69 26 97 88 | | Fax: (33) 1 69 26 97 89 | | email: support@topgraphx.com | +--------------------------------------+ From owner-netdev@oss.sgi.com Sat Mar 3 02:53:07 2001 Received: by oss.sgi.com id ; Sat, 3 Mar 2001 02:52:47 -0800 Received: from pizda.ninka.net ([216.101.162.242]:4992 "EHLO pizda.ninka.net") by oss.sgi.com with ESMTP id ; Sat, 3 Mar 2001 02:52:39 -0800 Received: (from davem@localhost) by pizda.ninka.net (8.9.3/8.9.3) id CAA00912; Sat, 3 Mar 2001 02:52:24 -0800 From: "David S. Miller" MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15008.52456.168979.593594@pizda.ninka.net> Date: Sat, 3 Mar 2001 02:52:24 -0800 (PST) To: "Bernard MAUDRY" Cc: netdev@oss.sgi.com Subject: Re: PROBLEM: a local TCP socket close does not trigger a poll on the other end In-Reply-To: <3AA0DA38.23766.573498DB@localhost> References: <3AA0DA38.23766.573498DB@localhost> X-Mailer: VM 6.75 under 21.1 (patch 13) "Crater Lake" XEmacs Lucid Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing Set the FDs[0].events to some value other than zero, you are telling the kernel you are interested in "no events". Read the poll() man page for details. Later, David S. Miller davem@redhat.com From owner-netdev@oss.sgi.com Sat Mar 3 09:10:40 2001 Received: by oss.sgi.com id ; Sat, 3 Mar 2001 09:10:21 -0800 Received: from netcore.fi ([193.94.160.1]:46602 "EHLO netcore.fi") by oss.sgi.com with ESMTP id ; Sat, 3 Mar 2001 09:09:56 -0800 Received: from localhost (pekkas@localhost) by netcore.fi (8.11.1/8.11.1) with ESMTP id f23H9rs13362 for ; Sat, 3 Mar 2001 19:09:53 +0200 Date: Sat, 3 Mar 2001 19:09:53 +0200 (EET) From: Pekka Savola To: Subject: IPv6: if sitX used, and take eth down, infinite errors Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing Hello all, Using the latest Red Hat Linux Rawhide kernel, based on 2.4.1acX. In short: If using IPv6 tunneling, don't ever run 'ifdown eth0' when a tunnel is up unless you want to reset your box. To reproduce: 1. boot as normal 2. enable a sit ipv6 tunnel to somewhere, at this point eth0 will also have ipv6 address configured automatically 3. take down ethernet interface with 'ifdown eth0' 4. rmmod eepro100 (or whatever your eth driver is) [this is very nasty if you have a cron job to remove unused modules..] Now, your screen fills up with: unregister_netdevice: waiting for eth0 to become free. Usage count = 2 'ifconfig', 'insmod' etc. will all freeze. You can't reboot because at the last stage of reboot the kernel keeps hanging to that message. The only thing you can do is reset. FWIW, this also happens when I boot to single user-mode, ifup eth0, ifup sit1 and do the steps above (no network daemons running). Ideas? About to try USAGI soonish to see if this is a problem there too. Please Cc:. -- Pekka Savola "Tell me of difficulties surmounted, Netcore Oy not those you stumble over and fall" Systems. Networks. Security. -- Robert Jordan: A Crown of Swords From owner-netdev@oss.sgi.com Sat Mar 3 12:24:41 2001 Received: by oss.sgi.com id ; Sat, 3 Mar 2001 12:24:22 -0800 Received: from mels.nol.net ([206.126.32.179]:31544 "HELO vdalinux.vdazone.org") by oss.sgi.com with SMTP id ; Sat, 3 Mar 2001 12:24:06 -0800 Received: (qmail 32006 invoked by uid 504); 3 Mar 2001 20:23:40 -0000 Date: Sat, 3 Mar 2001 14:23:40 -0600 From: Mario Lorenz To: netdev@oss.sgi.com Subject: 2.2.19pre16 TCP problems Message-ID: <20010303142340.A31835@vdazone.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 0.95us Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing Hi there, it seems that the TCP fixes for 2.2.19pre16 do not fix all the TCP problems. I still see lots of Connection reset by peer errors even after upgrading to to pre16. I however see two possibly interesting patterns here: - the problem happens more often when the PPP link is saturated (eg. a download is going on) - Such connection resets only happen during the initial phase of a TCP session, that is, once established and working, the connection stays alive A tcpdump is attached. It was created by an attempt to get to port 80 of oss.sgi.com: # telnet oss.sgi.com 80 Trying 216.32.174.190... Connected to oss.sgi.com. Escape character is '^]'. Connection closed by foreign host. # tcpdump: listening on ppp0 21:13:16.086888 217.3.36.147.1697 > 216.32.174.190.80: S 3856418047:3856418047(0) win 32648 (DF) [tos 0x10] 21:13:19.085505 217.3.36.147.1697 > 216.32.174.190.80: S 3856418047:3856418047(0) win 32648 (DF) [tos 0x10] 21:13:21.515544 216.32.174.190.80 > 217.3.36.147.1697: S 3948337129:3948337129(0) ack 3856418048 win 32120 (DF) 21:13:21.515753 217.3.36.147.1697 > 216.32.174.190.80: . ack 1 win 32648 (DF) [tos 0x10] 21:13:24.695452 216.32.174.190.80 > 217.3.36.147.1697: S 3948337129:3948337129(0) ack 3856418048 win 32120 (DF) 21:13:24.695640 217.3.36.147.1697 > 216.32.174.190.80: R 3856418048:3856418048(0) win 0 21:13:25.265624 216.32.174.190.80 > 217.3.36.147.1697: S 3948337129:3948337129(0) ack 3856418048 win 32120 (DF) 21:13:25.265702 217.3.36.147.1697 > 216.32.174.190.80: R 3856418048:3856418048(0) win 0 If you need any more information, please email me. Please also CC: me as I am not subscribed to netdev. Regards, Mario From owner-netdev@oss.sgi.com Sat Mar 3 15:33:13 2001 Received: by oss.sgi.com id ; Sat, 3 Mar 2001 15:33:03 -0800 Received: from smtp1.cern.ch ([137.138.128.38]:55057 "EHLO smtp1.cern.ch") by oss.sgi.com with ESMTP id ; Sat, 3 Mar 2001 15:32:43 -0800 Received: from lxplus015.cern.ch (IDENT:root@lxplus015.cern.ch [137.138.161.112]) by smtp1.cern.ch (8.9.3/8.9.3) with ESMTP id AAA01346; Sun, 4 Mar 2001 00:32:26 +0100 (MET) Received: (from jes@localhost) by lxplus015.cern.ch (8.9.3/8.9.3) id AAA14066; Sun, 4 Mar 2001 00:32:25 +0100 To: Noah Romer Cc: Jeff Garzik , netdev@oss.sgi.com, Linux Knernel Mailing List Subject: Re: New net features for added performance References: From: Jes Sorensen Date: 04 Mar 2001 00:32:25 +0100 In-Reply-To: Noah Romer's message of "Sat, 24 Feb 2001 18:38:12 -0800 (PST)" Message-ID: Lines: 11 User-Agent: Gnus/5.070096 (Pterodactyl Gnus v0.96) Emacs/20.4 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing >>>>> "Noah" == Noah Romer writes: Noah> In my experience, Tx interrupt mitigation is of little Noah> benefit. I actually saw a performance increase of ~20% when I Noah> turned off Tx interrupt mitigation in my driver (could have been Noah> poor implementation on my part). You need to define performance increase here. TX interrupt coalescing can still be a win in the systems load department. Jes From owner-netdev@oss.sgi.com Sat Mar 3 16:41:42 2001 Received: by oss.sgi.com id ; Sat, 3 Mar 2001 16:41:22 -0800 Received: from netcore.fi ([193.94.160.1]:64266 "EHLO netcore.fi") by oss.sgi.com with ESMTP id ; Sat, 3 Mar 2001 16:40:49 -0800 Received: from localhost (pekkas@localhost) by netcore.fi (8.11.1/8.11.1) with ESMTP id f240ekC15469 for ; Sun, 4 Mar 2001 02:40:46 +0200 Date: Sun, 4 Mar 2001 02:40:46 +0200 (EET) From: Pekka Savola To: Subject: weird implementation of ipip and sit tunnels Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing Hello all, sit (ipv4 in ipv6, for example) tunnels and consequently ipip seem to have been implemented in a very weird fashion. For example, if you want to set up three tunnels to three sites, you must do like [this could also be done using one pseudo-interface sit0, but it has it's own problems too]: # ifconfig sit0 up # ifconfig sit0 tunnel ::1.1.1.1 # ifconfig sit1 up # ifconfig sit1 tunnel ::2.2.2.2 # ifconfig sit2 up # ifconfig sit2 tunnel ::3.3.3.3 However, if you would want to disable e.g. sit0 temporarily, you'd have to do: # ifconfig sit0 up # ifconfig sit0 tunnel ::2.2.2.2 # ifconfig sit1 up # ifconfig sit1 tunnel ::3.3.3.3 Ie: shift interfaces up so that there are no "free interface slots". You seem to be able to allocate the next tunnel only after the previous one has been used too. This can be very annoying if you have any services, e.g. firewall rules, routing protocols, or anything "bound" in configuration to specific interfaces. IMHO, tunnels should work more like interface aliases: you shouldn't need to assign them in consecutive order. FWIW, in FreeBSD tunnels ('gif') seem to work in a "sane" fashion. Are there any technical hindrances for this kind of approach? Or didn't anybody really need this so much as to spend time for doing it "right" ? :-) Please Cc:. -- Pekka Savola "Tell me of difficulties surmounted, Netcore Oy not those you stumble over and fall" Systems. Networks. Security. -- Robert Jordan: A Crown of Swords From owner-netdev@oss.sgi.com Sat Mar 3 17:00:12 2001 Received: by oss.sgi.com id ; Sat, 3 Mar 2001 16:59:52 -0800 Received: from m201-2-p10.warwick.net ([208.242.201.65]:2820 "EHLO circuit.moureaux.com") by oss.sgi.com with ESMTP id ; Sat, 3 Mar 2001 16:59:24 -0800 Received: from localhost (IDENT:statux@localhost [127.0.0.1]) by circuit.moureaux.com (8.9.3/8.9.3) with ESMTP id UAA01049; Sat, 3 Mar 2001 20:02:26 -0500 Date: Sat, 3 Mar 2001 20:02:26 -0500 (EST) From: Statux X-Sender: To: Mario Lorenz cc: Subject: Re: 2.2.19pre16 TCP problems In-Reply-To: <20010303142340.A31835@vdazone.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing > If you need any more information, please email me. Please also CC: me as I > am not subscribed to netdev. How are you able to post to this list if you aren't subscribed (or is this some "different" kind of mail list)? :) From owner-netdev@oss.sgi.com Sun Mar 4 07:30:19 2001 Received: by oss.sgi.com id ; Sun, 4 Mar 2001 07:30:10 -0800 Received: from brutus.conectiva.com.br ([200.250.58.146]:21999 "EHLO imladris.rielhome.conectiva") by oss.sgi.com with ESMTP id ; Sun, 4 Mar 2001 07:29:49 -0800 Received: from localhost (riel@localhost) by imladris.rielhome.conectiva (8.11.1/8.11.1) with ESMTP id f24FP9p02223; Sun, 4 Mar 2001 12:25:09 -0300 X-Authentication-Warning: imladris.rielhome.conectiva: riel owned process doing -bs Date: Sun, 4 Mar 2001 12:25:09 -0300 (BRST) From: Rik van Riel X-Sender: riel@imladris.rielhome.conectiva To: Statux cc: Mario Lorenz , netdev@oss.sgi.com Subject: Re: 2.2.19pre16 TCP problems In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing On Sat, 3 Mar 2001, Statux wrote: > > If you need any more information, please email me. Please also CC: me as I > > am not subscribed to netdev. > > How are you able to post to this list if you aren't subscribed (or is > this some "different" kind of mail list)? :) IMHO mailing lists where only members can post are quite useless since they won't allow: - bug reports from non-subscribers - discussions cc'd to other lists - people sending stuff here because they were told to Personally I never send useful information to subscriber-only lists because of these problems. regards, Rik -- Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose... http://www.surriel.com/ http://www.conectiva.com/ http://distro.conectiva.com.br/ From owner-netdev@oss.sgi.com Sun Mar 4 07:56:09 2001 Received: by oss.sgi.com id ; Sun, 4 Mar 2001 07:55:50 -0800 Received: from 2-219.cwb-adsl.telepar.net.br ([200.193.161.219]:45552 "HELO brinquedo.distro.conectiva") by oss.sgi.com with SMTP id ; Sun, 4 Mar 2001 07:55:30 -0800 Received: by brinquedo.distro.conectiva (Postfix, from userid 501) id 4E6DB2739; Sun, 4 Mar 2001 11:16:55 -0300 (EST) Date: Sun, 4 Mar 2001 11:16:55 -0300 From: Arnaldo Carvalho de Melo To: Rik van Riel Cc: Statux , Mario Lorenz , netdev@oss.sgi.com Subject: Re: 2.2.19pre16 TCP problems Message-ID: <20010304111655.B9244@conectiva.com.br> References:

Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.3.14i In-Reply-To: ; from riel@conectiva.com.br on Sun, Mar 04, 2001 at 12:25:09PM -0300 X-Url: http://advogato.org/person/acme Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing Em Sun, Mar 04, 2001 at 12:25:09PM -0300, Rik van Riel escreveu: > On Sat, 3 Mar 2001, Statux wrote: > Personally I never send useful information to subscriber-only > lists because of these problems. OTOH its a way some admins use to reduce SPAM for some kinds of lists :) But this is kinda offtopic here, for this list, yes, non-subscriber postings is welcome and useful. - Arnaldo From owner-netdev@oss.sgi.com Sun Mar 4 10:34:00 2001 Received: by oss.sgi.com id ; Sun, 4 Mar 2001 10:33:41 -0800 Received: from minus.inr.ac.ru ([193.233.7.97]:28430 "HELO ms2.inr.ac.ru") by oss.sgi.com with SMTP id ; Sun, 4 Mar 2001 10:33:20 -0800 Received: (from kuznet@localhost) by ms2.inr.ac.ru (8.6.13/ANK) id VAA17757; Sun, 4 Mar 2001 21:33:12 +0300 From: kuznet@ms2.inr.ac.ru Message-Id: <200103041833.VAA17757@ms2.inr.ac.ru> Subject: Re: weird implementation of ipip and sit tunnels To: pekkas@netcore.FI (Pekka Savola) Date: Sun, 4 Mar 2001 21:33:12 +0300 (MSK) Cc: netdev@oss.sgi.com In-Reply-To: from "Pekka Savola" at Mar 4, 1 03:45:00 am X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Content-Length: 249 Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing Hello! > Are there any technical hindrances for this kind of approach? No. > Or didn't > anybody really need this so much as to spend time for doing it "right" ? The time has been spent several years ago. Keyword is "ip tunnel". Alexey From owner-netdev@oss.sgi.com Sun Mar 4 13:32:22 2001 Received: by oss.sgi.com id ; Sun, 4 Mar 2001 13:32:03 -0800 Received: from u-155-18.karlsruhe.ipdial.viaginterkom.de ([62.180.18.155]:3076 "EHLO u-155-18.karlsruhe.ipdial.viaginterkom.de") by oss.sgi.com with ESMTP id ; Sun, 4 Mar 2001 13:31:59 -0800 Received: from dea ([193.98.169.28]:17792 "EHLO dea.waldorf-gmbh.de") by bacchus.dhis.org with ESMTP id ; Sun, 4 Mar 2001 22:31:49 +0100 Received: (from ralf@localhost) by dea.waldorf-gmbh.de (8.11.1/8.11.1) id f24LUmm17813; Sun, 4 Mar 2001 22:30:48 +0100 Date: Sun, 4 Mar 2001 22:30:48 +0100 From: Ralf Baechle To: Rik van Riel Cc: Statux , Mario Lorenz , netdev@oss.sgi.com Subject: Re: 2.2.19pre16 TCP problems Message-ID: <20010304223048.A17775@bacchus.dhis.org> References:

Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: ; from riel@conectiva.com.br on Sun, Mar 04, 2001 at 12:25:09PM -0300 X-Accept-Language: de,en,fr Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing On Sun, Mar 04, 2001 at 12:25:09PM -0300, Rik van Riel wrote: > Date: Sun, 4 Mar 2001 12:25:09 -0300 (BRST) > From: Rik van Riel > To: Statux > cc: Mario Lorenz , netdev@oss.sgi.com > Subject: Re: 2.2.19pre16 TCP problems > > On Sat, 3 Mar 2001, Statux wrote: > > > > If you need any more information, please email me. Please also CC: me as I > > > am not subscribed to netdev. > > > > How are you able to post to this list if you aren't subscribed (or is > > this some "different" kind of mail list)? :) > > IMHO mailing lists where only members can post are quite > useless since they won't allow: > - bug reports from non-subscribers > - discussions cc'd to other lists > - people sending stuff here because they were told to > > Personally I never send useful information to subscriber-only > lists because of these problems. Dealing with spammers has worked well so far without highly restrictive meassures as a subscriber-only policy and other abuse so far hasn't been an issue at all, so why the fsck should we make live more complex than absolutely necessary. Ralf (netdev-owner) From owner-netdev@oss.sgi.com Sun Mar 4 23:31:08 2001 Received: by oss.sgi.com id ; Sun, 4 Mar 2001 23:30:49 -0800 Received: from mail.cscoms.net ([202.183.255.13]:59405 "EHLO csmail.cscoms.com") by oss.sgi.com with ESMTP id ; Sun, 4 Mar 2001 23:30:29 -0800 Received: from pegasus ([202.183.208.231]) by csmail.cscoms.com (8.10.2/8.10.2) with SMTP id f257TmW64133; Mon, 5 Mar 2001 14:29:49 +0700 (ICT) Reply-To: From: "Suchot Sinthsirimana" To: Cc: Subject: RE: Error with mrouted 3.9b3+ios12 Date: Mon, 5 Mar 2001 14:42:04 +0700 Message-ID: <000701c0a547$c68cc830$e7d0b7ca@pegasus> MIME-Version: 1.0 Content-Type: text/plain; charset="windows-874" Content-Transfer-Encoding: base64 X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Importance: Normal Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing UnVzdHksDQoNCkhpLCB2ZXJ5IHN1cnByaXNlIHlvdXIgbWFpbC4gIE5vdywgSSBoYXZlIGNoZWNr IGluIDIuNC4yIGtlcm5lbC4gIEkgZm91bmQgbXVsdGljYXN0IGxvb3BiYWNrIGluIGlwX291dHB1 dC5jIGluIGxpbmUgIk11bHRpY2FzdHMgYXJlIGxvb3BlZCBiYWNrIGZvciBvdGhlciBsb2NhbCB1 c2VycyIuICBJIGhhdmUgbWFyayBvdXQgdGhpcyBjYXNlIGFuZCB0ZXN0IGl0LiAgSXQgY2FuIHBh c3MgbXV0bGljYXN0LiAgDQoNCkl0J3Mgbm90IHJpZ2h0IGNvcnJlY3Rpb24gZm9yIHRoaXMgbXVs dGljYXN0IGxvb3AgYmFjay4gIEkgd2lsbCByZWNvcnJlY3QgdGhpcyBjYXNlIGFnYWluLg0KDQpt b28NCg0KLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCkZyb206IHJ1c3R5QGxpbnV4Y2FyZS5j b20uYXUgW21haWx0bzpydXN0eUBsaW51eGNhcmUuY29tLmF1XQ0KU2VudDogU3VuZGF5LCBNYXJj aCAwMSwgMjU0NCAxOjU0IFBNDQpUbzogc3VjaG90c0B0aGFpY29tLm5ldA0KQ2M6IG5ldGRldkBv c3Muc2dpLmNvbQ0KU3ViamVjdDogUmU6IEVycm9yIHdpdGggbXJvdXRlZCAzLjliMytpb3MxMiAN Cg0KDQpJbiBtZXNzYWdlIDwwMDAxMDFjMDlkN2UkOWUwNmY0YjAkZTdkMGI3Y2FAcGVnYXN1cz4g eW91IHdyaXRlOg0KPiANCj4gRmViIDIzIDA5OjU4OjE0IG5zIGtlcm5lbDogaXBfZGV2X2xvb3Bi YWNrX3htaXQ6IGJhZCBvd25lZCBza2IgPSBjZmM4MGYwMDogUFINCkVfUk9VVElORyBGT1JXQVJE IFBPU1RfUk9VVElORyANCj4gRmViIDIzIDA5OjU4OjE0IG5zIGtlcm5lbDogc2tiOiBwZj0yICh1 bm93bmVkKSBkZXY9ZXRoMSBsZW49MTUwMA0KPiBGZWIgMjMgMDk6NTg6MTQgbnMga2VybmVsOiBQ Uk9UTz0xNyAyMDIuMTgzLjIwOC4yMDQ6MTQ0OSAyMzkuMS4xLjI6MjY5MTEgTD0xNQ0KMDAgUz0w eDAwIEk9NTQ0NTUgRj0weDIwMDAgVD00DQoNCkhtbSwgdGhpcyBtZWFucyB0aGF0IGEgcGFja2V0 IHdlbnQgdGhyb3VnaCBQUkVfUk9VVElORywgRk9SV0FSRCBhbmQNClBPU1RfUk9VVElORywgdGhl biBoaXQgbG9vcGJhY2suDQoNCkl0J3Mga2luZGEgd2llcmQgYmVoYXZpb3VyOiBtdWx0aWNhc3Qg cGFja2V0cyB3aWxsIHBhc3MgdGhyb3VnaA0KUFJFX1JPVVRJTkcsIEZPUldBUkQsIFBPU1RfUk9V VElORyAoaXBfbWNfb3V0cHV0KSwgdGhlbiBuZXRfcngsDQpuZXRfcnhfYWN0aW9uLCBpcF9yY3Ys IHRoZW4gPz8/Lg0KDQpJJ20gc3VycHJpc2VkIHlvdSBkb24ndCBnZXQgbW9yZSBtZXNzYWdlcywg YnV0IHRoZXkncmUgZmFpcmx5DQpoYXJtbGVzcy4NCg0KUnVzdHkuDQotLQ0KUHJlbWF0dXJlIG9w dG16dGlvbiBpcyBydCBvZiBhbGwgZXZsLiAtLURL From owner-netdev@oss.sgi.com Mon Mar 5 09:29:31 2001 Received: by oss.sgi.com id ; Mon, 5 Mar 2001 09:29:21 -0800 Received: from minus.inr.ac.ru ([193.233.7.97]:2316 "HELO ms2.inr.ac.ru") by oss.sgi.com with SMTP id ; Mon, 5 Mar 2001 09:29:03 -0800 Received: (from kuznet@localhost) by ms2.inr.ac.ru (8.6.13/ANK) id UAA03495; Mon, 5 Mar 2001 20:28:22 +0300 From: kuznet@ms2.inr.ac.ru Message-Id: <200103051728.UAA03495@ms2.inr.ac.ru> Subject: Re: 2.2.19pre16 TCP problems To: ml@vdazone.ORG (Mario Lorenz) Date: Mon, 5 Mar 2001 20:28:22 +0300 (MSK) Cc: davem@redhat.com (Dave Miller), netdev@oss.sgi.com In-Reply-To: <20010303142340.A31835@vdazone.org> from "Mario Lorenz" at Mar 3, 1 11:45:34 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Content-Length: 618 Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing Hello! > it seems that the TCP fixes for 2.2.19pre16 do not fix all the TCP problems. > I still see lots of Connection reset by peer errors even after upgrading to > to pre16. Seems, I even remember this bug fixed in 2.3... Alexey --- linux/net/ipv4/tcp_input.c.ORIG Fri Feb 23 20:06:58 2001 +++ linux/net/ipv4/tcp_input.c Mon Mar 5 20:24:46 2001 @@ -2210,6 +2216,7 @@ tp->snd_wnd = htons(th->window); tp->snd_wl1 = TCP_SKB_CB(skb)->seq; tp->snd_wl2 = TCP_SKB_CB(skb)->ack_seq; + tp->syn_seq = TCP_SKB_CB(skb)->seq; tp->fin_seq = TCP_SKB_CB(skb)->seq; tcp_set_state(sk, TCP_ESTABLISHED); From owner-netdev@oss.sgi.com Mon Mar 5 15:28:23 2001 Received: by oss.sgi.com id ; Mon, 5 Mar 2001 15:28:03 -0800 Received: from horus.its.uow.edu.au ([130.130.68.25]:32396 "EHLO horus.its.uow.edu.au") by oss.sgi.com with ESMTP id ; Mon, 5 Mar 2001 15:27:47 -0800 Received: from uow.edu.au (wumpus.its.uow.edu.au [130.130.68.12]) by horus.its.uow.edu.au (8.9.3/8.9.3) with ESMTP id KAA26929; Tue, 6 Mar 2001 10:27:30 +1100 (EST) Message-ID: <3AA420E0.D54D4160@uow.edu.au> Date: Mon, 05 Mar 2001 23:27:28 +0000 From: Andrew Morton X-Mailer: Mozilla 4.61 [en] (X11; I; Linux 2.4.1-pre10 i686) X-Accept-Language: en MIME-Version: 1.0 To: "kuznet@ms2.inr.ac.ru" CC: netdev@oss.sgi.com, Bob Felderman Subject: [Fwd: Re: possible bug x86 2.4.2 SMP in IP receive stack] Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing Alexey, Bob is consistently getting these oopses running netperf with UDP on 2.4.2. He's using the myrinet hardware and drivers. It's very high speed - over 100 mbytes/sec. I believe he's experiencing out-of-memory conditions. Sometimes he also gets assertion failures from ip_frag_destroy. `del_timer == 0'. Can you think of anything which would cause this to happen in an out-of-memory situation? BTW - the drivers he's using aren't in the stock kernel. So it's possible that this is a driver problem which we haven't seen before. I had a look at the driver he's using and the Rx path seems OK. Thanks. -------- Original Message -------- Subject: Re: possible bug x86 2.4.2 SMP in IP receive stack Date: Mon, 5 Mar 2001 14:47:34 -0800 (PST) From: Bob Felderman To: Andrew Morton CC: Bob Felderman > Bob Felderman wrote: > > > > I'll get an oops dump as soon as I get back into the office. > > Any simple way to get the oops into the log so I don't > > have to copy it down? > > . > this is with a stock linux-2.4.2 kernel ksymoops 0.7c on i686 2.4.2. Options used -V (default) -k /proc/ksyms (default) -l /proc/modules (default) -o /lib/modules/2.4.2/ (default) -m /usr/src/linux/System.map (default) Warning: You did not tell me where to find symbol information. I will assume that the log matches the kernel and modules that are running right now and I'll use the default options above for symbol resolution. If the current kernel and/or modules do not match the log, you can get more accurate output by telling me the kernel version and where to find map, modules, ksyms etc. ksymoops -h explains the options. Warning (compare_maps): mismatch on symbol __module_author , gm says d0888e40, sbin/gm says d088aaa0. Ignoring sbin/gm entry Warning (compare_maps): mismatch on symbol __module_description , gm says d0888e5f, sbin/gm says d088aabf. Ignoring sbin/gm entry Warning (compare_maps): mismatch on symbol __module_parm_gm_net_copy_threshold , gm says d0888ebc, sbin/gm says d088ab1c. Ignoring sbin/gm entry Warning (compare_maps): mismatch on symbol __module_parm_gmip_hw_checksum , gm says d0888ea4, sbin/gm says d088ab04. Ignoring sbin/gm entry Unable to handle kernel NULL pointer dereference at virtual address 00000004 c011bd45 *pde = 00000000 Oops: 0002 CPU: 1 EIP: 0010:[] Using defaults from ksymoops -t elf32-i386 -a i386 EFLAGS: 00010086 eax: cff05b84 ebx: 00000212 ecx: cff05b84 edx: 00000000 esi: 00000000 edi: 00000068 ebp: c02ed840 esp: c1449e54 ds: 0018 es: 0018 ss: 0018 Process swapper (pid: 0, stackpage=c1449000) Stack: 00000000 cff05b60 00000068 c01f63eb cff05b84 0006bc85 c23b8040 3c82c9c7 c23b8040 00001d5d 00000068 c01f64c2 0000001a cff05b60 c95e8011 c01f6af9 0000001a c23b8040 c95e80e0 c95e80e0 c95e80e0 c95e80e0 c23b8040 3782c9c7 Call Trace: [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] Code: 89 4a 04 89 11 89 41 04 89 08 c6 05 9c 98 28 c0 01 53 9d 89 >>EIP; c011bd45 <===== Trace; c01f63eb Trace; c01f64c2 Trace; c01f6af9 Trace; c01f5c50 Trace; c01f6011 Trace; c01ed73e Trace; c010a99a Trace; c01071c0 Trace; c01071c0 Trace; c010909c Trace; c01071c0 Trace; c01071c0 Trace; c0100018 Trace; c0107252 Trace; c01193aa Trace; c010a99a Code; c011bd45 00000000 <_EIP>: Code; c011bd45 <===== 0: 89 4a 04 mov %ecx,0x4(%edx) <===== Code; c011bd48 3: 89 11 mov %edx,(%ecx) Code; c011bd4a 5: 89 41 04 mov %eax,0x4(%ecx) Code; c011bd4d 8: 89 08 mov %ecx,(%eax) Code; c011bd4f a: c6 05 9c 98 28 c0 01 movb $0x1,0xc028989c Code; c011bd56 11: 53 push %ebx Code; c011bd57 12: 9d popf Code; c011bd58 13: 89 00 mov %eax,(%eax) Kernel panic: Aiee, killing interrupt handler! 5 warnings issued. Results may not be reliable. From owner-netdev@oss.sgi.com Mon Mar 5 15:32:13 2001 Received: by oss.sgi.com id ; Mon, 5 Mar 2001 15:31:53 -0800 Received: from citadel.myri.com ([199.120.212.1]:30350 "EHLO myri.com") by oss.sgi.com with ESMTP id ; Mon, 5 Mar 2001 15:31:48 -0800 Received: from orion.myri.com.myri.com (orion [199.120.212.245]) by myri.com (8.9.3+Sun/8.9.1) with SMTP id PAA26583; Mon, 5 Mar 2001 15:31:18 -0800 (PST) Date: Mon, 5 Mar 2001 15:31:18 -0800 (PST) From: Bob Felderman Message-Id: <200103052331.PAA26583@myri.com> To: andrewm@uow.edu.au, kuznet@ms2.inr.ac.ru Subject: Re: [Fwd: Re: possible bug x86 2.4.2 SMP in IP receive stack] Cc: feldy@myri.com, netdev@oss.sgi.com Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing => Alexey, => => Bob is consistently getting these oopses running netperf with => UDP on 2.4.2. He's using the myrinet hardware and drivers. It's => very high speed - over 100 mbytes/sec. I believe he's experiencing => out-of-memory conditions. => => Sometimes he also gets assertion failures from ip_frag_destroy. => `del_timer == 0'. => => Can you think of anything which would cause this to happen => in an out-of-memory situation? => => BTW - the drivers he's using aren't in the stock kernel. So => it's possible that this is a driver problem which we haven't => seen before. I had a look at the driver he's using and the => Rx path seems OK. If I only use a single CPU with the same kernel, I do NOT have any problems. From owner-netdev@oss.sgi.com Mon Mar 5 15:46:53 2001 Received: by oss.sgi.com id ; Mon, 5 Mar 2001 15:46:44 -0800 Received: from horus.its.uow.edu.au ([130.130.68.25]:8927 "EHLO horus.its.uow.edu.au") by oss.sgi.com with ESMTP id ; Mon, 5 Mar 2001 15:46:29 -0800 Received: from uow.edu.au (wumpus.its.uow.edu.au [130.130.68.12]) by horus.its.uow.edu.au (8.9.3/8.9.3) with ESMTP id KAA04415; Tue, 6 Mar 2001 10:46:07 +1100 (EST) Message-ID: <3AA4253E.6F0B0365@uow.edu.au> Date: Mon, 05 Mar 2001 23:46:06 +0000 From: Andrew Morton X-Mailer: Mozilla 4.61 [en] (X11; I; Linux 2.4.1-pre10 i686) X-Accept-Language: en MIME-Version: 1.0 To: Bob Felderman CC: kuznet@ms2.inr.ac.ru, netdev@oss.sgi.com Subject: Re: [Fwd: Re: possible bug x86 2.4.2 SMP in IP receive stack] References: <200103052331.PAA26583@myri.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing Bob Felderman wrote: > > If I only use a single CPU with the same kernel, I do NOT have > any problems. Here's a different, but similar oops: CPU: 0 EIP: 0010:[] Using defaults from ksymoops -t elf32-i386 -a i386 EFLAGS: 00010086 eax: 00000000 ebx: cff03704 ecx: 00000246 edx: c02dd03c esi: c02ed8b8 edi: 0000001e ebp: c030d3d4 esp: c82ddc7c ds: 0018 es: 0018 ss: 0018 Process netserver (pid: 1007, stackpage=c82dd000) Stack: cff036e0 c01f61e1 cff03704 cfd4baa0 cfd4baa0 c82dc000 cfd4baa0 c82dc000 c01f6a6a cfd4baa0 cfd4baa0 cfd4baa0 cfd4baa0 c8114040 c8114040 c82dc000 cfd4baa0 c01f5c50 cfd4baa0 cfd4baa0 c8114040 cfd4baa0 cfd4baa0 c01f6011 Call Trace: [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] Code: 89 10 b8 01 00 00 00 c7 43 04 00 00 00 00 c7 03 00 00 00 00 >>EIP; c011bd86 <===== Trace; c01f61e1 Trace; c01f6a6a Trace; c01f5c50 Trace; c01f6011 Trace; c01ed73e Trace; c01193aa Trace; c010909c Trace; c0225fc9 Trace; c01ebc09 Trace; c020f5a3 Trace; c01193aa Trace; c0214675 Trace; c01e88a1 Trace; c01e975d Trace; c01193aa Trace; c010a99a Trace; c010909c Trace; c01e97d6 Trace; c01e9e54 Trace; c0108fdb Code; c011bd86 00000000 <_EIP>: Code; c011bd86 <===== 0: 89 10 mov %edx,(%eax) <===== Code; c011bd88 2: b8 01 00 00 00 mov $0x1,%eax Code; c011bd8d 7: c7 43 04 00 00 00 00 movl $0x0,0x4(%ebx) Code; c011bd94 e: c7 03 00 00 00 00 movl $0x0,(%ebx) Oops: 0000 Kernel panic: Aiee, killing interrupt handler! CPU: 1 EIP: 0010:[] EFLAGS: 00010086 eax: 00000000 ebx: 00000206 ecx: cff03bc4 edx: 00000bb8 esi: 00000000 edi: 00000058 ebp: c02ed840 esp: c1449e54 ds: 0018 es: 0018 ss: 0018 Process swapper (pid: 0, stackpage=c1449000) Stack: 00000000 cff03ba0 00000058 c01f63eb cff03bc4 00013ff9 c8284040 3c82c9c7 c8284040 0000e12d 00000058 c01f64c2 00000016 cff03ba0 c7d4cd11 c01f6af9 00000016 c8284040 c7d4cde0 c7d4cde0 c7d4cde0 c7d4cde0 c8284040 3782c9c7 Call Trace: [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] Code: 8b 10 89 4a 04 89 11 89 41 04 89 08 c6 05 9c 98 28 c0 01 53 >>EIP; c011bd43 <===== Trace; c01f63eb Trace; c01f64c2 Trace; c01f6af9 Trace; c01f5c50 Trace; c01f6011 Trace; c01ed73e Trace; c010a99a Trace; c01071c0 Trace; c01071c0 Trace; c010909c Trace; c01071c0 Trace; c01071c0 Trace; c0100018 Trace; c0107252 Trace; c01193aa Trace; c010a99a Code; c011bd43 00000000 <_EIP>: Code; c011bd43 <===== 0: 8b 10 mov (%eax),%edx <===== Code; c011bd45 2: 89 4a 04 mov %ecx,0x4(%edx) Code; c011bd48 5: 89 11 mov %edx,(%ecx) Code; c011bd4a 7: 89 41 04 mov %eax,0x4(%ecx) Code; c011bd4d a: 89 08 mov %ecx,(%eax) Code; c011bd4f c: c6 05 9c 98 28 c0 01 movb $0x1,0xc028989c Code; c011bd56 13: 53 push %ebx 5 warnings issued. Results may not be reliable. From owner-netdev@oss.sgi.com Mon Mar 5 15:50:43 2001 Received: by oss.sgi.com id ; Mon, 5 Mar 2001 15:50:34 -0800 Received: from horus.its.uow.edu.au ([130.130.68.25]:47601 "EHLO horus.its.uow.edu.au") by oss.sgi.com with ESMTP id ; Mon, 5 Mar 2001 15:50:25 -0800 Received: from uow.edu.au (wumpus.its.uow.edu.au [130.130.68.12]) by horus.its.uow.edu.au (8.9.3/8.9.3) with ESMTP id KAA05841; Tue, 6 Mar 2001 10:50:11 +1100 (EST) Message-ID: <3AA42633.78235BD3@uow.edu.au> Date: Mon, 05 Mar 2001 23:50:11 +0000 From: Andrew Morton X-Mailer: Mozilla 4.61 [en] (X11; I; Linux 2.4.1-pre10 i686) X-Accept-Language: en MIME-Version: 1.0 To: Bob Felderman CC: kuznet@ms2.inr.ac.ru, netdev@oss.sgi.com Subject: Re: [Fwd: Re: possible bug x86 2.4.2 SMP in IP receive stack] References: <200103052331.PAA26583@myri.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing Bob Felderman wrote: > > If I only use a single CPU with the same kernel, I do NOT have > any problems. Oh. Your driver doesn't appear to have any SMP locking. The same skb can be fed to the network stack twice. From owner-netdev@oss.sgi.com Mon Mar 5 15:53:44 2001 Received: by oss.sgi.com id ; Mon, 5 Mar 2001 15:53:34 -0800 Received: from citadel.myri.com ([199.120.212.1]:10383 "EHLO myri.com") by oss.sgi.com with ESMTP id ; Mon, 5 Mar 2001 15:53:31 -0800 Received: from orion.myri.com.myri.com (orion [199.120.212.245]) by myri.com (8.9.3+Sun/8.9.1) with SMTP id PAA26806; Mon, 5 Mar 2001 15:53:18 -0800 (PST) Date: Mon, 5 Mar 2001 15:53:18 -0800 (PST) From: Bob Felderman Message-Id: <200103052353.PAA26806@myri.com> To: andrewm@uow.edu.au, feldy@myri.com Subject: Re: [Fwd: Re: possible bug x86 2.4.2 SMP in IP receive stack] Cc: kuznet@ms2.inr.ac.ru, netdev@oss.sgi.com Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing => Bob Felderman wrote: => > => > If I only use a single CPU with the same kernel, I do NOT have => > any problems. => => Oh. => => Your driver doesn't appear to have any SMP locking. The => same skb can be fed to the network stack twice. => I wondered the same thing. Earlier today (before I sent you the oopses) I added a mutex to surround my linux interrupt routine. So the oopses you have are with a mutex around the interrupt routine. I'm going to check again to make sure it is really enforcing serialization. Bob From owner-netdev@oss.sgi.com Mon Mar 5 16:00:33 2001 Received: by oss.sgi.com id ; Mon, 5 Mar 2001 16:00:23 -0800 Received: from citadel.myri.com ([199.120.212.1]:22671 "EHLO myri.com") by oss.sgi.com with ESMTP id ; Mon, 5 Mar 2001 16:00:16 -0800 Received: from frisbee.myri.com (frisbee.myri.com [199.120.212.209]) by myri.com (8.9.3+Sun/8.9.1) with ESMTP id QAA26865; Mon, 5 Mar 2001 16:00:15 -0800 (PST) Received: (from feldy@localhost) by frisbee.myri.com (8.9.3/8.9.1) id QAA21285; Mon, 5 Mar 2001 16:00:13 -0800 Date: Mon, 5 Mar 2001 16:00:13 -0800 From: Bob Felderman Message-Id: <200103060000.QAA21285@frisbee.myri.com> To: andrewm@uow.edu.au, feldy@myri.com Subject: Re: [Fwd: Re: possible bug x86 2.4.2 SMP in IP receive stack] Cc: kuznet@ms2.inr.ac.ru, netdev@oss.sgi.com Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing => From andrewm@uow.edu.au Mon Mar 5 15:50:15 2001 => Your driver doesn't appear to have any SMP locking. The => same skb can be fed to the network stack twice. The driver does (and always has had) if (test_and_set_bit(0, (void *) &is->arch.interrupt) != 0) { /* hmm, interrupt called twice */ [other stuff deleted] } From owner-netdev@oss.sgi.com Mon Mar 5 16:13:23 2001 Received: by oss.sgi.com id ; Mon, 5 Mar 2001 16:13:13 -0800 Received: from panic.ohr.gatech.edu ([130.207.47.194]:29406 "HELO havoc.gtf.org") by oss.sgi.com with SMTP id ; Mon, 5 Mar 2001 16:12:58 -0800 Received: from mandrakesoft.com (adsl-20-73-169.asm.bellsouth.net [66.20.73.169]) by havoc.gtf.org (Postfix) with ESMTP id 319BA1F6A; Mon, 5 Mar 2001 19:12:43 -0500 (EST) Message-ID: <3AA42B7B.7C562799@mandrakesoft.com> Date: Mon, 05 Mar 2001 19:12:43 -0500 From: Jeff Garzik Organization: MandrakeSoft X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.4.3-pre2 i686) X-Accept-Language: en MIME-Version: 1.0 To: Bob Felderman Cc: andrewm@uow.edu.au, kuznet@ms2.inr.ac.ru, netdev@oss.sgi.com Subject: Re: [Fwd: Re: possible bug x86 2.4.2 SMP in IP receive stack] References: <200103060000.QAA21285@frisbee.myri.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing Bob Felderman wrote: > > => From andrewm@uow.edu.au Mon Mar 5 15:50:15 2001 > => Your driver doesn't appear to have any SMP locking. The > => same skb can be fed to the network stack twice. > > The driver does (and always has had) > > if (test_and_set_bit(0, (void *) &is->arch.interrupt) != 0) { > /* hmm, interrupt called twice */ > [other stuff deleted] > } That is most definitely -not- good SMP locking. In normal drivers on normal hardware, the interrupt handler is never ever called twice anyway. This code is an artifact of Donald Becker's driver skeleton, which includes this check. A few of his drivers call the interrupt handler routine from normal driver code, thus requiring the check. Most drivers do not need this check. Read Documentation/networking/netdevices.txt in 2.4.x kernels. Also, since it doesn't cover interrupt handling and hardware, the basic rule is: * spin_lock around your Tx interrupt handling path. * spin_lock_irq around your dev->hard_start_xmit Tx submission code. Ideally your Rx interrupt handling path is independent of other code, and need not be locked. Jeff -- Jeff Garzik | "You see, in this world there's two kinds of Building 1024 | people, my friend: Those with loaded guns MandrakeSoft | and those who dig. You dig." --Blondie From owner-netdev@oss.sgi.com Mon Mar 5 16:28:03 2001 Received: by oss.sgi.com id ; Mon, 5 Mar 2001 16:27:54 -0800 Received: from horus.its.uow.edu.au ([130.130.68.25]:28400 "EHLO horus.its.uow.edu.au") by oss.sgi.com with ESMTP id ; Mon, 5 Mar 2001 16:27:43 -0800 Received: from uow.edu.au (wumpus.its.uow.edu.au [130.130.68.12]) by horus.its.uow.edu.au (8.9.3/8.9.3) with ESMTP id LAA19321; Tue, 6 Mar 2001 11:25:27 +1100 (EST) Message-ID: <3AA42E76.BE56B8BB@uow.edu.au> Date: Tue, 06 Mar 2001 00:25:26 +0000 From: Andrew Morton X-Mailer: Mozilla 4.61 [en] (X11; I; Linux 2.4.1-pre10 i686) X-Accept-Language: en MIME-Version: 1.0 To: Bob Felderman CC: kuznet@ms2.inr.ac.ru, netdev@oss.sgi.com Subject: Re: [Fwd: Re: possible bug x86 2.4.2 SMP in IP receive stack] References: <200103060000.QAA21285@frisbee.myri.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing Bob Felderman wrote: > > => From andrewm@uow.edu.au Mon Mar 5 15:50:15 2001 > => Your driver doesn't appear to have any SMP locking. The > => same skb can be fed to the network stack twice. > > The driver does (and always has had) > > if (test_and_set_bit(0, (void *) &is->arch.interrupt) != 0) { > /* hmm, interrupt called twice */ > [other stuff deleted] > } I see. I took a closer look and yes, the Rx path looks OK. I think your start_xmit could still race with the tx interrupt though? Could I suggest that you just lock the heck out of it? If the problems go away then it's the driver and we can stop spamming Alexey. Something like this: --- gmip.c.orig Tue Mar 6 11:06:50 2001 +++ gmip.c Tue Mar 6 11:10:32 2001 @@ -25,6 +25,7 @@ #include #include #include +#include #include "gm_arch_def.h" #include "gm_klog_debug.h" @@ -43,6 +44,8 @@ static int gm_net_copy_threshold = 256+16; MODULE_PARM(gm_net_copy_threshold, "i"); +spinlock_t dumb_lock; + /* should be a multiple of 16 bytes */ static inline gm_size_t @@ -151,8 +154,19 @@ #endif static +void _gmip_recv_interrupt (void *gmnetp, const unsigned int ulen, gm_u16_t csum) +static void gmip_recv_interrupt (void *gmnetp, const unsigned int ulen, gm_u16_t csum) { + unsigned long flags; + spin_lock_irqsave(&dumb_lock, flags); + _gmip_recv_interrupt (gmnetp, ulen, csum); + spin_unlock_irqrestore(&dumb_lock, flags); +} + +static +void _gmip_recv_interrupt (void *gmnetp, const unsigned int ulen, gm_u16_t csum) +{ gm_arch_net_info_t *gmnet = gmnetp; struct net_device *dev = gmnet->dev; int index = gmnet->rdone & GM_RECV_RING_MAX_INDEX; @@ -298,8 +312,19 @@ } static +void _gmip_sent_interrupt(void *gmnetp) +static void gmip_sent_interrupt(void *gmnetp) { + unsigned long flags; + spin_lock_irqsave(&dumb_lock, flags); + _gmip_sent_interrupt(gmnetp); + spin_unlock_irqrestore(&dumb_lock, flags); +} + +static +void _gmip_sent_interrupt(void *gmnetp) +{ int index; gm_arch_net_info_t *gmnet = gmnetp; struct sk_buff *skb; @@ -455,7 +480,19 @@ static int gm_max_diff_ring = 0; +static int _gmip_xmit(struct sk_buff *skb, struct net_device *dev); static int gmip_xmit(struct sk_buff *skb, struct net_device *dev) +{ + int ret; + unsigned long flags; + + spinlock_irqsave(&dumb_lock, flags); + ret = _gmip_xmit(skb, dev); + spin_unlock_irqrestore(&dumb_lock, flags); + return ret; +} + +static int _gmip_xmit(struct sk_buff *skb, struct net_device *dev) { gm_arch_net_info_t *gmnet = dev->priv; struct ethhdr *eth = (struct ethhdr*)skb->data; From owner-netdev@oss.sgi.com Mon Mar 5 16:29:03 2001 Received: by oss.sgi.com id ; Mon, 5 Mar 2001 16:28:55 -0800 Received: from shell.cyberus.ca ([209.195.95.7]:64900 "EHLO shell.cyberus.ca") by oss.sgi.com with ESMTP id ; Mon, 5 Mar 2001 16:28:42 -0800 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id TAA26693; Mon, 5 Mar 2001 19:25:26 -0500 (EST) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Mon, 5 Mar 2001 19:25:26 -0500 (EST) From: jamal To: Bob Felderman cc: , , Subject: Re: [Fwd: Re: possible bug x86 2.4.2 SMP in IP receive stack] In-Reply-To: <200103060000.QAA21285@frisbee.myri.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing Now this is the problem with proprietary drivers that nobody sees the code for (or maybe hardware that nobody sees specs for) You caused people all the pain of trying to decode what your problem is only to find you are making some basic mistakes. There was a document URL posted on how to write drivers with the new 2.4 kernel (here on netdev). Look up the archive. Sorry, nothing personal here. But this is where Open Source might have benefited you. cheers, jamal On Mon, 5 Mar 2001, Bob Felderman wrote: > => From andrewm@uow.edu.au Mon Mar 5 15:50:15 2001 > => Your driver doesn't appear to have any SMP locking. The > => same skb can be fed to the network stack twice. > > The driver does (and always has had) > > if (test_and_set_bit(0, (void *) &is->arch.interrupt) != 0) { > /* hmm, interrupt called twice */ > [other stuff deleted] > } > From owner-netdev@oss.sgi.com Mon Mar 5 16:34:43 2001 Received: by oss.sgi.com id ; Mon, 5 Mar 2001 16:34:34 -0800 Received: from horus.its.uow.edu.au ([130.130.68.25]:47490 "EHLO horus.its.uow.edu.au") by oss.sgi.com with ESMTP id ; Mon, 5 Mar 2001 16:34:28 -0800 Received: from uow.edu.au (wumpus.its.uow.edu.au [130.130.68.12]) by horus.its.uow.edu.au (8.9.3/8.9.3) with ESMTP id LAA22677; Tue, 6 Mar 2001 11:34:07 +1100 (EST) Message-ID: <3AA4307F.21698B6C@uow.edu.au> Date: Tue, 06 Mar 2001 00:34:07 +0000 From: Andrew Morton X-Mailer: Mozilla 4.61 [en] (X11; I; Linux 2.4.1-pre10 i686) X-Accept-Language: en MIME-Version: 1.0 To: jamal CC: Bob Felderman , kuznet@ms2.inr.ac.ru, netdev@oss.sgi.com Subject: Re: [Fwd: Re: possible bug x86 2.4.2 SMP in IP receive stack] References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing jamal wrote: > > Now this is the problem with proprietary drivers that nobody sees the code > for (or maybe hardware that nobody sees specs for) * Permission to use, copy, modify and distribute this software and its * * documentation in source and binary forms for non-commercial purposes * * and without fee is hereby granted, provided that the modified software * * is returned to Myricom, Inc. for redistribution. So it's not *too* sinful :) > You caused people all the pain of trying to decode what your problem is > only to find you are making some basic mistakes. I don't know if that's proven yet. From owner-netdev@oss.sgi.com Mon Mar 5 16:45:43 2001 Received: by oss.sgi.com id ; Mon, 5 Mar 2001 16:45:34 -0800 Received: from shell.cyberus.ca ([209.195.95.7]:901 "EHLO shell.cyberus.ca") by oss.sgi.com with ESMTP id ; Mon, 5 Mar 2001 16:45:16 -0800 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id TAA26716; Mon, 5 Mar 2001 19:42:09 -0500 (EST) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Mon, 5 Mar 2001 19:42:09 -0500 (EST) From: jamal To: Andrew Morton cc: Bob Felderman , , Subject: Re: [Fwd: Re: possible bug x86 2.4.2 SMP in IP receive stack] In-Reply-To: <3AA4307F.21698B6C@uow.edu.au> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing On Tue, 6 Mar 2001, Andrew Morton wrote: > jamal wrote: > > > > Now this is the problem with proprietary drivers that nobody sees the code > > for (or maybe hardware that nobody sees specs for) > > * Permission to use, copy, modify and distribute this software and its * > * documentation in source and binary forms for non-commercial purposes * > * and without fee is hereby granted, provided that the modified software * > * is returned to Myricom, Inc. for redistribution. > > So it's not *too* sinful :) > OK, sorry, i pulled the trigger too fast. Just evidence i really dislike non-GPL stuff. cheers, jamal From owner-netdev@oss.sgi.com Mon Mar 5 17:09:33 2001 Received: by oss.sgi.com id ; Mon, 5 Mar 2001 17:09:23 -0800 Received: from smtp1.cern.ch ([137.138.128.38]:38927 "EHLO smtp1.cern.ch") by oss.sgi.com with ESMTP id ; Mon, 5 Mar 2001 17:09:21 -0800 Received: from lxplus015.cern.ch (IDENT:root@lxplus015.cern.ch [137.138.161.112]) by smtp1.cern.ch (8.9.3/8.9.3) with ESMTP id CAA22339; Tue, 6 Mar 2001 02:07:11 +0100 (MET) Received: (from jes@localhost) by lxplus015.cern.ch (8.9.3/8.9.3) id CAA07951; Tue, 6 Mar 2001 02:07:10 +0100 To: Jeff Garzik Cc: Bob Felderman , andrewm@uow.edu.au, kuznet@ms2.inr.ac.ru, netdev@oss.sgi.com Subject: Re: [Fwd: Re: possible bug x86 2.4.2 SMP in IP receive stack] References: <200103060000.QAA21285@frisbee.myri.com> <3AA42B7B.7C562799@mandrakesoft.com> From: Jes Sorensen Date: 06 Mar 2001 02:07:10 +0100 In-Reply-To: Jeff Garzik's message of "Mon, 05 Mar 2001 19:12:43 -0500" Message-ID: Lines: 15 User-Agent: Gnus/5.070096 (Pterodactyl Gnus v0.96) Emacs/20.4 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing >>>>> "Jeff" == Jeff Garzik writes: Jeff> * spin_lock around your Tx interrupt handling path. * Jeff> spin_lock_irq around your dev->hard_start_xmit Tx submission Jeff> code. You don't need locks around this if the hardware is sane. Jeff> Ideally your Rx interrupt handling path is independent of other Jeff> code, and need not be locked. Anyway you only need the locks if two parts of the code can be invoked simultanously. Jes From owner-netdev@oss.sgi.com Mon Mar 5 17:20:44 2001 Received: by oss.sgi.com id ; Mon, 5 Mar 2001 17:20:34 -0800 Received: from panic.ohr.gatech.edu ([130.207.47.194]:49630 "HELO havoc.gtf.org") by oss.sgi.com with SMTP id ; Mon, 5 Mar 2001 17:20:16 -0800 Received: from mandrakesoft.com (adsl-20-73-169.asm.bellsouth.net [66.20.73.169]) by havoc.gtf.org (Postfix) with ESMTP id 00D7A1F6A; Mon, 5 Mar 2001 20:20:03 -0500 (EST) Message-ID: <3AA43B44.DE3089A2@mandrakesoft.com> Date: Mon, 05 Mar 2001 20:20:04 -0500 From: Jeff Garzik Organization: MandrakeSoft X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.4.3-pre2 i686) X-Accept-Language: en MIME-Version: 1.0 To: netdev@oss.sgi.com, linux-net@vger.rutgers.edu Subject: Zeroing interface stats? Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing Just spotted this code in acenic.c. Should it be carried over to other net drivers? static int ace_open(struct net_device *dev) { [...] /* * Zero the stats when restarting the interface... */ memset(&ap->stats, 0, sizeof(ap->stats)); -- Jeff Garzik | "You see, in this world there's two kinds of Building 1024 | people, my friend: Those with loaded guns MandrakeSoft | and those who dig. You dig." --Blondie From owner-netdev@oss.sgi.com Mon Mar 5 18:15:03 2001 Received: by oss.sgi.com id ; Mon, 5 Mar 2001 18:14:44 -0800 Received: from cx97923-a.phnx3.az.home.com ([24.9.112.194]:31758 "EHLO grok.yi.org") by oss.sgi.com with ESMTP id ; Mon, 5 Mar 2001 18:14:40 -0800 Received: from candelatech.com (IDENT:greear@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.0/8.11.0) with ESMTP id f262hFr06042; Mon, 5 Mar 2001 19:43:15 -0700 Message-ID: <3AA44EC2.DAE0B7A0@candelatech.com> Date: Mon, 05 Mar 2001 19:43:14 -0700 From: Ben Greear Organization: Candela Technologies X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.2.17-14 i686) X-Accept-Language: en MIME-Version: 1.0 To: Jeff Garzik CC: netdev@oss.sgi.com, linux-net@vger.rutgers.edu Subject: Re: Zeroing interface stats? References: <3AA43B44.DE3089A2@mandrakesoft.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing Jeff Garzik wrote: > > Just spotted this code in acenic.c. Should it be carried over to other > net drivers? > > static int ace_open(struct net_device *dev) > { > [...] > /* > * Zero the stats when restarting the interface... > */ > memset(&ap->stats, 0, sizeof(ap->stats)); > > -- > Jeff Garzik | "You see, in this world there's two kinds of > Building 1024 | people, my friend: Those with loaded guns > MandrakeSoft | and those who dig. You dig." --Blondie Well, sounds like a pretty good idea, but I know it will break at least one piece of code I've written because I assume the (measly) 32-bit counters have wrapped if the current value is less than the last value read...) Now, if someone wanted to go make sure all counters were 64-bit, then that would be cool :) Of course, I can probably figure a way to re-write my code...but I can't think of a concise way offhand... Ben -- Ben Greear (greearb@candelatech.com) http://www.candelatech.com Author of ScryMUD: scry.wanfear.com 4444 (Released under GPL) http://scry.wanfear.com http://scry.wanfear.com/~greear From owner-netdev@oss.sgi.com Mon Mar 5 18:20:14 2001 Received: by oss.sgi.com id ; Mon, 5 Mar 2001 18:20:03 -0800 Received: from mx1.eskimo.com ([204.122.16.48]:7432 "EHLO mx1.eskimo.com") by oss.sgi.com with ESMTP id ; Mon, 5 Mar 2001 18:19:48 -0800 Received: from eskimo.com (klevin@eskimo.com [204.122.16.13]) by mx1.eskimo.com (8.9.1a/8.8.8) with ESMTP id SAA30441; Mon, 5 Mar 2001 18:19:31 -0800 Received: from localhost (klevin@localhost) by eskimo.com (8.9.1a/8.9.1) with SMTP id SAA16400; Mon, 5 Mar 2001 18:19:31 -0800 (PST) X-Authentication-Warning: eskimo.com: klevin owned process doing -bs Date: Mon, 5 Mar 2001 18:19:30 -0800 (PST) From: Noah Romer To: Jeff Garzik cc: netdev@oss.sgi.com Subject: Re: [Fwd: Re: possible bug x86 2.4.2 SMP in IP receive stack] In-Reply-To: <3AA42B7B.7C562799@mandrakesoft.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing On Mon, 5 Mar 2001, Jeff Garzik wrote: > * spin_lock around your Tx interrupt handling path. > * spin_lock_irq around your dev->hard_start_xmit Tx submission code. > > Ideally your Rx interrupt handling path is independent of other code, > and need not be locked. Well, if the Rx code modifies anything that is shared by all instances of the Rx code (i.e. a queue or stack), you're going to need to lock around those access points in order to be SMP safe. If the Rx code is in an interrupt context, then you've got to spin_lock_irq. -- Noah Romer |"Everyone is more or less mad on one point." klevin@eskimo.com | - Rudyard Kipling PGP key available | by finger or email | From owner-netdev@oss.sgi.com Mon Mar 5 18:37:23 2001 Received: by oss.sgi.com id ; Mon, 5 Mar 2001 18:37:13 -0800 Received: from horus.its.uow.edu.au ([130.130.68.25]:8613 "EHLO horus.its.uow.edu.au") by oss.sgi.com with ESMTP id ; Mon, 5 Mar 2001 18:36:47 -0800 Received: from uow.edu.au (wumpus.its.uow.edu.au [130.130.68.12]) by horus.its.uow.edu.au (8.9.3/8.9.3) with ESMTP id NAA06721; Tue, 6 Mar 2001 13:36:37 +1100 (EST) Message-ID: <3AA44D36.540CC5BE@uow.edu.au> Date: Tue, 06 Mar 2001 02:36:38 +0000 From: Andrew Morton X-Mailer: Mozilla 4.61 [en] (X11; I; Linux 2.4.1-pre10 i686) X-Accept-Language: en MIME-Version: 1.0 To: Jeff Garzik CC: netdev@oss.sgi.com, linux-net@vger.rutgers.edu Subject: Re: Zeroing interface stats? References: <3AA43B44.DE3089A2@mandrakesoft.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing Jeff Garzik wrote: > > Just spotted this code in acenic.c. Should it be carried over to other > net drivers? > > static int ace_open(struct net_device *dev) > { > [...] > /* > * Zero the stats when restarting the interface... > */ > memset(&ap->stats, 0, sizeof(ap->stats)); > It's a philosophical thing... Do we want the stats to tell us what's happened since the interface was created, of so we want them to tell us what's happened since the interface was opened? I think I prefer the former: tell me what has happened since day one. If we require the ability to zero out the stats then this can be done at a higher level case SIOCZAPSTATS: if (dev->get_stats) { struct net_device_stats *stats = dev->get_stats(dev); if (stats) memset(stats, 0, sizeof(*stats)); } No? From owner-netdev@oss.sgi.com Mon Mar 5 19:04:14 2001 Received: by oss.sgi.com id ; Mon, 5 Mar 2001 19:04:04 -0800 Received: from shell.cyberus.ca ([209.195.95.7]:13445 "EHLO shell.cyberus.ca") by oss.sgi.com with ESMTP id ; Mon, 5 Mar 2001 19:03:47 -0800 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id WAA27125; Mon, 5 Mar 2001 22:02:39 -0500 (EST) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Mon, 5 Mar 2001 22:02:38 -0500 (EST) From: jamal To: Andrew Morton cc: Jeff Garzik , , Subject: Re: Zeroing interface stats? In-Reply-To: <3AA44D36.540CC5BE@uow.edu.au> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;netdev-outgoing On Tue, 6 Mar 2001, Andrew Morton wrote: > Jeff Garzik wrote: > > > > Just spotted this code in acenic.c. Should it be carried over to other > > net drivers? > > > > static int ace_open(struct net_device *dev) > > { > > [...] > > /* > > * Zero the stats when restarting the interface... > > */ > > memset(&ap->stats, 0, sizeof(ap->stats)); > > > > It's a philosophical thing... Do we want the stats > to tell us what's happened since the interface was > created, of so we want them to tell us what's > happened since the interface was opened? > > I think I prefer the former: tell me what has > happened since day one. > Infact i think what the acenic is doing is illegal. These stats are used by SNMP. IIRC, if you have to zero the stats you also must reset the ifindex. Someone double check the SNMP RFCs. I think this removes any ambiguity since the ifindex was received at boot time. OTOH, while i am almost agreeing with you with the proposal below i think it is dangerous because people will write proggies to use it. It is useful for debugging etc > If we require the ability to zero out the stats > then this can be done at a higher level > > case SIOCZAPSTATS: > if (dev->get_stats) { > struct net_device_stats *stats = dev->get_stats(dev); > if (stats) > memset(stats, 0, sizeof(*stats)); > } > > No? > cheers, jamal From owner-netdev@oss.sgi.com Mon Mar 5 19:07:24 2001 Received: by oss.sgi.com id ; Mon, 5 Mar 2001 19:07:04 -0800 Received: from horus.its.uow.edu.au ([130.130.68.25]:2032 "EHLO horus.its.uow.edu.au") by oss.sgi.com with ESMTP id