From owner-netdev@oss.sgi.com Mon Oct 1 00:38:20 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f917cKP10651 for netdev-outgoing; Mon, 1 Oct 2001 00:38:20 -0700 Received: from chiara.elte.hu (chiara.elte.hu [157.181.150.200]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f917cFD10645 for ; Mon, 1 Oct 2001 00:38:15 -0700 Received: by chiara.elte.hu (Postfix, from userid 17806) id B89731FCE; Mon, 1 Oct 2001 09:38:11 +0200 (CEST) Date: Mon, 1 Oct 2001 09:35:47 +0200 (CEST) From: Ingo Molnar Reply-To: To: Richard Gooch Cc: Rik van Riel , Kenneth Johansson , "Randy.Dunlap" , Andreas Dilger , , , Subject: Re: [patch] netconsole-2.4.10-B1 In-Reply-To: <200109301225.f8UCPXl16936@vindaloo.ras.ucalgary.ca> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Sun, 30 Sep 2001, Richard Gooch wrote: > I usually think of "server" as the box that's running all the time, > providing a service to multiple clients. In this case, the netconsole > server should always be running, accepting log messages for storage. > The clients (which are transitory, otherwise netconsole wouldn't be > needed:-), initiate work for the server to do. > > Face it, Ingo's use of "client" and "server" is contrary to accepted > usage. You can't finesse around it. 'server' is the box that serves content. 'client' is one that requests and accepts it. in the case of netconsole, it's the netconsole-module box that produces the messages, and the other one gets them. it's analogous to a browser <-> http server relationship. The browser gets messages from multiple servers - often in parallel. That is a 1:N relationship as well. the fact that the netconsole-module box did not get any formal 'request' from the other side does not mean it's not the content generator: right now the 'request' is implicit, but in the future it might be formalized. The netconsole patch will soon be extended to do crashdumps - and in this case it will not only be a log message server, it will also be a crashdump server. to further underscore why the netconsole-module box is a 'log message server', it can produce messages to multiple 'clients'. (this is already possible by renaming the netconsole module's symbols and inserting it as eg. netconsole2.o.) but in the case of logging there is indeed another way to think about it as well: the netconsole-module box is asking the other side to store logs. But in internet terms it's usually the content producer that is the server, and the content sink that is the client. And i've been coding HTTP and FTP server software lately which influenced my terminology :) in this sense a browser is a 'server' too => the http server requests the served page to be stored on the client. but telling any side to be wrong is stupid - both are correct, and the correct meaning of 'server' depends alot on context. this is why i defined the terms. Ingo From owner-netdev@oss.sgi.com Mon Oct 1 03:19:45 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f91AJjR13570 for netdev-outgoing; Mon, 1 Oct 2001 03:19:45 -0700 Received: from emma1.emma.line.org (postfix@pD9E1EE5F.dip.t-dialin.net [217.225.238.95]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f91AJYD13566 for ; Mon, 1 Oct 2001 03:19:35 -0700 Received: by emma1.emma.line.org (Postfix, from userid 500) id CAFC8A2023; Mon, 1 Oct 2001 12:19:31 +0200 (CEST) Date: Mon, 1 Oct 2001 12:19:31 +0200 From: Matthias Andree To: Linus Torvalds , "David S. Miller" , "netdev@oss.sgi.com" Cc: Linux-Kernel mailing list , Alexey Kuznetsov Subject: devinet.c 4.4BSD compatibility patch to ioctl SIOCGIF* for 2.4.10 Message-ID: <20011001121931.B15943@emma1.emma.line.org> Mail-Followup-To: Linus Torvalds , "David S. Miller" , "netdev@oss.sgi.com" , Linux-Kernel mailing list , Alexey Kuznetsov Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline User-Agent: Mutt/1.3.22.1i Sender: owner-netdev@oss.sgi.com Precedence: bulk Linus, Please apply this patch to 2.4.11-preX. It's been in -ac for some revisions now, and no-one has screamed or claimed it broke anything. It's unchanged from my 2.4.9 version, it applies cleanly against 2.4.10, so I'm resubmitting. The patch is also available from http://home.pages.de/~mandree/linux/kernel/2.4/ Best regards, Matthias Andree ------------------------------------------------------------------------------ This is the second edition of my SIOCGIF* compatibility patch, against Linux 2.4.9. In contrast to the first edition, it only does the "search passed-in address" logic for SIOCGIFADDR, DSTADDR, BRDADDR and NETMASK ioctls as suggested by Alexey Kuznetsov. It keeps the "only do this if we got AF_INET" property. This patch allows the aforementioned ioctls to return the proper values for an interface that has multiple addresses with the same label, as configured by: ip addr 192.168.0.1/24 dev eth0 ip addr 172.16.15.14/16 dev eth0 Note that SIOCGIFCONF returns all these IP aliases, which confuses applications that feed the data obtained from SIOCGIFCONF back into SIOCGIFNETMASK, but do not validate the address via SIOCGIFADDR. 4.4BSD applications pass in the interface address alongside the interface name to select the alias. Remember, this patch falls back to interface-only match (return the "primary" address) if at least one of these conditions is true: - the address family is not AF_INET - no interface alias has the address passed in --- linux-2.4.9-f/net/ipv4/devinet.c.orig Wed May 16 19:21:45 2001 +++ linux-2.4.9-f/net/ipv4/devinet.c Mon Sep 17 00:39:41 2001 @@ -20,6 +20,10 @@ * Changes: * Alexey Kuznetsov: pa_* fields are replaced with ifaddr lists. * Cyrus Durgin: updated for kmod + * Matthias Andree: in devinet_ioctl, compare label and + * address (4.4BSD alias style support), + * fall back to comparing just the label + * if no match found. */ #include @@ -463,6 +467,7 @@ int devinet_ioctl(unsigned int cmd, void *arg) { struct ifreq ifr; + struct sockaddr_in sin_orig; struct sockaddr_in *sin = (struct sockaddr_in *)&ifr.ifr_addr; struct in_device *in_dev; struct in_ifaddr **ifap = NULL; @@ -470,6 +475,7 @@ struct net_device *dev; char *colon; int ret = 0; + int tryaddrmatch = 0; /* * Fetch the caller's info block into kernel space @@ -479,6 +485,9 @@ return -EFAULT; ifr.ifr_name[IFNAMSIZ-1] = 0; + /* save original address for comparison */ + memcpy(&sin_orig, sin, sizeof(*sin)); + colon = strchr(ifr.ifr_name, ':'); if (colon) *colon = 0; @@ -496,6 +505,7 @@ so that we do not impose a lock. One day we will be forced to put shlock here (I mean SMP) */ + tryaddrmatch = (sin_orig.sin_family == AF_INET); memset(sin, 0, sizeof(*sin)); sin->sin_family = AF_INET; break; @@ -529,9 +539,29 @@ *colon = ':'; if ((in_dev=__in_dev_get(dev)) != NULL) { - for (ifap=&in_dev->ifa_list; (ifa=*ifap) != NULL; ifap=&ifa->ifa_next) - if (strcmp(ifr.ifr_name, ifa->ifa_label) == 0) - break; + if (tryaddrmatch) { + /* Matthias Andree */ + /* compare label and address (4.4BSD style) */ + /* note: we only do this for a limited set of ioctls + and only if the original address family was AF_INET. + This is checked above. */ + for (ifap=&in_dev->ifa_list; (ifa=*ifap) != NULL; ifap=&ifa->ifa_next) { + if ((strcmp(ifr.ifr_name, ifa->ifa_label) == 0) + && (sin_orig.sin_addr.s_addr == ifa->ifa_address)) { + break; /* found */ + } /* if */ + } /* for */ + } else { /* tryaddrmatch */ + ifa = NULL; + } /* if (tryaddrmatch) */ + /* we didn't get a match, maybe the application is + 4.3BSD-style and passed in junk so we fall back to + comparing just the label */ + if (ifa == NULL) { + for (ifap=&in_dev->ifa_list; (ifa=*ifap) != NULL; ifap=&ifa->ifa_next) + if (strcmp(ifr.ifr_name, ifa->ifa_label) == 0) + break; + } } if (ifa == NULL && cmd != SIOCSIFADDR && cmd != SIOCSIFFLAGS) { From owner-netdev@oss.sgi.com Mon Oct 1 06:46:14 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f91DkEV17399 for netdev-outgoing; Mon, 1 Oct 2001 06:46:14 -0700 Received: from web11502.mail.yahoo.com (web11502.mail.yahoo.com [216.136.172.47]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f91DihD17225 for ; Mon, 1 Oct 2001 06:44:43 -0700 Message-ID: <20011001134316.86022.qmail@web11502.mail.yahoo.com> Received: from [134.22.47.55] by web11502.mail.yahoo.com via HTTP; Mon, 01 Oct 2001 06:43:16 PDT Date: Mon, 1 Oct 2001 06:43:16 -0700 (PDT) From: Guilhem Tardy Subject: Re: timing results To: Ben Greear Cc: netdev@oss.sgi.com In-Reply-To: <3BB1F858.51748064@candelatech.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="0-92941910-1001943796=:85401" Sender: owner-netdev@oss.sgi.com Precedence: bulk --0-92941910-1001943796=:85401 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi, I eventually packaged this stuff in a small Zip file. This is all relative to the Linux kernel source directory. It works well, except for the estimation of the i586 register ticks per second, which unexpectedly shows two modes on my PIII 800 MHz. I guess that I could modify the measurement application to filter out packets whose timings between the rtc interrupt and the i586 register ticks don't match (granted the former is non-zero). Guilhem. __________________________________________________ Do You Yahoo!? Listen to your Yahoo! Mail messages from any phone. http://phone.yahoo.com --0-92941910-1001943796=:85401 Content-Type: application/x-zip-compressed; name="perfIP.zip" Content-Transfer-Encoding: base64 Content-Description: perfIP.zip Content-Disposition: attachment; filename="perfIP.zip" UEsDBBQAAgAIACh/OSvG20BZ4AoAAK4nAAAHABUALmNvbmZpZ1VUCQADnOGw O2mjtDtVeAQAAAAAAIUa21LjOPZ9vsJV/bDdVd1LEiDAVvWDIjuJJpYlLDmE eXG5iQEvSZxJnN7h7/fIjvFNMryAz11HR+ci8eWPLxY6Zek2ypKnaLN5t17i XXyIsnhtbaO32HpKd8/Jy3+sdbr7V2bF6yT748sfmHlTMgtXt+Of7+UHEQg+ vljnTzEJhJUcrV2aWcc4K6kCYg8VEwgB0nQdg5bsdEiyd2sT/443VrrPknR3 LPAFj7Pijk+o40nk1kXmNG4araNfG5CTrk/w63ja79NDzUjK7MB1RGUoAJaO LwjzasAFQEuz+CF9io/H9GBl7/vYinZr6zlWVsYNs+jl7bhuToW4aiJK8HXu rooMvqXAehGAo3Slx41Nejm4iASUENKLv9JjFwapixs9HPuBYI4e90A8PCcc j3vRo17spW3Q++iTVWuNVUCGDzx8YP5ChGxRba9CEG/p8lkThilf4XkLuEK2 3YRMxAPiTRBnHNmFjg/T/Afh0HDmeBCuOBSceC7DC42dBaHSDKpC5M6YT+Sc NjW4wxAjPHdCMSdT+fO6jgsEgKVPvFl4VT+ECodcMvPUaQmHjXiTDARNkN6p BPsMM1u/nUoqFb4Rhzkca80yPTYnszl1aN2MM+hqphV3xo4NaIrkPHRo4CIJ 51dPIn29pYJy4woCHhKGOMGd/DLL0+FGAU77Kqt4jqwvaknEgy7dcUzqZPAZ ztiEMKG1pEDbxHew1AsDNPIeq/1WICWuCSkkNGEeonkSrLIqJGzDAUMa5XMm uRvMPnIkppigCxwd1pPTsZZ1a4tRFBpJ4lEsITXUbZkIO+QQgI4QIcLaxQMX lm4tYWPmO6HjTutyCiBigU7ChHhTKnNsJeYMLOQ0YZQIXPNhI4gRp51IofE2 PbxbMn563aWb9OXdsuPfCVQS6yuV9rdG6ZB2h51HEGYbqIDKkd0yxpHPmS9/ blsA8DPAKq+foYiSmX57Swo6xZe9BEgin/RSiMCbcN5LwuTc8XsphqPbq64z NqeXvPDyTfR+7kFO0JVAa1Bziccb7UfxnfNPNunTm7Uu/F9xTNxFaDvLcGo3 ou8MXdkmQ4khMSpOzO9DG/WiMYG47qFRym2E78aDXhKXMd5L4E3sXryPNFF7 2mTJj8JTZdRZX30ECV35313SZuR2A9eLs/+lh7dk91Jr3j58hxfNTFlAQkqR fi2QWF3iLUy4KXGlIaC6qLLn9MiqFiit1E14CPVEEoyE1MoFAmQvkYcdO/Qh eRjUAxkEoMluQJM+5MzXR5iyNnSwvtSJRy/EjC2IIzq7YtV3jXB9KwlH3NW7 2gadjt4fEzgOM6ej8D4VKmIuoG1+jpKD9fcpPsUQFHU71GIEtDTdIJLxJt6/ prt3S2hKyZx5XYVg/AUczAs6pRe+63ZTJiDLfAB/flcMeUzDb04+SxE1Zr6J oyMcjji27PTptI13WZ6JLpJ1/O/sn8x6hjW/xpv9RbJ7Tq10p9RZ60Pyuzkw lKLndtiXUAoSXQGrTLOJqHW3Z0ARx2q+cXTpraRaOp7N/F71mFEKPWxz3OqK w7ZBDzi4f31AM3UZ54+dXVWeeXpN9gAod/Ti1+nlOflHt0uY2uOrgc6KAhM6 3jw/uf3W5LWjZ63+X8PBYFDrEKplFB1etRGq1xJzBG0I8e+1HDZF7b6wxLLp dMKQr9v8ijFEgWQmyTqpyMHj0WrV6wIYGobXq8t+GmrfXA3u+iNHub1fFX68 HeHxXb8uLK6vL/vL4ZzLy09UKZLxuGdjOamXBvUVSigX3kznR0/c3lwNr3s1 chuPBp85m4nJVS+BILD8Yb+HhIvvBs543EskfTq663fjkiDYj5XBZIiqUA3j wpHCHJXniKxzKTBZTvTFBXAe8yCW+/NLnspEmYYFFkQ3aOTwThoZMavZ1DTq IWANbUyXq6pdYBbBlUHIxzvIUlUJqeqq366e5coDSh/r/fqEeTaEm9ZNzn0A x/IvQ78BcWrsJ8RE5avOEp3sNT4ok78OBxbULCCiv5LsW2ORoaNa9laThHwK x//mDnrUob6PnSPI5tRploxqjwJv5lDdFQEoLApSeAlVpzGgukO9W9yRAQ4D qjCg9GfpEl83z3M5zsNg4qzq++SqKmL093kFguLPSHyEDS5SpcPgWj4cDMy9 o2GMV7jWCWvimGrEe2MEtJbxUUUCdjyCDSFAEeQKfQAs7m6hr9c37rZtmDNd Q8/MDVOnaDHkhj8kB2guj0cL9tD6ukt3P16j7SFaJ+m3dm/qI5t0M4NM3+Kd 5asBR3PWpW908bR7nfQQ7axkl8WH56gl5wF53RZ3G2Xx6WD5ylpdIgOX620m B5g3v0I/eogO8fqbNgn6zZG04BO2B8S/ju/HLN42yBWmTc42awvbP3xGi373 cFQezvJc/T0nhTTecDO2IfWr5K5P27v9KYN5/6BP2x4PuhGLX6ND9AQu7Tbz y9pNz1JCW+sJ5joVTORNbuPZIoeUlPogK0iclYQjbegszzRQ5IREnq1v6dRc encbcvlYu7+rgGBE4Mmfo+vxR3/iEw/Gz0Ze4r22cg41S1Mcsb4qduOVgkMb M0wgKAuEo3vfUPBaMyVKwAcz9Bk3w0HYEVC7MxwCnqPufPhnCgGZPL2pzS3u ZaNt3L4q+0Dk8P+2WUR8SKKNlqmO6vA1b+bq6A/D7wkejML22HM+8dnT6zp9 sdRdaevESzy32cww+EvHDX1Da+AtW3c55cOCVLeBygunZPMaby0JSt+/W6PB YGh9qf3UGEIcCAkH+KeVc5blSBpuBvzLu7G+e4UOwCVQxDsumGbRPv5uQWWx njfpfv9uKUDZaRWJo+6YaduRpYJZ7cIPPgontUC3w8Y8qGBLoi+RCoeobcRB G27mc4mZL3+A0rV/fqPDgc9Q2lN9562QM0QpMmL94ehWr0QVMod51WWxguWe 2TZF0BnS3whu43US6crdElI3C3VZZZqo59e8bjTPRsCkwf8wNkxFONW5qsBd AbLusSmSLfKqsMCJ0EqSlDeFmAiJYHfj8aCl8k/mEsNNGCWQrE32LFedpVXH V5pxc25yiXohKYz7aBmXXIqWvVCLTQKgZI1a1OoxxmRJYE+NqI6KxgyVh4Lo hgJmNtKb5k3b61CQ5aWe1G4R2npKmDp8XnvDUa+xDVZBJ8Ytwly3QqgFWaIu /yz5vm+eDI58SdSz5Md1sa5ICpuJirRtjOcarHG7puB0d0w3Zf6sdz0z1G11 inMrHLd4GSyqXnrarSs+Af1G0z0KoN7IDWNEgXdoMBwshj0U00CYXmvPKsTV 6LZPgiOGlzeDzwj6JQhwCOsjocgR0me6QMrxBM/r+bMACgbj0JJMHNEjWaoX JE/26RaejV0kPiHhpM+N/Vc5BQ0T3UgKxKR73wHAeizAJ+SbSTDTtIjHp3iz iXZxejrmsjT/w1Owq+sTU0YBggm0yg/ElnOthRDv2SGFRqwe64otmKtLUVe2 zWVzTLSSindZC28imAqr49NgRkFrrGpgoWHy0czpekRLNfUdp9UX6aVxbCZC PbjzYKBd7Py0hZmTlDNn9Sg9J/mjdPFapJm8cmlxvI7X+QNHLovYJYf6JNvo xTAV5/uNTReh+aIx8jyDzf0tSC4bTRShjrksQtE62mfp4Vi3OF+XYccDIW5G A63I83gAEQiMatLUTE/5mjqPJh8itnBKappz+ELdtGys1+jprfVQRtFM/QfR o/Dv69L+D1BLAwQUAAIACACmYjwrLygBUnsIAAD6FgAAFQAVAGRyaXZlcnMv Y2hhci9NYWtlZmlsZVVUCQADd6O0O3ejtDtVeAQAFgRlAK1YfXObOBr/O3wK 7aQzie8Sx0lTJ81MbgaDYzO1DTWkSffuhsFCNmoAUYRjez/9PkLYBkO27d5l pi6g3/Ou50U6Vo7R2HshcxoSNGcpygKCXkgakxDhwEs9nJEU+eSVYoL8lL6S lLeVY6CasIz8hnSSkNgnMaaEIy8FDIsJ8pYZi7wFxV4YbtBsg04ikAFskpMz tAooDpAXcgZcUhKxV0EabxALfQHZ8Wsj3ZycOChZZmjDliliq7gCQAFJCTBZ xiHhHNHshCPOIpIFNF4gnhBMvRCdUoJiliEPtTESdrb2BqCr35ADFmsPI3Vg A/M5jWlGWSyNidkK0Rik0Iz4aJ6ySPgHaBNYjTMkrOJtwS7n6ASU5xIQZnHm UeAi3DlnOTTZ+RfEeMswQ6fgYH8FrFo5Bng8mBNnrFoPxqiP7hFOrt/ftJcx VRTTddTpoO+gu/s8Lm2mKGz27XxzhP55jyIStRnKso1LGTzELjzuPuAshOfU W8Fvkn+PKMf5p9hnkeB0jNQwRGyea3eagGPiDFzXQiCC4EyY4WWIrBOWZohv ohkLxS6QBoeUZ2BwBPHIPXSySEmCzkPUf7bMqePaX8c9c4T+0f53gP970lYU yecceHMk/u7uj2ZLHrElJ6AVuI6zUDy9kM2MgYfgkW94+h3+/49ydHRUqC9t 2RqBOAlBVQhd/pyC+lv81i+KEjH/nC9nPk35EUj9xjY8o/gFzTMvEds7QgmO YNMoijDqPIIgUQFUlE/9rxAXUPYeggeKQTiBIXzt6cKE+wS7Ql34ppkT24Tw ofudJYrdnxrqSOC2mikKnZPv6PTdqTrVhq0z/v5jp6UgtJcjXwr28LLnCy97 hgpkA503sVv/H/lF3dsXwQ7ROdgvaB+MgauOjYGq5DEseN8jL6KFI+AjCTmp rudfcwnox1K9NBJC6TyWXwuxluZKs1pnm1ypqpVb9s1kvTLR3huSqNmXwS84 cktayLSH7tDqdjqF0D0TVN5GZYaIYy8u9hIKEqB1d/7cS9pnScXgQ9n62NDV QvYxyt+gQEOiinoILGLImDznh3r3+rr7AVn2xdUu75BI05xyBhV4BWU9DKEg iir2aPd2MKgODM0ISsLlYgFlksbtnzT1GAV+Lndr7z6L/oaxfe39Zef6FzxN sCD4X9w77atjTbWdnxf6IzEHO7CQpfc121Edw5zsJe0Zvp1GBXlPhc7hjg3L /lnyaubYjxNXolpnrUYK6AWDJQ0DEiEHtsTmDF11Opd5h9pxmTqaqz3ajjlu iZ6VZtjFS57JDlRBfnFyxGsGW+IVuxynhMT71pB7tfTigrOzmSj3OQfh3Fa5 HVSZF4YIAe9Oi5cmhHt1eXX7IcfJoi0/NLNzbfXyEtK8hOae+PIGXB331DLY i2aebApxqb4Kv4tQmepUB7c3+ajUI9+dFlVRPvX0llLU3wqhpRlvUm7DWSEY qwNDgy5uTz9LlWUvroJUGE8MV7etD91POcrnyYfuyyFsamqf+o4YC+QeYPiF ZIegsfmsuvZYncKGz2HRGtzUiDImTn802sLY2jtE6cZAmpvg9bppsW9pMhIk wTVq7as2UvW+nQPwBoeeD/PeYUgdVagwkc7JoEg27DqjgqJvwTRzbD065qQv YckVRAd+I5gla640bEDfSj/CRMSi27pQoxBHcZ5mldW+bZVygfCkZtjXiTYy Jp+KqMc4pHEtohN3qI+0HBK7gR/iGherr4EQ41mykUM5rcUinyXc3qMxcoxJ OUlhoNgNTVXGBUdghRYkBgyUikbktNggKWUXaT4h/zUeSrtdZAkPzjmmP6bI 9b3s3pSSGt5qe/bLuH95fQPcpccOmaLXiEC5w4103atfpusBHUwQnZ8hVORg 3Oi12poxgW2ar25naOWwHhjPIwinPTYfbbmbvYyuw2LKr4JH5sCoQEO2oPsj QRVsTUXSTyUuKU/SFd1Eo8sPSGUlL76J7G0qchXxb4nWnVFR3LKwXtref+xe F3mQiufaDreskaEV3c9LkrApKcd2VZWIv6XM7ZV2c9lx99DvSSMuT6xSFCK6 8JqBooSXw+WltBmo90owf9YcJg3an2upelF94S3x/JrTHG07DNTK04OxW4Zj uUQchNqyNDmRVEvSl6kqvRy/pl60i3gF5Jj20CiacMZ4QGf13mFDznWKbiYe a/VVdB93OhnIKnsLBqfxojbOfDa0zhWc3y3psSz5TnHnqinlHnag/ERaW9en 0i44qdbWLG0Mdbbwdn6IPUSoA0uGbJHUkkZK3ieN/NsqcpH/vpE6Q9mDglr7 sSy9/0UqlPjktebe36Utf9RaylPv0XGKVhmv4PCR1Tvl5OlhpNrDAjMPPR7I mwwzDjdIXEKtvAwHPlsgOFAhvsSYEL+NnuCgkjI4rohjz/b+ZYcF+uKSC8GW 49mZgMU5lrN5JrjJ9TaSlx8R8WIORwN5QbXlB2x20n1xUQVHJgCfzFiashXx T/JDlLiqQinxONtLONQGylYYogUVd2QgQtJgJjRQDhPuSXW0oW4Oij2wqmWb qn1+NKZ990mXtdvD35c0JSs/q6f4FxU2tzbcY/1XL84IDhrQY+MZ6tqTTPWI rqGu1YV3O8/PO258hrud9bqB1xbRvLQdXmHVTURfriLyCd2t+AGQDVP7x5ub Ggy+1RJcpLSjmbv8Ps8wq1mvasPfd3pHHg7+qFtvmw9OVWKxn8SmVSjMVkuf wCDumJZuTFsX02UIk6a4X1SUuccznyR3igJnHvEpgM1+h0ovbawcvTsdmrYj +nzxlF9pttA5KyMPqJT6KQrfAYPSPWSrTKIctS/K3A6Q/2o4lTUKYXdNyJL9 hUcuYORcri+yTQLeCBRlf6wGNfcv8E85CpnnwztH5+fRS+bNQlJFgHYlcuVP UEsDBBQAAgAIAKeFOCt9gnBWhwUAAJsPAAAZABUAZHJpdmVycy9jaGFyL3J0 Y19jdXN0b20uY1VUCQADWpuvOyPpsDtVeAQAAAAAAKVX/2/aRhT/2Uj8Dy9M 6wyF4FTbGsGIlqZszZZARZJNmiZZh33ALfbZPZ/Tphv72/fuC8Y2TltpiYTN +37vPu8Lw167tWIRHYGQgR/kmUzi46DdIrncJGIEb1n0ERaU02/arXYrJoyD EldfHqjIWMJHjnes/xXNezk8OR2+8DwP9J/njU5O263eUDG/YjyI8pDCDxHj +YdhkPAVWx9vzg5ZcRLmEW1k3VPBadTIko8pzRo5LEkTIRtZq4DLZnOMs2aV NImaNbKU8SgJ7huZIY3IYyOHCsGT5jyQCO3VWCSL8UANxJwEAc2yBk72mEka a8ahk9LNW4GQrhinsLi98K/Pf5kvHOc7r8y4mM9uF/Mr/3r++u5q6s/Or6dO B82AuTd40IDoKBUrcn53+2a+cDsKSgZUPwh8PU7x9cfVZpAt18ckOCbyrNMd F1qvpzcXi8u3t5fzmdvgU0m2W5kkkgWwy70vNZTV67hg5jxja05DiBK+BpHF /l9stWI00xaGPSUo8sCoShZTAT31uiQZhQnM7q6uxk/JRCSTZRntMNgQAcs8 e0SOp508JCzUqiQMrboLhxZlDF34u91y2ApcY2ACJ10kOKlgXN67HYRLIo6O jv7kOleOjAdnnH4oBaGV97EZujHrlI5Vil7GSm0LNEJOIaeYhW0j4hwqtVvb XYnb0+NNKJgUWdfZUIqCkhDc6m1gPoSJLcnlEpOiCNAH78NLD/QBBZW54GhV cZF8osnbL3H4XjBJmzyig6p0SCSBT8dh6Vqyvw9kF5+i1+IqLp2Jd/6G8DDS 1473CEhBI1qgF9IHH5/9HR5SiZlaZ4gv9WmCqkar7OeZj4ZUAAr7GvHgPtuB X0emeIWkvXFzB7q4F9OfL29upwv/AnbiCjl7Fe1alYfjHEKV9HvLiUWcun7E xr6wnj8f7zTLSPYsCh1bGycaUw6xsSlgGsr7DY4ZcAkcVdBrrJX8wNkEyOCM 8rCQ0CJIW+U8cBXTXm3ZvZVaTiYV47VQSq+mDMaF2K7CFM8aKXjOYVswdFtf Nj5l8NB/SXe5d7fEW7uv29krLYsqJfVASQNxa1/qdpZKtpBq0LSKNb3lpFAi k6qGUdjCcAgHN1rGgafltVwVLhZauqgMznPegPRtpcWuqdQgRZibAkulsGXU c2vNoNfFcsMGNKmNhWp7McBnSSAj11YC4wnO0J5+FJWrliPoqc9Sf1HqQRz2 a2OIiHWttE1AKtgJHIaJ8mOThvdMBhtw0aZOor6HQOH1ZATDHuDxQR/fVKVj LVbPp6Bm+pbJvdZ/ofWzmn653CY6vgZ9XA1IHslRmTOYXs5+Ozf43+r+Yj5r aRU0ouj9CxNrclby3WgzSSn/Hwb3a8VeWtkUSEx4pl2sklTlQykrPPfrTwcX YMzmLAHdcnUyawwznpo4SiVkosRTMe4w2OAjjklqpHfHbxBaRXm22UvZxPef OkHtiZnRANTjC5dj3657tspMHotZ5CsR5BV1aodlBUxmNzLdlL7LaSbVqNRj /hTnYX109uHm3L+c4cha3L29Bfyu184BLG6uO/hNRQld207NvgTur9PFzEcN LTuCgHCeIEjomuFSLPQg/jrEZUqN9NOuWXIK/M7HRfdhK3en5AcbgUPbLTbk fkdv0SRFPYqWnu0A0u0e2rPZt7tJdRKf4ykU4c0fPm68P9mcPSX9yuaoaaa/ wgv5B4/0rbc38onZb7fLhjW7gx2IhNiIiMREwZuPOlcmyPL+41X6cIDQ4nna iJGcP51IqGSy+5l0lRPQcH54Bu6/KgVdc8iVoLQCMAOYz51fjRyVAbtxb3c/ GtYBYu+1EcUX31dQm175Pgx+x19u+IntgwVykIpEJvrnKQzm38MgqPzchkFS /p6YmfcfUEsDBBQAAgAIAMp7OSvgCG9gWXIAAJeOAQATABUAZHJpdmVycy9u ZXQvM2M1OXguY1VUCQADS9uwO9ajtDtVeAQAFgRlAMx9+3fbNtLoz865+R+w 3nM2cirJelh+Je6tLMuNGstyJblJt18PP4qiLNYUqfJhyU27f/udBwCCMp1V tv3uvT09sQgQg8FgMC8MwP3XopvM3ejKC+4/XlWdU9EWzU64yErFTacner3e /scr4WJh4CZiGnkPbiRmYSR8L0jXVfF6/+WL/dcvX+x8iLwkcQNRPzk5rMA/ J2LyKC7CwPan4tx17t2o+vIFvDeee7GIw1mysiNXLOxHMXFFGrtTYQdTMfXi JPImaYLPjhNGUy+4E0koAAORuNEiBhDhjB6/vb4V37qBG9m+uEknvueIK89x g9gtCy+AtsswshEQIO96AeITuTP4HTiugYscE/zCYSFgIsTuD2GUuOtdQmv3 PAwX0FFwtytiN/LcmGni2NE0Blg7fXcxcaNYSNTkO4CFn05dcWnHiUHZptM6 qe3jvw36t0X/HgEY7AvbZ+8C9ZvOSa1GaOCvllCd0gBcYafJHBCXlIxc25kj 8WJ4Qqp/EzuP/rTqhIuygNc6+wNo13EDICZi2107ru8jSQBbMVra8OPCTmzq rxcASRZ24oVQ5Xj4FtJfCNEJYVgnzVq1VRbfhtMpoCQbX/re3TwR3ANURq4b TFw/Ef0L0agdHdUJ8SvkHvEeucoX7enUwz7iU6h5+ULs1KonJ+++8u/hr6gA Og8eQK+KvgeYRlCC7APsWCYevei3RbqcwkzHZtt6tQZvfufOZuJbO/rNuxdv f7mjH98sYGyRfe8iGCTM19DtztBdhA+ugMclDHji+V7yKKbuzAtc5ox7RhZ4 JUZcxVvRqDaqa5z9nVvqn14L3BVUNKtrsQinqY9kBULMgDQ47Pf1ar3aEKW+ HTlzWCtloEmttgdVr8U1tMQB6QaiJDHeQ5oB1QW3b4pGS7SXkedT67Jow3ig cR84FmbqrU2Pi2/ScFV1p2nVTr/m5hXRd6M74I6Vl8yRDY9aljOpOqoWluur BJg3EcM1rIGl7ybEFhPF/YxblC4T4Qb2xEd+u1Ot46UX+KFzjy0eaPFYiQft cPkuozBxnUQspl4oZmng0HyrlrDqCRi0hrWs+4iFHYUp8CGU+ghFQs2Q8AIG IXSH3GeYJqU9URoVNHHsAMUNYWshuqW9PYXHyPURyZUXTMOVaG4OpLT3KhYr kHMuIvMhaFr9dsfqJJGvAPR0gzixo8RaLzxApCyItXBhE32gsWXPACFLrNxX UGP78OBO9ViWla+dNILB0CrERxjXLPX9qhAss0CO2UAfBBkhp0zcZAULbYMa JhIMapMaq7kHfLgCaeFNYcV6My9DA+TleXg3BXp1whhWl5NWN9hoYsNo4nQJ gjYh5o+9NS0AElFiFoEcfbAjL0xR5KcRiQ9F6kRhM7cfXGvpeMKbCfhj8boB JL0E2CtJI1hsv7lRKEozbw2LEYFP0lih6YHAj0hEheEy1nM5BrEmAEIytxMW nB0xh3Fef2j/SLjWa7XJGKVnGGQ8fAkDmAOlfFQ7IB4BzVFiJ2ncBeoBO7wa h6FYpEizMLqvVl+BwJiCvrE1OktgeFz/JyfALGqJiRIIMVe8A8aK9zQVR0sQ MqI3GiLLg4pbhYQYk0X8I1t2Jnb9wYXVu+7sX3Q7NPWaou8QbVyQTGlgKlxO RJiZ7fkpVFRNQM1O57IL6HXG4qp7IZahDzOVPOopBgEpEoNXLGAj744BIt8B D9qpn+RF08GXiyZsrDoFTSDcdRLZYrweuiiIslWfLoGJ4RGYQEiqzsNk6aco lQA12wcpQtiBsJlkY71Jk4zvZ6GVTHymt2VN3QdsCrLbNlYwDk42gJHDhCAH xJew/MS7dud9WYurGMtJSMAKBZCRJYUisNkK2E+xhE0vIdg5sC7aPdWsOwc0 Nlo/LB3CJTBUSsoZmqxsL7GAJSyHhTHNJS6qRhmIi8Q4wB/V/By0ROM4NweS 6AaRob9luHKjpaNVXOm7cB6Arg08UHJvf5nSj2/cJMrUpIhtjxAF8cnsX81E J7CqnLqpZ98FIDE8J34qF4miKBIjnl4Y9XgtYFwLG1SOD3OIigHk3ABNoJUH XS0AliKlM/eWMcjvFLj7UdwBBGQVIWV+NT9EQsfijqQyXtqRpsPQXfqwgKYg mhao5mDNu4I1FlgI3pK1JDRk6a9QKF3CkgxBCMNEeG4q3uKzAwXVIRV8E3hO dRZ9vfd08MjCwD6kZRe4vlCHOGEw857w8NRdugGKZLBt4BEklxWs7EeNRIJq wLEBcduJwlgKWyVv9jZY4lDUDsV3aVDMEOcgxoEMQECaj5ubjuIKTdDbGBe8 z5Sw4sfAYbU2VZoxJ1pWkb1cAsTUD0GMkkkVeTEMw7QlrGhd2jMXfm9kjQfD 6/bFoEwa2qPVj+oFpjBWMhysW+c+TheC1+YivtM0yXkbZdETVy7IXDLUQZyT /PH8+TexH1d9J66CPEMZlE2UZAiRBjgDq2xuiBHOL3vXo/3zy+7HMfNH6kvp GmreiGGgzuGBKN2g5fRPWL6O+wBmqosL+jtQNPHeM0sC2QKtF+ZWQ1cbFg17 JHunqrdpCNwPdrA3s1bATdavqZu6LCTBHsAuI9cJyWFD5qB+YGmVQCajr3Lj 4h83KCt4OfppRG+AeLjMQG4FpBUD5JRKOKsswGSOHkUJvSzwB4GeKMdCUScl WK+BJenEe/k1iUoOWrmiw1ocRbm47nXiU9GatI5o5IctIiFoF/HBvvOhaMMA PhIN4GYWb0+YmUduKvEYxC503Rt+H2dTraQPiIggXnjUCIZG4gTI5tuepJjB 2ADEiex4DhwCpBODQV9MU+QtNVkhLFo0PMG8vwfOWyxgAe+ZK2O4FqCyDT3l uwsFA8GV5q6/5OkqhIEOn7KztXmMwyDDGjxmXCkg2MEixFc802RnN6ZZq6Wj qppyxImMaxQnMHU5y9iazGEwyPywbrED4E57uk8WsEJoEIAc1jbxeH0J/t54 Dlw8D4GVwKLzErCDbMOFqIgQVyRNOstChQwagjAH7oYYP66BPQWGT7tT8e1H 5GYfrHNYmkkU+sr8lAzC/HEs6k3RBsOgiEF6oJwitJNgucQJDOkO5gLGyaYm 2Kt+6sJrto+DfSQ9FT24NBbJs5tmMOkPWzqx4MAo1p759p0UzOhZxQJ90+/v cyqTTVqaGKfVOkTtAhD7Nnjbfo5z8JVub9TeVBSl78BAH96Hkf1LfO8ZsiwA 5QCQej3rY+eHoXXzYUiLq9u9GQ76VqNJ9gPAuQvy7iC9dVGVYuCVCpDE+XWQ g0voa3MyP3u96x+6w7EFJia+WuZQj5ep6g3lY+rsOElns9zEnoh6A2zyZfHK z0NSwhT/TZHSOBtgWPQS8ndAcoaVcFn9jDxGNb1ppiRglxrCIf7C5jkrR0lk xf/vKAwzyYbBK9MF7lkFV16c3CQR2Ih3nkMhredRv4MJE+MwCmxQEeyL2Wj6 IN9oyzrU3mIeqRLHW6SvAlqBTAGY8fZHqzO4uuqNeoNra9gddccyYuZlK2Lo Tj25IFSskBYBxfbA2obxIV/GMErkAxSCzsLxbMuJK00MjdSqeT4DyQHreIXj LeMSNJxhDNnhQKCSrWsj0iJdQxQwK0C+LMiXXNjgLD7TBl9O0WqBIZ/b2glG 2Qf4z9GgCao5ZgQFVz96lhuZ++VqOz7vjfUibxUvcmWzgu70UObbfs61psAk iohzcWUvE3AU3qVR5Dk2GK2lqzAF9L91o4kd3eWMqnftkUUuLyz2XY4kdh4d sMtcUND9CRjT45vBbn5cdRSf1+FD4bgu0sXSdEKt21EX+OL2esxWG7CFBdW3 V11r8OG6O8yDboBxgAILINcVZFFqVA+qtQqYeXWNewd5H6MB7FJZknkm7gyF 90rLb1gnEWp2URqHEw/W9RDnCzTDIkcGCrlSdA2MNlhmj9LQn9scj5HrBzWy WxclktkorYf2k/lJ0C6fFvtmep7QVP2oWfkqDO+pEkYOghPZ/ubdj8B7U/Dz GwegBCIYSUk2EyGqkse96nPz2HRGg3eDeq1WGX8UpXNQXIlAJ+QeI62gkTLb jQN6M9DJIr6foGUg5RKsCjLkYJhLG4R/FuliE2aD05uicaTn7QnD+p4Dy+u/ Qd2AfAG74RUtepAC2B/G25Io9bXrwiEwNFnwfZE8Ll1apayF/jtcUmjwlR79 Lln8wgtx/S6VmkaBQmIm8aYTL4l3wayN0HwNUf2CEp+HIHBAQ/uheGs7C/cb MBhcJ/EebPRlqxPDQeOBIERSTnLNXg8yGScX3k1fsDKJN6SU79r39p2LxiYK qAhUD7BhpuPAlLgniR67IIbRXMb5fPeh837UV3LRsZe2DDsj+XTj9x6sBFPs ldrX7/dyJgiFx5xwCS0j+w6FR2b/4tv/ey8/nQei3hKX7iQ3nV2OXXwYXIEE 7MCsTYC4QGoW0zw5cjGuQv+V8qh5vjS2ZKIgKTmSnTNV2NZRQe0yCMBghoJw tIzA1s88oxlIfSbKq5gFNEysdMXQXIvZtUHRGKNgV2E97ciw91gW7gNIeVAa GOSLObYLYpIHMd1g8ZYgD9mVoul+ucgcIISTAJOmSCCKz0jPAhwF10PjcPKY 4NCubPCrOvMIUQzAtcrJIHBEkzCzWM3ofvaSEacAGsRO5PF6IK8t+s39RfSr 4n3022P8WxLOwpXn/JYRDgURzj5yM0b/LPJ9gZRg3rZvelnc0QWV61cc0MG0 El+BccLu9HEDJhYdlDtXhxAwWOuKeZIsT/f3V6tVNQve7f9LyvB92g7c/zvK 2XUF7HVaNuToTd3E9vxsxbR9EAyxi1aNk5KiwwHugx+LMSUg7D5LomqyBqse 9xd5h1G8Fpe9j/3uqTC37UD8pv7U0kTtj2+FA8x/B4DKYDvgtLkxiDxg6hF1 iob43F7YztyrOsJTNCMsqBcZvcO/7KlkBjCjcT0YAxbApo9hynGcV1J4Tl2w Vl4B+wK3xokNTFP6O4dShPmGqO1xCJXHEAPDBODOTR5F4556mKbkTk3BA6EA s3DRvQ4IR9oDQA9vhYIBFimJQbAaaK+OLEg3gD4T6XqQa8cACbYBCllY2ZjA vwnbabAkUwyLObiTCPMYJksMAeAE8my8fKEGdTH8wbpu97s7uzTxu/kaMPzR VtzZVUtso37Yvbpoj6GxsfJ2uQOcctEWM3fFDlnMcXy5yZmsQJ6ipca7z+K9 C0YY7YXguoy931wyeDHCSTuyMrDuzmYe7WI+ckuFy/ijNexdf2uNev/s7tQP s4qhWdFsZBU378fW+e2lNfrnzk691Tzc2dkBNEbQMfaHW7BAzwXuP0eP6PNP wJFxo6piZrH7PggnoLNoUPb0lxTUP+7oYJCe5gC1xAKjRrEe5Eg6EiTnJ6hR liFGZpTRgeWVENzxCkinx8osIjs7hjlcSFcBICRyO/1r0ABgjAFFSCe64MTL qHbMDCOxkYTyZmCqzIRlgViyrJcvMPINXghxOUpoARoP+ye0xBnO5Bto5vqx yzM57Iv4EYTiIlacqkJUE+oZxLot9/phKOA8VxY2innCm0I0QD9TRfJGe+zK 3XBRqYhocU/Y/jvkYORNwi6YejNGz8eIQizJgyIEXWMwFdGYIn4CUfK4tONY YRjgWvOzxAhomy6rxd0vkpR6JZpAb3177S0wgIlaBQQ78McSfb4kBo2VOFUK 4M15NwfYg7hJB3FyfRB029yjQREKnTUb3FUWGmcIsJZEaeGhr4ehuU1YKztx 5tPwDiAAtoiuQR777g5nSckNAJ1FllC42g/AO52bWzDwQGrZilkpdv8aWyxD oB9qUGgKnfp+jLqZ0iWINVA3F0StqDnKOHICBfIUdHcbyKWoIleSfZWUUiv1 uR2suhwb7g6RYUYdoIhhnQVSGlWIUoYYNq2dwrRjwLkMTgyIUA85oFoVh3t5 cZKT9PQvrSBcQD8MhuPuR+uie377bY7yuTZnufeyhfT8+3WDn83lOrgZ9/og vHDN/p01/I+os1DeoN/i+ZKAM/xFNh7PWhShsSRt8b/pxqhBaaMKE1p82jdS GS+86Yhwqv+mL6n1qLfdymC3msdd5sy8ZYuCzb7q/OunVWwqFVZxrkZhFQrE aXFNQkZ7URU5i4U1MNAgLKzxguLiEO2UYgR8e/IMKLX6i2rBSy6mj+c9A80r BgSCjM3u4pFKW/K5enAwQcs91xQt3406O17se9GvUMrq8xI45npogTM/EqjH lO4x3wdXL1zGRYDCgsKUA+JUIw0FyuEpTPIpsyEFdQuM84XCiIO9Anem3+m1 SVCzbSxkKgb8j8iqHV5ph1ZAvFWYC1XqAxlyVWlCaQvj9tpqj0vrPfAGPDBP YEF9JeBxr2gtTF3ffpSD2SALuB8hTE4xj0aJYzmwDMHtLeSH1vGhBYLlXhFK abC5Hal0p59+zm+Xi7OXL5TxJ3arzumuafGJXbFrWnjwlN9cIhuHjGsQrIZn oXPVlDugfIF5svD/K9gljSSjS+3b8bvBsLSbB/x2M+vt6929N7rNRXfUGfZA KA6uS7uU5kd26z5GXvBf3IySuXu82bavE//2VdiMxZcJ9QbMmxKJ4rLY9Z7U SCkKdfXKLpBR1o3GaFuWjveK2qCjb01T8E3WX9RuvrLUBukXdggq3nKSyP+i Vlk84IuamQZZIcmeGjWFr9E6/tXywOCYRp99I/r1c9Us1SxvWviSsoqKKomj 9NwTM7HKFz6Yd2Bs1SqHe8WNMragZqfiHF22WqV5akTGwO70EnFwiiaxYJOY HVsoPTkVFA9iNinuJM9HjJ/RSNm8JbAFS/VnEN3gKQbyTmayZPvx9ANNJrDo 7am9BFQJbK3yHGCT5xiq3OrLbfClaH+J0k37dtQlUftZmDmOlIQdY8AQZPoH cBrBQ6pcta85GrsVlhvMyogWOWGFDlgxzEL+ZsiLvHfAfsCUdtR1i2Kgm6uB 4VEa9/6ANlxwk5JyKUodelec9wYjvQ2NaPCO8t7nO6DFZEAffi+CFPOf/xxc cxVm0GV0tHfxHwI3Vi8DVRv+hmcklEsUgq4nMGQtDIDottzT1Q45++wqAwlX QBAmHHJSEQl2vcAYYB1CDiTaCckKR0LtpdUcRh54GQBeJ3z3ej2pgU7xDbIz Igw4Ltws9skJqmWVqFovI/QAfTR/ZT+ioKD0Gy9QdoortB7bwMaO2c3FH0mG MrjTMrCa7WsTrEwO6QWq0IrzbpD0YcaDcfuKIiiitm7UsvrzwaDfHbavv82/ clCTBMR4B5hE4IfjlgnNGgVbOU0jeDS27nAvyMPQCWVBLGFZLqQrKI00stBo 30+5OJryFzfHzeODGu2G+JjVwUn2kxDjCmgTZtvSBI/CcbGM9Ni4LisZIrTv QXORIGcm89UcHaJcXACtKjDNLYWmJbGfvjENw5vLjyIzsE6FERfDhPMdTP3H 5BtAQrMpVveqMNWYjd8xbVyskYnDtJqyMw9TN/buMMafO/yABxayMwhorOVK Xr74eFWmN1/FtEgx6ae2jxm1IGXGSrDK3XYbA764TuOM/Lj3CJ69TqaX6yHf rwztcNIL29UoCSPw+8A57V1gAgytENrdK9OfBv9p8Z+jMp+e4D8tJgPyi08H Q3CvAl6rt2grWfdAWNrZXg1Tq8yvVp2yOtcxfflCD0ieDpCmPiqXB9vzaUcF o7Ac/AZnZOrF1bt45lQDO7ard+HD6f4yncjwuQw073NHaPPSlKo5rcQYuJ8B E0m1HWM10h9XJU8t0USw15xGtDNhhlhA9EzCkGUfMXcQil/SxZLC24HL+VsY U6YEAQ7GYddqFhgKSeB4jpF39TIGzGJkJZZapBeux22BRTYl79sBW/yU05kG tNUm4aEGwWhCVU0Qb8HpYfJmQrZdiAwJRY47raAxU5HGDI48nHC6TjUHyGib oe27s0TFqWQys9jNtop2AUkflWWKUb+XL+rE3A2c2/ZtD0lM+4nIvZxF79gB rns6hON7MPXoWuKpB7nX1MOJvOCVF5prdsxb+ydr5Xvg/pUdGAIOdQYsNWgJ YswDbWVH6lAUiJEHSrAHCOsTEDMEQs7XMvIWGILOZTlgOBwW0R0oNKBLBY2D y97lIC5Lxgec6EBCgLyPY/fdCsp6YCjop8IJU0aMNovPlnBbJwBZuVfNxoUp BwXnp2CMGKSn+TNgZZgSMmoNvnwB+jlhOaH0N+9VkE7QW2VhhDI7T6F2/0KA ydfp7t908NQQeK7dDhiFvreUp5wSWLmjJXB/2JSE4x15zDZWHjBuPFP6iK3D Bz6gtUHI/JSRAp24LjbGIyB8hOD8Fe9ep8Cqv6a4s0hL7OULnElOf0bqDQKX sw1izri3afPSnj7gaYeYYdnMEqzBZzBHNL9zydRKDMN8RnJ/2Se1NXHJTteq nPKtA1P/08hljrGOua/wlAOpUrKeJW+IjDeE5I2FzTFcsIthGuQicmGqETfu B+f95QvTmtD96A0uHC4JEJR5uJlmbnJ5MAcOG1LUGwwykAno2fEWQHOJmU2A TLZNotM19FaN5N9ylrgPP6YYj3FAKDxSAqmK/MtBa553catRH8DJpZTiNOOs Mosmc0woRFLf0jYCnltTUGIeK9hzbKoQ7/OsTjjVE/NQtbIBM2MFDLCpPZmR BxFNAclaYizS0NJY8+UxzLxJiLMN4uX5teipCaTccWJrEvNOGkXAwODySIEO 7AkThQFzQYnHvIGqFiRn7xN/Uq7yVJMKFhJvgWTnGCmeJfMyFA15H03cgcG/ ZE0HSid0PDtT2yABeYbI/6LBfsCKzcFJdV/OxaSlUNodYqrbzY/nw277/W62 hza0aTjExDalM0yZSKRfEncDTwmbm6vzOHhgh3BBG3IqOGoaI+flGysRzvmo JkJyNqZ06Ejt/NGC1v7oKa49XLxzUGYUzARCT8H5nc1k6g1lbq+QFGgiIPmz za0n6ClJSpMmZBIuigDfF3Kf8QHwZZaQyODODzKmerb5Td5ZkxsMwDdmN0pL dqpi9Bg48ygMvN+klhxn2+XgDhG9UI9hCpU8KFFWuooXGpq0friKGQdifpJy gKIXy0O6wbTCbCeAocDSAJOIVbkbkHkRPwGJmpktqpcvwOCqfJ0AS8mEHZ6v ULIINhCyp0wgsTsfqX481QNOpOxCGmzqqFAWKtWnpplOP1TFdUjshNSxMYEg weQnWNN4XHfpJb/JQOsYOn8U/TRazslhIBMf5w+U14NHu65TjFaFS0yEePnC MKdbZX3wuCZ9ImlZYRpndkjaHCjae9QFgP8Ft5HoVSV/URi1R71OmaUjnXZG O1ypWVI5PoY8di+AS4NdoGvbPAnOo+HNX84lBZEOBKRkh8ib4gpO1GFLQMVe wKRRThSYEvda+tExRZJHKnVkAlTOdnRL9WrrPZ+avLy46InSAT1LfqEcg78p p0radORkMrA4A6QE28R1bGQftWHGKys7pYcmDioNqD94X80yX9iDTUh3U0+x piOlKyreZW8LxDielwJMJiG4ntESPR0yFeR5NNm9dDV4eimTxPfuMQ1g4kq2 YFakPZFZirkA0seRqLlBuqAETUrwtTAg+Qk8U0ALM0JHVm9wVqcD2vzY7/bP GuZzezTuDs8OyrJN++JiWDurreu1t29r/B4W1WVRPStqyKJGVtSURU2A9ge5 0YTdp53eyOIwBOICDzrmgLjAc+fHztXgugtoGKePzo4RKSNzl6CXcYuq/f6m f4qHoWIMZjRrKmpCwTUgwcJesrMi/Q7S+OT972DC382HodUZD6/OMBJSphzA fq93hkGPss7whMdj+dg5ty6vR9R9DXGSufSYew+gCAq8mc+wJ2hQWpCuTZBr xuAGl5dcXKdjYfnERwYvMeFURYZdEwaN5UY0ryRkgM47q9nBZNszQV3J54bx +8j43bLquSd8z3xu6lqQQMa7+NTIPeXfPDDg4HPLrAWyXhnPrRzcVg5u65wq cwUb9ZcfzedO9nBcM3+3DDAyf9caZ01brZbx+9D8fZ49HJlQ8LxGhjkfv9h4 7IzNgkMDLjz2+XmjxAB50KrpFUWiCPxb95StGjuKbEzdYT3OucwsMuwJnnZH 7pDipX87GlMWxwRlCB3bF3iEkC02ErsUY8MW3JrbKTHHkQ2CwGJwArpqJRM9 ZFgtTiIMChnsSKd9iSU5FYcCb69RY8B4dihoj5JLPUyjBytX4IUWSnl4/GPz /PDT3VDsR4XuUJ1Z4/Mrubx2arS7PQArz5WBX0pCTaJH47i35XqxzWfeSVx8 2uWUdemPca7+Ls6MMETs7xvylMQYSzzQ3yDj/ihrWA3Ws/XaArP+SbfuS/Vd xoBiJt+ME0sUlqA7IkStenDyjpH7T1E4YhToOpT/Jwi0MnpyqHL9p0j6BNzB fwbuWYAVEPV/AkM02jIvjwNY4+3gaU1ZFocHnwFJR0jA2JuEfx5uwcEUZoxC zsDjGf0t2UHq+d+Vjv0902llUW8cfx6RLxie2dET+J8baud/YqjPjvC8cnll 4ECsdnn1FwxRdtDK8cgXLTXNIL9L8+gJoxQBP/jLgJ8bdPkixLfjseKOcOtk //y681f2pPupXH7cHNPl+i+c6446argdTGllb4X9sV4pfxm+AJLpLc6R68cf 8SBy6eYxmYdBZbz31y1z44yWPqn3l88vnibcPA74ZZ0Yfs5n+zlU/fy5yd7s jl2c3/Oeze+8xUn/fRan8/9w8M+hxRbbdojlvaXP0e6oJX7qu3f23I2S335W 7IdZKDt4VPw8ZcPui4WXScyn/IfdZoLS7Ej8VT1hP9LROB9rGfNFXRXyuqS9 ySsG4fP+7vOEz+43UkehvwizQjlVgNnzHFLggn8eW/C+/mep+DyuX0JU9BK1 9vrqgxcswqm7+P8RY4nzR8ZZK6oCnGUKMtpcfCK9Ik+kt7a0u/4vs4sDvrl4 B6v75rqdieTPm5Df/5mhFGBRK/9RFtBZjW7mxDNleBTWi+UhFRU3yHvoGDLU mV763DUUPutX73wSGKE6PyrD3xZlc1Dg7/pHq3eR/12j/3UIitHMtW5s27pR 1Ppo29ZHRa1b27ZuWfXC9vWt2zc0++chNLaG0HyKwUltS9pTqK6wfX3r9o3C 9gdbty/Gv7V1+4MiCkLF4dYQWoUYtLdtj0HKIgitbeegVTwHrfrW7QvnoLUt DTF0WkzE1vH2IIpxaG8PALyfpxAaW3MyuDdPmx9v2/y4Vti6tW3rVhEBjw4P tuPCLMxcIE62nUh0MZ60Pty+9WFh68NtW58XoD5pHW3X/KiQfjD0rdsXSuL6 lu3ZAC0A0PgiAGDBPiVh67C2LYjDwjloHTa2BdBXEJ7COPgSGEXEBDtmu4Gg waPtDm3ofMb20Gd+fuh1uta4fX7VLYGhUd4wOvay7Q0z4ZuC+Xwttc6ixrQ8 mfBNCRiTR7kzz+cb+WRpGnjyIH0yT2PVgEC4a5lhSqlbu9X5rjowSdWXlHSG ux6c2B3rTCgE4tK92hs54CppT2Y2LWhPHa8cxQQzurg2S882UlKhf8x84oGZ Sdvdq+aH3vXF4EMJ4FtButjD07GrEt+M/IHzzb8SurYs+LgBlEFTq9O/2MsB wxKY5JqbLx2N2+PbkazQ6fLoyM8oAwYpuJIXyiehBkTpEGqcZbnHv5KnxOt1 bocvUWKkytine6PsJV5TY+u0cEyd4EwkaJedtpdd23giNuCkBbWRz7dJJEbq AOeD24E6rE15Ar4b3GG6EafFlP6F/C1CJ3GTmLMJME/f6JCuEyFIzUYFd9Ff XazCCI8cRka2EQNQxwtyO6+LKRnJ4zCxfb7Z60zgjjmo+dysnYm6LMVboTuh vcaD+lQEzYfrC3mr7Zlo8nvDtbwS5kwcqBLVQUu1u12O8Cg3FB3yO7fL2yCW RSUq2/uK3sRLzEYbFSCBsPhpi6bGCZPAoeKYoY8znE5USYZ4vabKFJ71ukL0 0r53e0ESYSGPWrSde1XSVARLsKQbTLDwQLU1boGmipZ+e7i+9DBXFksPjdLs CkSoODLgjPNVx7rNeE3zgoUn+n0suei3b5d0p4IxfVCIhEOKcTkSmS8q1gRq 1A04SZyRqdFQkMKl4oOMADygcw9RafBI/3hjrFJz1HIp0sWTS3kf+SzE6wIo Bcy34xgPsuh8Ed3wE03viG9dwTEjc/VTHxxF3ByEnrHgPALhIgsOsOAGHVpg BqExokNz8uztnfwYxIbgq2b9Z/ei4wuIBEz3FZ4QwkWzrpGD8i6Mky6dX5dl DeQofR++LKTMlfG6rWWqLD8um5fnc2G9RqvJjvxHWdLgfI4gGbq/yiLMAclu muYyziCgyQ5cXsHHvGSMHqDwBNddvqiumvaCG8xRxiRcJQJYdWICapYWSrlo CV0pk8aPdFXJTmcxfdK6IVsrmfy0mZ6bYV5XiboIaRM8LhtKTl4IGagrNXRm PMvpfFoybZ3LdpiDkDvuRMeqCD6ZFxX4Z2bMPYvBOgvLjxamvBOVcXIwv3Kj YE0sEHPJQVnzayqLYB7EGK8FwPSZNnKIWXvO3IG3oXJBhyIlQ7qCkNKRsYZv jYKxj/UVMdp6MXCuEc4fglrXXcICgFkhWVemeZDSvXaqco7Umsyzf9b8ggMr NJFF7fGGMz/ZXD7IqxGPEAZc63LjH2zfm8rsssxo2RxCzZIBMFKun7L8oyHm SCJ5jrPsow/D3rhLhQdZYXfYHnFhx8xf6n7oXp9TcVNSQwo/YCHMq6SralXy d70mFrHryOFoCBe9kVxsDEGJSYYt7xuUbfU943qI2CWTTaXxxQbLuURwi7mS 0+Pmj3Eb7KVa/QytXvnUaFKunHw6aGF2Wj+cun7v4ow0oT4T1YQSMJt7lx+d h6iHqWp4eAZq3LMTfPEaWkn4yMbqETuoZ4/QQ52SmPg0yjiF5vUm2NnyLO9Z vWWkeDGHNMSnHZNhGpqvGhZp2wGfZ2bIGxzc3GjcPMXrfOXlHmRzaWggVagU 6WN+YuPskJ9VN8c6M0kfKMSruUuUL12WkoAOSpOFtSfEf/EhrBL8lwby1Bte UL7Hbfb2xNdfixI3hId/wJt18fatKGkYe6Ii6vJuBt1p73pU8ucg0yL850m/ 5KhA1zvQLbyGcP+FKBSDpkKFwp74XbWMuOV2zXIXTLT7dLKy9ACYSAqRT9Xc y7/0oXcxfodv6ZeAIeobL41uut2L3EuglhvmS4Oi7g43XiJIV71xDlJ94y28 XZjAZK9g8PbAeKV9Ox6MulfdTh5S44Axz7HgAbJgxoEHpwKX0D6fDMtx4AGp ggvPvmOzA0uu3UQWHHIBLlawVfz+3SLBVYhlfYJFojm/AmRVJv/4cfR9N7Ma TPGFFXyDjr5scMbHziSS3L5eG99IAB3OT5XtfTw9OXFtPiP1iz2ZyIR9mQKU g3IV3GfWhokF3UqWgZLfTBl/xN3Ly4+bMM7xHYLDOaYbEuCIJIA5A0en4hzU ZT93mFhPwpHFFSiuKJVUZGVXbqAmQhVp3aulD9qGevfz359aJlQZFkobFtb3 iaHTkfnMi5ipDNjjMrLvFJZr3BzBAkYRnp3MEDANf6iZoc2W7wK0GJSZHTQz MSfN7yF/FGi8Ns68UeBDhik6dko2ddtfzm263xbGeCrz5fEEJHvIzQYx/d+k J0xndNHQOiajhGMIeDc4Jc4TzWxMiqQbsOly059+5pM56il/+PuqPRpbl8P2 t2KHko7pP57+K7TnkWD7SKSl7dGnTrLB5AFdXFudQf/mqjvu7rCBnkGixHiZ kK9P2uFVsngDGQh3Tg+lXSe8ft6K8MKs2KG5TYEAeFLxjeJKQouzMukIDZ2m qFVlfGQnhvfZbXgjW2PYQ7cGDw1eP2xS6b4vBxbrO8+qGRQOEOiWeMw9o1ZC R1o8h76ogBghmJy58QNfh6iOUwzXyt0h3A1mlqM1nR1wqJWTcEbExLVKvskF WbsyvxtZljBzN77WwYd+nCTFq+9IPpFEY/vwpjNH2wEAnYGf0GiBSdzZKIMl e3uxUUbJ4Kox2ZNUfGI016W0Pi6elNYzU4CvWcOd1NH7cyLpyOClgfXP7nCA B6nw/jd5q1pRbc28ksxkoeSvYSHVgshXO60rrijLS5HoflTKcq4KdQOjiSBC Y6wQkYwd8cHgMXj8g64E/qn+VY4mP2eXyn2utaKBFkAbzDd+lvmSp8zXGXak ba1PFoxz/HgsWQ9kg+IH5NIGC48yygzNElhxYFRorjhjBVSTsMZr9FtulywR zjJZxJoOD57TVfAggVDpc4xvYT9OOMobw7jMzWxo0sGTsfquTrrgGqOSE3mT CH27TR1l9fBWMToIJm9iVQfSDUohvA4eG/kEonsJGpGVkDwiAkU3q4iMDCaa jHzk092XYMajxPjECzevJiI6QZ8de/81tacVDDFWSMCrS0t2CiTlayXe32zU J6o+yeqnC9tCNrIS1cqCoo2aJF8jkZXX0ri5890V8+RnxQsq/HEeeWAyj3R8 b2Ep4csv/GTe3frzm4JXE/3q+JlXAzdR986/xoUuv0z3hljn+vbqCgPLxh01 hJF2LVBd4lf8onVZ8Nf83uyoASM0DgHQuWKWGU+bT70oeSQA/AtBSAi6HZ6l 5zsbEF5pb3M+s0FY/L0y+vcpQcRrJsgbKZpuWK/yUXEXL83H7Q7TkAIzSUqn jRkGKDTBKhdFHnydUsRJXUGEMmQDGtYracnMIU8u010WHCqM6SOjXvZh0vxo ZXaJeL2Ef3GQfOrDmVizwELbV4temfmjP0apJBntZlAvVQMR+laLvImYdmT0 MXjzTLqLbzCefIlvMq/QB5jwBitCU15HG9Bt7WV1M21ZHoe22JjhchBmcy/R x1EwDm550/UbhRJfZILxJLy1xo6ya02AHej8Po7ILavzgOzp8N0WT+jG3/VC M5J/vmEi9fNtuO7ZloA1CQcJAONda1yv5qcIIxdZnT++oCkir0J7o3f1bmM3 quDtH+SILLzYqepTlksjypJfLjRCC+mNx0BPwTaXnAyiZVrBa570NSCy+Y68 98OiGmgh+x8WyOz9LDahWhu3q53Wy3wliTWb4u/sHpFTCsHvAK9bzOundX3c Rd22YMtz//RFL32HP51Wz/WVAQFZQF1ulEbr0wZFGjPfR7VXl60Z3eNX3Yqu XMtdtsbN8esxAXSwAYbISzXZydccKG6tPl8naZFd0mYgY97Olt3SLwHgVylk Y/2ZCfUMbqqFsk9+yum0rgTPpW/fnaKMxs8jlPVnmcBB4J0KvLBD3lihWTF3 DCytH0qxID9DogqNjz2qIr25yrykcUAmMuJWim+hRd5MQJFW5z+NNzK4Sl9O LTMjmoYDdyhvAkH1q2h4/cFWC11fFEKslF8sJBSX88f4p8bPClG8ZEvqMa2O jd7wi7MRXlKl1ygG/QL1OUjjI2NkT61cI7FxB4/+2fImJKb8aJiBVgftNGR1 jzpKfzk/JRS1IN/wW5lyniVnqq+hgerBP4ZfFdGnJ1z1VTS6T5/G9w+BZveG 7bQJjL53piE+A5DuG9Mn9XN2Itk1UmqGeEeJq+6VWhQJVYEXsG5uTePnWgBl +rS0Ih2drKe7MVAf60sX1e1EMmCNoSJ5QJKszLXzACKDvH80EunLXTIYhIFW eoY28peswW+75Uoa+kmejClnsOS5EvkGHms+lL/pUPOxfOiuE6w7kY8XLH/P 6tlB03y6KMt01gOftD5XpzgLxD+6VKf1Q7bvaXcQPyLpknGeBek2Njd2QHze nx5n+bNjdY0LXzmH3yaKSSR7EpBWBdweDbrT4ze59salU3QBxyOZfdVM1uAn hIwmeMVf9n0j/uqN5BWSQdIRUSShdFlKkBXi086uPt0HHJSFB8voDm3OYal+ 8PrdP/f267U/6JMpn8Qu3ceGXAPtdXCyLKNe5kxh683GqbyLfbqLkWUYDW/o mKxUFhS3MfojXAhb2aRu8uHOztNuFNd91DheBfeIY22DL5ELC8fItU/bH2Rc Sz0/aUrHMEGOSVQP6puja0IL8TmC1OqfJ0ibv0MdJrjaZbPnOjLRqsCCois3 AL/PNsqwkzOpGkCLy8tnkcutSePmef5SVqnI7C7TropQ95DSWero1zIrBPpg N9+shdYsVyvbFlOxZFcPYZYdni5LBR4Z/H3mfQwB/vsWxmDom1fbdUFqgb5/ +2SQKHk24OuPZj4DnRuCJqZbT/G3MpQL+6XvbBaBelg+D4mfeIurmF78Cd/c npjA78htvG2a959926Cs8QX6TXcTHIOy2GqWsiDkXwJOohatS1/Y+zYtTLpq o6ik1gBXYxOaJrV66CuksXiN/27Cyrr/s+CMsTs+OCdbDmaaLpaWjNts2SRd TkGKcbwhv07+bfvnohavJeZ3UMFwt0OFPgC6xsiN+0UiwQudxH921cpyD9yO X4HOv0oxtpg+t8Kyjzhvh7btLD0Lcf8wuPpMk/+VXRBk3mD2Sn1EDw2P/9Pe t3e1cWT7/q1ZZ75Dmyyw2khYPOwQMM4lGGxuzGMBTuZxfbSE1IKOhaRRSwZP zPnsd7+qald3tSQcTyb33JM1k6Duqup67trP30Zkr1y+PZYbDWwmijucecak GmJZhY2PkaASoXharyMmpiR6tuhwbFCy8Km+/QbVzu+ODy/Oo01vmqWHf7fv iZeJ6nB3Tfn/vb9WSv7ONTS9nVwzGuD8t7RjZePf0oiTjR/cCq2dmXgkCpwE DjUw3i9JG2MBpnbJfmiSgaXsjAkrLNoJwuwQdFlScjCapUGLk40n2VgCmCx6 344Gg3FTI56wmk6MC6NUMlAxXnBd5bcW7G8UCRF9dn3NIIAXEiLlYNA1aHke aJxMrBhRVsLkEABxs4ud0OalvZPjg8PXzdOjcK1skiEkXVTCHE3EDJTEf/7T r2FlM5beiYYEMMfT28RwOFb9UTK2D9Uf98+OOUFQtOB/uLqYxf+nDzwk1UdR iV3cK2k3qmLTS0v8CkXfmC1IYXsC8BXoYVryMraNkOUIW0frBzJT0mqFk9PL fGMiwPZ1VUgXvNXsmn16j0Yr+L/k+cSluQ/Ps+R1Dk7zV59b/tgfaGodb+wm 1Jvu1tif7rKJZSMf6gztvnakgQuLl3pEJyGqvoRjE9cQXLVPCcEoa7sYhHOH scmQRgWMoyrecrJIikEwyu5R2/yFOO36HMKklRxO0n+h5pEEZ6RkWW8wtuCo RMUujRRNU/qIkJrgWYzzo+YFfiKZq0psAzuFApUwwQ4vJBjSPlg2RWR1SJNo qAw3SF/MJZ03NKoAlV+z2O9xtLNDtibqY4WwdvuTRNoke2Qi+XQx4WQHveVp qIevXKyIpB5O+ghe34keE3K7UX1gv9L+bdUGcjTu9jZhII9wTM87z8yYKr76 LSrvvexG1VfafV6HGUHYYLkbbZcFZUUwpF/Xat/eE9L/KPnoFPaOdkO3L/1u r8UvXmzG8KPwZj22J6nq2liKUPhtmNHiVfA1Rztqu+0qAjMuZU2Jjbl53yQn RHbOZGk5hyZWC2x/NzT8IB5OTSFUweVlKmkM/w8c4701hnH+WE6F8PC7mnrq 3dIy56GZKr/NRcEcRcX0IaGRqyHISS+WiuoFemOoZIEWPowKSlC7kwzTcRNd +sM8gl18yjeB4HXB4PknSX8sNNQSTV6hW1Q3TxjenBlKZ6OuVGQJitnVq3R1 ymrQ5q3vH56wGwluGvUilzCdWRtsES5KyoHAsrp5gzPFd66nCkILtncNf4Xt re44mpF7lUCYQ2wmwxy0oEPjZTe0S47to8UgNXp0JpcDLnE24axyuXzAWJgs eZdJxHvXQo63GPYV3jEvTeP18QuD22Saks3O4Rcr2zwWqciebHvG2YK2nX3b /44eY+9rLgsULA9br2w0AoiJfeMkRt1D8TkZbpeyZ/zGzgrxZEmnKQDBlkVI xh9bve3CCBz04xM5Ox/bKXRrKY/haGblvXMToE81xbhgmITc9+V0MKcYMat4 eHxwEi0sZsAYmlLbJYXO50t8DUymasJ9XXJ9CiFz/RWWNvpeThmyPvxiS/Ie ce68CrO/uezgaH4bdKuw6DF9leeWjv/xydH+0baZCyQPgfHvn51RwpqFCdMa xPgWp4/IfKSGPlIj9GW2Y7saQMHBZGxOLVxvJn3cyc/H+2eGd80x5ma2t9gW tpjh/4B/aNwt9jDLGeHP8D9ugvRDnqgFOP8LOD14JnUl2C/M26tnck9xZ4h1 9zh0+oGq86YwjI6dNUQPH47+YZ9wUmD49zY1iERQDM9IXLEHniUaSxgjOnI+ fgngYwxWSwwjW4VRNUwtsa6aVh2qaYVd1ZgYwOuCg4mkomfdgHF30coAd5vT lBpGVS5zaF05TcEHSmR/uFACL3iGPbYD1qvOGNtIxYGI9wZXQCO8TsjHkf5g R81lhKFiqIb/FN20Rh+SDqW+obbwEsC8N2wfHnCEGXn4ML+JToJX/QF51zDX wPElBs3bOBUInrb9jt3/OGmTTPHaJTKAXp2a2rjEmjoBAKc17/EgREG6K/e9 S3VAqTi6MEq8tVrkaEY8AIuV+FmziXJgQPxFvHZQEymOWjzNSqYpbEWLRmpY 6ckm3V3onNRvf9o2z/rJrXmGAZ8bm9KoL9XgAplSNPcuVwkj5Eo+GU5LzOXI REFNwdGlBMQuSxFpQiXHHudWQGrFmm6MjMPUBBPWf3KmtnZ7gilUuDFybqAc H2S0T+6GlHesMxlxjgOT0GNFEo+hni25o9Q1LozRNEZOWi7Trcw17p8mO701 0RdfOChcm7e7F/vHe39tXhweIVTTkppVkUIMX2dm4oWeZbMeJdT0hB2ncCBq 0heEz16QIVf3Dt5exDZZOrA1i3CcMCSeZxMGvdhZQSWJVFQ0WG+DmtezbVMY CpCVa74ZKDZxr1Qd4tBBXhzEcleX8PjgTyodemndPmJDIodOY2RJ4xHw15g4 YJSIey35AI8jFKfG0epzCqKIfP9abEscY7E5GCjfwsikoJjeH8s45TYO+OPG yIkqZ1UrAoH0GKg1drW0i6uTnKIl1Sl02Zx6+3sDMGw43eGKFpkbsv5ybMca 6lT0JPYaXPYGZidf+QvjDeb3NjzqkrmyawdDuU0ESoJEAMnuhuqHMRJ6Yh7T flFHiOzAE9KPo3sgpf/TdySQVX0TEZcktBPIIzYgC+wIqHiKEIgFatdGPoRF wV9R1IywTZMbFrDshYf/jgzwW4LJlCUhiTMGEepERDEzmUkgxXQJbr1RB624 QiGlsbeHb0+ix8TB7TzG18S20pXWH/TrNPgW2Zy4grjqSUy/31HD5JEcSRoA w3q8cEai2GvA2IhMSeLUjfLClqqvuj3ne4DCy2/N5pXiJLQ7BqVQvmoKLkXf kuptLY6AowLBD3iqyL5bfebpZXPNwIX9rbqtjb+i4d3pqTJd4Wdty+Tl7zFx VNy5eHqlV597Re+t1BKcXNFNwmttOLOzG720onWhg6t2vM7IVVbRFPCqKbNW ST1XwhdyqE3xrBUaoPpmCIXsFbtrJLqAHHlFOcp+2cb9PBSQUXHwNw0iQVaZ S67jU3gehfDII6KMIDucJcQq3aJpdVm/59XWYfLLFB/o6++KxaSPrL0mYTxK SV29gX8tLxsWAMfEftr0kzB+qLVllM+tSlJjDsgFSx6/yPmSBtddeWsR6taN yoOCXozfIwVuWB6H+sa8BIbKbgtf8ZI6S3/X65ZTmVDO+yo0b1gE0uB6elOv k3SENkkT71RFMFOYGktxCBVRX6RoWg23hjgJvuWkOKurmzyrWMqqQP5zJ7Kt s27B6kaqrpD6++XLaDPmnne7hmDZt484ODeW+OXXo9GoRiIL5dVL+y4boWQh JrHYTDcloMVeY3fXVsWKEOjq8jJ19gt6ey89DnR5aSl6FBQQBGI0jp0uoboQ PXny5PD4p923h6+ivTf7ez+evzuKFjdWNu7gBcj1VstEy5JfjXW3FNUq+hWL FY1M3yi882Jfj4FMVM2oYcFXG++D7T137ZkOLrYX11bW7qAnKZDdx1uPgfA+ jh6LOdB8CL5DDTpisjbrA3D+Lqv5RtQ5TGNtf242M7jK282mU4wsEA5EtJiR dbKJCvJmOh60qkb9QIodE/KXq9RxJk0saQKzkh7ma8SksRTcRglJPxLgB9RS DIkx45JTAay4/Wb0AsYaff5sm8ZDfnzWhOrn3sqTIPLz7tkxMGsRbYPo59YI cVK2pIeU864vKarE5PAIC1pBA7hZPQIXvqnWYYPZ/nEyLFiPVJA/7+3VhE0x 20HtU2RNV2QEWWys3eH/0VJFP1YWSZykN506yVDQwQX6hf+iGZdN+Pz9Eh4l 9fvly82aU7CubryvSQPY9Vpkdu8GFHwWEztii2+8h9/rq+r3y5ffGYg7p6uB ZQqqkBiwWMiv7+PX7QMj1zRRzxJBdRmIoOIQLSY/tonsGlOs9c39ZNsSUShs NVhzqn9XwxrAUX1l47pI54bhUENVlNCK5eIISTXc9CO/JbkylGAjAY58c4Rl 53w0mQUlWtxc2ezd1V8uDvVO1TKx7m6uJ/TVHBkhb/PipeWDsBDKCF76qw2P UfWosIefzaPuR5/Zkj2rlsBXe7UwHndbqNltte+xEbnueXq9fUzaiqlcJyMd 2Jdn0Ug9o2RpCpiJ7AW2uv5+W71ZU2+evXfqThdyowo8f+/0/a5px0n8Guba V7fDG4IAvKSUZN01Om8etPiFcD54lZwqAvkMaDv0j137F55trQOdWFjfWsX/ rPJ/1reeLdxve6eL/Z5QYZLbMmL2xr7nApSKW8jGKClTeagi3/zCX3GYxWjw Ieljlki8dp/a41n8JDGixGYw9A92oud3guMqXB+86+VlJCxMwFMnogS4lH5S WrcOEKiUQj4CZs9axhZUfAe8/uaOlFZStVY+OXHJqkdwTf0IN3D9FmXAs90j NEuc3W1hkDwuaQ1+L2bObXJFE4RNRPCxWD3ch9i+dfg88gbNFxjCTfYL1DW5 ltwWcjA7Uuu9LaSAc1SLLmbxKTXsGiUsHlPypY7kwXovDF+gZvQltGA1sDZg RbfzfsX4T/Fm8UIxYWPosj458gvuuCgjJ0wqW4mTud344HFxCnwJukyu1+Yv f/k5ssiW5qS7Zj44Dgi4AnQh85a++KGamrHiWz1vzPF0m5S/dCfQlLPdlxT3 JtPzZVMld2x4jOPmcu/Y9mTl5eH1p5pEI9wRx+uTJWbEKjdp2hyCZHaDJALt IukoMU4qU9/ZsApmEwjvadsKvpjmlrhs/OMFQt0Au2M68wI2BP7wZWH8GrMv 1O07Z4rgZXoSHYhkyymP/oLqwt5g8AFFYMp9CL3grPHONpSiRvJqkvU+mUZG CYfIcqpyplWnb/6KjbDrbQftalL6qdPo44CcNIsdJIsJ8zBWs0DDxed+OXxc l2PgdAfqLb9xU4CGSX+GsaydY+qSKo2zqyaQpb9uF68IkeJJjY7hprIIIGvy l++2ywwSEQWk6gPEHjo4U6LCWezUFpwqe8Gwn4sbQsm5065jngZB9XdJ4Dd9 jcG0/eesDOFi9bolWPYMOMVjSOqB8YIwIz9Q/jke5Ccg4xl4ZM3nblob7912 0J46TPVcnHBxYVUTCK3m1jfH7dilxOgAtJva8GI6FAevlK46s6qewveXdpAt baztNmQtVFhToD/8WzUQNvJ4Uo2vSURxtMD4LSlYFZZDvXALp543E1GsbXFa 8syhF6FfyiPCchPIWsrYZCYnvYE+YfjwRZsSPvMujKrjcUH0o5s7QWhZurNv rwe9pE7o0wtRvF3WtxFpnQsNoSZ5Lax0Frceti8wT8YWdFrre23OIH+TeoZe gl3gcA1KiaD0iqcPm2wmI6MYpFsFHZydcxn+0san4gTL7FNdDMNWAWLQTChu bJu0KH3lxv/m8PUbkAfFer5L6FYY95yOBdhWoHNQ5cAh3Ong5maiHFKhZwYP CASh4/2Lw4PmQfP8tVJDCNWpeqEmThG+g2aM2O7WvFTOuXcQD9LkC5rS0Gps D2tp3w5Pm3vn746cdlMRjZLJLAT05dgln0tvlPJJKC1n7dYYFvDpFSOcLxIq VGclun56a3V8mXnsTMm+Nw/Ogj/AJTX3fCz6C1sLnTRbmFVe5sOvpBhC9qlC fPodL4TOuvPYsDRXwD6yhToDDitzZeinfU9eFgYDmyFVdnQAmy3owslcU+6Z LXaLwNadwRW/wANvngAr8uZvcfSU4nw9m68RmZr9ZIyuYSZ8gD1XuJC5JT2P easqQbaf8sLjg7xx26BCPNjGXal8iYlbG91Dz4zNW8OHaP4/52gjo9ZO00H/ HZqxD4TvZCdQnwTrL+d70XZBWsacD2fiaCyKWZ52bAQWd0t5txoPSB0fg44s f/4TIgs0gRduthleDTs6LQIZIxedI3HKtx+pceCNqIade5vL9RBSa69xYITl r0nFRhabYDOcCgIVVh7aON/LMlRPXXTyY409czppp/94THQZJu3VACk2mfoQ fwZkP3IUQr8f2ISCI5PvK4W5f5XeGlmiVGFRQgoNWnfjbrGxcQdCI4gUICWC +NDOlBON0W6zypAWJYU9v9pw2k2ZJ2aOjN2sYYhYYf+Fvg4zSrB3smuSR5FF /F68K2jadV9mzFcc3KUzw/sL8UHGq8B+ZnuaB/PcAVYlejR1FgxFCLOZsU66 N06AbuMsoo0u6XxvAJTEFWSIvh50MSS2RfSPJ2S31wPCjPmYKOixHxgShJLR tHrpP5FnFC3GWDBmPhpXEZQuC6Zs1gHOo3D7QtUH7qTpyo+Q3kPtoYAKpKg7 +mqakMiM0emD1OHX2qOQwOaGPCGRBhUfhqclmJoaLb42qvEgdQ/l1OY6bRUp RUEOkaRSOKikYVCJfHRMN7Fm2CViUi1Gj5XGQl+zpbhDYi1+FJUoft1q6Mbe ryB8jlDs/GfKqhBy6pTZLR/plpg5hTSqKS79lt0mQWZ3ukZs+vJLaeme7thD +iV8NEYFMbwGOebRn84VjrD3jMfpTnT27ri5e1EtaxtZgFxdSamZQ78XN2tX zNrzdjyAEJb/2FmDEx4Jgh+QkmCvAyPSECK2c/rh7D56pVVXPfDBMrFktWB1 FtsBLughU1iN27SYlZGsWUtqHJc8g5F2ZfJoMmcRMKYH3aIBvd8OD+h52YDc 3RpvRSYNBl7jSJqNnYQpvzJ70GeA7+vZvky/KX6rmhi1Z8BHr9bMX8/CimLj yfXD0fkZyiFXICyT8gGhLoEdH0UIq65hslgzhyVnKL1WnSoavz+j9DNninA1 jL4zwmFH/nPkL5kwOpC+gQbbI19KhtkXjEXREssw7I3lPkgKy1UTtYpNWjJe P3hllW/Ym1ylxp5oOUXhSVUvoBIDH+PFhSnr37zSVryQ0ZMmJQ8JSf6T/gc3 TC8fYs9TXAXM0zeLHafaJRueN1lWN/dJXisLH4MqRsjb1qzH+mImwCcr4UtE r7fSBfvb1Lyx1uKaGEvzpmNYg4Iq9ftoAX+Tvu661esuxEGrrYu3TZjV06At l6lh9EhMq8z7dfg4upiixyi8JrcSUYtgRBIsB6ar+57stKqMNK/bESbYLb48 KOyKmJrDjPfYHk+cR1VMhpd4Ctkusp6KcCtalyNt5H6hFlrLLXmjX0hmJgIg 2eSoaGmhM1fIrImXNe4zpxgqpkucOegCMZxjFtDXCo6NRg/k01F2ndnQ3Lzd WwAT4/COzHvVoiyJvCDqeiVN1xplmiS+AXN7uXiIr+0exw2IXL+GlXfWYuuO 0ih60zn1vvO4yTk+idfSfx93m9JLWwAZY7ltOVBaQgP6fVxuJ86tRD8nRisO 92QP7mE8F9Gzxk22siKImHwETKrJ8MYvdudRkU8o7H9quVqyTym9ksO//GzB LGNDysqFGIsf6s2ladk/ACBj4a4e2GNg8tLV6BAwU0IgJPTu8hO5RMP9o6m2 zstYMkNu9M+Dp2S14Y6J59DIZ8KbJdEG+c/WjL/ncXK7lUu3d5tw9Bxlh+Eh 4dsfWp3z81eMg2AhVPPL5HXGfWVlBadnwJ9xINXwJYlbxOcJeoQxccnGaTvT +Zn4NEu+xKD75nZxkawjEAYBfisBLJRucJIlhQF8m6MPReMZbM2oJMGQYT+J sEzE0kbHWsD8rTsEhpYapU4iyVw4R6TtbotMgIObm6TfsYTT3i46zehyVF19 tv785cu1uGQnMV8vySddEZuBSBXKachVaZudKNae36XGOQZgD+ZhcooJM1Nj f6bGbqaClPG8+cPJydH+2e7x69gEMVxWT3+8aP7w7qB5/jdyorUdz6Vgircp HSN0zCDREPL+iov63bPb/eyuZvNoWDT23DH0Is3oQCqFIAaCIrYEAn6kfW3d rKHCV/PaJp1S+n7F+mI0tkNfvCh80TbiMluQx3tDrb6aEZXLyi6mXO1idiZQ t2QrGhL4FZzOT5LoNbqsE6YFJTkAVpmTJqhIVo26aAwgjuJxfrHQNsVFuUDb hngT8/E3iRiePi3nv1wG3c+wxb/tdt03TR7jsg/uO8JjBq5pzcWc1cV270iA pL64NQzYpYBRUyxy0leurB7GPKxYjnO0CWk/c8LYzzZN7GeXl5Yut5KTCBy4 lzN2K1KJa+VeLKF0UFVllt1SuW11PeUp8H1k0tVuidTHgokFzOfxmTTLnyOb gfez7tWsTjVCfXHpc9WkUbt2BPQ9mEMs/jma3nkorMb+2ZtDu7sK61dCfsm3 gHHJEKSPsqp8RGQUZhjwYhJVviQAg54jK623oklYXT5p9gox4+TIgWCPbPfV 4kzh03KcssM5P9072jvchVENYGYodjXtpAO8sE0MEvr34NXD2cZyjvvLot1i 8Dr2L6B0KNZ2msP/s4ajabDNM8Bt5jYNGRuQjzkzl0XoK5iEymxC78Tvqj+4 rVvYt+wa5p6Cc6IUN6mFmFGxOlVl6GcYChD6qk76KztvrO5YCoARw0lZyuMd AznebZ6/gZ7kLSBxHBDlrTlyj0QKnCiE2x/BkPPxSb60GoSR+a2MW0AUXyv1 OcdeRwdpjzy5xMHpjHmFlSJQ5Dx8gxh8AvDW2yEugTJJ7UTt4aQ5HjR7yfpa 9aFR8tV0eTWOt2fwII4nMtZhpYMqVOQUcrmOOd4MyJPNeShfRpTaHR+nVlXw u6e5GzsxuG7UiIZLkcBPPowgtyBiBCa9lF5DedIpCNqMZEJpjT5ElE4SF5XS k4P4RlwcU5gVVb0pO7VK+ONrbMbdReCH6PAU2RgDBnGJzpstcoQLT5nYt/WE IYG4aQ2baGnqadpA/R4D1a9Fbo4YIQMuMJjWk6NX+z8d7u3HzuzFQI8oYXtI C8oJ+peiPW4fGGzZ5v0BSM43gxHjAI14k4f2OG/yX3iT/wKbPIX/uJ2tsSRk IX95b1+iiq5JrjO0AwrltpXiVb+wfK4J8L2fEvVlKAb79ggsE9fARESjFgPE jZy47i9WfXWeg6c0BkWwVgPnajTqni+V7ho6/Ji/FaGWdsQjaC5TlyWz+vqM 2etoS+CNSEFeoLfOA2kO36MpSQXQuDcLmTd0oceC0PsV7vNZziR4Dgi2CvPx YJbhxpM3f7NQeB/MNnP5ZkzUgI4gAAGzydqgUu3u2nSV9lFpjqDoGgML+4hh 9eXWyeB3Az6DO3ov+K/KwmUIO8dey3BhfBT3AaDrwFiTcyJrvLwdzRypnbe8 xtUwoxSSu74d0Dfp1Sjqa7USL2O1kKd45NUgoVYnQoHLXT8TI1vo6cHdljUM 6p4sRV6GbEPnBg4NZF6fscLuWMwo67HN0A0bwrNpzb8lat5ejrX7yYMi8Ap9 M5bNQPd+W+8snoNbiKPDwy21LqhBpiX5dWZ0S8gePGuF1masUC54BScjbIfx HUXywSq5UBs24jY23F3qjOhzmq7FxdHzRXAhPNROMYCHAa0tWo0puWTs0NYv vZK3UnsGZ3N5BwJM8KO5UJOQ9dmBvlTKfSrP85Zegm8hrTOuy4J1KV5A23KZ NRmRzfJLVTAU+5bVnFnX1fIWwsyQ5WVm23jpn0JUb8Ugt1SqM628sjqzLb1c 7uuae3nRy2y+FbcnwlQwfMzMKksHJSubadhFSWlBHNrdcpYr1LlccJ5kTJfc FQMBtfqKXQSi6BRzt65uNtxC3BfYzHuPJEXGKWzLfPyEwhxUhJQ4UJFgAT0Q x/1sigjKcyHIAbOpruiQKP2TT3Z/A9F1JNEo/qNHwBPljaaeEy052Q3kSD/U NfA+EofE6As9Eq31IGjzlERoohN4PQAygaCTA7N+ZjmiaC43wehLL0vH5iEv jqEKOedC5dICS0xOvl9ykcZFv9Iv5T38LiMz2gembTz6hN3WUZnF/j64wxXN j09zd3RSHJuIc4zYv84uHMRcmA/h4IEugPM65/EMTLP0A5m21nkg1efjwZAM 9TwXYQvig/b37QjzmGJkwSY5Hbo+5h0OHX0+OPzL0T50RlNoIM1LSCQRlBfp 8fMGXs0Yhn3XXhl2ujYAUS2BEyToA6KXzcsbf/5TQTzZKpPWiK4/SFjrpv00 uy7IaQ/zIb0ZFH1ra8b1156L2FPSm1S9zvnloPUhQcNBiXbfBo/khfn505d9 LXX7DPG8LI7FGN9o6juo2qlRInrhuBG0K8x++7TI81kgcyLefaZgQSBVwS0F cbTQTfhQ2rrqD8iiuYUTSH6JxnGcwm9uWjmHtfxXtZtDqauWq6nojjPzG2vU eS+9uhb8qsveoDUW2+Qkw807ShPgUD5p00F4dtjbcFO4fPijVO104SyklIqG vEdgtTAV9qDXSzPCTaozWhb7bwo2PrDlaN2SLCLfB5SOhSQ2frCWMZdNMz4c WovGEBYJeTPxiO0kPZTh4InXNzRNXKK2g9m4FhRFLo9PRaiPRgWddpIWU7MV +gZ62dBSAOt2i6BrzBxiJvjkVoyEhhX5NYdK8QMBi6rs2wZ7Im+SiToDREEH Fgdz9Rng42rsYU3kAL8M+HylgljmPQKVy4BgVulNCCzA9/8QDXwggaWnx6wZ 72AFTlHIoFlSw3UMLgtE79V9u58eoUysqU5u6QL/5/EFVbb7bAWaoMwvGSNE zIxYL7lJqPGxyawnTiewH2keggYla5llB5q670DzEoFrUbb3aIH2/FCYF9b1 Z+wZj6pei4ue4wmaj6ZE5RoOtdTnZEb3fS+XWOU4wwQ3OSvxv8ZNyN6h2PV3 nBqx5BrVjLW3MzqjAWLBCXhIfgC2/7hdmTQfZtmEll98S+ZzRrHR4UhnBUhv J/ol7XbThM7xV/CK08l03jCZmiAC6c2g76hQJEkXDAh+ioQnSxB+Gm78EQwE E1RCB26ARbqB7UPNcaacNvw7vRm22uMVP0GOZ1ygszY1olnk1t+JL2H92IAo CTGtcKhqzL82yR9azAaWwBLem+NQdkzauTktBXlTDs9HXJNh23hdzdwoSd4R RSscXShfml8rJuIGXXsXYM9h8wuSysJcRbr3YdbAXHqnyn8MBl0XXpLciMm/ FO6HhNwe68ypTzPF0xnGyBb3/UfMeZBKKk/kQyGcBY6Es3aRl580aY8G8Y6l 0rbtg7oM1bxQv9aCaReoD6fA0Mi0iBbrJs1ukFGBo4OZgqyqcuEhOYNENRW6 25Qp2PWT4E43YoYis1Srm3YH3qUWqLW+ma9FSX6Sjl+RSG7DI7N6iwTabdhd OL77363LS1gJxN+6e9fvII3p25hW78yt5i0XuWYbm2GMFYRI3zt5+/bw/PDk uHm2jzjc2AFMgtOCg2GZ0ylfrfinHRXjm9tyjrCcxf3jtDUu4ArTnwL73dHp bOxdYvWXR7kjwr75BZ/DylzXRI4zsjMkzmM899T1u4ighRh+F10hrD3cJlG3 OyvnoiaNlV2W+W9b3z278vs3w/GnghtoRecGQ1e9V2nn4ro1LlNQbExVwL3D TOUUUU8bOMRhecnMDdqHHTT0883u3o9bkbjxa6/1Vsagyf61qHyM6XbkBDOc WQZEgWtgsGlxb0d45zil8SNFFNW4icogkyduHKUykJ7gMhiywKTIWIyaj223 SsuHshDHOJQMN+gi4tgKYwYxTu3OMbSaD15AHZ4bRdneLvibIuiYrZb//LdS Sc2oEK370C5lp0qDxX7GModg9xW5H+WuP6+b6EO8Mgu9sz6vRiUPojWRcVQ9 hCNr7euQzfzg8ODEhFSUXKH4SXN9YnGl5VCX6LTQM9sDpwvstIaUthb2HKXY YVi9DK6Np+hcR/QUrTajBAPMPW5hmtDFmRwmmeZbgioSm+mAXVbxH0oVI17C dMdZgkvFNnQx4LyvsIdesTxpikt4ApxUbAQdE2Ve8U+jvUL1UJ6zc0OKVQqt wz77rKFJvIaailthsTHTC0KjGje0UBbuKRLwAEQgvtI+MxDwdigftW90g7Og wxNbGSbTzhjGhzytRNJNMzKArkSHwBRCF/F+vLQUU13sbtvaMOPSm7lYft2h Os8V0embaGELdtMe4fVx+BfQ71EiModyDgzEQFTmi0g48275XFBC4aK1p36+ 215NkczBLG3HZ8faKBo1i8ew0OplmqFpgnCZy7eDn6sGPGRr0b9dPY3izgSx zHmvdAYTmCKEao6ukxbwrisrzqG/VyU/zl6igzcv/kJk19Pou8lz+Q3NblTw kZhrdQTMquxMSUVHpfzwJqSDPfI9tD2AL6/TBbvuzEwqmEgScE13SOUEvjQc 44l6cSJ+qCI+qevl2yYjaO5KYk3ZVv5suGJvk76Ds+ROKe9fF21KARy3/bJd CTNX2HdxdIs4j5dIHHtolReqZGJB7NUu86cVPRTSuEK3kZvniHJhJ5QBp6X2 gArgy9TFI0vuzWJuYV6+NMkQPP9YKuaOm3eBsyorJk+P9eeG3JVFWThyPUo6 6KncH3+fk0XcxWI4nUnmEiLS5Y/eByMTVAeiUx2Vg53c7rN9GAxzB99xgRcP C22895EayzRgfD5dZJ+T9+E/bRNqY9FJrPC4bZ6g5/76mlyy4ptQr6eoaags LVWqs3UisQLHDAm+e4a33IU6dcdtc15xkhVGKIVZd5QpvmmlBksYt7AW2kLm OTpN03IE+o0uaTO0B2X6g9nqg1INgXR2LuQG57gz1+0lhVlz4SM3OOUF3tWn g6GNAc9tJCtCKOfuwsUWAqn9I95tnPGl155gvk0OPkIPeNhKnDN9iAFrCPVr bYaYJBx/i8OMqPh9A4Jy6vazQw5HQOdM9SVllfi7shcgZq7f3ntPy+qZscoA lEoNMsGVia2WLdDY+lRdwwV7x8CdkCWIZC5kkdSODCHvJU4tQuTxoAOO1yXG n52crebXkm434un6ABjBI+zkGXYfuTlUbXfZRcmMBvepmKiK3Y+V4SNE9uV8 5JL+uTBm2gcu2MfPDSpvTUBGgxGeo1cnzb/tn53snZz+1YjJeKOmwLxMbtBJ 4NGOTTLWfPOzFz2tm7X0XEd62MsZwy+RDX83RBVewgYja8b8zc3BA2CPLvZO 965N3jOTih2ZsuwaAX+IB4CJHjW7I7R9OgtU7tv4+u+NL4g0IpbEeXJK3FSY 0fOYtNDXg6Fhagb8wDDNfCg8zt9neHX7uvmvGKzffFySu7F0of14QSiFT5vj 6An+F6lmoSZV46yEZfuTOp4ur5pplGmZZzL1DDJO/SDtPImHrSu+RJIsq2Lz 6DR8BYzpcuR+NQfdLtzUhTa4BOdED86+5aNSVISWzlXd+hJPHXNwuVwfPueD Fp3Dwm9q12NiTZK+QINftrOjWac18KWHHdKvQjZVxj6VmHv0D/I4sYm7a5Fz 8MBYPcQdMklHyThPkIBiyhROpJQ/REHxHCuJy5ZhOWDfhKP7fM8IZk+mekAo 56SpDhh8iks8MOb4zhRcD1odunA7TeY4mK12Fy2zD77nSpk/ieYoMKuM9Dx/ tzuvDM+SZaUvnOt0MMmU2lybfvEC7yTdFJg45PttoSawYOkV2UANUtWrySU2 tBIddjEGgJg2P0B9oUsKhIWo/andI3kes2hKiuvbBP2hyHUTzRukMb0l79P2 gKu/Om7unRydvt2/2F+Jjgffq1TXescY2cTfNP9Vvs/pvM/pWkIHYtI3R8L4 OgVPxXQfkLwoIhgObrYMtCv7i8F5GnSN1nN8TWl9ySPPgFz1s2gy/POfgFRz /ISI1VzUAreww4hxBsEyh+dn9txKNpEsoejo9nU6zMj5IwplVEFr19JSFEpo Aq/EZyToxeqcyYiRILgDfI1SVDPtWAELttkoucqiJ/jvWeGqXPerx6YanxKn f8BfOPNNghHaQe2KOhf4Rlx8piGlW78/u3PkDg3HTSoD4hQBSmfILUx1vKJ8 UjbYdOHZKshMr+1r4qhp/dIojlR2ZTO5S8fbBkXVfmVLYV+gwWNC8BhpZhAz wiY8Y0ueUGoU7Tmtc9LRA+O/U7CZW+xXToSICUww5hEWMiEil6FdoHo9GA97 k6vYaNSKQyqb3Y1pgMUKbyMHkAqbq9/+hBCQ6B5eGiZs4pV9VRWBM8vqmNCd QNeeTY8TOdSm4N5gMKyVBkCGOuVkbOWdYF2UtMXJuSAUangoR6UBL1MGUrn4 C+szL4HNuG3ZuK0V52vDBHSUPBbNZ6rUoQwtRiF9QUVozpCjuluqlpri/ngf mAFRYevR56y8Rr+u3blXlX3MAO6tNhqNoGJeK8MQ4YgA/NBp2cUuIpM66QfZ VN/EUIuq7kHOLhHgXrct0rpTiVOUh2vEGNQoZ4tzmpFjzVoLT5c6hw7dOmAT B+FFrhwiytvjMaFmfkB3DkT8IVq0EiGHgh3gdbuliGByDI06E3SHdE2m/Qw6 lrZTpB60k3Br1eRydXBPhCtEtots0m7jl+hahlvbtVW0dth3JqCx1BBXccr/ yFP8k9b/ARr/L1bsT7cWFGIwkb/E/FF88mBeis6pGVp3Bv2252Roz0tVw4Y5 LC0NLBYAE/PSvxRuBg1EowkXO22ysKYD790xrtfdjf9iav5A0TAOBtHNpH3N LFraD9wQOpTQ0uEy8kv8tfg1aeAyN5kuucRAJ3u09yZcrMqEUgkCI1f/y6uy FBV8ZWwYcekuyVHSaq7Fxh0iApbr+k2cabWcD8LSXiiJJv8S8kVHEQgG2+4D sxQO5xL2HJ2cnrwxUEduv9zb1HQf+oNbuH2umJABs+MhgnrocLArFUxcqcf6 7HGbICc8HAp1Tl3G8ZdxL8j3eJupNhdStdufmoHa8oUkn7+9n0MGscr+ucWQ 1XIxZDUohoQCYf7bSiJTBRHS3WBE5RN09MmS5AbRUYcJ5XMYiusELidFNxEn n7AvqcRLtTk+FL1pW1etFDOoF0IVuX28EEOGHHr79PeQicImJeVT8PvLR/3p MtKOwv94iFjzoPDg0GlYjbciFxFBXbCMdkhmCvkwfx3x7o8khv2hhDAHiGqh jPy7xxUovW4fNo7ARgFqYh+O7pwspp/OkIsUlKsZiAdgoTCotQbUWBxyg9bN lQ5b7lpPu6oj9TxQH2P6Lgm8EyPnqkaZTnDDoQ4OvSjpOmJ6KWrPusvWa0Wd cp100arxAKW3VNbYi4cmmy4IDCggXQKXhAG97SSzUQXfOHMKUb1ys8KS1sr6 ebsf/lGToRf+8QEK8wZoJ/8F/DNko+TrFK3R6kMOXbZiYBPTHTT7vdiZYfVz wDilorVvSkNtNGql28NJtdRUxRam+MuqsskoXmrcHRwc5JoISe5qtecYxYxO cL+nWbq2c0tdVBpYV7oCjLr1P2jkpGLT/bmoFkFoQosqg7txSYK96rlBsfiM /k8RguL2KRw96Q1unURtCIOXkr6Sg89XZKsSCLZ1gbY7ZYadSjl/84CBo26B 9R5q7FNUWVak/7cK9TOF9f+R0/9HTv895PTfiL8+HwD7w/UBeor/+AoAFTbo LAf/Zp9Ke/9zoqmRcjgOT+izGfkeYUhOhhIxZOSAcArz6pZ3WS2uhpZZPrvT wDFmg4zuSrBWz0KezpQ93vPf3WATAyXQYFdkjrOUQMwCIgkFzI+EDue9q884 Qj2LvwSuk4JcsfpW0RvaflGFFnIHR3c5H2UZIvePsS+92GjMT/oRN+uMWmv5 WsxXzay3ka8HfNlNMrPaZr5ae9SeVQlDMWf3MR82cOHCI7j4Fiw5Knw2Vp79 +MjFycGRGH4YNzk+xd80q91ud3sKPH4pkLxpcNmioJZHCof2yBmFYBEWC48A pZ8ol5gyJ03LN9X5i32g+kcCVG8u5BwYvX34GyDmseRjbGI4GVfjx4wkk6m0 ZUB0Bl07jZR8Fy6Q1oofXuAnSZFQ40rl0VzGRMpEqfDdb1pE/prjaHZMEfWb hi2z6ckmepID0PfbzvTUq/pprELRR4Y7qYZMj9MCkiq58KN3wynWLKGf886b Ge32TEEpohHOMR852SXt61CkMxWKNGXyI3WivOgkP8ji7A64Ugw4LQt0pOAJ +D+Q4LzFkFYBZPXxoD1AAP8E6AuGwTfJfsv9UnZ/I07gne7kOLpxezC1nHdN +UHlibmTuOzBId/CVtRLb1KUvyiDJZycqwH+mxwE8322rrtrDXLefclOvPV6 rICPo0dTocr2bjqH/dPR4Ar9Za0qRZsPETMSGKdJYpjiMqDuRoCgHZ9cwF5g BuzYZnTAODUklngBtyw5AMqgoTyF5hUkEbM7HBPszayPfjQl3Fb2iopvmC9u 5l/PxoVCWUaokjvLhbLMw+9pJs88IPGRNlo+WeCy9w3xx+TvP5xJ9OYsxyfS qobZQQOKFmD98mqZUUhdR8ICrK9VuP7qBGd/8C/MnnXbPc86QjsG4uCLmUfX HqYTeP4/XOP/11xjiCmZsa9bwjX8W1lJM0WsHEu7OgA5zTj4LekPJlcE9vYL JXLl9I236fiaLaTUfHswNJFpLVSPDxNUkLEjD2tAPRQLM88vsDtYlw4rAStN 573XYskF/6/keJFDwuXMPvXbJUzSjJRJ21+JdYa7tT38VJ3KSUV8JEY5VbMk dyrsgFixLjL5Vv/rsXXIW7WyjM4KdpcahzsevbI/mfS7JlGZ5rnk/hkVzRjF rEtWLY5ijFq9wmi3Z7oBPmBdpA/9gTd6l85rPp4RStuQBMymBzdE5xZ1Z208 TdkEHTm7AyUHIag9PCYc8AKJSRKGf1Hhyq6wlZfUsx1gPQ4lqu+nVi/FSD8b 5kcPVFKNaHq9d6+8ekrWykU8qoDHd8fH+3v75+e7Z3/NZdTCb12nihO2PnYU LPDNN99gttfoRWdEiU1G7Y823KbzkplkCxn8pGqjkOD4fycYwavkOVOtIjrR kzjyyqyBqPj31fdY8Ho8gDWD4pfr38mYRuN28yoZk2q2Cg2srxUb2IxZUVX+ mtqPsJ302eZzAtDOqorrDIoS0ySJKYIETZ5hHavLy457iwPso6TgPUu6qKBW R1QcVQ1rI5nuFSta97nGlyh36Cc2ULDsCtTsrW2m2MNA8jZDBXxtxlzp/PIp ++y9wHy+H8dNkw/zrs5Y1Wjk6/Yt+eo2EJ30b3HQPucZYkT8Ie1r6yoJQpVV KqbtggR5nzNWBxeDdv1ZAce2YGRAVg+N6B7E+5u/wVBsdsYZ2Qzv/1g5Db80 aPEBeQ3LLiSzq+/zqfvy0Ymz8lqSGuPdcAbwbzCuilwHD7vRbfL4I9wvpEsY I3hWdHJyhAH0kjVUH/NbYOAGGHPVb5MgnoCo32EfRDjyGUhvsmVXIk4qc5uy VzsafSjxCwJ7EbQvcygCriU+D0FIXb3vcvn6WqOrL0rXB/W+ih9iPhTUt+o8 +NxZyEXym08zB3XDjjI3rQ+wk2GOu0nSi64Gg47sgqBfUdDbbSq6t57paDHL 6VFoP08dTyF15/fRgjj+Jx1KOwW7kEAmFuy+zEUOltvFPHxlAoP7t2NplQaV UpRjjyeSGP0iBZVQyGIh+5bvWs42DzehwrrcPd89hQP2c2LUCvaC95wwXEJ7 sb2HyQN/x5jnNaAbo0YVMFyd6nRKJJCA2Mz6cmnuI8kCY6Jo7TRg4ArmpqCc 1DAJr9FM/khTQ5sppuybRSRPSlrXKK0wFWF1Si5nGzKtm3431MHPM7ITFGrn gqdn5vm2cMI2cRxehkvRXmt4ejs6uroZswq31R5yru+fT97OSmve7g2y5I9i mZYZ4KPIHWmKAzCNI1ZgwBpC8oEEUoOZ8/jjnBXbYXXlwKryKVzCmvWy5C6l 2VBdhyLqEOsWiYiz6Ik6G5HB4U+VGGR8x+SqgyXQsjFK6kaqvKGnZZ33RFv7 0/3wovjde5HYnAgTxr/JS3hGMK0W8LDf7J433/wMkuKRgScwRa9vm2Yo2d95 3486zbRzRwJAfTWU3l1x3FE2GSIZyorydlYjcgjc0igh3E/BF7pObh6F+PLg F045zC9LgOEaj4dbT5/e3t6uTIBkw02/0po8/S/4xii5vXn6z2REs7xyPb6x yQLvtY/g1LzL8ySZx9DBYKZ53BS+QDdHXnjPg0/nQHfw8nn1SsUx1mX6zJDL 53xqmGl5utP3cVBhlCofSg2vXIraO30aL2ZO40VhGnUYsOfjqSZyHu/e9H0Q Z0o0RR+sVZJ69AGdej9Mder9oJx65/PpLfXLTcUn90PInXeeWuWevLMceado +KZ898HOu/7eU/be/Br5203BRKr0643tcksjM8VeIoWp93LQCOspXn7DlT3z zq7Mj4ItTucS2EWgp6Vo2GIpIkn5dJBlKfKxo1YbhNAfQGpKx4+ziGILeMjZ mA7NUzkBgURoB4TIp5FrFwlZHuQc+Ku62IktpDL/VNaQ8ND0W+PUXIumJC7S FdgdulYOUii7KzASm0CkBzwjg3NHH4EaLQ59JLzSSAtXxMM4nNLz97MAun1Q owfQxNAAFztb0f9aHEZifJMhFsDIU2WqWPJPd62ETGLRWbTIAqjhPOl4gel1 dR3l8T+9Uqb5QVGlMdAen49Y+bXMBuwxpEYTkwLlYIkHaAAfe9RIswz0L+b9 p+JjlnP6nBCFo43eZYxufJkIwMBo0u8jvyYpOvIAXlEYq8gX/XyKpkw1Yfij kjaVNm7JCu4K7ojTW+hEHStQ412/l35IyDJKMjospoVvQG70djAafUJI/8lY hx20r4EngXFDC5Szg3KO2iTS3dHgJkIk6FE/aUHDGORnMv6hOZaDk64mwAtD AyaStcWBraj6SymfKHWqxfEK0nkX1TcxyTrg+9gLBtGycc4Uk3D5idlsLibo 1rvnR7C7Y04AeZX0E8zO1UHdcIadGQ9uUAVPn8u4C4+Xdx7jXOVUiJG3iOqS +pdD1uL15VK9lkUDuL2tyzrgHjKhp0OQTTIMnMD8oaMo+YUQwQxjaVOkkr2Q 9wpO4nr72d133kYZa4VS5iaQnGfQ/Oosrl4yNoMhnMkmqkmGi1YH0VRGn1Ax c1VIyvbc4u/Z2CGQyEapdRmpVJZzztKNYpXrpDUaX8IWNZXydVYN3t7RpDdO MczAZezEPiEnoYqv5T7hCleK/Vkv9odnoHQEG7kaIw9uulj+WfELYqULdOf5 jMJ+GtTvkEtGLOoXL6INmSPrzJhVcGZyc/NtvA0XEU0lnHVaX0QsJAuE2RUE qoxBMa1eFgXa2DTL8YrAZi458yidUZtz5TvcQFFyN076OPPoq2DeZdHzpW9X KKMxGyiASkQTgceBV9HHVm+SMEaCRwc4cceQiBKQwo8Jt4GUpv8Rjc+yP72l QUtRFi3nDudqYBuWlFwzoz1ObreiE/Zd+IlVM4aSwkSZScS3P7Q65+evIskx Wzg1vIH8Hb4mdIIT52yi50T+EKw7PMXi6KpQgRjlLm0G8SoLj8+U7fLGWV1z xudQDmvyUlsPpGr2glqAyKJclIyvx4NBr5kO2phgoiRFIsNNYKpfEr5mE2Sf gVhfQ9+K9k3HiB3k8TAYfmrildfEdqtLXKIWma/UTMwwv4hZdyhXdX3/YPfd 2wtag0j+yZg4mvLIpppXmNQm2r94c3Fy8rb5+tXZT4fHBydbnqRlJqIz+ogS N/lzwEB+zde635ZK6KxDTh/s2gCc9NlPzePdI+H/dQl4jYeKi/y0f4Z54/xI rSHfcxW/Hgot+AcLG0Oa1Kw3GDedXstyuRmx4918xYX9w/NdTKveQ7xwI2ME WCeb5rziVgdYX1obtyRL3KqsDP6QdSkuTEVLybxdzXJI6lJb5eT0+OTi/N3p aXnw1fT9Kc9TEPL/ET1BxRs54Nx0fjcFuDRyk6bc1SZ5kDyhf7tv5F/HS6N/ oLFlRL8NjzK8Nq4V8FcGYo34JVoP4GQM5FMMnrTpq2bDV2inw/7ak327pRYi eORp/qp8wONId0conG3x9dHh4embv26RBeg1ZqxilGa8K+BVBO8wONUlg7U1 X+3/dHp2+NPuxT5XRhnzMu23gEVGcbQ1RoT4G7ggsP7ayjPhorATNAfNFPUN 8EegR2f7r7dM/tGO7Ye5u8p7srz6oL4UMqFx52Admsjj70Q3nXTQxPuZZ9Tr Oy+feQh9a/YnN/LUQtpDU1b/ZPyZddfP9WB/xhDNh4x27UGjJUmWbFS9pLq3 e9o83r9o7r46Ojy27mK2x/X90/2zo6I3bmHGKjRFFFz6wDkyT3G2U+MfmJuy ez1vQOC6LeA9t7y59QmNKu2FD9jTZSTA01FS3xMsYIJ54qSL/QHwQZxlFXj5 G+R0YeLHkuwLaBIz6ajqIsrnShh5L83Y94LMuxSJrVpBx2qMyd1FT2Z4Qm1Q +jAUPhBHrxUho8+wusaRI5M0T5hGDK5SljxuWv1PETF9ovnW0pjOOzZV/prH BNhPbqmlnDXZZtc+OGienp0cHZ7vPSi9hI5COZfJGwLnkGbtCaJC4yeDiSNN f6DDUO/s7oDW5jMHncKNDH8dmSmHv38YwfDk71No3wFSs/MItX7TbqL6Lo6i z5+jwAB33749ghvwMLZY11/WB/fx8lYIoEDaob91bWNbN1XDuiezy5GWkHOB OBtIdjB0KsLWV2j/EY0luxvRH9xpWM9JCmyFywbdMVrr6k5PgDBprR41cvTq 8CQybreSudi0BCJ1O+3CXIwpgS5Qq1enm+ubG41ddpnOrpOEjwJSsk6Czl+Z w40m//+b1l16A7SDKrQJE4wyicNxQwp3dP1PUngknE8cCo7TGzmOk2yCSaz5 vMJRuvwUXYJsVh8P6vhfyrR4+PSEMHISp51J+zA81MR0kh6fwxabAtDAiwEZ 1At0tIGPZJNEevwNY5bz1UFVqzHpfumBcNquGM5b8/zN4cFFc+/tjxUM/si9 fHV41vz57PBiH19u5F/uXuzy20ZEOYpgw/i14tIaq1Rjbb4aZ/u7r7ADa7mX +8c/NA+PK/iqIcv1WvYHxyIAOb4hYwEn8OzQGmPKTtg5EbrkXI8G/fSfvNtb fZ7fVtRNblGtg2jibv8GKB3NKnn2eEonyvKJ9nBF5uwCoFCnU5yeAkMGu7OH rhrWWWc/QzVaml1TJ3HTZIKo0RLiHYEExPma4RIx5kebEIw+T9F/Dtj/tppf gprrFRM3vW22S6uZNbMbZ2Y7xXyKirkp48CFbyU4QsoXOGgb3P2vp7xL5w6a g642gSfG1kFufk5iM8zu56gq7AY8eIa/TT89Rbdjsg178dBd4TTal9esd6bK Ci4QLxQUB8yub5pdH9s1ob1qtun6mvMOO79Ou+KsiQQZ4W0IuBC3EWwC7faN lpzVDT+61KXHQRLJo6yaKVuqrr54kcboLVjYSlsFSuL2nTQ15y41H/6SzWmZ fYJnvpXUlaT7rqGvMWcxoVsK5rOednrko5sVpuU7mhZ/VtwZYnI1ezx2o1Tl L9xrtNUobtdVx8zxHpHEOcZJbeSPL3/6yyZHGz+Ce8/nc4nDXqNcxN+Lijva kncvX66y3okj4vKW7kjx8w8lC/yLFIe/q48a9VYoQ+PuWaPBQ/fowto6rZ7p Kq3nJj6i7j6YGmjfuN+LLMyiCOurc1AEO1f/b5CEtwlKZWOTnaSLRjE8+4Vj Hxr6w4/973ta+fj9BzFNu3unh1vRbudjq9+Gy2pv0O+mV5ORZYuiU3TGZfh1 nAWeAMkB/HPrQ1I/6dff7h6T2EQVDIbu5QCzfcP0DaJX63AksJ06umnGOlm1 JgGeo+ofIpsyjX0yBEZrK1rd2UHbuyTHOxDBfG1n56iFvNgpm11q0cbOzlsE 4GcPloL+/1vnQr2mhCjgY9vG1nBBURkilN2o2FmcVpgbK88rV/CHCIPT3LrP pucmJXtlqy/oYrSmvJoRLfN2ZOpTDh/KRoDwe0m3C5KfdBgdtXCVqTZZbrWv VuNuc7WxHhfMDLRDmrgXEHLLwmiRvqmJ2hSz7hSEm3yMngwL+6UYQoKF0feh M/qIJKM6NHb/0h2m0iBiWc/ldKHYK8nqjASDlOf8eQfD+MO71zr+r2Dv4Dn3 8jyAjJhdEwg1qo4ITDoE0gYLSO5w+efbDE6NcTsWiHrSN1J3k/W6NtKEbgJG vE5QmiVbXcj/J6YW1ZYsePbu76O6pnl8crZ/vn+BF0D1AmOSOM87B+kjPqF5 VMsdy9xWxKUjP7w2CPfY9/646jkqBlBiR4ISC+PXXq7WzXB5GrQsIrgH3Lh0 jFfoGUaEe/EAeKK55zjpkk1sxFlC5FHBHQXr4W1tEvaRB6LnzJ939aFjwHGy RifGv3Zoy6Jaa6tSWVhvP/vujiw5w9HgEh/ZNEqwxrCDa3Rt4H5279z+xrdp p0meH1vmNX56fNlj7ysQ2qO9k+ODw9fN0yM8WOzpYAvLb/5MNrnxPoO/lSvV /XbB8ihF8STgd7dLXyZp1ipWb9I4IzVmNl8Yn8o+T+SoXYuwAfiDzyQ/NAhS g84EmCusDZeunmxOycoV0QojeJ74gEorPs20aOCEcwMjXkOKmgZfBoriOynr uPNqvrHlwtTgcWzA8avvH6Pjaxn51bSXukEKY7gh9ayV8e9PxjfDWbQ1B//P OeUw5n0wGDf1hxm0wWk+HDEu2oynpwMoUj+FBoukTNOpMp87//xGlqX+6eTs Yv8vzYuTi9231qETK8BcKOgJHiSecgR3kh0lJemsR1Wo4GlUpi1NaFWUdtzs A8bqIb9pOwNCJUIbOd8C7Ru1Af0NoSiTPiDqqGEJeYWdr/qd54n6DxvO+hZE qB6ITaMUiU22RQ+jdh3TV/fH9R4wBr2taMM8hkVO23VO7WofA52q36YdRHLh J/tAiSQw9f8CUEsDBBQAAgAIAJyFOCu/Fgh06wEAACkFAAAaABUAaW5jbHVk ZS9saW51eC9ydGNfY3VzdG9tLmhVVAkAA0ibrzsaTrI7VXgEAAAAAAB1lG9r o0AQxl8byHdYKAeuV4mmaa65cj2ssZdylwjGUHpvlhDXIDTbQzelUPrdO7Or 6fnPN8k8v3lmZmfFsywVCU8Ji2Kf+Zt1HC7ZYjg4Ay0TvCkPByPrEyJb/DUM 48qdjZsy88PVnWE4r+OL4cAa9Xonzmza751U3pZv7Eyu+n2Xvf1cB4v2+aZd /dYB8PkaE5xOwrw/XrQE7Nbx8n61iQNlHHeSk/GiMVK4iZRt0qGXJqCXdTr3 Hll4xx6C4DfSaSddhqt4AfRbYyAlg95Y6mPgRSjP6nIU/Lpfx0HEPIReD7xF eNsDfYR+D5wjnFeXWMj8uJMklzsmswPPydtwYBxFke0FT8jTs9gTLpJrEDMh iWmlR7GjxCQvz1lCLEKR6P/JVm4xalW0BH+VSN6vsSsEPBe6ACZtk6RMNEnb Kw+qh35zmtY9lyrvNM8/mav0z6NXpZIsTdmh4DuzkNtcnsOh4Bim6ziOZZoY 3WhCf6rILqPv5a+tVEpH+rWm2OM0R9UA5qivTvcirX02JQsGa2lHEAntuJAE +6krgY5Q7UaPCIKhEPmBPWylYhp/Kvj/0ISnVhE2YbuUElsPTMlXLKAXaeBs YFLm8vRYVI8HxRT4Uq7FUhuFp9oTpL6r+4CCMO7Iqn/3CF7sB1BLAwQUAAIA CAB8hDkrkSzy3pwAAADoAAAAGgAVAGluY2x1ZGUvbGludXgvaTU4Nl90aWNr cy5oVVQJAAOs6rA7Gk6yO1V4BAAAAAAAbY7BCoJAEIbvC/sOA15UKCFJBKNL l6Rj3RdxxxrSLXbHIKJ3b128FB3n+/6Z+SPqjMYOVL0uC3Wqd4ej2ksReUYG f7EXZNp+1Aibxg0LyssiG5xdXrZ/FT/v6IKUwnHD1AKZPhxWY76CM7Ki6YM3 Vwfx40Y6keIlBcwJi1xN2wBWs2v72IOkCjPyaM0ceIduaDR1kKXftSHNpPgA UEsDBBQAAgAIADd8OStCX0nvqxUAADs3AAATABUAbmV0L2lwdjQvaXBfaW5w dXQuY1VUCQADGdywOyPpsDtVeAQAFgRlAMVb+3PbRpL+manK/zDZ7DmkTEMP 2zlHsrWWKcnmWiJ1JBVvrc+FAoEhiRDAcDGAKG7i//2+7hk8+JCcuqqrU2KL BGZ6evr5dc94f+/778Se6PYuRo3GWSLCeBHJWCaZl4UqEWoispkUo87NfvdG LFKVKV9FQudhJsVEpfz2qtu7/QeRaTTUQqaYmUyFXulMxo5g0iLUFWUZiFzT EJor3g3PxVD5c5kZCiEGpBPPl8LTPCKWXqKJEV/FcZ6EvuFsGWYzfp9rmYpI 3snIIQqGyggvukQpkZm4Kdhudm9aIlZBHslyrPhVphoEjxt/7QbHIly4YbLI M8dv34lD5+VzcXRwcLB/eLR/+Eoc/nJ8cHj88kAE3p2MxcX9Qvy1pHOWZzOV 6uPGQGkt3oWpaovX4zD9z7eRjLwkcIaZl0BkgXMR5KeGzcsUwug54s5LxEcZ L2SCOUsvyn6Tb/NP+O30rpzrfIpdOP3BezvrXCVeFIh3ElJLaRH+8FbnEL6j 0qkddoZVRUfdYwR9dPDxbRQm+X1tzCD0Z14aiNskkOlSqcA8HmZygsnlCpq/ j9+uZBQpZwHlOYG0JP4OalJ0luEcA3+jL28jDzpxtJdhubkD5guO0iQT7/No nEIemjfrTfnr2+Ru6kC5mZMoO7gU7GV4L/Xx+p4ax40OzMFYkwfTyGFcZCRx mMAqx2HGJqPzdBHlGu8DuUWA9jwJE7LE7o172R98Ohuci0BJnfyUiTDxozyQ bGLVdPw0f8t1JnSmFppXjhdhBAtceimRajlb61ymXiy1MdjTN9dn/3AH/dvR hUilD2sQqcozqdugmIZ+Zr8KbCJSSkv7vVzdS6X4LU/m2DYcbSbJA7ESKKX5 IjNehb/1g2xkSoy9QIxT5QW+RzvJx9AQ9gLKQQ4bDMrFPrF3BTQHru9LzR4Z iwT+VxGALssZY7UiZcA6wVYmfA/zxcy7U/42Q3A1uMOSzE8TAyTpSSrxV+RN BZbAg5JwIpd4iR2IMBP/ymUutSOGWRhFwk+9hYAP0FrleIzy1SKUBVmaOc7B bIZYQVyHWTm2ULnEy1jGKl1hx2qb4fcykdCRWZ6NgvZeTJmQmTo1v/61dGpx 3ED0BBNTslhIfLFQaSaaY2hNLTF0kqpYIE4eXbQMgfcyZb/8qGRKWoX6rKgK KjyLNUq6l34WrZwH5ooGuxCpMWY+i7DujdWdFK+etR6aSWzXAnIQpLABMjIQ sXOuEFG0GKn0DjFJQ0rXKiWrHcNLErIYf4YooreEeXEHucQ01gw4ttpMVEaR 3xCgT9CkyqezjC1324uHCMXwXzGTHkKYWCjml3NT5OHTDrMTasEpBFZHu9wO DLBZGGem9D79yqJipN7lVNMqW47VVOkwW8G1Y0i2covmCE46Zw1ce+nceZfr 2dtFOnXUveP5Tj63aj+Pwwym9F6lKkBgRkCB6QwljAwa8+DxS7HwKFeS0etZ TQ11iUCMK2wA9j4VyxnEjFdgJ41kMs1mJVOlPWZ5igybLxzRU0t2Qb3me2zv O9W9D/aujfnDq+YeEgHEMKkLZYs9bCuIKIN8w9LJzODRWjDwCFVKgkVsy6wW RlLGuRh4WCeRSU0XFKvGgBr6AQ/2EDLAbZIv2owpMEWzI6cSVhN72bZEe9c3 iM/Yj85Cf9sKCMNY5xApwAXsJQZfGOqIs0grfK+keXt+Y6ENbD+kHJIgHq2s G+Qxnum8FsgmYqVypCUb2Nlxl6mCdJbeanuLZxEea4HUBnG0BdthPb9lyqa0 ZBJOHRjYBOlrk4ogxzdeAgYDk/Hmz04LPdhlb2SQKjFQMBCSPwJLnOX7xsyE RGxLET1EFed3cHunwoBG+LO5SyI0BrsAhArHFqVtg5QGB6Y4jyBiyK4ypjo8 IBUT1FxUAwXbHvDe9vDB2SehGYaS7HgGEMU98U/iIg1DSQVHjfd5wsn32lvJ 1O6dxrGlpJLCO+fi1UI9pCFQniK5L8TR0QvnAP8d8sgbL4d3SHEdBoEk/ETU OWGRkKyJ5IsAsc3IalK6S2kz156GSlKPfchmm+0tX5diAdJYjBFXrM4odq4z VZ8GW7q+vRp1O2fDkXvV7xeG4uyMowjn4SSsRP4OCDERF3OA3IjEbgIlb442 1uRMqKOZ7/iUhFs7Z10Xcybg3kyEWYcLZyaaYULi+6G1A8gKIY6F+TExFXig 24HCPvSHIzG4OO8OLjojnriJVA1cGedhFLj3cbgdIm5A3tgPIzYSvO/5s+18 Rak4WB/TpuCBhzaF8bNtaV4VSgrCgNDKUqVzkYaUGWFKcMtpiLJArLEpnjEs gax26L+fIE1oEgQJwS32LxBxdOrvB5Lyb2oQsCb8RFXItjkkiKEGQxd5dSPC 1wMpxKPZzTGMshnEgGKrQDIIiyhhyNl2TAXgBjatA1eVp74FyELeLyJFddwO 3+7gUxBqb0yRuZpUeszIm0s9C1GGwu/ISNYdqCZvQ/ucIMc7lSTtul7BEngn nFflL3JQVKbpRkBb88IqfaIIyCAIxB18ikPtIxIhh093itLYURXamuMVVlnk kZdSAKN89qrVemBajUWaJ0ss9qcpbKRE8bnL+Wn+hcdvJvXGGgA2YKnGA2x0 a5lzeRdCURF8SnhTL0ywxwUBjAkC5C47vHE/nA+6vc4VRvhhVGapT9QESJGH YWFjifKU4vU/JRJXCYAg5J0JwfAMy7cpbde4fp5NldlEKik+kWlvBQqn53zM /011lrozzPZvRt1+b1gWA9ksJZhLMYkcbk5cR4/E7CKCzJQq0PXflRa/Kt0g 6Snh+SjEEh6zNyZkI/cQWqLIDRO34NUIiPHiR5WsgKNIQpg/QtzTC3i/gVP3 q1omKQtzUX4aKarSbY2+pemlRwgLKwI6cNCj1IxyRVCXZnDZeXX4UnjRlJDF jOAPrNqfGXuUk0noh6BkSJNNmvJyLEWMaCngXKtqVNUVQpAJiw6WVlDhXZhm OeKjrdZmiKSMxxFwSjXxxAgmVza/PKhyMjFVNpWzjAYSWv0nJO9l8pOA+2RL 5GoKeMZAMwo0RRjqlwFuSyCcxGmWFzGaK1i3K8IBQc0AdEVIGMxQ7YUgVuuU QZSUC1Hrrmoy1MowXCzLhuIR1qQQx4OpIKCEA4Tqs2BMYWHbeIZrPWNZE57G nlYEQrmxstzA+FRVE+eAtvvkNxGhIoe6hpYExc9afb4xOVkVSLZo3sEGsOgU vkfmwE0BrSYZ4oU8YThMmkCNHVLTZEzxP+Q+xL6y9V6sgnCyoqc5gUbmDtKK dVH6vu/diqIauMnHEcr6K8SbpGCRUhE91jPKOSuec0l8DC0f4hLeFfAOToQM GQ3emVaioXBULGXptqmn04SkwH9qU2WLNm/q1GIyy2H/++++/+7HogX12tPx vu2nzk7rL7idtw+UCXvb9cZEkZ2vSHLIK7teAQUmaucbWzXQq+2XBA92rsSw 6MFXodrNe5g88PgBWngecM7YvSdSUO19fQRm7uskXmxMpMfhrodFG3zHKwYj u7c6J7/eMYVksOOxl+7kx9/JJqDUjqdFufCQuFD5wfDccHH3YueQ+OHdYDb3 djdFSZY6XmVSQdbpbv29fPWzixw213byvvF6BhS1mLdebrNDwGRznwuhOBzT r2rM597A7dzcDveOvpzUiN7YruWANgIMkGUUDBFzSrDKxBFQEUy5CKEEmXqo tIA4mnZBPXdJd2IPOgQe+v377xoVK8VgsZd6WLqRv6qOSd5g6vjZaTJzwsWM qmfz/KQiQMoXexEl9Deid3t1xdw3UgRml6BP84lZgj63aB7VZs3Uw+ja2ifA 0vQHT1MP68n7rCWIzfVl9NwO0HOzTGN/T3QnRd2CSDummEbpBuG17PshcFGl YKpamoWET3HNdqJQLlCzlWCS6WNy5ilnO2bGPv3C0CaYePKEmwkJyrY3bypp 8UD84HWzSQOYGxdO62IiRh60eEzjjz/EzvcsbHx9doqSE2H/vtWyYrAr13RB KchVUOoTMctQNTSprL38g4DZ5eXwYlTObGCakbZpqRAZ1kRFldYm3ZVTjP7y ZKcGzXvquYlD+/0r//packr2UK2/bYRHxrJcH8BBEgdt8f7yxj0b9a+7nQ3m jlrFmt7STf07Jt4W/Oaktq61QTYN8+wr28gGP1tUDJFHN1zf7dfCuh8YbMce 4PNX48jiAzdtqLzQXLIjrVKJ3yazw6MUydhhA+Nw4AvrymmeILSRde3yYz5v MS48Qzm6h19tlkP5HCHOLW1zz1BqM3H4oC9dQlE2GtBDMA7xHRjHAlD6vTB4 M7PyfrKWtXBQaPphRZ+UllEQYwgHP1hjxVrrn7GOcMK2IX7YsFyzi8Iui9Vs 26y5aTT8lxmEWc3dkmuJggzFJbYB1MVhJO1mSh6K0McGgF+FCZRq5b6Y1S6B 9MilDhlwk0vHeHr2SLzmWbOtiIxHey9OTAqbwL9Fp9+77L53exejy+7V6GLg nl+8u30PAsnE5WLc3Vy7iAc/ygSQU+zv7SZhMhgF3JucakQ6irIlSiReCcqZ XI5o6i1wwYLyy6ftlgFSGytnO/hhQRqOvZWLej0ySsZmnnLB3mhMMYPWIGm7 Lg1dG9Z6bMuGS8a6qUQhIhOwIb00Wh2LpUQNQGA+kdxUYvhKhVqbG+h4AJaQ UqufmYpMd08lSZZS+8qcv6Ow96jnFdpD2ExSQcj7g6jL0eA7s4F74mdse+W3 KmM+KHsj9YIXEr5iS0hsFxwwABDeo0KjLbjLvaAwSK9MM86wVBDgpWcONa6K bIPZFnCIH3/8kXQqXne5M4KtIlDyAXEkMxmciiyEWoneVjaqx4aD+8NDkwPz w5/JfcqhWPfp0UGr9fnwCw00eQvDx89/YbVjxvOjzRniqTiCVXx+jjliCs+s AFjTutzvFgr0VCaPxZAUDzzps0pLdNkWqOx+feF+GJ29u7pwh91/Xrzhw2y6 3+HeDPqj/rBI86Zp8E0MxANndI5VQwHY+CbVZ+LQRJ01NEOZyCKapXv3wp1x 3f2Z6H2pj94ZyMvl6ax5DQyZcxmY8VjaEzcLj2D7MVkIo2rwBCXSWeVSWkhk HMOn0hDlv16DPU3LbBHqOIRuboDvnRgfrYmszRKymvozwbZ8oGuy4CN1k56Y n43QW6KkglGLaGCHJimKtSD+yOu6JW8muCJP2r2wxPloY+FlM8ex7l9moR3Z p1VOXUsTlI+EjLQsFrLb3UABW5ItkvqmejZynDWOM8TtMhYbM5CBhbfGcqht YUDxkpstZa+6zdbjrcQeTGaPoHS04iMLwlAhBVgCzZYUN0q0aGKtO7ojM0Wh 3hJnA4oqiloY5JdedeA6KXpazdABEuK2nuGpYw6jsFYMi8RgTRHA3F/gkzG+ wUWtvjjU3HxwnNajhit+J0kMZ+GEu1IMGslH+BKWVV8BEc3UGkhskCdxRDev zFOrOc5qpDdao8xA5liWG6R8NqWLRagQdunwwqiVTzDOL4Yj97Y3uDjrfLCP KIJUjw5aJ43vv0NePGYac2osUXasMH2Fe7eBKHc6G+cm7VPquPHMeaFNJrNw Sk2g0gO4C2b6gLUKcwM8PIhYaL0GHX5KT2sZ09lFrbFKZBu2LP7fFzcPljas DMNL5WsHJ5vC6V26H/r9j82bS47XbXqARa76nbMrPGHVt8uyrM0m1C7KPA4a O3Fc61HkB9P6Nt6zbygWmqaP2MNvUdWIJ/UivqgA1pPViUVsRg9dYJTQi0Jt DsSKdjKFLnNiV16HNCHAEd1MBFL7aTim+1tEY6aW9coZnncH28fGdBhQnzDJ 74llOmkyZz4NYzqVhgOdbVSbpipwuVVTTyFsCAGdtdnPuvYZyaEN5u4MWjRw MUjV4kQUKt4Ch27namgurm3wA2o+4gC2sN5zIKYy1/PxcU9nNhrbB8AkL3/e 03Bhe69MpW4YNI3xEYgJg/tSWfUlbGL/jPdPDu4vL784yjXS1E+f7nrJwPrp G6YUyaQY0sSY09PDn1t2YLhNZXvIDlpfC/BZOmJRV4hT8XJLIMWVij18qKEO PoGf8/knteXnYRS1ixtt5kKUjelC1O5lpPJfeZiW1hR7ydSeuvLId3lmK2Qy OUSRkA50yUgTtWwLqRfSh0FTIvLmfBZqEjlhY3t2ZE89VDwOk/JIpOLAzuAY nSd8kVInIR9eYFXghFTGkjpHhIkA6U2549QCwDd+nj07630UzV9eHbw6fM6m aiNeYYAoFJZlXncJsqdKxRzHtkzbwKedPm5qCmoo5nGMzFgBJfYtu13X3vVs chjjtFZbI0wQRVxOUpYopoHSEwTfzjtm6dkpHlUBFl/glWm6Uf+HSRmyzEdy Hf7gInk3yWtr/QB+UQEsitvdHjLhr+6wfzvoXBiXLcZVHSoaaQde9d+712eD UfesNywJogCh4Jl6KMzCOMyaraKP1FgAj2Tz5seLQQ9B/rIv/rJ2Em8vAfxH 7hT/i2en9a//nfylXdCyiu51b/7r9uy8WQWqVnv9IUeyVtlAswKhaGc5Lt6s K7zobj044WuttVIqmlIMFNOsVLxBdgsmVJGqDMEGq9bMgkAHsmO313GHo7PR 0H33odlddJMPQXrB2IZm0BLHVcW5BVGKF0UGRmAe/MM9H/RvNoHKNTWl4awD 6UskV3tgLDfxCCG1x/pjGym0fG6ijktnT2IPZr3RF38kpT5UKv8/F8hHDxfI 4PgTHftSIK2uw4badmfoqoZDfQ3ZZhOhcG0u2aXegpO47YlT9cPKoOyrOLJn fNkYMdSLVoSCsxLYVUl2Mbdyxp5uzjofofP+6MPFgK4tVe0ea5wPGJm1Am13 RLRN65ojp54hRJvrZVsNw1YJODY6S4+0onT4b6kmzbo1tGqdqa1Y+Q3sNbjs HB4eHR2L586hc+Qcievb4QiLIAlndCwN+dNtE2pGlVc2zPm3F0Ym/5VnYUzR QjrzqugAUdaTC+4i/K026tBpXJnrH+Xdcb4RhS3yDQG64Gi7RWbCkdOw/6aE 3r8wD587jY5loX5jW3weLqiVRpEHZlTejDFnwpRkQLy8nldsQn8xRF84jXN7 d31GfUOPbiDnWhQXfdcqhBKYvBYvqXXM3+2hM9V3L3brRzyq6KqL+qh6TXgt b+vBD1/BDW0Nbii0iIeDx3jg1OW65MLYIOwFSGWmmxbQZi4ethjAVp5D417z aGzYfCnl8Pr10eP5G17fz219zBUooEGYx1zHs7gX5tpe7c4GHMNcoka5PwfM 4n9lwLV4yPeY8ZjvkKRhXFSNcJHyfpG9MMAGXDQWqClnWgtEz/xjqHJv1F3V u+SwfepmZ5zSWgUMcF1WKbFj1MkiXD8wq6GiN6Lz4aLzcXh77X74VJ03rCOn ckiv37vYnSwfKBtvBgatdHvvbeX4UNFYlYD/N0nWdAi+lWr/B1BLAwQUAAIA CABrfDkrNdBUZnciAAC4ZwAADgAVAG5ldC9pcHY0L3VkcC5jVVQJAAN63LA7 I+mwO1V4BAAWBGUAxFt7c9tGkv+bqcp3GNVtKaREUZKTdXySpY1MUTbPEqkV qSR7qRQKBIYkVngdHpS1a3/3+3XPDAACkDa+u6pzdm1yZrqnp9/dMzzc+/Yb sSfGk9G807kIhRfEvgxkmNmZF4UiWopsLcV8eHs4vhVxEmWRE/kizb1MimWU 8Oz1eHL/K6HpdKJYJoAMVyJ9SjMZDASjFl5aYpauyFNaQrDi3exSzCLnQWYK g4cFydJ2pLBTXhFIO0yJECcKgjz0HEXZo5eteT5PZSJ8uZH+gDAoLHNM3NPE pZ3Zq8QOxK2hvXt/edsrloqfZZIC30nnT2P3RORuPHD6G3E8+Pc34tXR0fHh 0feHR6/Fq+OT4z+fHB8J197IQIw+xeJPBYqLPFtHSXrSuYvSVLzzkqgv3i68 5MeffOnboTuYZXYIZrmDkZufKwKvErBhMhAbOxQfZRDLEDCPtp/9Xf6U/4J/ B5PrwU2+CmU2mN6911AXSZiJ97m/SIA2ZRh7xV9/CjerAbiTDcLILMbeYhh9 wiL6OMDHn3wvzD8NomR1XlB/5X2S6ck2SOeks5GJt3yy7ETa3Z5wbN9P62tE J82iOMZBHD9KpXhce74UXkgigcCWwnOCWAHhTyDT1F7JdCAmUSZssfQ+iUVO nxZR5pAs7axYTKoFWQg7g2jtNCMF+m5j+5773aBBKp3A5c3EGnzxSbegqlBF /6lJ8zBKEulkQiYJNqGNog3pwD+Aw9Xa0jhqZ+65Hhak0gcsGJLKwA4zz8Fx GmtBuAXsWLVkygwtfWH7aSTC6FEUB1UIBcgW4LQrHu0HCU1nEv0nAUVnOtPW TaABrqgISgQ4iiuySNibyHNhOgGx76EBDBOASENslEOhvCylpVHS4FaHdyB8 EGr4AMoJJgxBHNmgY0PSaXEWLBoNp5PJ3ejqfja6ZBUgCyXJ9EnWxUo3kqmY TOdKcZoSncG6nbU6SigfRfpgLXJgY/FCWKRDOGwixcJ2HvxotdM8IruYlQzB HqcQLKh35UCMNjIUaQClhosg3hOZt6PRx4LCRb6CoIQfhSsscRI7XYNkmFfF fH8ubJf0aiKzVxKeMY6jJGO9Sh8OzhdJZLsONHjQQiEwpg8La5lIaRkKG8su XBd7rWR2mMpMpHCVUZyZfZpY35kN6fjkJCPYGCBIXoQgkVmewKGOLobD0awJ D8aSDuaxsvmBGIdw5KSZkjWXbJu4RbPE/LT1YMKLrSxKmbf0MfObMp7c3Iob b9FU7ZvZe+tyOpnfTe/noz7jOBrwf7Vj39hZJi49H1LSSu3LcIW4AN1ppWwW 2ElGIl9aHjwrHaY0z/aDkPo9yCSUiHqei6jkugk8GbjbXD8OteVqLpP5Zkke ImRVnAsmHelttNrXXbogGWDPwr7p+GkG7W9sN7RxTuGTe0wrEbSy4g7CJ1Fh nZr7D1B0K+1s6UnfJVZ7IRRVLpee48nQeWKvDKOjnbOo2zOM9uCVZmvEvoSw Xg2Pj1+9epbLkyhc+CCINFB7We+TWvYLhPUkPkbhUyjDhUxW5Fpx8jSG/wIj IItPT5w+bAua9r9xru2VHepj0aLFEw6eJ44+3KW9gdObDaBVZNiakYo1sOTo AUptJ84auYsD8Ri/gz/XJQ8Vt0iANlTEpSzEqzku4gPC1tpbrfEXhA3xalxT 316Kjx42we6XUfgdNgYaO/HIJuCdHcF2GLpBWihQ6Hrioy/JjXRmUQCBwWmH eZz2NTGuJHNWeQ+4ZPy0jpPaIetA9LOX2WDxaCCu7U0SbdoYjNNtPIoU9pKs 4Qn0UXZlG25Lf4MoPgvgPUjNSMwC9FqhHZATyIovMDeK90Zb+gVdWv9H8PEU Edgdwg4UpRRHmdep6N5OZ+Nfe1o57RAu48pOHuwU+7rMPlf6IDURQe5n3mHh TlOOSsL1lktJBysdN0Kae5BFBy7OiCyytIxqdoh0Apxga8RHcsBYuMwewaRT 8RTlHB3h4700S7wFGZHHQfowShSKIMLWTzSK7aTKg8HLIDUp8/vJvXhP0cf2 xW2+8BGErkFPmGplhV7FNJxSmIMiE8wV0THTdIgrOgkL/VRIiAK7bFSyqjC8 MltpvH0BJneRMYH+RLv8Hoh+govISmAW8iH++vabf/NCx8/h1N7aaXCoM/b1 eX0itx0HLq9lxouQpdQnOMc8zJ5i2T6zdMLMb51Rsnp2yovaEXph6zBcTxi1 0+YFMmmdCYLWYSju0ls9s/kzFGNcqWBtNqRIHgZxy7DXNmiKrna2PFBe1LYD +NUyTPVNy75OKzkJRY+25fjLomosCltmVVTIFSPrCrN4yiQKoWe47/35zWsL SfVDqoEPlcHOwC5xM35XVJwc6u0nmWhd/vYb2GmOuEtRM/AWHZUeUxGbUpL+ 2+TOGt7ez/Ze/X5aWU1MEnu0dI387jdgtT7ML95dj6zZ+D9HtDR5pEBmKcS0 yKLv4kzc/WJdT4cfrfsJ/TO6PFXkUpBMlEFvfjjcvOaCkklEXctIKKpZCZUc mhLQ6Agzu/nBQqbHi7pbRKYPfXia1FtRTErXlGKmYR7Ac/7z2286jwliGpNm Ldbd3S1ie9in4y1Fl9aLszNx1BMEQ5W2WCCyWFT7WGlkLe2kzyN9uL4U/rYv PCayw/Db1ItzKvFh/rSJ7esZG+nyb8e/i8+fCUrgTw3q7bNQR7/3CKZTAzh7 AYBO1qkfARDfv/rx9Y/FJAbUcfChIQEVQrse5o5OhQf6alqAwf19w5D9fc27 Tk04jIk5SpRopnXSB70lq5cmYpc7ENUtxIE47v2uUYDRO+mD2Ya/a8CXGK5Y R3FXH/R5pol9vbbTNZgPXhKK2DXr2wnvKcI7K7gpgb+8TA18USwAO5i3/M2N qufa3+fZ87OGGprjMM5QfjIodYuh22XOUn1Fkz2xcyYm99fXmpQWjdBS6dQ0 gocIxQl/YpILFtJKoyFVBekei7dvxfHrnjh8UVnE/ll93oj166T6B4Sq+bUv vlKmMNCXhMraSArsMzycdCr1Bj1N3CKR9kPJPYLwSKQvcElBsmyXtucDmNXm hIab1s9eqyKvL0L6SKaZkTUrfKUNj00aX6vWx3hetD3yVoKgtDZtD5K6vTL6 VkpRDSrHqrZQrK3ggq5uDx6cc5JqIT2wgONMafLWWAWA3AFAEmdjpVR8at+q KNshyPapGpTepRjote0hqbr//FmjpW9GzFVpKVHT/9kCWT509iLQYDSOUWPQ pmyXLdKCuGLA7X6FgAh31xg9YAlFaftMZ5fHiu3FrlnOCPSe2l1XyMRQrMYo hFLGBV13LNJ2XoWBXjm/jnwX4zzypYi+efhC/NX10BErKLHx5KvgjvH5SyVd 4O6ezhcIoJ4r6LTg3f37bu8F0Dx8AfiP5RSGhyan0GNsJjoGqm9VXuvPRibb U4W0qhOFPZYad1QTmCvbBGZZakWefZ3Evuh8jjJNVIhUwqOqtv1H+ykVj56P IjRxUezlGSekj/Y/oqhP5WBgP8iUWocp1WRUpep6XSAzVJUiSssMFehAHFza G3mjc9hGTgoBqaaFRV1IbNzNv38l2HiRCx6/Fik5SnzEqFuOumqUchGUqFqY jVxyrwgqhrW17HKtuBxm0TrtMk4ldEoabTdEPYjZg2PlcSlCbic76xetmXyi 8bKiFsu1HnULSZ8xLYXTpRzLodZvkVXoxRXXViQZ2xO0J3PK5Beo6zIENZ0a dBjv/n41f9Eo3Da8rsGZ/s9wMlPrOLl7TDh59mtxVoNIHfVWgCFOsHZ8BX7F 9TPxQ4G5zEx0BlxJB0ygLiDPjeK8BF6oFsNUKDAhR3vEIh9gQ7XgrqnPZlni eTP6vzMfpfR0X8P+os05FtbQtGJNgt5eb623pS1L39prDTm8r/JdL4QMpvLL VolJHGrlT0AdNYvMr1F0lnmiYPZgG6vkG31j+uvLkiCrLKNvjWUvcbhQiZec kiGFT00eCK4E/yPYmi+htEE5kx3tTJiMskilaWXNu7vCfN45M3T3mouNmZqD GkAzftSEKd2QXrzllwwre3WwWl4o0nZLVqa8ZcjGFoUgy8HfRjXK+MaBiduh CffUJQU7utopm5Lj4c2tCCI39+mGlxq+Gd1GpZBVwL1MfMCZo6Xu9YME16Pe I9398yBqpiNCFTI+3WaHQHPfZfiFVPeAbnEVpzApctUtoLnpL5GeiyOG9rLv UvH3PM2Ku0ZBvUeqPt6Iz2pA3fpxg1x8gPkgDMcRFDA1qL1YrNW47qkqCmKb G5LiF8lXqwwfhRjm3SiEz+lYXb4kULdJQOj4thfgVEvyerZLtNG7h57KAtq2 X3oJML4R1B4rOsh0/61Wg/Rf6CZKKn5RJu5pTtkxXV8lnp1JUVyYcEesSPTo KtqYl75HhWUvlGl64TLaNkEvXkMj9/APLKlbHdzrAYyMI7NPy/XYggHy6no1 uNfrFhD7XeA4OPfW/tu3r3pFLsGiImvHsvWAhIVFNGgWkOgaC2jwtNUvM4zK tBLzVX1sdcqKKOP+6SO3v3g0LUfVLVNfUQG7wzlIAvJTxVnXih0yG2s8GVqz +cV8Zr370B2D9nE44uv8HpslTLKwytMOss1JZMyDSlilgyTNDptrh9ReZz36 iPornY2vzEWXeKdIcOXSRoykSgP+XRmyNR/fjKzRr8PR6HJ0yRW3wjL6MJ3N 7yd3o4vhhy3nUYGdTe/vhiPrr/ejyfADw3JVCMexve724u7iZjQf3d3eTd9V 98D3+ZSxl+QfP7fb5Wg2tzRFJ8aPK204Uyuu7i7eWxM+CU4swL9bO1uLm/k9 PCLyh41MnhT7ysKEKgPS+YG+Ah7EQZbTavKjY9AO6MvxbMgX0EWioum/mb3n Lo8arJ9hO/kpexyaP2roGW4XZ3sLJbqz+Hh63hBRbkdGQFZtwdXijNlvBPn7 YAkrU+V558V1fC9SUlQQrXSMYkIHTob/6JveEzH9CB90a6f0ZIEDgnqVws8v +CYujn39MqtPV6YxXBzj6fwwOB58j//426G2lp1WUSTSAZlJJWzvmEN//swV AqUykiQ1H95a0A/UFePZB4i/we1ql8iLLcMKrlPYjvt0hC2bByl90c3fwGvl 633dAzNdDsVR7VT0QJRYiVStek69sDUpaq3gZLYWWVgtoSFnxBcm3YY3VTmo T4+8CiBKIYsUdmvQbRtcwJi0c1dOpuukeWBlTsz3JPbKc0w2quF5u/EtG6qF 2q2vcFSaCIbKJepconQrdmjq8zXnbibdNl9c88UEFb4P38M/ZsUj84I2O61c /wyj+IlTA3O1JGyuy1V0FsskCtQrwDSmd4NgG10NU6BD6CxvO2tXLchhlom9 gtmFiL0cLffiPvawE46xFWYSSLRcpjKrDRICsKzX4ILhzl6+rIXFYqYXm9ih UJ+dHVU0nwUV20nm2T7sN36y6JjMMRm63Szi3nm03NabHuhbUpSNNn1DcdG9 F4bcg2dAdxlWyUDnkzpjPBhdXdxfzylodTqVVThaldBuV3GvhyV90b5LWQJU d2N3xAP5erCFektZeYXR/2UZuQukqjIwiEg05YZbat3YnG+4tgmgLF5dSdUI O1AOP5CBEz91WVWePbDCXun/sUv5QyJuCvPgX/FUS7hNlg1RVon60jC3P2xh xSMzY52BeRv1ktlZYYSl/7/G94z5fb1YFcT/1kL/lYEW8ejFXb9SacpNCx9v JKVfKTXvofUA5piX+LeIVTraMAp6HHRGg2K/nYfVUBA7AWwgevCoFnNOW+WJ g1UmksxeoCTdS6pNTKUXsmgOqgrCvDrSg1uxKH8joG9po1Y41HWxsnl8mE6u /8bVHQ9Qfm4n0PxAknZTrrn0o0fKcgSViPSMfSWT7k6POKDePNOD1R1xJ21f I3n0fJ+q3wBFnoL0Qnouucg937U+BagpDw4uJh+ZIMy+m/+i64OBIw7O6Rnj nvo0GAy486yeScV+gY7fTAuVANLrn5RFYWUEBBq5tL25n81RbMe2l1AZuxzI gQJf0Eux2Lcdemxng1ZVmQbq8b1GpJ4ub+iBG5YoQKpMCY4euOrmAOvBWU0R uKzsKZixWvOI7PH+ejp5b91c/HpwfEQUPVKrgNsEyO9Re6PipZ8OaN50dMHL xkwoqN2AhJE+noujT1f4s+0ByzxeCVplquqdG5fjvr1KBzXMUPODc3r4xrO7 9Eh1On3XI/gbjys1+imDqtn0W3d6VBfD+y0838tMKVJQMb2dTOez+9vb02ri 3XkvFU/Vu25V5Ks3pwNRJNGiThS9xmtesxEgcmu4VPrFRcUXVib3eltIimC4 NarYqoW3R8jqCcJ48vPFdRlJsQKR2gutpR14/hPl7BdXFv0WpHp7+syy+8ns djQsbr8bO6hKhcLyQHXtzkSJiQYGqWXsm1chdrvq3r9cR4l7NfKXi4q439x6 q6zQJeXLRUmBQz+FPK1TXtwktFBb3AjwHDRtGkMQS2o/xVTqVp5/Fo8qB7p5 qN7+souhJ5sL7p+58NTL8lcVUSjN23B4uNQAK0Wt+s3joo2IasoZVEgvM3xF uWqVmElNuwajh+sVX00jdP/duP4+rWs3tTWTyOdo/8+yjIardGiaIlWXw9Au kJb5HBWTVSHo8k09UlD0qHkdMlRa14gXJmHbqcKU52mtZTGlzk1s0TfwwrDu dJuNhQ4brShzE7PL7q7QH8HU7RK5cqHVYipuZWuGXhaq1npOerR/Ju7m1nw6 a2+YYMXW9S+/L+GnguR029xk8Za/RyuUt28/mpda9ODWyRhfTx+TaPpMRE0t BOHx5OOz5GuybpDgjocXs3lXMafKLq11imHbKtg4auBY3OUrlGankGZRGBTC fQ7Di/w2pYCZUnEqq3hqleZQ69Vy08w0C5CFHfVKNUmyWvexsA9mpIW/qCGx W7lwKkjvU/rTF4YvTeOptFVK1Afq9yPF6gTSSzIt9Lv58Mp6dze9uCQhkIR3 tn4L0xPN7liTDR19ZLglPjAf34fLonOgHhtgoEiKnw/ScLpX47ubXtGvdCOL 3/Mm9GCFGlicQpuxk4bV6qOliXNadQSFFlSdecWqNZirHpFp76hy4nWGRKyb 66TbzJnq8siMIYfH1+JIpk+CmcfqWpXD1Fp1SbTxEETUT3UqhRmCh6OykS6l fT3TrGOXDzSqXffDQIzpWkW9e84TzrT1fVCl1Sc2nk1RQ/8oqdsr0HRvLv7W U7/+oSf83Ijm2w4AqY0HJrcqVLXMervm4lHdgdFDgMgqinJ6UjCc3d9Yk+mv N+O5+Ev55qlRYIqT1sleiX6Xa7xcFc0QX1+QkWzrkbI03d4jo8q4v2eeREAj KICwMjzQpyK4FApTtjWJ+vJa4H42uuveu/E0z8wvVdOt0hJ01co+U6GUakxU sXGor1XzKAhoswxqzyufrCo3bR8NkzjdvnqomG21bTCeDufXqAT+K5f08wyt JqQ4+pKLGwr6MXvZHjDVJv+UoFlrcgEXuPXOpp2szK0V33t0sYa9CvOYLxFm IAhB56+sAeXbETvg3zyeIQlC3exYdJve5WdijyioLZQwkc4gDMMhanpYlHQV aB+hCwj2ekRDNR0tdh1Pqpu2XL0p9LUTMXLzcrigUj9xSf+7tyvtaSPLop/9 LyogRy5iGJvOZNImtESwE1ASQ5PQk1FPZAEuQw3GVeOlDa3u/953fUvVK2Oi 0Ui92K8Wv/Uu595zycFipWyldPpf7q6wyAYw34tkx4T/UWxecsBskCfJbeBm J7kT733maY4aOyJ4OrbQOSQDMpuMH5TTRjKAumduI7cQKTUMF4F1fZUQnxet SnYK+UZqU68XZ35HL3GUxo6bIm0oLbersQOTY4yTI8lcj0/PE1fViaJ5xjxt 98NPXf90NlrFfDsnNQV/YDAwkD/Cf+ggjpN5EgrQesg9uChe5CAm9HDEaSGX 5oUNimy0mmb+5BPeHT/Ste/sGL0fhCIqGOgd7KbDo97hBxTQ5/1+D4yEzwdn /5Js08rhw1sLWQnM0uI0Adwu4LA8kPeCTCjKU8AEBNh3XEkAmpYcoNeOpbCo GV5ZIjaxBAWGu8AGyHXMGHZ6Ks7VVALFJKO38gUSrfxxi7xrg4hFUdAtr3TK o7JX7rzCFyVwgWGuPEX/zkGwXFwDeWA5BvCGCicgc9uJyTkdrtVM9/fV50eX 3xqcNNLoeYRqpHd29vN577znYiwpT6sTcqPZE4uHQmgqpOg+5cbSrTKNZmqf o/bcs6HDy7gQe6Y54PGvKTrY2qQHfoqsW2neIWq35mtN9EJwxF/OzvuHvuHp n4L9/dAZ8GxzOrUI+uvQBwTi8gGuQI5dU9DLpZLFjx2IosIapq47/tCqA+mY /1d8EbqrPvT/ZxShCQOrckD9+c7fjB1OE716Xz3mVQO2S62OkW5A4u/LrkYV RBsaKYYgZu9yxiUk6Bwbc52CLB62Zw4injS1ZIr4mEHR9ryrlHVmkmRsxop/ l4OM6c0T8DJMrosGtcjh0od+T7CcScvOrHshjh1cqCrD4sra0RKNpxacJp0c mhhZZ94EanHT5HZY+hVLJTScWWXj3Oa4qRT01hGVuG9/vz1C69vNypHlGSMn mm0HuzKO6dzrfXBodFd0u8lJX99YUxtsf7XBprbZYIB3LVBr3/J+r7TutEtt m+nxJDuJJCS+gw8F+TX466qq5UCsXBRrJGz3Dt4fHPf3aoUYk3j8lbpXVWK0 tWCsgkx4T7UGFetilWaNFoVEBBtKKoWBCsqxBIIznFVG3uS5lTh44dGDd/0T DAecnH1RHUn4B4iHRJNJXH+ZoR2dQcJ2gih40+K01u0lglAY3fvDR/WaJXjW aFCVhD68Svm2K/CgOAQIaaKO71o7k6NoE2eypiPOITJYHMkH+e5jNoSbn+dD BOkFnBaZqwe74bOoYuWZOOSpAghks/+LOI/N4Q/FGmy4oBQt0MulFUnRHvlP OhqlyczbF4qLFdz9kiOiZ41y4iqOm7Fe5Uyp7dhutX7YaV+DKUVpYhEYkNlV SuDPjheMKo7s8OPJ515xqlrFGTINPiFOYreEW+BV9NNQMs2ef0bO9dvjfveg 2z0jBrbKxuKCtQxdyG3ZhJeCR4eVPRoEDL4fHJ/+8opAkED74NNJ9/wjW7ai GIVRBtvUXaVXiqWComy/ivfWeMB01nloM5kM05EbcqgaP0qJ0vhx78DdwuoS QW7iMC7MHxAtpXSQImGMcrTLW4fQCzR3QFiY8D8MV2whDDQlbipcIR+E9M6A JgOVi/v2yFUEZf/T+jfgs1OBDgGaOA7WjIbTLM/RPWSnkdU51ftYjMdOeDew Kd4df/zSO4sds2aUjrGIBibgr+Xwrmtji2Z/zCyBW46L+cRwQxfONBLRFNLx ih7M7nKkxmHtjgxU37ARb+1+26HHuKjKbHt7r0K7W9HrGhD+yMMDly2mW1mn EHdDYa1vm/hrb1pBZDQ0BY/PwHdOQGD83vD/rDQcfdS2IqPqEyZlcNUahMSd IjbXGW7a5OLqJhpDp7FejK1i2M/mSUfZGATBUSIWbuW3R1oLDTH6eXI/59o7 swyBDq6cg3wBfDtVrMCHbqiI2Y7hGxYrTzANSAruhCkD5SxVAfBcbpWm11SS p2rC/XkKj4pYhZLeJzm78Uq68FDieoWM/SI5y+U+3br5wMNA+r/kyvokrSAV XSjKEusmIDGLKgHhNvfU1o2Qx0N9ZG5TuadOnueKPgsb0PJza/bnJbpGC/3+ 3eng4MvJp+NDdZf5wcu2PBaQ3Wz4tw0MTW81dF4p3qA/bp388AFcj+XmnLTo eJJiKiWWlKD6hlqHhghD9ykGXilvCR3X6LeLMeiCxmxBpY24clUYuKT6o02G ridSdZDfHWGVJgx4pNNkSMcqOlG8sUkIQiSyHmNlciNmlF3YcJxA5pfZ8IH7 wCVx8JLl/5DUQKyXlBnpOYPqVhxno23SSTp/0lnmWFkhTZ3T0KvPOPk6SMyx mbNqlqyrMixsVXzGeejon099r61hwaHzhYn2+ZnvDkzuqj82I8Fd6/aQTZ9P YYJvGx96Z/0BtUQbuEy/vYxulrziSO7H2nL/nmzEaoBVdbV/0u+5CSZrWhbm pfSDfp70JOOYgDe2RTmtn9wD9IxOp9kleL8PTVQcgrabPHtnCzbSuQPGpxOt KlijPUvwMgbAOOiLYAC2gifzIFt8FjXeRNN7hHceyJeIrQNRoTYPxmOqOil6 DAQo2G7Xydwz8oQyOMHLF86hL0cYQU5VRzN8gF70lG20hyTASl0IWhxKTC2n caA+kpwAPU+VaJw5ZsU7TMKYxCMc2FsrUW1ubqKjG71BYQjSAUYFM/ATmepc kUp3HCOGmvvGyQmt+/blDz/SWWgQcXorlruhC9OLZfQi2n0dx7++/AY/jSWr bO2uRqyQfNBeE8sr9oMUv2CeKrqPlDrKQlHZoLawp90ykl3MBoEzChthIJHE OaD4X3DwFgUA5ya2OC6t5IB/2D6fo1acT1OJq1HGxopnYMjEgjT98daCHClz tLA06A1WTdbj8cxBgAur8qywKuiFBIQ8dTEo3mLMibUdJ4FBoRkdqofXoFnl QzZ/0FeTVuURCsLGo/bF60UlCzNAtSxxMoPcS8u+9KPYldaJU8bFqQlS5Gew aDRszJ2oC66kDW/D/M7HHIw06wmiaTlFV9is4iOBleBKhJyMfnYK+0yS7pDH RumONMMloqQ0ETpgmlr+WYuO7u6YUXyNhbgnLtODeB0EFsAnsNfAL1mKkGen YnkxmVyIr4Jc7OsJ2kNUF1lOZsiUs4gUGCrOqUFAfpVuhZ51RMjKA1F9+Lf6 EJSrbnM6kntrAPuPdsyuRcdJCy8kVqm7SYLp8mJY3BTEBceaY/A/fh5vnCTz ZTYlGgG4a1dYTniIquthefFgUrAk9crkR5kZfXSGsB9W871DJ7E+3NF/OnVS n4UWEoY8k2Tv9I9Pfz4/6PJRlPwo627xqYz9W4fBW8kz48bF/2pxSngUahzi qGSWOimK29CI5nc5KHnGNlOxUz0CkcgVhnQHA6r+AU05tQlBiOQvivTcUbpw OdJGA+Ptyd25VUu5FnmRh/LIuzaz12hlRw3t8kb95RC2euv110699fKr/bSB 87oB33dN4+uv9BU/jbkxqv99GNVfw79grMFS13Ne45QGxqNryuh1wLnAt8yl 83Ojci83qhmVrk6dq/h4C2TOR4Q1jaxNB4sUZFYeF1rTSUatwR+dJlgqNsbe lcBsMjkmo0wYh0xF05XfgqFgOCQbjQaWPCamEpgRDkuJzQjosBSV4pKXNX4y p2Tolm24TK5TMvXod3i1fm3v/vityCYDfdsG+4gmU5kvssba1Y36dnv3HzNz ACUqsxFFs3EUcdU+DVRMcX7lM1wH6XzP6i3a8J6davMcewfLhoYxHKTpZHY3 R7cOjo5itTAf2TDZoP3HA4Ue7z2Gx6xTLTOMiPi1+VxUJ6WCUIVCULQitt6m gWFNGO1UwmgMR/gVjHA8L8yA+Glse7MvuyH4lC9UwF4wEkQgjeBKviA15Kym PGbhFjJC9+3eq4UqA/xpEkwfQT5oDwqxbpuHta2jwuu8+bF+JdMzX9hdS4/s 2+/aN7drhrUHDXs+qcq5Dqtv/TZ2Olx+OKV5crFV+AT3o4dVw0SmDogv2DO0 5ymg0GFjjT5TI4eo+A9cyJcmwXkz75L9jlcpe1ReRZ+x0WYmdwgYNl/x4rV/ 8dq7KNRH+1c2ME+BBkypYtwuX7Bd/goFGpudsu3ZpHodsxvpn9Tsa5I68tv5 u3SPooadWqEscJPJ8X8BUEsDBBQAAgAIAGyEFiugfzfnUQEAAHYCAAATABUA cGVyZi9nZXRfcnRjX3RpbWUuY1VUCQADixeEO5LWsDtVeAQAAAAAAH1SUWvC MBh876/4UBxtaa2+7GHOgZsTBqIgbjIYlC5JayAmJUkFGfvv+5LOjSIbhH7N 5XJ3HOlzSURDGdyak8nsqWZmuL8L+h3Y2MJ20ZJIKy6JXJEL2FLB3y8wrrpQ IzmiXYxpLT0t4NLCoeAydD+FrnKSANkXGuLY7Y4RfAQA7rCkCWhmGy3zYyEm iDbS8EoyCkLJCqxHHbmEMCwpTEHVTIa9jLJjpi3JSWOsOvQSWOeb+W6TwCiK YDqFdNy6ANQancqwh/mUbq9HvWjiz1pvJLvt54/RbyQ09C2FLuk4gSuX6H+H lh/dwIBCOKDJeUUY0neUwOP9bL7AsZg9L7c4V+vt9hXn0+pltvwz2tnmodGa YXmWHxi6iOZNorQP5rsiQhmGgb3Qt8xoEqBGFkNFCKS7Qgj8Gqs5sWmtlVX+ LUG6voaK2dw16+SHBOIs+AJQSwMEFAACAAgAi446Kyq0tMruBgAAhRQAABoA FQBwZXJmL21lYXN1cmVfdWRwX3RyYWZmaWMuY1VUCQADFU6yO/ROsjtVeAQA AAAAAM0Ya2/bNvCz/SuYFA0kS7YsyQlWJ/bgpgkWLIuD1F23poGgSJRNVJYM iXKTFf3vO5J6ULKc5eMM2CSP9+LxeA+/IZEXZj5GZ+lzatDnDU4Hq2n3TQ2c xt43TFvg1G1AAy+i4S4iib0dMPVD8rgDI3EdlEUEoHXY1g2zppo4SaIGaYQp ga9Boj3wTTs88/mGtBOSKHsyEuo5XpbSeF2nE7vk+JcThxLvWyqIfRwAN/T+ 0+XHqy8XyDweDksgR3No7KxT7ClgxYTqOPL1xKUkVjuKYg6Hw56iBGHsUhUp CmyqU4Go/spX/Xw1zsc+h8LHEEy6XRJRtHZJpLCJmywdT0feyk1Qr8dWWxX9 6CLENuFgga8jdsts3HjfQhzBGCcUTdDJu5N3p4CZRSlZRtjnJBEgwd5QBxFP Yn7MkBwng23bcijiegmL6AiUK6ZRPop9kC12+aS33dCE8eEnz1GdDU4cMFQF Z2YbmjofTUuMli1GeyThkcgRuKDfH7O/Lq/ns4Vegk2rFWzZrWB7JIElGe6T JOPqpqDKwULGDljI2AELGTm4kpFm60oG2LxYc+bSmnOV1pzdkPFJaZJ5lF+x 6/uJQyL0SCL/Fq4Y0El0mrsCTOHugSol/+A4UHbp1JovcIfauCw83Oeu/iDJ I5uVDw4HPxIQ3heHZhzMBAdI4Q46NYVTCpCJDkCP1HOjgG9v780HHR2+9Q91 dMScUy2wEdokoH2gHEIYiJMmkXrIlWYf/ESo0jfz9c8u+9Y0sF7WwOIaZEwD 4fiv08F6hQ4FeZa6SzxGa+ymWYIdsJZDEzcIiIfu2akf0P0bYfH04Wsk2Ba0 j9lyjDw39LKQRYEohUtFtgV3TVP2oNOUPIYYxVucgGd9Pzg4+Bo1eSTYixOf RMv8XlMURyIWvPXh4ikJ0dsMARYmW+wPBgNgoLN9PbdIeacKDyzgTPEGR8qh 4eOtFETBhnPn7sPnOx0NwYqTCeqX1183JScHwtKGsgVrtjuPo4Ass4RpT1cY eSE4rlCRIzOt+iaTxTOSkkc+CCVHZThS25UQBOqYWUGJA3TxfvbhEhY6uric fbpeiOnNfLH4W0yvbv6cXcNUhYPy7KQLmoKgwC5QW8+G5DAKllxi6lSZRuFY aYjxRjH/64B5kP2fHa9MDHsOFxWbSoU5lY3yawXvy/CxbLh+JUfiWiQWxr1M tmJLNZTcXiVXYbseulucO799kV9MriNwQ8AtjvwxUuCJGPBOGAFMJ+jtwArY QynTXwt7PeetN/STXpTI0Sw+85JMmV06VzcXYOyP8/PfnbvZZx1d3d7ezRdz 59OH25efVc5i/6sqMsQAsoITuGsSPk9ygafNbRYAJisKIUfhoXkHgaWPQcqH ybA8kPBVhqgU5ceRnJj2JaE9XswZ7T/Q9xWB6Kew4uUsD1ZStFdE4QPGheC2 DZJ4XeokIqFeVHM6S7JHcCrxC0RNSzcVKzm+kAeEzi++B47Ckid4rJxie2qu ouCV1VFEwgUcRSAhDSkA6E/JKuyN1IIvs0FE41WqZHzXxym8B8iA/EJzpcG9 oUrOcE5jGOVBN0lMIXxkOlMQTCLmPiMWU1AjB/OHwGUwotiLw3zJFWLu39Ci DuL2BgWMHjrPkgRD3RIQHPo827GwD64P9niGB+2Pu51Ox/aO3z11yAY8Z5PR DijS7bBExAxMyRp37ocP3U5l6g4UDR1I2ujefuh07kcP3Z4hzsvqU2ZZqc5l duUG15BZmFIUa5N6nd9TGLVmqno+s1R956EX1KbVTm2V1PYL1JbdTm2X1KMX qO1ROzXQVHX8bogyerUXOWQ3jfr9KTL5TGEhkPNnKaMyBqpOltstV6VgxHlY +3nYatEJtBHa+wlHatE6tBGOmoTca6WbV+WuRtgN+BSewl5TXrOfSY2IWm9K xOS0STGV2gq13mLIFFJfoLXzgg7hTOp31HrvIyY7FFOpe1HrnYxMIXUhWjsv 6EfOpLZKrbdYYrJDMZWaJLXeMMkUUs+jtfOC7udM6t7UeidX9Yo1iqnUi6n1 vkymkDosrdrhWyy7aNpOafo1SnCahVBOu1B9Q4nvs7oaPEzE5LEoDkT9LHki YlmDlQhLjIaIxvwxDWzhkugHm+oc8JMzqDzCYLx0ydt02af2yjCZDOtVMkyr IYO34pXn7JVhMRn2q2RYdkMGb/Mr/9grw2YyRq+SYY8aMuyR3JGLmF5dI0KL +WJ2/SJrpboHrTKXVp1Kq4SrOzelVQbVqnNrbeox5NLiWmUYrak/dEIpzusZ bjUB4P0BXyeYZknE/i34yTPr0oPa9AoZWZoYaeIZ+b9c+Z9e/c9uGMIvVBjE o32exvmfh6g/P2nrXQfALW5taiFg/gtQSwMEFAACAAgApZg5K8YBptm6BAAA LwsAABIAFQBwZXJmL3VkcF9zb3VyY2UucGxVVAkAA6UNsTvuDbE7VXgEAAAA AACtVm1P20gQ/hwk/sM0sWpHEAhURYeBU2nh4HRqiwj0TupVkRNvklXtXXd3 XRed+t9vZtZ2zBH1y5VIeHc8b/vMM7MePNsvrdnP9DzJ9mdS7RfCZDCqtre2 twZghUrh/uIGimT+WTgLTkMCK20dLLQBJVylzWfIRWJLI3KhnLcb8Q/gjVYL uSxN4qRWuGe5V/lzlTiQ6HGF/wqjlybJAZPIRLrLzoUx2oxyYW2yFBYSzGQh MzFSCcq2twKygTMIy7SYWl2auQhPWseFNo6SLa0gVdqdwdHx0TGrXOsKZnKJ sUV7MrvSZZbCjPRzu8yEWroVGr0cj1ubPFEPrQFGScDJvDZQZY7aB61uptWS MqgS6dCrq4RQjS2lxCt+yTE4She6D8JYAu1aWqfNQxe84Gv9jo4/3jsIa1Nc wiG5Ojj8BZwQqJ+kqUg5S+uSvLBebQwHx8fH46PxYa0mlXQyyaB2/DiTiXBl 0Y1vxJdSGgEv98bjQ4yNIMOVcLpwcTxxaS35mhgLX6oIAnwxTf1j5R+5fyj/ KPyjgmFtOtEETrN5sHF8jZSjwteyOzwQCuWt8CFKmwlRsD1lfoOhBVd3rnMs WjrKpMI6LTlJG4XpKldxEVdx6E3kAqJULFCpyXMI/8Dzkqh3At9hAEqDQRyM 2qCcknKQilm5ZAqggcgwgY5wjMINljlbdunmxZu1VaPtueZlm1ULVq157yWb FSuv2GWjl7M6oq0y7ECIgsH57dUHOMOj/ACagEfDGTaTXDh4RSZ1Sd4JjGi0 zrm3LTJIqLkYcFsTPZGCnXbsxF2D8yu8GD/B6wUhC48J+zaRigr+aOAEEnvB oAWSgLKcPRCfolVNrGhITKChgj3LSrystWjIMFUGCJV3Y5GjtJxKFTHQu+BD DAEPmEoBfR5RcVczhuDZ36qPjixTPILJ+zd/XN7tws1v09/f0YIE04ur2/O3 6JBz2OhQuLWvAc7tNGo8+QyfGJHO2oTgmhSiIsxz0CUiHsx1qRxzlQopbQMY IummidN4TkLriWesvy3nK74WMAI9fBBysgkuaPBqghCy1QqHO0QHWOHtrZ5P ZmfnhNYKx+kZsyQasgAJQDz1Ce+FEO4BK9GyVdjD4fhXCN8ecWgEfsEyLnmP +BgF8gyJFMjTur1ovbPjc+k1DRPRdbjGGTV3YeyP4TF/WinRBb2HHdV7ziMh 6uMr16Xy7MHhMGu6myc2XTF8rsgfddgfAjWwnyqcPG/X3UvtUc/CjpRbmbtZ fMMOr2+a0f/9A54BgCukczmrt4RYYSQebnJ3cXl7C6en/cv3F9P7yfnVZf+k ubibKyxorxw2R7DG8BEHEIrsJ2i6k+gh8MapX0BiRIxxRmlPqGSGvGFElhKv XORyQXTujVY9vNMr/4GxEllBshzyPO/xZ037ZaEXEKI0rEvwMchxfsRxW5pP ZKhAKeUNQ1yFa/P2O6BriTVkswKKoujRlcWzOMRd2OqRiLUqqKqqx+M3xBU6 l3OjrZhrldr/fjyszdsCo5MWYvTXVvn7Tyq0v8XaQvstFTp/8N01xHZ8NWVe 9ToMRTY+okL/4vL1/VXMLKeWIF7+zDyRSzOEbZ1pI/hhrrXS02w3pPkvUEsB AhcDFAACAAgAKH85K8bbQFngCgAAricAAAcADQAAAAAAAQAAAKSBAAAAAC5j b25maWdVVAUAA5zhsDtVeAAAUEsBAhcDFAACAAgApmI8Ky8oAVJ7CAAA+hYA ABUADQAAAAAAAQAAAKSBGgsAAGRyaXZlcnMvY2hhci9NYWtlZmlsZVVUBQAD d6O0O1V4AABQSwECFwMUAAIACACnhTgrfYJwVocFAACbDwAAGQANAAAAAAAB AAAApIHdEwAAZHJpdmVycy9jaGFyL3J0Y19jdXN0b20uY1VUBQADWpuvO1V4 AABQSwECFwMUAAIACADKezkr4AhvYFlyAACXjgEAEwANAAAAAAABAAAApIGw GQAAZHJpdmVycy9uZXQvM2M1OXguY1VUBQADS9uwO1V4AABQSwECFwMUAAIA CACchTgrvxYIdOsBAAApBQAAGgANAAAAAAABAAAApIFPjAAAaW5jbHVkZS9s aW51eC9ydGNfY3VzdG9tLmhVVAUAA0ibrztVeAAAUEsBAhcDFAACAAgAfIQ5 K5Es8t6cAAAA6AAAABoADQAAAAAAAQAAAKSBh44AAGluY2x1ZGUvbGludXgv aTU4Nl90aWNrcy5oVVQFAAOs6rA7VXgAAFBLAQIXAxQAAgAIADd8OStCX0nv qxUAADs3AAATAA0AAAAAAAEAAACkgXCPAABuZXQvaXB2NC9pcF9pbnB1dC5j VVQFAAMZ3LA7VXgAAFBLAQIXAxQAAgAIAGt8OSs10FRmdyIAALhnAAAOAA0A AAAAAAEAAACkgWGlAABuZXQvaXB2NC91ZHAuY1VUBQADetywO1V4AABQSwEC FwMUAAIACABshBYroH8351EBAAB2AgAAEwANAAAAAAABAAAApIEZyAAAcGVy Zi9nZXRfcnRjX3RpbWUuY1VUBQADixeEO1V4AABQSwECFwMUAAIACACLjjor KrS0yu4GAACFFAAAGgANAAAAAAABAAAApIGwyQAAcGVyZi9tZWFzdXJlX3Vk cF90cmFmZmljLmNVVAUAAxVOsjtVeAAAUEsBAhcDFAACAAgApZg5K8YBptm6 BAAALwsAABIADQAAAAAAAQAAAO2B69AAAHBlcmYvdWRwX3NvdXJjZS5wbFVU BQADpQ2xO1V4AABQSwUGAAAAAAsACwBlAwAA6tUAAAAA --0-92941910-1001943796=:85401-- From owner-netdev@oss.sgi.com Mon Oct 1 07:53:44 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f91Erig19103 for netdev-outgoing; Mon, 1 Oct 2001 07:53:44 -0700 Received: from smtp04.primenet.com (smtp04.primenet.com [64.211.219.53]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f91EraD19097 for ; Mon, 1 Oct 2001 07:53:36 -0700 Received: (from daemon@localhost) by smtp04.primenet.com (8.9.3/8.9.3) id HAA18621 for ; Mon, 1 Oct 2001 07:51:50 -0700 (MST) Received: from usr06.primenet.com(206.165.6.206) via SMTP by smtp04.primenet.com, id smtpdAAA7_aWtK; Mon Oct 1 07:51:42 2001 Received: (from im14u2c@localhost) by usr06.primenet.com (8.8.5/8.8.5) id IAA08877 for netdev@oss.sgi.com; Mon, 1 Oct 2001 08:13:12 -0700 (MST) Date: Mon, 1 Oct 2001 10:13:01 -0500 From: Joseph Zbiciak To: netdev@oss.sgi.com Subject: Leaking skbuff_heads in 2.2.19 due to pumpd? Message-ID: <20011001101301.C8657@usr06.primenet.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i X-Operating-System: BSD/OS 2.1 on an i386 Sender: owner-netdev@oss.sgi.com Precedence: bulk Hello, Alan Cox gave me this address to report a problem I'm seeing in Linux 2.2.19. I have an old 486 box configured as a masquerading firewall/web server/squid cache. It sits on a cablemodem and runs pumpd to acquire a DHCP lease for our IP address. It appears that every time pumpd renews our lease, the kernel leaks about 200 skbuffs. I noticed this because the number of active skbuff_heads shown in /proc/slabinfo seemed unusually large prior to my last reboot (approx 180000). I rebooted, and then set up a cron job to monitor the "skbuff_head_cache" line in /proc/slabinfo every 5 minutes, so I could correlate the increases back to a specific event. (Prior to that, I had tried to trigger a leak by exercising the various services on the box, such as the squid cache, etc. with no success.) I discovered that the number of active skbuff_head's would increase by 200 or so around the same time as pumpd would renew my lease: /var/log/messages: Sep 30 23:32:33 asylum pumpd[215]: renewed lease for interface eth0 Oct 1 04:47:37 asylum pumpd[215]: renewed lease for interface eth0 /var/log/skbuff_head.log: (my adhoc "grep skbuff_head >> log" script) Sun Sep 30 23:30:00 CDT 2001 skbuff_head_cache 1010 1058 Sun Sep 30 23:35:00 CDT 2001 skbuff_head_cache 1066 1081 Sun Sep 30 23:40:00 CDT 2001 skbuff_head_cache 1211 1219 Sun Sep 30 23:45:00 CDT 2001 skbuff_head_cache 1288 1311 ... Mon Oct 1 04:45:00 CDT 2001 skbuff_head_cache 1290 1311 Mon Oct 1 04:50:00 CDT 2001 skbuff_head_cache 1299 1311 Mon Oct 1 04:55:00 CDT 2001 skbuff_head_cache 1355 1357 Mon Oct 1 05:00:00 CDT 2001 skbuff_head_cache 1447 1449 Mon Oct 1 05:05:00 CDT 2001 skbuff_head_cache 1475 1495 It appears that right after the lease is renewed, other traffic seems to leak skbuffs. It's probably worth describing what other sorts of connections are open on the box most of the time (and most importantly, are live across a DCHP lease renewal, which might be why I leak skbuffs after a renewal): -- I typically have one or two rsh connections open to the box from another box on the local LAN. -- I have a Jabber client (Gabber) running through a "redir" session on the 486. Redir is a simple proxy program that accepts a connection on the "listening" side, and opens a connection on my behalf from the firewall box. (I do this so that Gabber doesn't choke when I fire up the VPN software on my other box. I don't run VPN on the 486 in question.) -- I often have a remote telnet session into the box open from work. (Our firewall has a telnet proxy, but not an SSH proxy. *sigh*) -- There are a bunch of connections over the loopback interface that come from Apache. I imagine these are for managing the child threads: tcp 0 0 127.0.0.1:1032 127.0.0.1:1033 ESTABLISHED tcp 0 0 127.0.0.1:1033 127.0.0.1:1032 ESTABLISHED tcp 0 0 127.0.0.1:1030 127.0.0.1:1031 ESTABLISHED tcp 0 0 127.0.0.1:1031 127.0.0.1:1030 ESTABLISHED tcp 0 0 127.0.0.1:1028 127.0.0.1:1029 ESTABLISHED tcp 0 0 127.0.0.1:1029 127.0.0.1:1028 ESTABLISHED tcp 0 0 127.0.0.1:1026 127.0.0.1:1027 ESTABLISHED tcp 0 0 127.0.0.1:1027 127.0.0.1:1026 ESTABLISHED tcp 0 0 127.0.0.1:1024 127.0.0.1:1025 ESTABLISHED tcp 0 0 127.0.0.1:1025 127.0.0.1:1024 ESTABLISHED So the question is, what might be causing the leak? Where should I look? What information (aside from the above) might be relevant? I presume this email address goes to a mailing list. Please copy me directly on any replies, since I don't presently subscribe to any of the lists. Thanks in advance for any insight. I'd love to help stomp this bug. It doesn't affect me horribly (that is, I could live with it easily), but I'd rather not have to reboot once a month when I could instead help make Linux perfect. :-) Regards, --Joe -- ------------------------------------------------------------------------------ Joseph Zbiciak http://www.primenet.com/~im14u2c/ Not your average "Joe" R$+@$=W <-- sendmail.cf {$/{{.+ <-- modem noise !@#!@@! <-- Mr. Dithers swearing Zbiciak <-- Joe's last name ---------------------- Member of the Intellivisionaries ---------------------- From owner-netdev@oss.sgi.com Mon Oct 1 07:56:52 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f91EuqO19269 for netdev-outgoing; Mon, 1 Oct 2001 07:56:52 -0700 Received: from waste.org (waste.org [209.173.204.2]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f91EunD19264 for ; Mon, 1 Oct 2001 07:56:49 -0700 Received: from localhost (oxymoron@localhost) by waste.org (8.9.3/8.9.3) with ESMTP id JAA27910; Mon, 1 Oct 2001 09:55:39 -0500 Date: Mon, 1 Oct 2001 09:55:39 -0500 (CDT) From: Oliver Xymoron To: Ingo Molnar cc: Richard Gooch , Rik van Riel , Kenneth Johansson , "Randy.Dunlap" , Andreas Dilger , , , Subject: Re: [patch] netconsole-2.4.10-B1 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Mon, 1 Oct 2001, Ingo Molnar wrote: > > Face it, Ingo's use of "client" and "server" is contrary to accepted > > usage. You can't finesse around it. > > 'server' is the box that serves content. 'client' is one that requests and > accepts it. in the case of netconsole, it's the netconsole-module box that > produces the messages, and the other one gets them. Server is the side providing the service - direction of data is irrelevant. If the service is logging, the side doing the logging is the server. If the service is console message generation, then the machine generating the messages is the server. Client/server architecture generally implies the possibility of multiple clients per server, and that seems to make more sense with a 'logging server' than a 'console message server'. -- "Love the dolphins," she advised him. "Write by W.A.S.T.E.." From owner-netdev@oss.sgi.com Mon Oct 1 09:30:32 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f91GUWp21268 for netdev-outgoing; Mon, 1 Oct 2001 09:30:32 -0700 Received: from vindaloo.ras.ucalgary.ca (vindaloo.ras.ucalgary.ca [136.159.55.21]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f91GURD21265 for ; Mon, 1 Oct 2001 09:30:27 -0700 Received: (from rgooch@localhost) by vindaloo.ras.ucalgary.ca (8.10.0/8.10.0) id f91GSOc28559; Mon, 1 Oct 2001 10:28:24 -0600 Date: Mon, 1 Oct 2001 10:28:24 -0600 Message-Id: <200110011628.f91GSOc28559@vindaloo.ras.ucalgary.ca> From: Richard Gooch To: Cc: Rik van Riel , Kenneth Johansson , "Randy.Dunlap" , Andreas Dilger , , , Subject: Re: [patch] netconsole-2.4.10-B1 In-Reply-To: References: <200109301225.f8UCPXl16936@vindaloo.ras.ucalgary.ca> Sender: owner-netdev@oss.sgi.com Precedence: bulk Ingo Molnar writes: [Much rationalising deleted:-] > but telling any side to be wrong is stupid - both are correct, and > the correct meaning of 'server' depends alot on context. this is why > i defined the terms. Hey! I didn't call anyone stupid. I didn't even call the idea stupid. I think your terminology is backwards to the way I think and what seems to be accepted practice (and I notice I'm not alone in that). However, "he who writes the code gets to choose", and I'd be a hypocrite to say otherwise :-) I was just hoping to change your mind. Regards, Richard.... Permanent: rgooch@atnf.csiro.au Current: rgooch@ras.ucalgary.ca From owner-netdev@oss.sgi.com Mon Oct 1 10:03:02 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f91H32C22305 for netdev-outgoing; Mon, 1 Oct 2001 10:03:02 -0700 Received: from osdlab.pdx.osdl.net (air-1.osdlab.org [65.201.151.5]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f91H2uD22240 for ; Mon, 1 Oct 2001 10:02:56 -0700 Received: from osdlab.org (dragon.pdx.osdl.net [172.20.1.27]) by osdlab.pdx.osdl.net (8.11.2/8.11.2) with ESMTP id f91H1IS31748; Mon, 1 Oct 2001 10:01:18 -0700 Message-ID: <3BB8A09F.DAFE41B3@osdlab.org> Date: Mon, 01 Oct 2001 09:58:07 -0700 From: "Randy.Dunlap" Organization: OSDL X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.4.3-20mdk i686) X-Accept-Language: en MIME-Version: 1.0 To: Andreas Dilger CC: Ingo Molnar , linux-kernel@vger.kernel.org, linux-net@vger.kernel.org, netdev@oss.sgi.com Subject: Re: [patch] netconsole-2.4.10-B1 References: <20010928010819.A434@turbolinux.com> Content-Type: multipart/mixed; boundary="------------00741BC33BCD832C57A99F68" Sender: owner-netdev@oss.sgi.com Precedence: bulk This is a multi-part message in MIME format. --------------00741BC33BCD832C57A99F68 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Andreas Dilger wrote: > > A few minor changes to the code, after testing it a bit locally (note that I > am using kernel patch C1): > > First, a patch to change the MAC address kernel parameter to be in the > standard XX:XX:XX:XX:XX:XX form, instead of separate bytes. It also > fixes the output of the IP addresses to be unsigned ints. Isn't there > a function in the kernel to format IP addresses for output already? Not quite for formatting, but NIPQUAD(ipaddr) passes 4 octets of IP address to . Here's a patch/replacement for the IP address printing in the netconsole kernel module, using NIPQUAD(). Against netconsole version C2. BTW, in linux/include/linux/kernel.h, isn't HIPQUAD() totally useless (and also unused)? It looks very little-endian-specific. Well, it can be used on little-endian systems if the ipaddr is in host-order. ~Randy --------------00741BC33BCD832C57A99F68 Content-Type: text/plain; charset=us-ascii; name="netcon-ipaddr.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="netcon-ipaddr.patch" --- linux/drivers/net/netconsole.c.save Mon Oct 1 07:43:31 2001 +++ linux/drivers/net/netconsole.c Mon Oct 1 09:28:57 2001 @@ -263,25 +263,20 @@ printk(KERN_ERR "netconsole: network device %s is not an IP protocol device, aborting.\n", dev); return -1; } - source_ip = ntohl(in_dev->ifa_list->ifa_local); + source_ip = in_dev->ifa_list->ifa_local; if (!source_ip) { printk(KERN_ERR "netconsole: network device %s has no local address, aborting.\n", dev); return -1; } -#define IP(x) ((char *)&source_ip)[x] - printk(KERN_INFO "netconsole: using source IP %i.%i.%i.%i\n", - IP(3), IP(2), IP(1), IP(0)); -#undef IP - source_ip = htonl(source_ip); + printk(KERN_INFO "netconsole: using source IP %u.%u.%u.%u\n", NIPQUAD(source_ip)); + + target_ip = htonl(target_ip); if (!target_ip) { printk(KERN_ERR "netconsole: target_ip parameter not specified, aborting.\n"); return -1; } -#define IP(x) ((char *)&target_ip)[x] - printk(KERN_INFO "netconsole: using target IP %i.%i.%i.%i\n", - IP(3), IP(2), IP(1), IP(0)); -#undef IP - target_ip = htonl(target_ip); + printk(KERN_INFO "netconsole: using target IP %u.%u.%u.%u\n", NIPQUAD(target_ip)); + if (!source_port) { printk(KERN_ERR "netconsole: source_port parameter not specified, aborting.\n"); return -1; --------------00741BC33BCD832C57A99F68-- From owner-netdev@oss.sgi.com Mon Oct 1 12:31:00 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f91JV0327185 for netdev-outgoing; Mon, 1 Oct 2001 12:31:00 -0700 Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f91JUxD27182 for ; Mon, 1 Oct 2001 12:30:59 -0700 Received: from localhost (IDENT:davem@pizda.ninka.net [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id MAA01036; Mon, 1 Oct 2001 12:30:24 -0700 Date: Mon, 01 Oct 2001 12:30:23 -0700 (PDT) Message-Id: <20011001.123023.83622655.davem@redhat.com> To: matthias.andree@stud.uni-dortmund.de Cc: Torvalds@transmeta.com, netdev@oss.sgi.com, linux-kernel@vger.kernel.org, kuznet@ms2.inr.ac.ru Subject: Re: devinet.c 4.4BSD compatibility patch to ioctl SIOCGIF* for 2.4.10 From: "David S. Miller" In-Reply-To: <20011001121931.B15943@emma1.emma.line.org> References: <20011001121931.B15943@emma1.emma.line.org> X-Mailer: Mew version 2.0 on Emacs 21.0 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk From: Matthias Andree Date: Mon, 1 Oct 2001 12:19:31 +0200 Please apply this patch to 2.4.11-preX. It's been in -ac for some revisions now, and no-one has screamed or claimed it broke anything. It's unchanged from my 2.4.9 version, it applies cleanly against 2.4.10, so I'm resubmitting. I've already submitted these changes to Linus with some cleanups from Alexey... Franks a lot, David S. Miller davem@redhat.com From owner-netdev@oss.sgi.com Mon Oct 1 17:44:26 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f920iQm02153 for netdev-outgoing; Mon, 1 Oct 2001 17:44:26 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f920iMD02149 for ; Mon, 1 Oct 2001 17:44:22 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id UAA27968; Mon, 1 Oct 2001 20:41:20 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Mon, 1 Oct 2001 20:41:20 -0400 (EDT) From: jamal To: cc: , Robert Olsson , Ingo Molnar , Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk >The new mechanizm: > >- the irq handling code has been extended to support 'soft mitigation', > ie. to mitigate the rate of hardware interrupts, without support from > the actual hardware. There is a reasonable default, but the value can > also be decreased/increased on a per-irq basis via > /proc/irq/NR/max_rate. I am sorry, but this is bogus. There is no _reasonable value_. Reasonable value is dependent on system load and has never been and never will be measured by interupt rates. Even in non-work conserving schemes There is already a feedback system that is built into 2.4 that measures system load by the rate at which the system processes the backlog queue. Look at netif_rx return values. The only driver that utilizes this is currently the tulip. Look at the tulip code. This in conjuction with h/ware flow control should give you sustainable system. [Granted that mitigation is a hardware specific solution; the scheme we presented at the kernel summit is the next level to this and will be non-dependednt on h/ware.] >(note that in case of shared interrupts, another 'innocent' device might >stay disabled for some short amount of time as well - but this is not an >issue because this mitigation does not make that device inoperable, it >just delays its interrupt by up to 10 msecs. Plus, modern systems have >properly distributed interrupts.) This is a _really bad_ idea. not just because you are punishing other devices. Lets take network devices as examples: we dont want to disable interupts; we want to disable offending actions within the device. For example, it is ok to disable/mitigate receive interupts because they are overloading the system but not transmit completion because that will add to the overall latency. cheers, jamal PS: we have been testing what was presented at the kernel summit for the last few months with very promising results. Both on live and setups which are experimental where data is generated at very high rates with hardware traffic generators From owner-netdev@oss.sgi.com Mon Oct 1 18:04:56 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9214uC02535 for netdev-outgoing; Mon, 1 Oct 2001 18:04:56 -0700 Received: from lacrosse.corp.redhat.com (host154.207-175-42.redhat.com [207.175.42.154]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9214pD02531 for ; Mon, 1 Oct 2001 18:04:51 -0700 Received: from toomuch.toronto.redhat.com (IDENT:wH46r5IGkiQKVtJUcYm18UBt0el5o0nT@toomuch.toronto.redhat.com [172.16.14.22]) by lacrosse.corp.redhat.com (8.9.3/8.9.3) with ESMTP id VAA15475; Mon, 1 Oct 2001 21:04:45 -0400 Received: (from bcrl@localhost) by toomuch.toronto.redhat.com (8.11.6/8.11.2) id f9214j818626; Mon, 1 Oct 2001 21:04:45 -0400 Date: Mon, 1 Oct 2001 21:04:45 -0400 From: Benjamin LaHaise To: jamal Cc: linux-kernel@vger.kernel.org, kuznet@ms2.inr.ac.ru, Robert Olsson , Ingo Molnar , netdev@oss.sgi.com Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: <20011001210445.D15341@redhat.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: ; from hadi@cyberus.ca on Mon, Oct 01, 2001 at 08:41:20PM -0400 Sender: owner-netdev@oss.sgi.com Precedence: bulk On Mon, Oct 01, 2001 at 08:41:20PM -0400, jamal wrote: > > >The new mechanizm: > > > >- the irq handling code has been extended to support 'soft mitigation', > > ie. to mitigate the rate of hardware interrupts, without support from > > the actual hardware. There is a reasonable default, but the value can > > also be decreased/increased on a per-irq basis via > > /proc/irq/NR/max_rate. > > I am sorry, but this is bogus. There is no _reasonable value_. Reasonable > value is dependent on system load and has never been and never > will be measured by interupt rates. Even in non-work conserving schemes It is not dependant on system load, but rather on the performance of the CPU and the number of interrupt sources in the system. > There is already a feedback system that is built into 2.4 that > measures system load by the rate at which the system processes the backlog > queue. Look at netif_rx return values. The only driver that utilizes this > is currently the tulip. Look at the tulip code. > This in conjuction with h/ware flow control should give you sustainable > system. Not quite. You're still ignoring the effect of interrupts on the users' ability to execute instructions during their timeslice. > [Granted that mitigation is a hardware specific solution; the scheme we > presented at the kernel summit is the next level to this and will be > non-dependednt on h/ware.] > > >(note that in case of shared interrupts, another 'innocent' device might > >stay disabled for some short amount of time as well - but this is not an > >issue because this mitigation does not make that device inoperable, it > >just delays its interrupt by up to 10 msecs. Plus, modern systems have > >properly distributed interrupts.) > > This is a _really bad_ idea. not just because you are punishing other > devices. I'm afraid I have to disagree with you on this statement. What I will agree with is that 10msec is too much. > Lets take network devices as examples: we dont want to disable interupts; > we want to disable offending actions within the device. For example, it is > ok to disable/mitigate receive interupts because they are overloading the > system but not transmit completion because that will add to the overall > latency. Wrong. Let me introduce you to my 486DX/33. It has PCI. I'm putting my gige card into the poor beast. transmitting full out, it can receive a sufficiently high number of tx done interrupts that it has no CPU cycles left to run, say, gated in userspace. Falling back to polled operation is a well known technique in realtime and reliable systems. By limiting the interrupt rate to a known safe limit, the system will remain responsive to non-interrupt tasks even under heavy interrupt loads. This is the point at which a thruput graph on a slow machine shows a complete breakdown in performance, which is always possible on a slow enough CPU with a high performance device that takes input from a remotely controlled user. This is *required*, and is not optional, and there is no way that a system can avoid it without making every interrupt a task, but that's a mess nobody wants to see in Linux. -ben From owner-netdev@oss.sgi.com Mon Oct 1 18:57:52 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f921vqQ03581 for netdev-outgoing; Mon, 1 Oct 2001 18:57:52 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f921viD03576 for ; Mon, 1 Oct 2001 18:57:44 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id VAA28206; Mon, 1 Oct 2001 21:54:49 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Mon, 1 Oct 2001 21:54:49 -0400 (EDT) From: jamal To: Benjamin LaHaise cc: , , Robert Olsson , Ingo Molnar , Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <20011001210445.D15341@redhat.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Mon, 1 Oct 2001, Benjamin LaHaise wrote: > On Mon, Oct 01, 2001 at 08:41:20PM -0400, jamal wrote: > > > > >The new mechanizm: > > > > > >- the irq handling code has been extended to support 'soft mitigation', > > > ie. to mitigate the rate of hardware interrupts, without support from > > > the actual hardware. There is a reasonable default, but the value can > > > also be decreased/increased on a per-irq basis via > > > /proc/irq/NR/max_rate. > > > > I am sorry, but this is bogus. There is no _reasonable value_. Reasonable > > value is dependent on system load and has never been and never > > will be measured by interupt rates. Even in non-work conserving schemes > > It is not dependant on system load, but rather on the performance of the > CPU and the number of interrupt sources in the system. i am not sure what you are getting at. CPU load is of course a function of the CPU capacity. assuming that interupts are the only source of system load is just bad engineering. > > > There is already a feedback system that is built into 2.4 that > > measures system load by the rate at which the system processes the backlog > > queue. Look at netif_rx return values. The only driver that utilizes this > > is currently the tulip. Look at the tulip code. > > This in conjuction with h/ware flow control should give you sustainable > > system. > > Not quite. You're still ignoring the effect of interrupts on the users' > ability to execute instructions during their timeslice. > And how does /proc/irq/NR/max_rate solve this? I have a feeling you are trying to say that varying /proc/irq/NR/max_rate gives opportunity for user processes to execute; note, although that is bad logic, you could also modify the high and low watermarks for when we have congestion in the backlog queue (This is already doable via /proc) > > [Granted that mitigation is a hardware specific solution; the scheme we > > presented at the kernel summit is the next level to this and will be > > non-dependednt on h/ware.] > > > > >(note that in case of shared interrupts, another 'innocent' device might > > >stay disabled for some short amount of time as well - but this is not an > > >issue because this mitigation does not make that device inoperable, it > > >just delays its interrupt by up to 10 msecs. Plus, modern systems have > > >properly distributed interrupts.) > > > > This is a _really bad_ idea. not just because you are punishing other > > devices. > > I'm afraid I have to disagree with you on this statement. What I will > agree with is that 10msec is too much. > It is unfair to add any latency to a device that didnt cause or contributre to the havoc. > > Lets take network devices as examples: we dont want to disable interupts; > > we want to disable offending actions within the device. For example, it is > > ok to disable/mitigate receive interupts because they are overloading the > > system but not transmit completion because that will add to the overall > > latency. > > Wrong. Let me introduce you to my 486DX/33. It has PCI. I'm putting my > gige card into the poor beast. transmitting full out, it can receive a > sufficiently high number of tx done interrupts that it has no CPU cycles left > to run, say, gated in userspace. > I think you missed my point. i am saying there is more than one source of interupt for that same IRQ number that you are indiscrimately shutting down in a network device. So, assuming that tx complete interupts do actually shut you down (although i doubt that very much given the classical Donald Becker tx descriptor prunning) pick another interupt source; lets say MII link status; why do you want to kill that when it is not causing any noise but is a source of good asynchronous information (that could be used for example in HA systems)? > Falling back to polled operation is a well known technique in realtime and > reliable systems. By limiting the interrupt rate to a known safe limit, > the system will remain responsive to non-interrupt tasks even under heavy > interrupt loads. This is the point at which a thruput graph on a slow > machine shows a complete breakdown in performance, which is always possible > on a slow enough CPU with a high performance device that takes input from > a remotely controlled user. This is *required*, and is not optional, and > there is no way that a system can avoid it without making every interrupt > a task, but that's a mess nobody wants to see in Linux. > and what is this "known safe limit"? ;-> What we are providing is actually a scheme to exactly measure that "known safe limit" you are refering to without depending on someone having to tell you "here's a good number for that 8 way xeon" If there is system capacity available why the fsck is it not being used? cheers, jamal From owner-netdev@oss.sgi.com Mon Oct 1 20:07:06 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f92376Z04753 for netdev-outgoing; Mon, 1 Oct 2001 20:07:06 -0700 Received: from emma1.emma.line.org (postfix@pD9E1EB05.dip.t-dialin.net [217.225.235.5]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9236vD04749 for ; Mon, 1 Oct 2001 20:06:57 -0700 Received: by emma1.emma.line.org (Postfix, from userid 500) id EEEEFA201F; Tue, 2 Oct 2001 05:06:52 +0200 (CEST) Date: Tue, 2 Oct 2001 05:06:52 +0200 From: Matthias Andree To: Linux-Kernel mailing list Cc: Alan Cox , "David S. Miller" , "netdev@oss.sgi.com" , Alexey Kuznetsov , ak@muc.de Subject: 2.2.19 backport of devinet.c 4.4BSD compatibility patch to ioctl SIOCGIF* Message-ID: <20011002050652.A30832@emma1.emma.line.org> Mail-Followup-To: Linux-Kernel mailing list , Alan Cox , "David S. Miller" , "netdev@oss.sgi.com" , Alexey Kuznetsov , ak@muc.de References: <20011001121931.B15943@emma1.emma.line.org> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline In-Reply-To: <20011001121931.B15943@emma1.emma.line.org> User-Agent: Mutt/1.3.22.1i Sender: owner-netdev@oss.sgi.com Precedence: bulk Here's that 2.4.9 patch to devinet.c backported to 2.2.19. Same functionality, but caters for CONFIG_IP_ALIAS: The patch is also available at http://mandree.home.pages.de/linux/kernel/2.2/ Alan, please consider this for inclusion into your current 2.2.20pre* series. --- devinet.c.orig Fri Dec 22 12:36:33 2000 +++ devinet.c Tue Oct 2 04:29:03 2001 @@ -20,6 +20,10 @@ * Changes: * Alexey Kuznetsov: pa_* fields are replaced with ifaddr lists. * Cyrus Durgin: updated for kmod + * Matthias Andree: in devinet_ioctl, compare label and + * address (4.4BSD alias style support), + * fall back to comparing just the label + * if no match found. */ #include @@ -405,6 +409,8 @@ struct device *dev; #ifdef CONFIG_IP_ALIAS char *colon; + struct sockaddr_in sin_orig; + int tryaddrmatch = 0; #endif int exclusive = 0; int ret = 0; @@ -418,6 +424,9 @@ ifr.ifr_name[IFNAMSIZ-1] = 0; #ifdef CONFIG_IP_ALIAS + /* save original address for comparison */ + memcpy(&sin_orig, sin, sizeof(*sin)); + colon = strchr(ifr.ifr_name, ':'); if (colon) *colon = 0; @@ -436,6 +445,9 @@ so that we do not impose a lock. One day we will be forced to put shlock here (I mean SMP) */ +#ifdef CONFIG_IP_ALIAS + tryaddrmatch = (sin_orig.sin_family == AF_INET); +#endif memset(sin, 0, sizeof(*sin)); sin->sin_family = AF_INET; break; @@ -473,6 +485,25 @@ #endif if ((in_dev=dev->ip_ptr) != NULL) { +#ifdef CONFIG_IP_ALIAS + if (tryaddrmatch) { + /* Matthias Andree */ + /* compare label and address (4.4BSD style) */ + /* note: we only do this for a limited set of ioctls + and only if the original address family was AF_INET. + This is checked above. */ + for (ifap=&in_dev->ifa_list; (ifa=*ifap) != NULL; ifap=&ifa->ifa_next) { + if ((strcmp(ifr.ifr_name, ifa->ifa_label) == 0) + && (sin_orig.sin_addr.s_addr == ifa->ifa_address)) { + break; /* found */ + } /* if */ + } /* for */ + } + /* we didn't get a match, maybe the application is + 4.3BSD-style and passed in junk so we fall back to + comparing just the label */ + if (ifa == NULL) +#endif for (ifap=&in_dev->ifa_list; (ifa=*ifap) != NULL; ifap=&ifa->ifa_next) if (strcmp(ifr.ifr_name, ifa->ifa_label) == 0) break; -- Matthias Andree From owner-netdev@oss.sgi.com Mon Oct 1 20:31:17 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f923VHE05296 for netdev-outgoing; Mon, 1 Oct 2001 20:31:17 -0700 Received: from dea.linux-mips.net (f-123-218.cvx-stuttgart.ipdial.viaginterkom.de [62.180.218.123]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f923UvD05293 for ; Mon, 1 Oct 2001 20:30:58 -0700 Received: (from ralf@localhost) by dea.linux-mips.net (8.11.1/8.11.1) id f923UsV15391 for netdev@oss.sgi.com; Tue, 2 Oct 2001 05:30:54 +0200 Received: from grok.yi.org (IDENT:root@cx97923-a.phnx3.az.home.com [24.9.112.194]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f91F0xD19451 for ; Mon, 1 Oct 2001 08:00:59 -0700 Received: from candelatech.com (IDENT:greear@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.2/8.11.2) with ESMTP id f91ExIT01521; Mon, 1 Oct 2001 07:59:18 -0700 Message-ID: <3BB884C6.63476FAD@candelatech.com> Date: Mon, 01 Oct 2001 07:59:18 -0700 From: Ben Greear Organization: Candela Technologies X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.4.3-12 i686) X-Accept-Language: en MIME-Version: 1.0 To: Guilhem Tardy CC: netdev@oss.sgi.com Subject: Re: timing results References: <20010926214925.7742.qmail@web11502.mail.yahoo.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk Guilhem Tardy wrote: > > > Hrm, my tests, where I use mili-second timers in data sent > > from userspace shows that I can run 100Mbps of TCP, UDP, > > and custom Ethernet traffic with less than 1 milisecond of > > latency. And that is end-to-end (user-space to user-space). > > This is 100Mbps tx and 100Mbps rx at the same time, too... > > > > Maybe you don't mean miliseconds?? > > > > ping also shows about .3ms for round-trip on normal fast-ethernet > > and modern computers.... > > > > Ben > > I run over a 100Mbps ethernet card from 3COM (hence the 3c59x.c driver), and > stand by my measurements in microseconds. Yeah, you're right this was > microseconds (sorry for the heart attack ;). Really, this can go as low as 19 > usec and as high as 1300+ usec for the total. > > I included the application that receives and sinks the UDP traffic for your > consideration. One thing puzzles me: my calibration results in either 350 or > 800 megaticks per second, changing from time to time on the same setup. Would > you think of an explanation for that? Maybe your CPU clocks itself down when over-heated? (Is your cooling adequate?) I have no other ideas as to why the timing changes.. Ben > > Guilhem. > > __________________________________________________ > Do You Yahoo!? > Get email alerts & NEW webcam video instant messaging with Yahoo! Messenger. http://im.yahoo.com > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > Name: measure_udp_traffic.c > measure_udp_traffic.c Type: application/x-unknown > Encoding: base64 > Description: measure_udp_traffic.c -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From owner-netdev@oss.sgi.com Mon Oct 1 22:14:28 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f925ESb06572 for netdev-outgoing; Mon, 1 Oct 2001 22:14:28 -0700 Received: from lacrosse.corp.redhat.com (host154.207-175-42.redhat.com [207.175.42.154]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f925EFD06569 for ; Mon, 1 Oct 2001 22:14:15 -0700 Received: from toomuch.toronto.redhat.com (IDENT:UtTtZttYsm9Z+ZLt7MKJcaFEiwVC8nuE@toomuch.toronto.redhat.com [172.16.14.22]) by lacrosse.corp.redhat.com (8.9.3/8.9.3) with ESMTP id BAA05177; Tue, 2 Oct 2001 01:14:00 -0400 Received: (from bcrl@localhost) by toomuch.toronto.redhat.com (8.11.6/8.11.2) id f925Dp520068; Tue, 2 Oct 2001 01:13:51 -0400 Date: Tue, 2 Oct 2001 01:13:51 -0400 From: Benjamin LaHaise To: jamal Cc: linux-kernel@vger.kernel.org, kuznet@ms2.inr.ac.ru, Robert Olsson , Ingo Molnar , netdev@oss.sgi.com Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: <20011002011351.A20025@redhat.com> References: <20011001210445.D15341@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: ; from hadi@cyberus.ca on Mon, Oct 01, 2001 at 09:54:49PM -0400 Sender: owner-netdev@oss.sgi.com Precedence: bulk On Mon, Oct 01, 2001 at 09:54:49PM -0400, jamal wrote: > i am not sure what you are getting at. CPU load is of course a function of > the CPU capacity. assuming that interupts are the only source of system > load is just bad engineering. Indeed. I didn't mean to exclude anything by omission. > And how does /proc/irq/NR/max_rate solve this? > I have a feeling you are trying to say that varying /proc/irq/NR/max_rate > gives opportunity for user processes to execute; > note, although that is bad logic, you could also modify the high and low > watermarks for when we have congestion in the backlog queue > (This is already doable via /proc) The high and low watermarks are only sufficient if the task the machine is performing is limited to bh mode operations. What I mean is that user space can be starved by the cyclic nature of the network queues: they will eventually be emptied, at which time more interrupts will be permitted. > It is unfair to add any latency to a device that didnt cause or > contributre to the havoc. I disagree. When a machine is overloaded, everything gets slower. But a side effect of delaying interrupts is that more work gets done for each irq handler that is run and efficiency goes up. The hard part is balancing the two in an attempt to achieve a steady rate of progress. > I think you missed my point. i am saying there is more than one source of > interupt for that same IRQ number that you are indiscrimately shutting > down in a network device. You're missing the effect that irq throttling has: it results in a system that is effectively running in "polled" mode. Information does get processed, and thruput remains high, it is just that some additional latency is found in operations. Which is acceptable by definition as the system is under extreme load. > So, assuming that tx complete interupts do actually shut you down > (although i doubt that very much given the classical Donald Becker tx > descriptor prunning) pick another interupt source; lets say MII link > status; why do you want to kill that when it is not causing any noise but > is a source of good asynchronous information (that could be used for > example in HA systems)? That information will eventually be picked up. I doubt the extra latency will be of significant note. If it is, you've got realtime concerns, which is not our goal to address at this time. > and what is this "known safe limit"? ;-> It's system dependant. It's load dependant. For a short list of the number of factors that you have to include to compute this: - number of cycles userspace needs to run - number of cache misses that userspace is forced to incur due to irq handlers running - amount of time to dedicate to the irq handler - variance due to error path handling - increased system cpu usage due to higher memory load - front side bus speed of cpu - speed of cpu - length of cpu pipelines - time spent waiting on io cycles ..... It is non-trivial to determine a limit. And trying to tune a system automatically is just as hard: which factor do you choose for the system to attempt to tune itself with? How does that choice affect users who want to tune for other loads? What if latency is more important than dropping data? There are a lot of choices as to how we handle these situations. They all involve tradeoffs of one kind or another. Personally, I have a preference towards irq rate limiting as I have measured the tradeoff between latency and thruput, and by putting that control in the hands of the admin, the choice that is best for the real load of the system is not made at compile time. If you look at what other operating systems do to schedule interrupts as tasks and then looks at the actual cost, is it really something we want to do? Linux has made a point of keeping things as simple as possible, and it has brought us great wins because we do not have the overhead that other, more complicated systems have chosen. It might be a loss in a specific case to rate limit interrupts, but if that is so, just change the rate. What can you say about the dynamic self tuning techniques that didn't take into account that particular type of load? Recompiling is not always an option. > What we are providing is actually a scheme to exactly measure that "known > safe limit" you are refering to without depending on someone having to > tell you "here's a good number for that 8 way xeon" > If there is system capacity available why the fsck is it not being used? That's a choice for the admin to make. Sometimes having reserves that aren't used is a safety net that people are willing to pay for. ext2 has by default a reserve that isn't normally used. Do people complain? No. It buys several useful features (resistance against fragmentation, space for daemon temporary files on disk full, ...) that pay dividends of the cost. Is irq throttling the be all and end all? No. Can other techniques work better? Yes. Always? No. And nothing prevents us from using this and other techniques together. Please don't dismiss it solely because you see cases that it doesn't handle. -ben From owner-netdev@oss.sgi.com Tue Oct 2 02:39:11 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f929dB211033 for netdev-outgoing; Tue, 2 Oct 2001 02:39:11 -0700 Received: from sgi.com (sgi.SGI.COM [192.48.153.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f929d8D11029 for ; Tue, 2 Oct 2001 02:39:08 -0700 Received: from dea.linux-mips.net (f-158-218.cvx-stuttgart.ipdial.viaginterkom.de [62.180.218.158]) by sgi.com (980327.SGI.8.8.8-aspam/980304.SGI-aspam: SGI does not authorize the use of its proprietary systems or networks for unsolicited or bulk email from the Internet.) via ESMTP id CAA01551 for ; Tue, 2 Oct 2001 02:37:44 -0700 (PDT) mail_from (ralf@linux-mips.net) Received: (from ralf@localhost) by dea.linux-mips.net (8.11.1/8.11.1) id f929GiW16542 for netdev@oss.sgi.com; Tue, 2 Oct 2001 11:16:44 +0200 Received: from grok.yi.org (IDENT:root@cx97923-a.phnx3.az.home.com [24.9.112.194]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f925uED07114 for ; Mon, 1 Oct 2001 22:56:14 -0700 Received: from candelatech.com (IDENT:greear@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.2/8.11.2) with ESMTP id f925tVT06254; Mon, 1 Oct 2001 22:55:31 -0700 Message-ID: <3BB956D3.AE0FCC54@candelatech.com> Date: Mon, 01 Oct 2001 22:55:31 -0700 From: Ben Greear Organization: Candela Technologies X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.4.3-12 i686) X-Accept-Language: en MIME-Version: 1.0 To: Benjamin LaHaise CC: jamal , linux-kernel@vger.kernel.org, kuznet@ms2.inr.ac.ru, Robert Olsson , Ingo Molnar , netdev@oss.sgi.com Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 References: <20011001210445.D15341@redhat.com> <20011002011351.A20025@redhat.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk Benjamin LaHaise wrote: > You're missing the effect that irq throttling has: it results in a system > that is effectively running in "polled" mode. Information does get > processed, and thruput remains high, it is just that some additional > latency is found in operations. Which is acceptable by definition as > the system is under extreme load. So, when you turn off the IRQs, are the drivers somehow made aware of this so that they can go into polling mode? That might fix the 10ms latency/starvation problem that bothers me... Assuming it is fairly easy to put a driver into polling mode, if you are explicitly told to do so, maybe this generic IRQ coelescing could be the thing that generically poked all drivers. Drivers that are too primitive to understand or deal with polling can just wait their 10ms, but smarter ones will happily poll away untill told not to by the IRQ load limiter... > That information will eventually be picked up. I doubt the extra latency > will be of significant note. If it is, you've got realtime concerns, > which is not our goal to address at this time. I'm more worried about dropped pkts. If you can receive 10k packets per second, then you can receive (lose) 100 packets in 10ms.... > > > and what is this "known safe limit"? ;-> > > It's system dependant. It's load dependant. For a short list of the number > of factors that you have to include to compute this: > > - number of cycles userspace needs to run > - number of cache misses that userspace is forced to > incur due to irq handlers running > - amount of time to dedicate to the irq handler > - variance due to error path handling > - increased system cpu usage due to higher memory load > - front side bus speed of cpu > - speed of cpu > - length of cpu pipelines > - time spent waiting on io cycles > ..... Hopefully, at the very worst, you can have configurables like: - User-space responsiveness v/s kernel IRQ handling, range of 1 to 100, where 100 == userRules. - Latency: 1 who cares, so long as work happens, to 100: Fast and furious, or not at all. In otherwords, for gods sake don't make me have to understand how my cache and CPU pipeline works!! :) - Another Ben -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From owner-netdev@oss.sgi.com Tue Oct 2 05:13:56 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f92CDud13641 for netdev-outgoing; Tue, 2 Oct 2001 05:13:56 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f92CDhD13637 for ; Tue, 2 Oct 2001 05:13:43 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id IAA29642; Tue, 2 Oct 2001 08:10:00 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Tue, 2 Oct 2001 08:10:00 -0400 (EDT) From: jamal To: Benjamin LaHaise cc: , , Robert Olsson , Ingo Molnar , Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <20011002011351.A20025@redhat.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Tue, 2 Oct 2001, Benjamin LaHaise wrote: > On Mon, Oct 01, 2001 at 09:54:49PM -0400, jamal wrote: > > > And how does /proc/irq/NR/max_rate solve this? > > I have a feeling you are trying to say that varying /proc/irq/NR/max_rate > > gives opportunity for user processes to execute; > > note, although that is bad logic, you could also modify the high and low > > watermarks for when we have congestion in the backlog queue > > (This is already doable via /proc) > > The high and low watermarks are only sufficient if the task the machine is > performing is limited to bh mode operations. What I mean is that user space > can be starved by the cyclic nature of the network queues: they will > eventually be emptied, at which time more interrupts will be permitted. > Which hardware flow control has been doing since 2.1 days; > > It is unfair to add any latency to a device that didnt cause or > > contributre to the havoc. > > I disagree. When a machine is overloaded, everything gets slower. But a > side effect of delaying interrupts is that more work gets done for each > irq handler that is run and efficiency goes up. The hard part is balancing > the two in an attempt to achieve a steady rate of progress. > Let me see if i understand this: -scheme 1: shut down only actions (within a device) that are contributing to the overload and only they get affected because they are misbehaving; when things get better(and we know when they get better), we turn it on again - scheme two: shut down the IRQ which might infact include other devices for a jiffy or two (which doesnt mean the condition got better) Are you saying that you disagree scheme 1 is better? > > I think you missed my point. i am saying there is more than one source of > > interupt for that same IRQ number that you are indiscrimately shutting > > down in a network device. > > You're missing the effect that irq throttling has: it results in a system > that is effectively running in "polled" mode. Information does get > processed, and thruput remains high, it is just that some additional > latency is found in operations. Which is acceptable by definition as > the system is under extreme load. sure. Just like the giant bottom half lock is acceptable when you can do fine grained locking ;-> Dont preach polling to me; i am already a convert and you attended the presentation i gave. Weve had patches for months which have been running on live system. We were just waiting for 2.5 ... > > > So, assuming that tx complete interupts do actually shut you down > > (although i doubt that very much given the classical Donald Becker tx > > descriptor prunning) pick another interupt source; lets say MII link > > status; why do you want to kill that when it is not causing any noise but > > is a source of good asynchronous information (that could be used for > > example in HA systems)? > > That information will eventually be picked up. I doubt the extra latency > will be of significant note. If it is, you've got realtime concerns, > which is not our goal to address at this time. > You are still missing the point (by humping on the literal meaning of the example i provide), the point is: fine grained vs shutting down the whole IRQ. > > > and what is this "known safe limit"? ;-> > > It's system dependant. It's load dependant. For a short list of the number > of factors that you have to include to compute this: > > - number of cycles userspace needs to run > - number of cache misses that userspace is forced to > incur due to irq handlers running > - amount of time to dedicate to the irq handler > - variance due to error path handling > - increased system cpu usage due to higher memory load > - front side bus speed of cpu > - speed of cpu > - length of cpu pipelines > - time spent waiting on io cycles > ..... > > It is non-trivial to determine a limit. And trying to tune a system > automatically is just as hard: which factor do you choose for the system > to attempt to tune itself with? How does that choice affect users who > want to tune for other loads? What if latency is more important than > dropping data? > > There are a lot of choices as to how we handle these situations. They > all involve tradeoffs of one kind or another. Personally, I have a > preference towards irq rate limiting as I have measured the tradeoff > between latency and thruput, and by putting that control in the hands of > the admin, the choice that is best for the real load of the system is > not made at compile time. > > If you look at what other operating systems do to schedule interrupts > as tasks and then looks at the actual cost, is it really something we > want to do? Linux has made a point of keeping things as simple as > possible, and it has brought us great wins because we do not have the > overhead that other, more complicated systems have chosen. It might > be a loss in a specific case to rate limit interrupts, but if that is > so, just change the rate. What can you say about the dynamic self > tuning techniques that didn't take into account that particular type > of load? Recompiling is not always an option. > I am not sure where you are getting the opinion that there is recompiling involved or how what we have is complex (the patch is much smaller than what Ingo posted); And no, you dont need to maintain any state of all those things in your list; in 2.4, which is a good start, you have the system load being probed via a second order effect i.e the growth rate of the backlog queue is a good indicator that the system is not pulling packets off fast enough. This is a very good measure of _all_ those items on your list. I am not saying it is God's answer, merely pointing that it is a good indicator which doesnt need to maintain state of 1000 items or cause additional computations on the datapath. We get a fairly early warning that we are about to be overloaded. We can then shut off the offending device's _receive_ interupt source when it doesnt heed the congestion notification advice weve been giving it. It heeds the advice by mitigating. In the 2.5 patch (i should say it is a clean patch to 2.4 actually and is backward compatible) we worked around the fact that the 2.4 solution requires a specific NIC feature (mitigation) among a lot of other things. Infact we have already proven that mitigation is only good when you have one or two NICs on the system. > > What we are providing is actually a scheme to exactly measure that "known > > safe limit" you are refering to without depending on someone having to > > tell you "here's a good number for that 8 way xeon" > > If there is system capacity available why the fsck is it not being used? > > That's a choice for the admin to make. Sometimes having reserves that aren't > used is a safety net that people are willing to pay for. ext2 has by > default a reserve that isn't normally used. Do people complain? No. It > buys several useful features (resistance against fragmentation, space for > daemon temporary files on disk full, ...) that pay dividends of the cost. > I am not sure whether you are trolling or not. We are talking about a system conserving a work principle and you compare it a reservation system. > Is irq throttling the be all and end all? No. Can other techniques work > better? Yes. Always? No. And nothing prevents us from using this and > other techniques together. Please don't dismiss it solely because you > see cases that it doesn't handle. > I am not dismising the whole patch. I most definetly dismiss those two ideas i pointed out. cheers, jamal From owner-netdev@oss.sgi.com Tue Oct 2 08:52:32 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f92FqWv18544 for netdev-outgoing; Tue, 2 Oct 2001 08:52:32 -0700 Received: from ganymede.linux.org (ganymede.linux.org [198.182.196.48]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f92FqSD18540 for ; Tue, 2 Oct 2001 08:52:29 -0700 Received: from linux.org (ying.invlogic.com [198.182.196.14]) by ganymede.linux.org (8.11.6/8.11.6) with ESMTP id f92FqSQ26683 for ; Tue, 2 Oct 2001 11:52:28 -0400 Date: Tue, 2 Oct 2001 11:52:28 -0400 Message-Id: <200110021552.f92FqSQ26683@ganymede.linux.org> From: URLWatch To: Subject: URLWatch Notice for www.linux.org X-Mailer: URLWatch Sender: owner-netdev@oss.sgi.com Precedence: bulk Hello, Your email address was registered for the URLWatch service. This is a confirmation that you will now receive notices every time the document below is updated. The request came from 24.50.228.116. Address : netdev@oss.sgi.com Document : http://www.linux.org/footer.html If you have any questions about this service, please address them to webmaster@linux.org For general information on URLWatch, please visit their web site at http://www.urlwatch.com/ Thanks for visiting our web site and using URLWatch!! URLWatch, linux.org ------------------------------------------------------------------------- If you did not sign up for this service, please reply to this message copying the lines below, and you will be removed. We apologize for the inconvenience. URLWatch notice : Signup URLWatch password : yFYnfK URLWatch message : 20011002115228 URLWatch address : netdev@oss.sgi.com URLWatch document : http://www.linux.org/footer.html From owner-netdev@oss.sgi.com Tue Oct 2 09:51:47 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f92GplA19820 for netdev-outgoing; Tue, 2 Oct 2001 09:51:47 -0700 Received: from texas.dotxtra.com (texas.dotxtra.com [209.15.112.65]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f92GpgD19816 for ; Tue, 2 Oct 2001 09:51:42 -0700 Received: from mcns76.docsis184.singa.pore.net (fluff) [202.156.184.76] by texas.dotxtra.com with smtp (Exim 3.12 #1 (Debian)) id 15oSle-0007q2-00; Tue, 02 Oct 2001 11:51:39 -0500 Message-ID: <003301c14b62$80119a40$4cb89cca@fluff> From: "Sanjeev \"Ghane\" Gupta" To: "URLWatch" , References: <200110021552.f92FqSQ26683@ganymede.linux.org> Subject: Re: URLWatch Notice for www.linux.org Date: Wed, 3 Oct 2001 00:51:33 +0800 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2600.0000 X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Sender: owner-netdev@oss.sgi.com Precedence: bulk Hi, This is a mailing list, for Network Issues in Linux. Please do remove this notification. Thanks ----- Original Message ----- From: "URLWatch" To: Sent: Tuesday, October 02, 2001 11:52 PM Subject: URLWatch Notice for www.linux.org > Hello, > > Your email address was registered for the URLWatch service. This is a > confirmation that you will now receive notices every time the document > below is updated. The request came from 24.50.228.116. > > Address : netdev@oss.sgi.com > Document : http://www.linux.org/footer.html > > If you have any questions about this service, please address them to > webmaster@linux.org > > For general information on URLWatch, please visit their web site at > http://www.urlwatch.com/ > > Thanks for visiting our web site and using URLWatch!! > > URLWatch, linux.org > > ------------------------------------------------------------------------- > > If you did not sign up for this service, please reply to this message > copying the lines below, and you will be removed. We apologize > for the inconvenience. > > URLWatch notice : Signup > URLWatch password : yFYnfK > URLWatch message : 20011002115228 > URLWatch address : netdev@oss.sgi.com > URLWatch document : http://www.linux.org/footer.html > > From owner-netdev@oss.sgi.com Tue Oct 2 10:00:59 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f92H0x420019 for netdev-outgoing; Tue, 2 Oct 2001 10:00:59 -0700 Received: from robur.slu.se (robur.slu.se [130.238.98.12]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f92H0rD20013 for ; Tue, 2 Oct 2001 10:00:53 -0700 Received: (from robert@localhost) by robur.slu.se (8.8.7/8.8.7) id TAA18748; Tue, 2 Oct 2001 19:03:07 +0200 From: Robert Olsson MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15289.62283.695135.525478@robur.slu.se> Date: Tue, 2 Oct 2001 19:03:07 +0200 To: Ben Greear Cc: Benjamin LaHaise , jamal , linux-kernel@vger.kernel.org, kuznet@ms2.inr.ac.ru, Robert Olsson , Ingo Molnar , netdev@oss.sgi.com Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <3BB956D3.AE0FCC54@candelatech.com> References: <20011001210445.D15341@redhat.com> <20011002011351.A20025@redhat.com> <3BB956D3.AE0FCC54@candelatech.com> X-Mailer: VM 6.92 under Emacs 19.34.1 Sender: owner-netdev@oss.sgi.com Precedence: bulk Hello! Jamal mentioned some about the polling efforts for Linux. I can give some experimental data here with GIGE. Motivation, implantation etc is in paper to presented at USENIX Oakland. Below a IP forwarding test. Injected 10 Million 64 byte packets into eth0 at a speed of 890.000 p/s received and routed and TX:ed on eth1. PIII @ 933 MHz. Kernel UP 2.4.10 with polling patch the are NIC's e1000 eth0 (irq=24) and eth1 (irq=26) Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flags eth0 1500 0 4031309 7803725 7803725 5968699 22 0 0 0 BRU eth1 1500 0 18 0 0 0 4031305 0 0 0 BRU The RX-ERR, RX-DRP are bugs from the e1000 driver. Anyway we getting 40% of packet storm routed. With a estimated throughput is about 350.000 p/s irq CPU0 24: 80652 IO-APIC-level e1000 26: 41 IO-APIC-level e1000 The for RX (polling) we use only 24 interrupts. TX is mitigated (not polled) in this run. We see a lot more interrupts for same amount of packets. I think we can actually tune this a bit... And I should also say that RxIntDelay=0 (e1000 driver). So there is no latency before the driver register for polling by kernel. USER PID %CPU %MEM SIZE RSS TTY STAT START TIME COMMAND root 3 3.0 0.0 0 0 ? SWN 12:51 0:11 (ksoftirqd_CPU0) The polling (softirq) now handled by ksoftirqd but I have seen a path from Ingo that schedules without the need for ksoftirqd. Also note that during poll we disable only RX interrupts so all other device interrupts/functions are handled properly. And tulip variants of this is in production use and seems very solid. The kernel code part holds ANK-trademark. :-) Cheers. --ro From owner-netdev@oss.sgi.com Tue Oct 2 10:40:13 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f92HeDZ20793 for netdev-outgoing; Tue, 2 Oct 2001 10:40:13 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f92He7D20790 for ; Tue, 2 Oct 2001 10:40:08 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id NAA01300; Tue, 2 Oct 2001 13:37:11 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Tue, 2 Oct 2001 13:37:11 -0400 (EDT) From: jamal To: Robert Olsson cc: Ben Greear , Benjamin LaHaise , , , Ingo Molnar , Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <15289.62283.695135.525478@robur.slu.se> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Some data on lesser worthy cards (i.e 10/100) on 2.4.7 can be found at: http://www.cyberus.ca/~hadi/247-res/ cheers, jamal On Tue, 2 Oct 2001, Robert Olsson wrote: > > > Hello! > > Jamal mentioned some about the polling efforts for Linux. I can give some > experimental data here with GIGE. Motivation, implantation etc is in paper > to presented at USENIX Oakland. > > Below a IP forwarding test. Injected 10 Million 64 byte packets into eth0 at > a speed of 890.000 p/s received and routed and TX:ed on eth1. > > PIII @ 933 MHz. Kernel UP 2.4.10 with polling patch the are NIC's e1000 > eth0 (irq=24) and eth1 (irq=26) > > > Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flags > eth0 1500 0 4031309 7803725 7803725 5968699 22 0 0 0 BRU > eth1 1500 0 18 0 0 0 4031305 0 0 0 BRU > > > The RX-ERR, RX-DRP are bugs from the e1000 driver. Anyway we getting 40% of > packet storm routed. With a estimated throughput is about 350.000 p/s > > irq CPU0 > 24: 80652 IO-APIC-level e1000 > 26: 41 IO-APIC-level e1000 > > The for RX (polling) we use only 24 interrupts. TX is mitigated (not polled) > in this run. We see a lot more interrupts for same amount of packets. I think > we can actually tune this a bit... And I should also say that RxIntDelay=0 > (e1000 driver). So there is no latency before the driver register for polling > by kernel. > > USER PID %CPU %MEM SIZE RSS TTY STAT START TIME COMMAND > root 3 3.0 0.0 0 0 ? SWN 12:51 0:11 (ksoftirqd_CPU0) > > The polling (softirq) now handled by ksoftirqd but I have seen a path from > Ingo that schedules without the need for ksoftirqd. > > Also note that during poll we disable only RX interrupts so all other device > interrupts/functions are handled properly. > > And tulip variants of this is in production use and seems very solid. The > kernel code part holds ANK-trademark. :-) > > Cheers. > > --ro > From owner-netdev@oss.sgi.com Tue Oct 2 11:33:30 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f92IXUK21926 for netdev-outgoing; Tue, 2 Oct 2001 11:33:30 -0700 Received: from uni04du.unity.ncsu.edu (uni04du.unity.ncsu.edu [152.1.2.81]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f92IXRD21922 for ; Tue, 2 Oct 2001 11:33:27 -0700 Received: (from ankulkar@localhost) by uni04du.unity.ncsu.edu (8.8.4/EC02Jan97) id OAA04626; Tue, 2 Oct 2001 14:33:23 -0400 (EDT) Date: Tue, 2 Oct 2001 14:33:23 -0400 (EDT) From: Amit Kulkarni To: Subject: netdevice ->tbusy and qdisc_wakeup function Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Hi, Sorry to mail to you uninitiated. I note that in hte recent versions of linux the net device no longer has field tbusy. and /net/pkt_sched no longer defines function qdisc_wakeup We had written a driver (compiled on 2.2.14) that used both these functions . Now we need to upgrade to linux 2.4.5 and hence our code does not compile w/o warnings. please advise thanks in advance rgds, -Amit ***************************************************************************** Do not Worry much about life.. anyway you won't get out of it alive...! ***************************************************************************** From owner-netdev@oss.sgi.com Tue Oct 2 11:48:50 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f92Imok22342 for netdev-outgoing; Tue, 2 Oct 2001 11:48:50 -0700 Received: from osdlab.pdx.osdl.net (air-1.osdlab.org [65.201.151.5]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f92ImmD22339 for ; Tue, 2 Oct 2001 11:48:48 -0700 Received: from osdlab.org (dragon.pdx.osdl.net [172.20.1.27]) by osdlab.pdx.osdl.net (8.11.2/8.11.2) with ESMTP id f92ImRS08814; Tue, 2 Oct 2001 11:48:27 -0700 Message-ID: <3BBA0B33.9B2A2BDE@osdlab.org> Date: Tue, 02 Oct 2001 11:45:07 -0700 From: "Randy.Dunlap" Organization: OSDL X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.4.3-20mdk i686) X-Accept-Language: en MIME-Version: 1.0 To: Amit Kulkarni CC: netdev@oss.sgi.com Subject: Re: netdevice ->tbusy and qdisc_wakeup function References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk Amit Kulkarni wrote: > > Hi, > > Sorry to mail to you uninitiated. > > I note that in hte recent versions of linux the net device no longer has > field tbusy. and /net/pkt_sched no longer defines function qdisc_wakeup > > We had written a driver (compiled on 2.2.14) that used both these > functions . Now we need to upgrade to linux 2.4.5 and hence our code does > not compile w/o warnings. See http://www.firstfloor.org/~andi/softnet/ and/or search kernel archives for "softnet". ~Randy From owner-netdev@oss.sgi.com Tue Oct 2 12:48:34 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f92JmY123533 for netdev-outgoing; Tue, 2 Oct 2001 12:48:34 -0700 Received: from webber.adilger.int (h24-64-71-161.cg.shawcable.net [24.64.71.161]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f92JmQD23528 for ; Tue, 2 Oct 2001 12:48:26 -0700 Received: from webber.adilger.int (adilger@localhost [127.0.0.1]) by localhost (8.12.0.Beta19/8.12.0.Beta19/Debian 8.12.0.Beta19) with ESMTP id f92JkQL7009591; Tue, 2 Oct 2001 13:46:26 -0600 Received: (from adilger@localhost) by webber.adilger.int (8.12.0.Beta19/8.12.0.Beta19/Debian 8.12.0.Beta19) id f92JkH3I009589; Tue, 2 Oct 2001 13:46:17 -0600 From: Andreas Dilger Date: Tue, 2 Oct 2001 13:46:17 -0600 To: Robert Olsson Cc: Ben Greear , Benjamin LaHaise , jamal , linux-kernel@vger.kernel.org, kuznet@ms2.inr.ac.ru, Ingo Molnar , netdev@oss.sgi.com Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: <20011002134617.J8954@turbolinux.com> Mail-Followup-To: Robert Olsson , Ben Greear , Benjamin LaHaise , jamal , linux-kernel@vger.kernel.org, kuznet@ms2.inr.ac.ru, Ingo Molnar , netdev@oss.sgi.com References: <20011001210445.D15341@redhat.com> <20011002011351.A20025@redhat.com> <3BB956D3.AE0FCC54@candelatech.com> <15289.62283.695135.525478@robur.slu.se> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <15289.62283.695135.525478@robur.slu.se> User-Agent: Mutt/1.3.22i Sender: owner-netdev@oss.sgi.com Precedence: bulk On Oct 02, 2001 19:03 +0200, Robert Olsson wrote: > Jamal mentioned some about the polling efforts for Linux. I can give some > experimental data here with GIGE. Motivation, implantation etc is in paper > to presented at USENIX Oakland. How do you determine the polling rate? I take it that this is a different patch than Ingo's? > Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flags > eth0 1500 0 4031309 7803725 7803725 5968699 22 0 0 0 BRU > eth1 1500 0 18 0 0 0 4031305 0 0 0 BRU > > The RX-ERR, RX-DRP are bugs from the e1000 driver. Anyway we getting 40% of > packet storm routed. With a estimated throughput is about 350.000 p/s Are you sure they are "bugs" and not dropped packets? It seems to me that RX-ERR == RX-DRP, which would seem to me that the receive buffers are full on the card and are not being emptied quickly enough (or maybe that is indicated by RX-OVR...) I don't know whether it is _possible_ to empty the buffers quickly enough, I suppose CPU usage info would also shed some light on that. Cheers, Andreas -- Andreas Dilger \ "If a man ate a pound of pasta and a pound of antipasto, \ would they cancel out, leaving him still hungry?" http://www-mddsp.enel.ucalgary.ca/People/adilger/ -- Dogbert From owner-netdev@oss.sgi.com Tue Oct 2 13:59:08 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f92Kx8625680 for netdev-outgoing; Tue, 2 Oct 2001 13:59:08 -0700 Received: from po4.wam.umd.edu (po4.wam.umd.edu [128.8.10.166]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f92KwvD25675 for ; Tue, 2 Oct 2001 13:58:57 -0700 Received: from bay2-21.dial.umd.edu (IDENT:vgb@bay2-21.dial.umd.edu [128.8.22.85]) by po4.wam.umd.edu (8.9.3/8.9.3) with ESMTP id QAA22701; Tue, 2 Oct 2001 16:58:53 -0400 (EDT) Date: Tue, 2 Oct 2001 16:58:41 -0400 (EDT) From: Vijay G Bharadwaj To: , Subject: TCP SACK bug report Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Summary: -------- Linux 2.4.9 seems to have a bug in TCP where it does not handle duplicate retransmissions properly, causing TCP transfers to stall. Short description: ------------------ The attached tcpdump extract shows part of an HTTP download, right before the TCP connection stalled. Client was on PPP dialup. This kind of stall has been repeatable for me for some time. Turning off SACK through /proc fixed the problem. Long description: ----------------- I've been having TCP transfers stall on me for no good reason for a long time, with various versions of the kernel. So I usually use wget for transfers, and I keep hitting Ctrl-C when it stalls, and restart the transfer. Stalls happen about every 50-200K. Today I tracked down what seems to be the problem. The attached tcpdump extract shows a transfer between my machine (my_dialup_ip, which is a dual-processor machine running Linux 2.4.9 SMP) and a web server (web_server). Except for the IPs nothing has been edited. The following sequence happens: - The transfer is proceeding normally, until segment 157498 gets dropped. The next segment, 158946, is received. - My machine ACKs up to 157498, with a SACK block for the 158946 segment. - Two more data segments arrive, creating another hole in the segment space. - My machine now ACKs 157498, with two SACK blocks. - The server's retransmission (to fill the first hole) arrives. - My machine fills the first hole, ACKs up to 161842 with one SACK block (for the second hole). - My machine receives a repeat of the server's retransmission filling the hole at 157498. - My machine now adds a SACK block for 157498, even though it is less than the sequence number in the ACK field. This is definitely wrong. - The server goes crazy retransmitting the 157498 segment, with increasing backoffs. This looks like a bug too, on the server side. (I don't know what OS the server is running.) - The transfer totally stalls at this point. I repeated the transfer many times with SACK turned off, and the stalls went away. Sorry I don't have the time to track this through the source code further. Do let me know if you need further information or testing (I'm not on these lists any more, so please cc: me directly). Thanks, -Vijay 15:10:51.924697 < web_server.www > my_dialup_ip.34186: P 156050:157498(1448) ack 135 win 10136 (DF) 15:10:51.925015 > my_dialup_ip.34186 > web_server.www: . 135:135(0) ack 157498 win 63712 (DF) 15:10:52.004706 < web_server.www > my_dialup_ip.34186: P 158946:160394(1448) ack 135 win 10136 (DF) 15:10:52.004795 > my_dialup_ip.34186 > web_server.www: . 135:135(0) ack 157498 win 63712 (DF) 15:10:52.114694 < web_server.www > my_dialup_ip.34186: P 160394:161842(1448) ack 135 win 10136 (DF) 15:10:52.114769 > my_dialup_ip.34186 > web_server.www: . 135:135(0) ack 157498 win 63712 (DF) 15:10:52.254671 < web_server.www > my_dialup_ip.34186: P 163290:164202(912) ack 135 win 10136 (DF) 15:10:52.254744 > my_dialup_ip.34186 > web_server.www: . 135:135(0) ack 157498 win 63712 (DF) 15:10:52.824666 < web_server.www > my_dialup_ip.34186: . 157498:158946(1448) ack 135 win 10136 (DF) 15:10:52.824746 > my_dialup_ip.34186 > web_server.www: . 135:135(0) ack 161842 win 59368 (DF) 15:10:54.864609 < web_server.www > my_dialup_ip.34186: . 157498:158946(1448) ack 135 win 10136 (DF) 15:10:54.864699 > my_dialup_ip.34186 > web_server.www: . 135:135(0) ack 161842 win 63712 (DF) 15:10:58.914469 < web_server.www > my_dialup_ip.34186: . 157498:158946(1448) ack 135 win 10136 (DF) 15:10:58.914560 > my_dialup_ip.34186 > web_server.www: . 135:135(0) ack 161842 win 63712 (DF) 15:11:07.034200 < web_server.www > my_dialup_ip.34186: . 157498:158946(1448) ack 135 win 10136 (DF) 15:11:07.034287 > my_dialup_ip.34186 > web_server.www: . 135:135(0) ack 161842 win 63712 (DF) 15:11:23.273663 < web_server.www > my_dialup_ip.34186: . 157498:158946(1448) ack 135 win 10136 (DF) 15:11:23.273741 > my_dialup_ip.34186 > web_server.www: . 135:135(0) ack 161842 win 63712 (DF) 15:11:55.792591 < web_server.www > my_dialup_ip.34186: . 157498:158946(1448) ack 135 win 10136 (DF) 15:11:55.792672 > my_dialup_ip.34186 > web_server.www: . 135:135(0) ack 161842 win 63712 (DF) From owner-netdev@oss.sgi.com Tue Oct 2 15:03:04 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f92M34q27825 for netdev-outgoing; Tue, 2 Oct 2001 15:03:04 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f92M30D27820 for ; Tue, 2 Oct 2001 15:03:00 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id SAA02394; Tue, 2 Oct 2001 18:00:05 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Tue, 2 Oct 2001 18:00:05 -0400 (EDT) From: jamal To: Ingo Molnar cc: , , Robert Olsson , Benjamin LaHaise , , Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Ingo Molnar wrote: >> Silencing a specific target cannot be done by IRQ masking, you have to >> ask the controller to shut up. It may be the default "shut up" handler >> is disable_irq but that is non optimal. >this could be done later on, but i think this is out of question for 2.4, >as it needs extensive changes in irq handler and network driver API. This already is done in the current NAPI patch which you should have seen by now. NAPI is backward compatible: It would work just fine with 2.4 and drivers can be upgraded slowly. If theres anything that should make it into 2.4 then NAPI it should be (with some componets from your code that still needs to be proven under different workloads). >> And how do you select max_rate sanely? [...] >> Saying "hey, that's the users problem", is _not_ a solution. It needs >> to have some automatic cut-off that finds the right sustainable rate >> automatically, instead of hardcoding random default values and asking >> the user to know the unknowable. >good point. I did not ignore this problem, i was just unable to find any >solution that felt robust, so i convinced myself that max_rate is the >best idea :-) if you havent taken a look at NAPI please do so instead of creating these nightly brainstorm patches. With all due respect, if you insist on doing that please have the courtesy of at least posting results/numbers of how this improved things and under what workloads and conditions. I do believe that some of the pieces of what you have would help -- in conjunction with NAPI. A scenario where we have an appearing ksoftirqd, then disapearing and then a new kpolld showing up just indicates very bad engineering/juju which seems to be based on pulling tricks out of a hat. Lets work together instead of creating chaos. cheers, jamal From owner-netdev@oss.sgi.com Tue Oct 2 18:13:18 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f931DIb32627 for netdev-outgoing; Tue, 2 Oct 2001 18:13:18 -0700 Received: from osdlab.pdx.osdl.net (air-1.osdlab.org [65.201.151.5]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f931DFD32624 for ; Tue, 2 Oct 2001 18:13:15 -0700 Received: from osdlab.org (dragon.pdx.osdl.net [172.20.1.27]) by osdlab.pdx.osdl.net (8.11.2/8.11.2) with ESMTP id f931D4S11442; Tue, 2 Oct 2001 18:13:04 -0700 Message-ID: <3BBA6556.E27FC534@osdlab.org> Date: Tue, 02 Oct 2001 18:09:42 -0700 From: "Randy.Dunlap" Organization: OSDL X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.4.3-20mdk i686) X-Accept-Language: en MIME-Version: 1.0 To: Andreas Dilger CC: mingo@elte.hu, linux-kernel@vger.kernel.org, linux-net@vger.kernel.org, netdev@oss.sgi.com, jgarzik@mandrakesoft.com Subject: Re: [patch] netconsole-2.4.10-C2 References: <3BB77591.C1349C09@osdlab.org> <3BB77FD7.696CFC58@osdlab.org> <20010930220950.K930@turbolinux.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk Andreas Dilger wrote: > > On Sep 30, 2001 13:25 -0700, Randy.Dunlap wrote: > > I'm interested in using netconsole early (during boot). > > Any problems doing that, other than getting module parameters > > to it? I can fix that part. > > I was thinking about this as well. It should be relatively easy to allow > a line line "console=eth0,XX:XX:XX:XX:XX:XX,a.b.c.d" or similar. The only > slight problem might be in configuring the interface early enough in the > boot process. AFAIK (which isn't much in this area) the serial console > has special "console" code which allows it to be used very early in the > boot process. > > You would obviously need to have the network driver compiled into the kernel > and not a module. Maybe Ingo can hack something where it is possible > to initialize the network code very early in the boot process? Well, I have netconsole-in-kernel (+ AD patches + RD patches) built and parsing parameters, but not getting active nearly soon enough. I'll continue to work on that...hints appreciated/accepted. ~Randy From owner-netdev@oss.sgi.com Wed Oct 3 01:36:55 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f938atJ08602 for netdev-outgoing; Wed, 3 Oct 2001 01:36:55 -0700 Received: from chiara.elte.hu (chiara.elte.hu [157.181.150.200]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f938aqD08598 for ; Wed, 3 Oct 2001 01:36:53 -0700 Received: by chiara.elte.hu (Postfix, from userid 17806) id 5E15E1FCE; Wed, 3 Oct 2001 10:36:46 +0200 (CEST) Date: Wed, 3 Oct 2001 10:34:23 +0200 (CEST) From: Ingo Molnar Reply-To: To: jamal Cc: , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Linus Torvalds , Alan Cox , Simon Kirby Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Tue, 2 Oct 2001, jamal wrote: > [...] please have the courtesy of at least posting results/numbers of > how this improved things and under what workloads and conditions. > [...] 500 MHz PIII UP server, 433 MHz client over a single 100 mbit ethernet using Simon Kirby's udpspam tool to overload the server. Result: 2.4.10 locks up before the patch. 2.4.10 with the first generation irqrate patch applied protects against the lockup (if max_rate is correct), but results in dropped packets. The auto-tuning+polling patch results in a working system and working network, no lockup and no dropped packets. Why this happened and how it happened has been discussed extensively. (the effect of polling-driven networking is just an extra and unintended bonus side-effect.) Ingo From owner-netdev@oss.sgi.com Wed Oct 3 01:41:01 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f938f1n08734 for netdev-outgoing; Wed, 3 Oct 2001 01:41:01 -0700 Received: from chiara.elte.hu (chiara.elte.hu [157.181.150.200]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f938ewD08731 for ; Wed, 3 Oct 2001 01:40:59 -0700 Received: by chiara.elte.hu (Postfix, from userid 17806) id 40E4B1FCE; Wed, 3 Oct 2001 10:40:56 +0200 (CEST) Date: Wed, 3 Oct 2001 10:38:34 +0200 (CEST) From: Ingo Molnar Reply-To: To: jamal Cc: Benjamin LaHaise , , Alexey Kuznetsov , Robert Olsson , Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Tue, 2 Oct 2001, jamal wrote: > You are still missing the point (by humping on the literal meaning of > the example i provide), the point is: fine grained vs shutting down > the whole IRQ. i'm convinced that this is a minor detail. there are *tons* of disadvantages if IRQs are shared. In any high-performance environment, not having enough interrupt sources is a sizing or hw design mistake. You can have up to 200 interrupts even on a PC, using multipe IO-APICs. Any decent server board distributes interrupt sources properly. Shared interrupts are a legacy of the PC design, and we are moving away from it slowly but surely. Especially under gigabit loads there are several PCI busses anyway, so getting non-shared interrupts is not only easy but a necessity as well. There is no law in physics that somehow mandates or prefers the sharing of interrupt vectors: devices are distinct, they use up distinct slots in the board. The PCI bus can get multiple IRQ sources out of a single card, so even multi-controller cards are covered. i fully agree that both the irq code and drivers themselves have to handle shared interrupts correctly, and we should not penalize shared interrupts unnecesserily, but do they have to influence our design decisions too much? Nope. Ingo From owner-netdev@oss.sgi.com Wed Oct 3 02:24:47 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f939Ol809700 for netdev-outgoing; Wed, 3 Oct 2001 02:24:47 -0700 Received: from chiara.elte.hu (chiara.elte.hu [157.181.150.200]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f939OhD09697 for ; Wed, 3 Oct 2001 02:24:44 -0700 Received: by chiara.elte.hu (Postfix, from userid 17806) id 0F6F41FCE; Wed, 3 Oct 2001 11:24:40 +0200 (CEST) Date: Wed, 3 Oct 2001 11:22:18 +0200 (CEST) From: Ingo Molnar Reply-To: To: Ben Greear Cc: Benjamin LaHaise , jamal , , Alexey Kuznetsov , Robert Olsson , Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <3BB956D3.AE0FCC54@candelatech.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Mon, 1 Oct 2001, Ben Greear wrote: > So, when you turn off the IRQs, are the drivers somehow made aware of > this so that they can go into polling mode? That might fix the 10ms > latency/starvation problem that bothers me... the latest, -D9 patch does this. If drivers provide a (backwards compatible) ->poll_controller() call then they can be polled by kpolld. There are also a few points within the networking code that trigger a poll pass, to make sure events are processed even if networking-intensive applications take away all CPU time from kpolld. The device is only polled if the IRQ is detected to be in overload mode. IRQ-overload protection does not depend on the existence of the availability of the ->poll_controller(). The poll_controller() call is very simple for most drivers. (It has to be per-driver, because not all drivers advance their state purely via their device interrupts.) but kpolld itself and auto-mitigation is not limited to networking - any other driver framework that has high-irq-load problems can use it. > I'm more worried about dropped pkts. If you can receive 10k packets > per second, then you can receive (lose) 100 packets in 10ms.... yep - this does not happen anymore, at least under the loads i tested which otherwise choke a purely irq-driven machine. (It will happen in a gradual way if load is increased further, but that is natural.) Ingo From owner-netdev@oss.sgi.com Wed Oct 3 02:41:15 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f939fFc10140 for netdev-outgoing; Wed, 3 Oct 2001 02:41:15 -0700 Received: from chiara.elte.hu (chiara.elte.hu [157.181.150.200]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f939fCD10134 for ; Wed, 3 Oct 2001 02:41:13 -0700 Received: by chiara.elte.hu (Postfix, from userid 17806) id A78A81FCE; Wed, 3 Oct 2001 11:41:08 +0200 (CEST) Date: Wed, 3 Oct 2001 11:38:39 +0200 (CEST) From: Ingo Molnar Reply-To: To: jamal Cc: , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Tue, 2 Oct 2001, jamal wrote: > This already is done in the current NAPI patch which you should have > seen by now. [...] (i searched the web and mailing list archives and havent found it (in fact this is the first mention i saw) - could you give me a link so i can take a look at it? I just found your slides but no link to actual code. Thanks!) but the objectives, judging from the description you gave, are i think largely orthogonal, with some overlapping in the polling part. The polling part of my patch is just a few quick lines here and there and it's not intrusive at all. I needed it to make sure all problems are solved and that the system & network is actually usable in overload situations. you i think are concentrating on router performance (i'd add dedicated networking appliances to the list), using cooperative drivers. I trying to solve a DoS attack against 2.4 boxes, and i'm trying to guarantee the uninterrupted (pun unintended) functioning of the system from the point of the IRQ handler code. Ingo From owner-netdev@oss.sgi.com Wed Oct 3 05:53:15 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93CrFU13873 for netdev-outgoing; Wed, 3 Oct 2001 05:53:15 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93Cr8D13868 for ; Wed, 3 Oct 2001 05:53:08 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id IAA04626; Wed, 3 Oct 2001 08:49:51 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Wed, 3 Oct 2001 08:49:51 -0400 (EDT) From: jamal To: Ingo Molnar cc: , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Linus Torvalds , Alan Cox , Simon Kirby Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, Ingo Molnar wrote: > > On Tue, 2 Oct 2001, jamal wrote: > > > [...] please have the courtesy of at least posting results/numbers of > > how this improved things and under what workloads and conditions. > > [...] > > 500 MHz PIII UP server, 433 MHz client over a single 100 mbit ethernet > using Simon Kirby's udpspam tool to overload the server. Result: 2.4.10 > locks up before the patch. 2.4.10 with the first generation irqrate patch > applied protects against the lockup (if max_rate is correct), but results > in dropped packets. The auto-tuning+polling patch results in a working > system and working network, no lockup and no dropped packets. Why this > happened and how it happened has been discussed extensively. > > (the effect of polling-driven networking is just an extra and unintended > bonus side-effect.) > This is insufficient and, no pun intended, but you must be joking if you intend on putting this patch into the kernel based on these observations. For sample data look at: http://www.cyberus.ca/~hadi/247-res/ Weve been collecting data for about a year and fixing the patchs and we still dont think we cover the full range (hopefully other people will help in that when we merge). You dont need the patch for 2.4 to work against any lockups. And infact i am suprised that you observe _any_ lockups on a PIII which are not observed on my PII. Linux as is, without any tuneups can handle upto about 40000 packets/sec input before you start observing user space startvations. This is about 30Mbps at 64 byte packets; its about 60Mbps at 128 byte packets and comfortable at 100Mbps with byte size of 256. We really dont have a problem at 100Mbps. There are several solutions in 2.4 and i suggest you try those first 1) has been around since 2.1 is hardware flow control. First you need to register callbacks to throttle on/off your device. Typically the xoff() callbacks will involve the driver turning off the receive and receive_nobuf interupt sources and the xon() callback will undo this. The network subsytem observes congestion levels by the size of the backlog queue. it shuts off devices with when its overloaded and unthrottles them when the conditions get better 2) and upgrade to the above introduced in 2.4: Instead of waiting until you get shut off because of an overloaded syatem, you could do something about it... use the return values from netif_rx to make decisions. The return value indicates whether the system is getting congested or not. The value is computed based on a moving window averaging of the backlog queue and so is a pretty good reflection of congestion levels. Typical uses of the return value are to tune the mitigation registers. If the congestion thresholds are approaching a high watermark, you back off and if they indicate things are getting better, you increase you packet rate to the stack. since you seem to be unaware of the above, i would suggest you try them out first. NAPI builds upon the above and introduces a more generic solution. cheers, jamal From owner-netdev@oss.sgi.com Wed Oct 3 06:06:24 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93D6Ot14177 for netdev-outgoing; Wed, 3 Oct 2001 06:06:24 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93D6KD14174 for ; Wed, 3 Oct 2001 06:06:20 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id JAA04686; Wed, 3 Oct 2001 09:03:19 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Wed, 3 Oct 2001 09:03:19 -0400 (EDT) From: jamal To: Ingo Molnar cc: , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, Ingo Molnar wrote: > > On Tue, 2 Oct 2001, jamal wrote: > > > This already is done in the current NAPI patch which you should have > > seen by now. [...] > The paper is at: http://www.cyberus.ca/~hadi/usenix-paper.tgz Robert can point you to the latest patches. > > but the objectives, judging from the description you gave, are i think > largely orthogonal, with some overlapping in the polling part. yes. Weve done a lot of thoroughly thought work in that area and i think it will be a sin to throw it out. > The polling > part of my patch is just a few quick lines here and there and it's not > intrusive at all. NAPI is not intrusive either, it is backward compatible. > I needed it to make sure all problems are solved and > that the system & network is actually usable in overload situations. > And you can; look at my previous email. I would rather patch 2.4 to use NAPI than see your patch in there. > you i think are concentrating on router performance (i'd add dedicated > networking appliances to the list), using cooperative drivers. I trying to > solve a DoS attack against 2.4 boxes, and i'm trying to guarantee the > uninterrupted (pun unintended) functioning of the system from the point of > the IRQ handler code. No. NAPI is for any type of network activities not just for routers or sniffers. It works just fine with servers. What do you see in there that will make it not work with servers? cheers, jamal From owner-netdev@oss.sgi.com Wed Oct 3 06:28:33 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93DSXR14712 for netdev-outgoing; Wed, 3 Oct 2001 06:28:33 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93DSUD14709 for ; Wed, 3 Oct 2001 06:28:30 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id JAA04882; Wed, 3 Oct 2001 09:25:32 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Wed, 3 Oct 2001 09:25:32 -0400 (EDT) From: jamal To: Ingo Molnar cc: , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, jamal wrote: > > > On Wed, 3 Oct 2001, Ingo Molnar wrote: > > > > > but the objectives, judging from the description you gave, are i think > > largely orthogonal, with some overlapping in the polling part. > > yes. Weve done a lot of thoroughly thought work in that area and i think > it will be a sin to throw it out. > I hit the send button to fast.. The dynamic irq limiting (it must not be set by a system admin to conserve the principle of work) could be used as a last resort. The point is, if you are not generating a lot of interupts to begin with (as is the case with NAPI), i dont see the irq rate limiting kicking in at all. Maybe for broken drivers and perhaps for other devices other than those within the network subsystem (i think weve pretty much taken care of the network subsystem). But you must fix the irq sharing issue first and be able to precisely isolate and punish the rude devices. cheers, jamal From owner-netdev@oss.sgi.com Wed Oct 3 06:36:07 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93Da7714843 for netdev-outgoing; Wed, 3 Oct 2001 06:36:07 -0700 Received: from robur.slu.se (robur.slu.se [130.238.98.12]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93Da4D14840 for ; Wed, 3 Oct 2001 06:36:04 -0700 Received: (from robert@localhost) by robur.slu.se (8.8.7/8.8.7) id PAA03723; Wed, 3 Oct 2001 15:38:10 +0200 From: Robert Olsson MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15291.5314.595897.458571@robur.slu.se> Date: Wed, 3 Oct 2001 15:38:10 +0200 To: jamal Cc: Ingo Molnar , , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: References: X-Mailer: VM 6.92 under Emacs 19.34.1 Sender: owner-netdev@oss.sgi.com Precedence: bulk jamal writes: > The paper is at: http://www.cyberus.ca/~hadi/usenix-paper.tgz > Robert can point you to the latest patches. Current code... there are still some parts we like to better. Available via ftp from robur.slu.se:/pub/Linux/net-development/NAPI/ 2.4.10-poll.pat The original code: ANK-NAPI-tulip-only.pat ANK-NAPI-kernel-only.pat And for GIGE there is a e1000 driver in test. Cheers. --ro From owner-netdev@oss.sgi.com Wed Oct 3 07:53:44 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93EriV16104 for netdev-outgoing; Wed, 3 Oct 2001 07:53:44 -0700 Received: from chiara.elte.hu (chiara.elte.hu [157.181.150.200]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93ErfD16101 for ; Wed, 3 Oct 2001 07:53:41 -0700 Received: by chiara.elte.hu (Postfix, from userid 17806) id 8D33B1FCE; Wed, 3 Oct 2001 16:53:24 +0200 (CEST) Date: Wed, 3 Oct 2001 16:51:02 +0200 (CEST) From: Ingo Molnar Reply-To: To: jamal Cc: , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Linus Torvalds , Alan Cox , Simon Kirby Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, jamal wrote: > You dont need the patch for 2.4 to work against any lockups. And > infact i am suprised that you observe _any_ lockups on a PIII which > are not observed on my PII. [...] as mentioned before, it's dead easy to lock up current kernels via high enough networking irq/softirq load: box:~> wc -l udpspam.c 131 udpspam.c box:~> ./udpspam 10.0.3.4 10.0.3.4 is running vanilla 2.4.11-pre2 UP, a 466 MHz PII box with enough RAM, using eepro100. The system effectively locks up - even in the full knowledge of what is happening, i can hardly switch consoles, let alone do anything like ifconfig eth0 down to fix the lockup. Until this kind of load is present the only option is to power-cycle the box. SysRq does not work. (ask Simon for the code.) and frankly, this has been well-known for a long time - it's just since Simon sent me this testcode that i realized how trivial it is. Alexey told me about Linux routers effectively locking up if put under 100 mbit IRQ load more than a year ago, when i first tried to fix softirq latencies. I think if you are doing networking patches then you should be aware of it as well. your refusal to accept this problem as an existing and real problem is really puzzling me. Ingo From owner-netdev@oss.sgi.com Wed Oct 3 07:54:35 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93EsZo16197 for netdev-outgoing; Wed, 3 Oct 2001 07:54:35 -0700 Received: from chiara.elte.hu (chiara.elte.hu [157.181.150.200]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93EsDD16159 for ; Wed, 3 Oct 2001 07:54:13 -0700 Received: by chiara.elte.hu (Postfix, from userid 17806) id A146C1FCE; Wed, 3 Oct 2001 16:54:11 +0200 (CEST) Date: Wed, 3 Oct 2001 16:51:47 +0200 (CEST) From: Ingo Molnar Reply-To: To: Linus Torvalds Cc: , Alan Cox , Arjan van de Ven , Alexey Kuznetsov , Subject: [patch] auto-limiting IRQ load take #2, irq-rewrite-2.4.11-F4 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="8323328-2125670340-1002120707=:7342" Sender: owner-netdev@oss.sgi.com Precedence: bulk This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. Send mail to mime@docserver.cac.washington.edu for more info. --8323328-2125670340-1002120707=:7342 Content-Type: TEXT/PLAIN; charset=US-ASCII the attached patch contains a cleaned up version of IRQ auto-mitigation. - i've removed the max_rate limit and have streamlined the impact of the load-estimator on do_IRQ() to this piece of code: desc->total_contexts++; if (unlikely(in_interrupt())) goto mitigate_irqload; i dont think we can get much cheaper than this. (We could perhaps avoid the total_contexts counter by saving a 'snapshot' of the existing kstat.irqs array of counters in every timer tick and comparing the snapshot to the current kstat.irqs values. That looked pretty unclean though.) - the per-cpu irq counting in -D9 was incorrect as it collapsed all irq handlers into a single counter. - i've removed the net-polling hacks - they are unrelated to this problem. the patch is against 2.4.11-pre2. (the eepro100.c fixes from the -ac tree are already included in -pre2, i only included them in this patch to make patching & testing against 2.4.10 easier.). (i'd like to stress the point again that the goal of this approach is *not* to be nice. This is an airbag mechanizm, it can and will hurt performance. But my box does not lock up anymore.) Ingo --8323328-2125670340-1002120707=:7342 Content-Type: APPLICATION/x-bzip2; name="irq-rewrite-2.4.11-F4.bz2" Content-Transfer-Encoding: BASE64 Content-ID: Content-Description: Content-Disposition: attachment; filename="irq-rewrite-2.4.11-F4.bz2" QlpoOTFBWSZTWbeC/UIAEZXfgHw0e//////v//6/////YDNcdD0AZJ77vr2z Tbeve+ufb2824r7syBbW+7uyyzt33Nvd2tWHu2TnjD43x8z7vT7M+dNDBbPj NdOn16B0rU027rjp0Llz3nvVnusylN92ube+GV3bVBXmrWlG1vbuirNe7rsO xildtqLCbKzbWmnkblojrRpusp3PXor197ehKaQRoAQ1NNJoMgGICZNMmVH6 k/SelP1T9UeU9TaeqfpT0m1NBpoECaFGIZU9ompHqZlGjQAMgAAAAAAaYiaK kybU1PJPKPU8jKeppoA9QaAAAMQHqAABJpJETTIhhNKPKemmmkZI9NMoeo9T IaABoDQAHqBEkIQmmQJ6BNDUejU1PSn6jGpNjIiNPUabUMg0HimgEiIEE0BA mIaAmqeQ9FPU9TynlMahtIaNDTRpoDQ2OLP4y2CyKCyA+RQADx9+vFwRsEEQ wTLgUVf/Epl1lRBCpbBqLY1lRttK2WllmyYQuA0daq/T0YPDhxizdETIKClh gtBpsAgsMMsFEzKW4ZVVTEuY2GCzMwMbUrglyZiIWha1p6wpkx0stiyjhTLi mKWimGZMuTDBqquRWWYqWioZglarGlMaqCuZblEMLC2VIQKKDLu7/R8emmdT SQk0vYKb/0fCe7D6F0PvQ/SN1F0YH9JTaxFnnDZeTzQtoxVLQpLxwKNBFCoi g1hiCinDWYGnRcyosDRgrYFQbaxuQyFMKI4lyzLljXAUtMKYNDLLWDWrXWaN DdWZS5blstaJRbNGXKM1HI3PO6LVsF0kRNRlYYWUWTemODrIjhlQKtzMZiFQ ylmCNKhiYorBVyrlBRxwcmJmDbgXC2phcMAYXFKOWTJv9O78stvbPu0aPWXD WASG2+7jwPb4kNrWdFYvdkM6IIKw8AaoIGKoINaY8FjqN7d4fwFdhRLO2/sz QJYW74jDGH/J4rOCrXatFXZrczL5iwq5rAKlbdZrYdYVMamTC5dsvbQ0wNkq B1MRA2rYjYZHettG2grQ7jmkJBIFFGCL5qmUCpRWmMGtnJm6gkWkCcb36dnr cfEJ4919RHA1Ky9/jZ9AoGG6qHO28Zam5WoUEqh5EoJUEoXVVdUqFHJJlDkk UPnNM+UfM8Mq6hDSbEXngzkqCgo8qeQ7IPlH8uGzMHfcsfv0s15Pr/PkjpHe VmKbBWouvf8xQTYsT4RnIHQSQOsVOo5gj7unT1Vtdpy6+MKw6kenXcK5sV5q TuFMIbxVoYIoxcrLQFmnwuyBJGwxkhoogUGcjJ0WfP65Y6Mss1Bv2IkGpvtw MzlhHN5cp9pJw0iHFDztT2PjtfDqMXwmpbw1jvta4wMN0ePDDrjsgjxyqIuG 7cn2/ZOt2NGDZjBUp9YqmQENNYg4rAYBYb0YL2b++OdfrnrHk8idY1S6cNaq NULRY6zC5qjdaRWjWTLQx+9+izo79Mze1VBjIpFavvKATiuTDjTLLH+XJIFs i7rp7USH9/3qp9V7VtK8fNDmOevRx2eC6I7M1Hf37cgnoaJL8Z6IRkSW0dKX NW5rMYFtUNpMDqT5GKfc3b5eXw9fu4bvN73nt8s4G8LCqAX3axLi1ln2rz2q GFXWrIn2wMEBEQnJkgpJNk0/Mkgd7AwikTPGov15Ud2QVErMMvQS0A4gRH3O Q+mFb8587k10qupT1Pv8ruj84OLFmGHZ+54jhyjLvuW4o/en1CTQjHxwKi7/ Z4+bot4I4iH63aFtw/OU/Xa2t7jdqPrNKCStRkjOLD+HR0qhyfyIh+yGkkmL CVIBemfuprYSs72ZnydvdfgIqtttt+LI3ApUqVmFri4539/4zDTu0aO7XEco KHj9Usgkyosh7epHjj4QC47HCB9Pcj2zO6tEFXPxZ73RieyrgRo+TGxPLYfy whD8nqyZ51lVD0y1a6XZ5riczScPGcRdBc0twv++nNSQFoDtPR4mQYWiri4g zVP9/hmL7cQPBykemiEPC20V2kPXfMyQKMpYSIPBjwqpLVP3YqZQmL3eTfxf rOW08z7vuEPEiKdibk8UrP12YaV9JIQ7D7ffsPoOM8trwPmO8Co/AjUeHCWN MYWOZvBaZ1MI3ft7Pj+4t59j0TKwxU/EZlaMbDAfXgGGhyazogckocp45Rao S6lLW1PPoBrBi3i1+nLnNbfghez5hA5dWY7XPDrdBNvualcfUP0Vn3/xVgQs pWuJJYKZI1Ra45TyNpiRJYD/3vw6WC+TGmB5Z1GDNnVn8qyANjJVhBW6Z91f bWUKzTZ3eddvu+TvQEDKJCVHyX0iYoGiUaUgsTM3Iow1fTdAd7s8XBxF/X1h H3dtdyCGlt6vRYTkiiPMwdktgR6TpOoDxHVr/KGukxPVUZ3WMD4SyKApKntj troDkH7idvl3aLyZrc6xmQEmA6SpRr0BYR1CMUasl6+UHmb6btlRZg7ax57e NFWSKi4wh0oiWoSJBNVQaNjg5wKi+8nMmmxC+rjvOnvdZVn4jp38I3FRl+EH K9+FSpJPNVhEP30sFbqw25f15S1BYHw7/1O2pI1xNR5mxzSnuDR/g+1mt7Jz 9+saZhWdaD+NbOvvDMsadCt9yJngpA9HlrMcVgqn8fs1nKyYZF8kcygm/8uC SOpXqxqFj5afZIIjWtF6nOV2MVKj1BY/L70HLrdpGZxdckXsrD3OwRw8I2Ep N4lOdw7fkrrTlLGIuqu+bRcexm8ITLO6YQ7abZycZjgpEda8a+FkRyhxl9um 6TebZXVkoMpI2ki4i0Y585cVDpIyuq7fFfs/rXRmM4mWNmwaG/Fi4DXGZPAo 8XnVOGzac2/AlMRYFZDswDG4fzDD4duIrFOhWgMTi0JV19zHmBfKKgX1VJDO Otb40+qhRs7H2Suvp3b8zu+Yq0E7DSYRCcECIBpDorTRiDiImTg8GkhrNkKq Pf9H+OV5PXJY4BuRqTHm37d9tncWBtNj1odhLzrJ5DGPOEQK+TwKqIApf2/8 HhNPBurC5/S6obj2dz15T49+88L0Yfizbx27erlnho8feqqqqqqqqudZzD4z sw7wjsYb+feNWmY+nGOwRmZVu7rP1eTZw0OBvs29r17ffhWcacI0j7OPhYwR WcboS6GV6zT10NPGlcJjPyQQpbjpPAfMhRagYNhgLAeg4zfsE+OnaIjWhTjo 9rXannfE447qDiY93H3Nh47N9c21fNHLlQXpjuzezx+TjNC9Bh5byMQdNH0c DsaWZ5VtiQFs/tILjSydceqVpVN2RIbm2/bYkPSgHsWgrRQvGo269Ym4t7e7 DfIZEiUwFBEY+0rzrBheV1KpGLQItOI4zU3xJZnQR0lY+gMNx8aY6PLcKkta XNzUoGiE1VNSJEqFKS8d0qNecemz9Lc9t72gqlCSm3d5UfAocFWrFa6LtVcW VJKka411+pWX9MPpTtq7Y8ujsZQyO+91Ti2ib1rRayONJUiaQiRjXLxQ4GnQ Q0DlL3dMn69/gRPkNIYsnqPN69t8t1Ovp/XknQ5SInSzcGJlpaY2c+QPZ5B3 LOOcu7zps4bDcRWqXjpTw+NJ04WyX91o8jflp2C+PS0wC7oEdrlgatEIH0dQ wfgfCgVFtXFE0EfqOh6vu+zzcYyJ1/yHlCqRqBzP8/E3GH3/KPrO5T29eQ9/ Cc7ZUEqs+idGT8wbB+Rcu7TFKNpG1S2wqKVW1GlYVWKPq/D9GA/yLB9mbgnT rCRDYHiDQSCffMKS3TRQkXm5fy+ItdKNn9cluh9tKnAhm18nLQUqMdPiOUng 0CjHzYpFCp373OqkhQebOvPfEUHHF97rnT9Bm+BoZuM3Tp7kh871noLKw2/+ 6YnP+nq2wlhbg7mL4MZAdDQm3u5FC6PQjnnQV97bR3NRkWXsWTDieZViMywP Jaoh+8PqkIvvp9piqZ2JFs02oRRnbNfq1R28Ktlyg1LDcUj+HsPiw5oeoNw0 XSmWrelu30nKw7B8LmbnSn+fEFrvojHX3kltVv3G9Gati6q0Zs3G0WVUES31 zmGwpO6doWuzhs1t3yY4d3M58jXtjL1UK0+FL6S5uA+1S/KaDY/micIeVIoO eWcYLKr8xc/F2tun3Y+/WkSQh3TaYHBMdfN9OubG/Q0tuud3vxIEIj6StKLX 0j59D8FgYX4+W2l22ZDPKcJsem438llO0buOZf7jLnwwtt1suy4nJXjSp6s9 FXl34S63PDl5Kv+IRYijmhWp6KMgcPFBwQDRPQZpw8re63LHq4azb9rSLRC2 EenWglxJKxZnp8LPHgeomflcUuiuKN18TrXoRcXhjC3src6sc5efCnDwxI+Y lmbq2pV3YQ7j1QxkRjU+g673Ftld/nPHGEQ/LQ+GnCnnW+ey+dzPwZj0U+9a 2o48VG2a27mffk+5MY8IHCJwKaMJF94HaODerbabEVk6d2nWYFM17I7y2o23 4muiT1brniXmIm101VRFpoqS/VzbLo2RmVvopI4rpFSi0CZhlhuxFTKvk4Sw zC/IOu3SwH17SZ0CYlbELahaWCiqqwYrEEFBiDBgyIyI41gjERZEGKDJI4gv c2CkRLMkFYRIRAwERPsBUZASeuAEF/CytDnBe445OnAvby+jcdOFPA8eMF54 oG0cIdkUToNOvwKPLbyfWfw68j/KFJJG7s8e2fz3mLeRd9m57fEzdbuXg0h2 cJ+ATt+vGZmQgM8S5qrWs6bSyiU8WMHu88efJ/jlLQzAm5rbtulXvPCNwIWZ /7PM51eGyZb5JOQNOy+DamyZrWrApAERneY6zhFSBSrrF0+LZQSK+LErMpYF hvQxYmO6PIjZ08fLxPtDmHVKGJqgO2JKyUr+WxWq0XQnKms6zl1kq1k1TFG4 rWZmbyGM4l5xWU5sVnWlJVUtLSl41Uu5U5ObQocXciBAAHdEfqFhQDQB5Shb PvlLAT5RiKe7s4eRIBPgeSlVD6u8tRUNrwsCaRWahdpQ8KgB6CaUJKe6qVxK QaGihU8+qNg9ZyUfJbyo/T7uQXR8gMytSgY6WSCGLJiU8NN1p6aRSK8nZ101 ZvfqLGpoWBv+Tf9d19mLUjxagHcuJGFjYoOSD13H0BWvfp0VGKu8AOmpGV8l YMUObPwe3DtsvDEkBPc7kjIQoP0r7hEOcPn+Ayq8oSiqKPyoD4FqGTGPwtlD BJAjEgYxVLwPX7Oo2H24+X/D4nsA+WniOCUfxnjLIG3lNYPkUsSYcIFn4Brk M/EVHQftpQ2LbWkcqV7PvD/Xro3VtS7jE/eRNJAAtAd+JlwyKBL1w4QOz1F0 naKytzbcBUusoJRkWJBoS3kfrK0mEJSR6nGsUJVaV2VFn9OKV24F5rlXOWT/ XhKWNi/VMskZAJVtnjMGeJKBkvEx2m2wGtpsz2vzXZEbIFD5tsTl27wAThDa GTmbh6pEgmRuVMkcAKPRIWRv1IEzlkzNGY9oCoS1WJEVFpDanOIw2C7x1zeM ZEWkYPuwUq2wZtCfcd4UCneYN0g9Uu4c6wv5pXQ7e/u7c5JxaL9LQAGesFOm 90IiiRSXiKVTrEWiMDuVZFWPFjmkQocBQ2iEgJYTwUZyEEcxjDxXtqKZXcwH jW37O6423tsrIagD+XMYffHgGyLaOh4aXOtJOT8YGFUKq8a73HkTEevft27l R8aQhJIXDA406CNNQ8AlUXDBULREGqb/UjeOO1Oi62G+4WYuVbXVem+Q95DO 6BFVgWq8RCiCFK0h7F7RFeBZQMN7GGGh7QBLGZ5uN0y77TLS11Q1nNiuBAbX 7HrD0hdlWji5AVcYRiiByCUGGhkAchBtEWnUl3s9ZDvIuKOHRpo/YrXQlA4B QC42cG07uMSoa4Wx/KmVZbKwuXCon8y2BopBijujwrXLGVwqi1IUS2ooLNoL 6wKTIBK/GnUff0eLcRXfjvaHmbT4zv3mCYluUNY9XXgZiKnZttTiNtHhYDWK VfJVEz08UOjdrKQFSD1IasQkdZRens9cj4IrQOKqM+b2ysFYj0daUGvZdmp/ QmPapLoTYaP7ZzOaItXDNBFNs1e234wnzP/OkocEMxP+HGa+GUG70HSF6XbP bhA/RSyjhv1rKuVBejcr/hOEGycHWRoX5qnj582Y5ltHQeRxykb25zwI14ns q/HpVuwiQQilYhhdnW0U15Tce8f0efEoz/VBZMxtBIzg9iu9Lm5B35X1xZBm 5XW5/yuY2uVCPwQ83Z8KKBmoV8cAdcUnbNIUe4ccf6SDjW9Y3J+wi/sUxR1j wlVnM+55Pb736z4xhJzUGyK/V/zQBR+4lrCVAP+CGZQylrFJIAh/msn9Wr/f x/hkmPzKUHFH4ACHl/Dov+AqsVX/JVftev98fvx+38T/b6TB75YvgLpYqPHx Oo+4AsfwJ/fRFLdpt8iVQPw2RkH2Hy7LeNGpwP7lS6SZl+wsr/DB8hrXceUU oiaqsoLpmflGCpixmIh8hE1A7KB2CWA5xsPH+BHgOSgzzNI3ipB1swZNdMBA mHVC3eaiuXLoi3BF+H+uisGXRvpcQesZVQFRDCEKW7uQu6ZuITxxDQLK9GYP y4tmsSaBTFwo12r6OcDRUEw3ww5BuY2feRD0SuWLnWo0lpO5+rrkDfF+m1lC xPxwQgdMeAt2BREuTszIziE2Bkj/GVAdXsJvitbwcugSUO5aIX4USqLpmcgO 8Y79XLDxEQu4qkdRuMFaaHYceIuLCrXnGLdAMCU/VhagKP1U+N2gqdc5ge4+ Uncejv4naC5AxYzSIpYcGqgAg6d6PANGmwN0FaJMoCDUgt5mmgcXZBo6kNIF C5gVkPrwHMVbRX25H7uCmuFVejxSUe3IrPV5oOuiW/xLJXfLYg7MgsCZtcOr v0N4TzLJFMYA2LNA0jlvMF2Y2suQskmioSB3UAg2FeuZQG/S+TaHr3mVuaQd IXRJPC7ANWHwOKS5GEw1wWDQANUr9FKBtG4wGjm/EqVMWR07Elz8qbLTmVNU Blqff0Ov6MlzQhHA3GNYOAeZQdJPOXu1REGYIlnCuRq1FBYunINMBwfTHX5s ko8lQfvFI9Az9uGOWcWqJPrXAZMjOAg64RJahKQVKGqlCotaEpJ9zNVdDHW6 vUD5eJI3ouGsSTcrgTiJgbQ89gOYO7eciHMobGZKLNanINgR1JJhIeTXQYfm ZAmtUHBj2B7djB18C8vHBbkQYcoBaWmFDHbCtkEIrKGrCGemaFrJKlYFQ5mj GMfvkxqkGdw0i0+FofYH4/bAcZCR2vxIAyPf8q/g6PwmN4KcX7P9wui30d/9 fiofDt/L2d5sctdn+I1/1QUSEdgSsQeqMFwRqPvIVQsnnzSu8DZBPa+nvGus 0wYPV8mfiNhciG+06SjzboG+xvTRKaIHR5xrGi11k2Q3sNv/dQNmx1Urgf0v IxrEzYl0PUekJMZmOhutJFhmRC5gwlQLlIBqB9C6OyRCwVWg1XOMlpwOK/Yn wo8vr8Cj6T2pDQ8pQyxP4sPxi2GMZCEIifmYVNktYJaiIiVolUKWdYHOzcwf MTsvn1eUihIQGgGGaapjN2Igq02s+iBhv8wbMV4GpOfCIWbUY5w6mKon0kG7 SksgJiJmZmxomMG/+UsxLruv+tkvgG8VBBtuCsIyJyqKwNbmFypVfWt1NrIu 5uNsG2rp8A+zNP0U3DvxDeiZOTQc7ECR+3KdFKzXeb+PR3V0OC+p/fC0C8HG aZMVsmstBkgtlQhYRQurlQaIRkMPiXGS0WNgyIxjlHBSUWDoYgnPcBP52BlU EabySiggQh2TiCuSZjIfHJ3XXzAagZtbXpQTcZS6ZC3d2EGaXO23RKevuz7H hFYFkN6SoDrYA78Smuo09kk00EMb6KUuypTgMXYbbQ+MKg5TSUIgzh70YJ6F UIslQ3AU2CK22wLbuMklxpx2g2krmGz3qRZokTNcid70IZBeIbbEJJJCHClp Rq+CbS/W1JFg2qqLkDh/ZkmGmVPYJNtUD0Tu9B5CNOqzFiDMpZBBiMREVFRE RXIYd67ngBaZgS5Ux+a1IsJR9Z/pKIGWbQgw9cGNWy3OhMygSEYJrKFzsHCg DLyHINdIrvet6qK5lHsXp4gqiLXVwuSuolIaL20IdxAcmFpcKN/QgCtNhVCQ hdX8z1dIGxFkxsCJ74uUB5GtKIgzFvokpscDiGlFQMWnQgSuwzgS381c/SmZ K6YQBo4rgy36abGQZGDZC+iSJZlmz1hxe+nIDgCqqP0NEGb0IE/XPMB+hg0i 8pbkIsN2NxoJUVINRCTcAYKUIBskeGB6nhjSBWAsUGYalC9fQ25RkkLJFHeb 01nHrGTe3sosJhEQM2mZwQJgNjGLVMbMYfbBx86s5T9BNNYESDIT/LaJGHqh BwPloS5iBgn3vjVeTSQFT11pT3nt/NX3ai84Z0RlFPUtci4HgnNYKG6qWLpc KYSVRBNIwkNiaCpklQiIPtL3HAmwDKbIatpExiKhDW8mTRIGIdkTtjGZM5Q4 5XniJe0qAmJkkagG0OX8M72tr6kzgNCz2dR2IVxHaDNzN60QFgMKhcz8UJDN h34czSaSGWUhWExAOByHkFsLO0JOZC5fKGGFs9GerwHWKbopKIwfTSs6gupR JuuTcKFWJXyFVKQfN7IBk3gBoStICRliFSjf8KYCTEiVP3gCtzAOaMQFEzDD izIecf5ZCeIAdoxCTNELi2mIGxXpDoY1z0b36ZKxF8zmyFakWFGTrmYswSFp DPDR8QhNCkE0zHBkIJlIR/5y5ZuwczI5thDUYvy4bSOKyMBfEEuDhLF3PPDd yKnamcSiU1JaKxw0BkjWIDDBKIWROZgCoijAqxkSBSJRUUe8yN/v2XmPD0C4 9IqpAu1zD9HnOKB0qdCRAmWQnGcxyHqI7PyBzIEkIiwO5hRrNemSTeD7TTVG ZxUcggtB6ioT4JGgph0RS5V34BPdsFh37xObTVqqgTdgVMOD2G11rx46YUYj Z2wiydj2pJ2GPTELuRQG0ooIRmqccdNUIsB4ZPdmW/wiDtQnCfKSMDFfxSUc xzlVrYPNXALa+xs3g5kqeNeO6+LvLuo4koqt3TcqChzdNcQm1aW0Gd6qbnnR 3th6OiQaIDFgqKOipm2bNcjXETQOID2DoCtFCkGMMGJpRZH6GGbxJuOJE1ey Pcg6XlngZBUGmtstCQNVXOq4uXtGZFGUYlbUksILoqoM1M2g1LSWxSGFc4i4 xtdE3BQMMSM7RGWLGtxm4F+DAst3Hjlcwk1e7N2ob6JKiE0eLRhyaQQPXgWG FA1yg1nfkSGVtYijWgBAKARVYSZdoILhUmkI5gAOk5hNWlA8UFKJLixlyQpc lIgEtKCsA8982XIM2JohENsA0MK8IG8umA4xiQEJuNWHUE+ch/iR0KzN6sLN 1uoxOq9LYQKmsRF9iIxAYSBLrtV3K2sCyDTKlW99hcQMCDAL794VLt4poru2 +07idHDXvAsRAa6Ujrih/fbdfkglizR4H9UnwzwB65lkczsrHiehrBJ2levu Bo0Dzomw6KbM9DXXs2cCah6oci5xKRPR0aE4aWFahsmHw9hIj36sriT0lDI6 dZwNTcdYmiEeEN18O3n7M5qx+kHEcKUNw0WFY3OhQiAlXSChlkYXVkYzy0hS hHgI5HuaXl3RUBcBtXQZd/NKNqEN8Len3RbGWG6lDHj0yjSndrpyEgKsyg8h ymg31W8ctAbwNCURlNiiR69WxuPnjK4QaSklIJIREqcLKtRQ2XTbvKC5Vlnl KMhLZoPNuBdi1QgMeO9Skr0NinlEdgHnYxiawREZ6lFEySbew8SSGCEnpwqj VrGS0LGVv5qi3XxaPZ9aDrEMBcNQ6WLIccwhc5Vw+ABEzMjcUneHqvf1QCbg DT3mhUFgogX56HUHVTehEMZpe1kIcKaB4xEcVaOjxFJs1yCWSRsmMYJBDAwO Nmid8gX37RZYGEJtJop7CgRNZqzTQT2TGbk5FEZBJCLAu/0Ad3uW0s6X8DcU vsE7bNwNTR6ExS6PpdV47sc5YTQQ9J05hcYPuiEiD0vCDeZol4BDEPHBV2Ww XxQp/jdDBkRNmMiEsdn0RRNJ6JOPae6nMMCmzouFmEfPijw6q7w5Ghx3mTuW gi6kIJTESI+OVjZzUbyvPdVNSE3kdOyEilYOyuXn86ISAfHAWlIsorPirjwh qDITznbACDARC+AZju00aBBT8gcqFA5JLMlG6ebUpgckFAgiQiT7O+zY+OZw A4wUNTXj5ec9ptfsuYQjdqHx51aGzr7yNkGIIS79N9tDok7QDTLJKoB2khFI DrgFA0WJw65whIl/H1uORETPXs8IG4NDag78QACUu6owYCtm47MOPOSduKAn ZHhC9LD0NGgevunRoLsbTEUpUPFoMBZDI4gHBkNeaxffxpOw3EpcCR5l/Ybl UtYvlz7vNWRu1lFETp1CRkUh6wYFQUKkUrty8H0nKFPT6uf5Eh6ehyE8Wc0V cLGJAYIIyIiJVh22gq1aUj54xOFKVP1TX1qm32QEShNODVJmUBqOphKbtUUj lFqgKW2FCLMhYgFksKBy+bfad0N0FUFCEhIwDB9kBfDeo461A6epOK/THJld 9TthzEpEtqBlGN5PuEbi8zRHc/dUL65kZ/TOaNmkaBuaiPVFMiOf4szSpjL2 EqmE4VKREKKqkEaDCLbKsFkRmq2xIwUNho3U+Vtoms9yDSq7u0vdiQKkLWjZ b0Td/TUmhzSrpnk7rwocSFg5vTmmkGIKUmB4Q3QcAQxUIXa6BWGNmyB6XPMd j+q9WNDW6gfoorPwI7tgVFiqYlRiAjQQyyilLvr9m0XU7bme0p5w1WddOLUG J+bOAg7BkEGa20NhFTi/e3BAwVAOemg3Rd5bsf2pigD5EpKlllz4fApShTD2 DPqNpsTM8PMiAn6fTPMmozVO4NrMQw4exH7V/CYHQ++drmFMgTwO/Z0KonsB JfQLA1jlGO2eNzNDJxSHEVUlQYw9Z7XtnnS+qFXcu2Bsq5AZpNAht+plZlal rqnjF3eNRdyub3cgWHDa/d67Gd/Pf2CJazJzO5daoitw7jap3alI1pI6OENl DHQolSjWrLHWI9SYFrdi54gyqZHx5zQQXjYIQeVOHqDKQ+fCnmYRBLOqGhs0 SGjontNuLlYsnfBSXbIVKhOBmOLNKZBzIYnz6hrz6K4Ab019wHlhpIgkN0D4 xFDE4KixNwdpnGWBRmwYAhKEkDu8/T6vOWrW8PZNNm05u0TIJURQMWsLC1kc Q9xYawwNa0SGkkZJ/QO5qAIkgsIrEDicyEChElDmrYR58rFOwKBD75JIPhGA GMMy4mJoG0igKsFQiMAVAZD7RlzXL74z6MkXfnS1Bc2UpIQsFbNOosbyxq9j Th6+4leDKG0NmrSO5NfENNDUdtqMohAMDsKHizUlLCEWEDaIYBgyHuyUlZhE JeJli0TngRgOLolNNLS4I0FIzHJOR0z9ppmpAnl7Rh94Vh3pFkKnLLDlI5RH 352j24tyjcSzGzBJ3AQWFwQgCQAiI9nfzwkvtzOMA3hdd9gNwiOs9wgmfqKe KtA+YvQmZyocaG5A2GYNokURXkMhzSTzTuoGJsbCJtOB2AI41XtuVAR73ruw stg3EFYz9bCKxTAZHUkUZUeJI+u8KYgZ6Y1SPd+XtNanLCCSVHxEs8GqK2nt +9NU+400WVTheMzwY23zWyUSCm38TPHJ7M2QdvAQit+YKi7+OHYkDNdvQf5O h+K1y9O9Zj4B9rIG/Z7+9HzUGnwHl595JMzWfwYTgwWQQ0Gp8x9IccOvrPoT q/Ym0yHgPUWiKtrK0xIRebiNMDsiUMSSQaJQ5cHiNXjDa0lwhhAqFsawGoEr 9st/XjqW470By1AK7oDQouJnu3R9b0+bMlsWSQdWrMyKGq0SyvxFUKEPmIFB GEIRSer5qTV1psALOiWNLq5CEBq0ez2mplo9BegBCDGpq1csYcGGrrj5rWvP IGBiQ/Wl+Dp1SpWFYtps5jyzMoawzBZo6qVnTixYUwpREjgEaHBG2m0ycZBo BWG4UpmGQEFMIzLXQ1Eh9cN+0ue6ajT6rZSyxkCYGgxpwKZTH5LZWuayA7BT EPk6gWekQ9c+EOxGMb0ebXIyFZDXqN7nnxIpmUEsfRXQ/jIMC5bCxEOqk8Cy G3JS8fsjez6+PbDqrgLXqMi2ZZov0SMG3UfN4iKnUclph7arxoqC0BUTGNgD ExEheVgQldpMOpB+zr4rXBv8FRZbAQgdCHZ0JOwwICdlk0YREjUsiYCSLlEu SjQBVLa2VTzXh4ApZl1ImxENB+jLS3pgqMtqkSehz+vaT31g8fUryZjDIbU+ w4S8UeYUlbqwmK1DoKjLsrKlCrqom6sMkaKthFKpQoRDiFfCyGnQJNEFwXLM Id8OIKBWYmFS2pTWhj13xjOc3JefkOqBhQp82pAy8QEaTaabAbBaEksunVRQ L0hIaoSXyNB5BNQmPUWKCnuqdDZrsCImJ3Ug3DJSFRWve0aBc8Clw43q5IaY QrmSm1mnc8M3n4sMYgGHBqG2u5z7QkRkkR/n6tWNuERahsOUthJkEXq3lBoo EZIZjKRx5/kq1TQO03jIMmYrsh55BPRAdVCuglLwDL9zT5On05rhXDOtkAxY ZIu2SeEBLlMcxCbvuQeKIwnl4CpI9PIlM66XZBSRA4HrSeYMhyycDvoVPjZo Qg2pErH0D84Uo9o4qK4K2rSpSliN15Jv+XvoHHfpIS7IotV48OI2wqOhDEZQ khowDC5CgBxChMZ+lTsxuUUAOeVF5sqEmSL8jrkdIMo1o+aeZ70he8ZoTSH6 8xSVTYgmtaB+NiyqQwyNmHgMSwQkgmYpizi5CdClKWNfVFymlXKSrSVC05BR QO1YU1gA3DsnM6bFrWgJsF6kyCIIMdl40RMIalJYIYpegAUMcpeKxqWMlcig zWI1mI+vLI+0C+uZRIwBvyiZRChPmMvxShkY2dKsGXHCsQREKCAaLWqHCwVR 5CcLn5iYS9WhJZLWdMBr6gqXA06tFgpqp7huo73wwd3IYWUUYCSrpXkU0FVG rgRkMCYIEA+aklFKKXIIfa6blQ0YmYRCEZsBMED9ljbZ7X3JVQGobCIiiKDO +kjZaw7oxZcJRTmsXeclXGmOSoBlyG1m1KUqBviVCAVhQEm9N3Okzpluszlv hnrEDfzQgoFS+l8bDbiwCdTi01dskQirRTd88wxUYml9szXppJuN+aSRHWul cgsZInYwsHppmwZ8Qe8pnPwnL2+rp0rJ8XYbEiIKjJFgdpfxggTMWSyx4Sa1 0wiaMQckMQ2whkAZIHyvPY3IjhyRbcXtupv0RkhJDQlkG7f3G5bktqXKBhYj SbFZEiIdgK9+hNKi63EuAoIDYF499tum9IVxDuikp2blT1xFbB2woUeXwnr+ jUcoKnSi7dcrcfpGBpanOHBZ7GKAPsZvcg0D3xhrELAbzwYsOPy+wgzoMNBj bpoJsQ/bbu6kb5n3TgYIsITKEgVVEGhoiV+kpZGZDcCFEeQ7/Lc6vKXDuOw4 SQ8dR3wHrMaHdAH4ZI0jlzlVAdlVkWXfvpvr190TyaPW659VFRSzttZdqqco AHfE+I5joN5ixJnXbUqI1OUfbmJCKlgNTl5M2nB1cbdbm3drCTQlJvBJPxfX W3kwuXMwy22IVAkS0brRqePQ8k+eGWeqLn0tBWcaVLSwFBRpoZw7IFTRXkOp ELmML5ePD2nhvROfxVT+4csvIazXGLJcN0EXrvcg9dUXyyBsxRDPM06DKS3J SIRqdTICeggosTKcYUEMb6TpF6nCaRvEBMyeIajbiDSIMXnc2ZRmYgdZIEF6 oHXCQjIRaZCP8iMRkPwsmAhsqAWoAxhXaJGVthaP4kcRH/8XckU4UJC3gv1C --8323328-2125670340-1002120707=:7342-- From owner-netdev@oss.sgi.com Wed Oct 3 08:01:12 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93F1CP16367 for netdev-outgoing; Wed, 3 Oct 2001 08:01:12 -0700 Received: from moonstone (zaqd37ca49c.zaq.ne.jp [211.124.164.156]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93F0lD16351 for ; Wed, 3 Oct 2001 08:00:48 -0700 Received: from localhost (localhost [127.0.0.1]) by moonstone (Postfix) with ESMTP id 189ED29B; Thu, 4 Oct 2001 00:00:21 +0900 (JST) Received: by moonstone (Postfix, from userid 1000) id E48EF2510; Thu, 4 Oct 2001 00:00:17 +0900 (JST) Delivered-To: rei@oucc.org Received: from orochi.oucc.org [210.199.215.5] by localhost with POP3 (fetchmail-5.9.3) for rei@localhost (single-drop); Thu, 04 Oct 2001 00:00:16 +0900 (JST) Received: (qmail 13931 invoked from network); 3 Oct 2001 14:56:02 -0000 Received: from vger.kernel.org (199.183.24.194) by orochi.oucc.org with SMTP; 3 Oct 2001 14:56:02 -0000 Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id ; Wed, 3 Oct 2001 10:54:04 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id ; Wed, 3 Oct 2001 10:53:54 -0400 Received: from chiara.elte.hu ([157.181.150.200]:43784 "HELO chiara.elte.hu") by vger.kernel.org with SMTP id ; Wed, 3 Oct 2001 10:53:44 -0400 Received: by chiara.elte.hu (Postfix, from userid 17806) id A146C1FCE; Wed, 3 Oct 2001 16:54:11 +0200 (CEST) Date: Wed, 3 Oct 2001 16:51:47 +0200 (CEST) Old-From: Ingo Molnar Reply-To: Old-To: Linus Torvalds Cc: , Alan Cox , Arjan van de Ven , Alexey Kuznetsov , Subject: [patch] auto-limiting IRQ load take #2, irq-rewrite-2.4.11-F4 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="8323328-2125670340-1002120707=:7342" X-Mailing-List: linux-kernel@vger.kernel.org X-Loop: takahashi@future-s.com From: takahashi@future-s.com To: rei@rei.sh X-Virus-Scanned: by AMaViS perl-11 Sender: owner-netdev@oss.sgi.com Precedence: bulk This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. Send mail to mime@docserver.cac.washington.edu for more info. --8323328-2125670340-1002120707=:7342 Content-Type: TEXT/PLAIN; charset=US-ASCII the attached patch contains a cleaned up version of IRQ auto-mitigation. - i've removed the max_rate limit and have streamlined the impact of the load-estimator on do_IRQ() to this piece of code: desc->total_contexts++; if (unlikely(in_interrupt())) goto mitigate_irqload; i dont think we can get much cheaper than this. (We could perhaps avoid the total_contexts counter by saving a 'snapshot' of the existing kstat.irqs array of counters in every timer tick and comparing the snapshot to the current kstat.irqs values. That looked pretty unclean though.) - the per-cpu irq counting in -D9 was incorrect as it collapsed all irq handlers into a single counter. - i've removed the net-polling hacks - they are unrelated to this problem. the patch is against 2.4.11-pre2. (the eepro100.c fixes from the -ac tree are already included in -pre2, i only included them in this patch to make patching & testing against 2.4.10 easier.). (i'd like to stress the point again that the goal of this approach is *not* to be nice. This is an airbag mechanizm, it can and will hurt performance. But my box does not lock up anymore.) Ingo --8323328-2125670340-1002120707=:7342 Content-Type: APPLICATION/x-bzip2; name="irq-rewrite-2.4.11-F4.bz2" Content-Transfer-Encoding: BASE64 Content-ID: Content-Description: Content-Disposition: attachment; filename="irq-rewrite-2.4.11-F4.bz2" QlpoOTFBWSZTWbeC/UIAEZXfgHw0e//////v//6/////YDNcdD0AZJ77vr2z Tbeve+ufb2824r7syBbW+7uyyzt33Nvd2tWHu2TnjD43x8z7vT7M+dNDBbPj NdOn16B0rU027rjp0Llz3nvVnusylN92ube+GV3bVBXmrWlG1vbuirNe7rsO xildtqLCbKzbWmnkblojrRpusp3PXor197ehKaQRoAQ1NNJoMgGICZNMmVH6 k/SelP1T9UeU9TaeqfpT0m1NBpoECaFGIZU9ompHqZlGjQAMgAAAAAAaYiaK kybU1PJPKPU8jKeppoA9QaAAAMQHqAABJpJETTIhhNKPKemmmkZI9NMoeo9T IaABoDQAHqBEkIQmmQJ6BNDUejU1PSn6jGpNjIiNPUabUMg0HimgEiIEE0BA mIaAmqeQ9FPU9TynlMahtIaNDTRpoDQ2OLP4y2CyKCyA+RQADx9+vFwRsEEQ wTLgUVf/Epl1lRBCpbBqLY1lRttK2WllmyYQuA0daq/T0YPDhxizdETIKClh gtBpsAgsMMsFEzKW4ZVVTEuY2GCzMwMbUrglyZiIWha1p6wpkx0stiyjhTLi mKWimGZMuTDBqquRWWYqWioZglarGlMaqCuZblEMLC2VIQKKDLu7/R8emmdT SQk0vYKb/0fCe7D6F0PvQ/SN1F0YH9JTaxFnnDZeTzQtoxVLQpLxwKNBFCoi g1hiCinDWYGnRcyosDRgrYFQbaxuQyFMKI4lyzLljXAUtMKYNDLLWDWrXWaN DdWZS5blstaJRbNGXKM1HI3PO6LVsF0kRNRlYYWUWTemODrIjhlQKtzMZiFQ ylmCNKhiYorBVyrlBRxwcmJmDbgXC2phcMAYXFKOWTJv9O78stvbPu0aPWXD WASG2+7jwPb4kNrWdFYvdkM6IIKw8AaoIGKoINaY8FjqN7d4fwFdhRLO2/sz QJYW74jDGH/J4rOCrXatFXZrczL5iwq5rAKlbdZrYdYVMamTC5dsvbQ0wNkq B1MRA2rYjYZHettG2grQ7jmkJBIFFGCL5qmUCpRWmMGtnJm6gkWkCcb36dnr cfEJ4919RHA1Ky9/jZ9AoGG6qHO28Zam5WoUEqh5EoJUEoXVVdUqFHJJlDkk UPnNM+UfM8Mq6hDSbEXngzkqCgo8qeQ7IPlH8uGzMHfcsfv0s15Pr/PkjpHe VmKbBWouvf8xQTYsT4RnIHQSQOsVOo5gj7unT1Vtdpy6+MKw6kenXcK5sV5q TuFMIbxVoYIoxcrLQFmnwuyBJGwxkhoogUGcjJ0WfP65Y6Mss1Bv2IkGpvtw MzlhHN5cp9pJw0iHFDztT2PjtfDqMXwmpbw1jvta4wMN0ePDDrjsgjxyqIuG 7cn2/ZOt2NGDZjBUp9YqmQENNYg4rAYBYb0YL2b++OdfrnrHk8idY1S6cNaq NULRY6zC5qjdaRWjWTLQx+9+izo79Mze1VBjIpFavvKATiuTDjTLLH+XJIFs i7rp7USH9/3qp9V7VtK8fNDmOevRx2eC6I7M1Hf37cgnoaJL8Z6IRkSW0dKX NW5rMYFtUNpMDqT5GKfc3b5eXw9fu4bvN73nt8s4G8LCqAX3axLi1ln2rz2q GFXWrIn2wMEBEQnJkgpJNk0/Mkgd7AwikTPGov15Ud2QVErMMvQS0A4gRH3O Q+mFb8587k10qupT1Pv8ruj84OLFmGHZ+54jhyjLvuW4o/en1CTQjHxwKi7/ Z4+bot4I4iH63aFtw/OU/Xa2t7jdqPrNKCStRkjOLD+HR0qhyfyIh+yGkkmL CVIBemfuprYSs72ZnydvdfgIqtttt+LI3ApUqVmFri4539/4zDTu0aO7XEco KHj9Usgkyosh7epHjj4QC47HCB9Pcj2zO6tEFXPxZ73RieyrgRo+TGxPLYfy whD8nqyZ51lVD0y1a6XZ5riczScPGcRdBc0twv++nNSQFoDtPR4mQYWiri4g zVP9/hmL7cQPBykemiEPC20V2kPXfMyQKMpYSIPBjwqpLVP3YqZQmL3eTfxf rOW08z7vuEPEiKdibk8UrP12YaV9JIQ7D7ffsPoOM8trwPmO8Co/AjUeHCWN MYWOZvBaZ1MI3ft7Pj+4t59j0TKwxU/EZlaMbDAfXgGGhyazogckocp45Rao S6lLW1PPoBrBi3i1+nLnNbfghez5hA5dWY7XPDrdBNvualcfUP0Vn3/xVgQs pWuJJYKZI1Ra45TyNpiRJYD/3vw6WC+TGmB5Z1GDNnVn8qyANjJVhBW6Z91f bWUKzTZ3eddvu+TvQEDKJCVHyX0iYoGiUaUgsTM3Iow1fTdAd7s8XBxF/X1h H3dtdyCGlt6vRYTkiiPMwdktgR6TpOoDxHVr/KGukxPVUZ3WMD4SyKApKntj troDkH7idvl3aLyZrc6xmQEmA6SpRr0BYR1CMUasl6+UHmb6btlRZg7ax57e NFWSKi4wh0oiWoSJBNVQaNjg5wKi+8nMmmxC+rjvOnvdZVn4jp38I3FRl+EH K9+FSpJPNVhEP30sFbqw25f15S1BYHw7/1O2pI1xNR5mxzSnuDR/g+1mt7Jz 9+saZhWdaD+NbOvvDMsadCt9yJngpA9HlrMcVgqn8fs1nKyYZF8kcygm/8uC SOpXqxqFj5afZIIjWtF6nOV2MVKj1BY/L70HLrdpGZxdckXsrD3OwRw8I2Ep N4lOdw7fkrrTlLGIuqu+bRcexm8ITLO6YQ7abZycZjgpEda8a+FkRyhxl9um 6TebZXVkoMpI2ki4i0Y585cVDpIyuq7fFfs/rXRmM4mWNmwaG/Fi4DXGZPAo 8XnVOGzac2/AlMRYFZDswDG4fzDD4duIrFOhWgMTi0JV19zHmBfKKgX1VJDO Otb40+qhRs7H2Suvp3b8zu+Yq0E7DSYRCcECIBpDorTRiDiImTg8GkhrNkKq Pf9H+OV5PXJY4BuRqTHm37d9tncWBtNj1odhLzrJ5DGPOEQK+TwKqIApf2/8 HhNPBurC5/S6obj2dz15T49+88L0Yfizbx27erlnho8feqqqqqqqqudZzD4z sw7wjsYb+feNWmY+nGOwRmZVu7rP1eTZw0OBvs29r17ffhWcacI0j7OPhYwR WcboS6GV6zT10NPGlcJjPyQQpbjpPAfMhRagYNhgLAeg4zfsE+OnaIjWhTjo 9rXannfE447qDiY93H3Nh47N9c21fNHLlQXpjuzezx+TjNC9Bh5byMQdNH0c DsaWZ5VtiQFs/tILjSydceqVpVN2RIbm2/bYkPSgHsWgrRQvGo269Ym4t7e7 DfIZEiUwFBEY+0rzrBheV1KpGLQItOI4zU3xJZnQR0lY+gMNx8aY6PLcKkta XNzUoGiE1VNSJEqFKS8d0qNecemz9Lc9t72gqlCSm3d5UfAocFWrFa6LtVcW VJKka411+pWX9MPpTtq7Y8ujsZQyO+91Ti2ib1rRayONJUiaQiRjXLxQ4GnQ Q0DlL3dMn69/gRPkNIYsnqPN69t8t1Ovp/XknQ5SInSzcGJlpaY2c+QPZ5B3 LOOcu7zps4bDcRWqXjpTw+NJ04WyX91o8jflp2C+PS0wC7oEdrlgatEIH0dQ wfgfCgVFtXFE0EfqOh6vu+zzcYyJ1/yHlCqRqBzP8/E3GH3/KPrO5T29eQ9/ Cc7ZUEqs+idGT8wbB+Rcu7TFKNpG1S2wqKVW1GlYVWKPq/D9GA/yLB9mbgnT rCRDYHiDQSCffMKS3TRQkXm5fy+ItdKNn9cluh9tKnAhm18nLQUqMdPiOUng 0CjHzYpFCp373OqkhQebOvPfEUHHF97rnT9Bm+BoZuM3Tp7kh871noLKw2/+ 6YnP+nq2wlhbg7mL4MZAdDQm3u5FC6PQjnnQV97bR3NRkWXsWTDieZViMywP Jaoh+8PqkIvvp9piqZ2JFs02oRRnbNfq1R28Ktlyg1LDcUj+HsPiw5oeoNw0 XSmWrelu30nKw7B8LmbnSn+fEFrvojHX3kltVv3G9Gati6q0Zs3G0WVUES31 zmGwpO6doWuzhs1t3yY4d3M58jXtjL1UK0+FL6S5uA+1S/KaDY/micIeVIoO eWcYLKr8xc/F2tun3Y+/WkSQh3TaYHBMdfN9OubG/Q0tuud3vxIEIj6StKLX 0j59D8FgYX4+W2l22ZDPKcJsem438llO0buOZf7jLnwwtt1suy4nJXjSp6s9 FXl34S63PDl5Kv+IRYijmhWp6KMgcPFBwQDRPQZpw8re63LHq4azb9rSLRC2 EenWglxJKxZnp8LPHgeomflcUuiuKN18TrXoRcXhjC3src6sc5efCnDwxI+Y lmbq2pV3YQ7j1QxkRjU+g673Ftld/nPHGEQ/LQ+GnCnnW+ey+dzPwZj0U+9a 2o48VG2a27mffk+5MY8IHCJwKaMJF94HaODerbabEVk6d2nWYFM17I7y2o23 4muiT1brniXmIm101VRFpoqS/VzbLo2RmVvopI4rpFSi0CZhlhuxFTKvk4Sw zC/IOu3SwH17SZ0CYlbELahaWCiqqwYrEEFBiDBgyIyI41gjERZEGKDJI4gv c2CkRLMkFYRIRAwERPsBUZASeuAEF/CytDnBe445OnAvby+jcdOFPA8eMF54 oG0cIdkUToNOvwKPLbyfWfw68j/KFJJG7s8e2fz3mLeRd9m57fEzdbuXg0h2 cJ+ATt+vGZmQgM8S5qrWs6bSyiU8WMHu88efJ/jlLQzAm5rbtulXvPCNwIWZ /7PM51eGyZb5JOQNOy+DamyZrWrApAERneY6zhFSBSrrF0+LZQSK+LErMpYF hvQxYmO6PIjZ08fLxPtDmHVKGJqgO2JKyUr+WxWq0XQnKms6zl1kq1k1TFG4 rWZmbyGM4l5xWU5sVnWlJVUtLSl41Uu5U5ObQocXciBAAHdEfqFhQDQB5Shb PvlLAT5RiKe7s4eRIBPgeSlVD6u8tRUNrwsCaRWahdpQ8KgB6CaUJKe6qVxK QaGihU8+qNg9ZyUfJbyo/T7uQXR8gMytSgY6WSCGLJiU8NN1p6aRSK8nZ101 ZvfqLGpoWBv+Tf9d19mLUjxagHcuJGFjYoOSD13H0BWvfp0VGKu8AOmpGV8l YMUObPwe3DtsvDEkBPc7kjIQoP0r7hEOcPn+Ayq8oSiqKPyoD4FqGTGPwtlD BJAjEgYxVLwPX7Oo2H24+X/D4nsA+WniOCUfxnjLIG3lNYPkUsSYcIFn4Brk M/EVHQftpQ2LbWkcqV7PvD/Xro3VtS7jE/eRNJAAtAd+JlwyKBL1w4QOz1F0 naKytzbcBUusoJRkWJBoS3kfrK0mEJSR6nGsUJVaV2VFn9OKV24F5rlXOWT/ XhKWNi/VMskZAJVtnjMGeJKBkvEx2m2wGtpsz2vzXZEbIFD5tsTl27wAThDa GTmbh6pEgmRuVMkcAKPRIWRv1IEzlkzNGY9oCoS1WJEVFpDanOIw2C7x1zeM ZEWkYPuwUq2wZtCfcd4UCneYN0g9Uu4c6wv5pXQ7e/u7c5JxaL9LQAGesFOm 90IiiRSXiKVTrEWiMDuVZFWPFjmkQocBQ2iEgJYTwUZyEEcxjDxXtqKZXcwH jW37O6423tsrIagD+XMYffHgGyLaOh4aXOtJOT8YGFUKq8a73HkTEevft27l R8aQhJIXDA406CNNQ8AlUXDBULREGqb/UjeOO1Oi62G+4WYuVbXVem+Q95DO 6BFVgWq8RCiCFK0h7F7RFeBZQMN7GGGh7QBLGZ5uN0y77TLS11Q1nNiuBAbX 7HrD0hdlWji5AVcYRiiByCUGGhkAchBtEWnUl3s9ZDvIuKOHRpo/YrXQlA4B QC42cG07uMSoa4Wx/KmVZbKwuXCon8y2BopBijujwrXLGVwqi1IUS2ooLNoL 6wKTIBK/GnUff0eLcRXfjvaHmbT4zv3mCYluUNY9XXgZiKnZttTiNtHhYDWK VfJVEz08UOjdrKQFSD1IasQkdZRens9cj4IrQOKqM+b2ysFYj0daUGvZdmp/ QmPapLoTYaP7ZzOaItXDNBFNs1e234wnzP/OkocEMxP+HGa+GUG70HSF6XbP bhA/RSyjhv1rKuVBejcr/hOEGycHWRoX5qnj582Y5ltHQeRxykb25zwI14ns q/HpVuwiQQilYhhdnW0U15Tce8f0efEoz/VBZMxtBIzg9iu9Lm5B35X1xZBm 5XW5/yuY2uVCPwQ83Z8KKBmoV8cAdcUnbNIUe4ccf6SDjW9Y3J+wi/sUxR1j wlVnM+55Pb736z4xhJzUGyK/V/zQBR+4lrCVAP+CGZQylrFJIAh/msn9Wr/f x/hkmPzKUHFH4ACHl/Dov+AqsVX/JVftev98fvx+38T/b6TB75YvgLpYqPHx Oo+4AsfwJ/fRFLdpt8iVQPw2RkH2Hy7LeNGpwP7lS6SZl+wsr/DB8hrXceUU oiaqsoLpmflGCpixmIh8hE1A7KB2CWA5xsPH+BHgOSgzzNI3ipB1swZNdMBA mHVC3eaiuXLoi3BF+H+uisGXRvpcQesZVQFRDCEKW7uQu6ZuITxxDQLK9GYP y4tmsSaBTFwo12r6OcDRUEw3ww5BuY2feRD0SuWLnWo0lpO5+rrkDfF+m1lC xPxwQgdMeAt2BREuTszIziE2Bkj/GVAdXsJvitbwcugSUO5aIX4USqLpmcgO 8Y79XLDxEQu4qkdRuMFaaHYceIuLCrXnGLdAMCU/VhagKP1U+N2gqdc5ge4+ Uncejv4naC5AxYzSIpYcGqgAg6d6PANGmwN0FaJMoCDUgt5mmgcXZBo6kNIF C5gVkPrwHMVbRX25H7uCmuFVejxSUe3IrPV5oOuiW/xLJXfLYg7MgsCZtcOr v0N4TzLJFMYA2LNA0jlvMF2Y2suQskmioSB3UAg2FeuZQG/S+TaHr3mVuaQd IXRJPC7ANWHwOKS5GEw1wWDQANUr9FKBtG4wGjm/EqVMWR07Elz8qbLTmVNU Blqff0Ov6MlzQhHA3GNYOAeZQdJPOXu1REGYIlnCuRq1FBYunINMBwfTHX5s ko8lQfvFI9Az9uGOWcWqJPrXAZMjOAg64RJahKQVKGqlCotaEpJ9zNVdDHW6 vUD5eJI3ouGsSTcrgTiJgbQ89gOYO7eciHMobGZKLNanINgR1JJhIeTXQYfm ZAmtUHBj2B7djB18C8vHBbkQYcoBaWmFDHbCtkEIrKGrCGemaFrJKlYFQ5mj GMfvkxqkGdw0i0+FofYH4/bAcZCR2vxIAyPf8q/g6PwmN4KcX7P9wui30d/9 fiofDt/L2d5sctdn+I1/1QUSEdgSsQeqMFwRqPvIVQsnnzSu8DZBPa+nvGus 0wYPV8mfiNhciG+06SjzboG+xvTRKaIHR5xrGi11k2Q3sNv/dQNmx1Urgf0v IxrEzYl0PUekJMZmOhutJFhmRC5gwlQLlIBqB9C6OyRCwVWg1XOMlpwOK/Yn wo8vr8Cj6T2pDQ8pQyxP4sPxi2GMZCEIifmYVNktYJaiIiVolUKWdYHOzcwf MTsvn1eUihIQGgGGaapjN2Igq02s+iBhv8wbMV4GpOfCIWbUY5w6mKon0kG7 SksgJiJmZmxomMG/+UsxLruv+tkvgG8VBBtuCsIyJyqKwNbmFypVfWt1NrIu 5uNsG2rp8A+zNP0U3DvxDeiZOTQc7ECR+3KdFKzXeb+PR3V0OC+p/fC0C8HG aZMVsmstBkgtlQhYRQurlQaIRkMPiXGS0WNgyIxjlHBSUWDoYgnPcBP52BlU EabySiggQh2TiCuSZjIfHJ3XXzAagZtbXpQTcZS6ZC3d2EGaXO23RKevuz7H hFYFkN6SoDrYA78Smuo09kk00EMb6KUuypTgMXYbbQ+MKg5TSUIgzh70YJ6F UIslQ3AU2CK22wLbuMklxpx2g2krmGz3qRZokTNcid70IZBeIbbEJJJCHClp Rq+CbS/W1JFg2qqLkDh/ZkmGmVPYJNtUD0Tu9B5CNOqzFiDMpZBBiMREVFRE RXIYd67ngBaZgS5Ux+a1IsJR9Z/pKIGWbQgw9cGNWy3OhMygSEYJrKFzsHCg DLyHINdIrvet6qK5lHsXp4gqiLXVwuSuolIaL20IdxAcmFpcKN/QgCtNhVCQ hdX8z1dIGxFkxsCJ74uUB5GtKIgzFvokpscDiGlFQMWnQgSuwzgS381c/SmZ K6YQBo4rgy36abGQZGDZC+iSJZlmz1hxe+nIDgCqqP0NEGb0IE/XPMB+hg0i 8pbkIsN2NxoJUVINRCTcAYKUIBskeGB6nhjSBWAsUGYalC9fQ25RkkLJFHeb 01nHrGTe3sosJhEQM2mZwQJgNjGLVMbMYfbBx86s5T9BNNYESDIT/LaJGHqh BwPloS5iBgn3vjVeTSQFT11pT3nt/NX3ai84Z0RlFPUtci4HgnNYKG6qWLpc KYSVRBNIwkNiaCpklQiIPtL3HAmwDKbIatpExiKhDW8mTRIGIdkTtjGZM5Q4 5XniJe0qAmJkkagG0OX8M72tr6kzgNCz2dR2IVxHaDNzN60QFgMKhcz8UJDN h34czSaSGWUhWExAOByHkFsLO0JOZC5fKGGFs9GerwHWKbopKIwfTSs6gupR JuuTcKFWJXyFVKQfN7IBk3gBoStICRliFSjf8KYCTEiVP3gCtzAOaMQFEzDD izIecf5ZCeIAdoxCTNELi2mIGxXpDoY1z0b36ZKxF8zmyFakWFGTrmYswSFp DPDR8QhNCkE0zHBkIJlIR/5y5ZuwczI5thDUYvy4bSOKyMBfEEuDhLF3PPDd yKnamcSiU1JaKxw0BkjWIDDBKIWROZgCoijAqxkSBSJRUUe8yN/v2XmPD0C4 9IqpAu1zD9HnOKB0qdCRAmWQnGcxyHqI7PyBzIEkIiwO5hRrNemSTeD7TTVG ZxUcggtB6ioT4JGgph0RS5V34BPdsFh37xObTVqqgTdgVMOD2G11rx46YUYj Z2wiydj2pJ2GPTELuRQG0ooIRmqccdNUIsB4ZPdmW/wiDtQnCfKSMDFfxSUc xzlVrYPNXALa+xs3g5kqeNeO6+LvLuo4koqt3TcqChzdNcQm1aW0Gd6qbnnR 3th6OiQaIDFgqKOipm2bNcjXETQOID2DoCtFCkGMMGJpRZH6GGbxJuOJE1ey Pcg6XlngZBUGmtstCQNVXOq4uXtGZFGUYlbUksILoqoM1M2g1LSWxSGFc4i4 xtdE3BQMMSM7RGWLGtxm4F+DAst3Hjlcwk1e7N2ob6JKiE0eLRhyaQQPXgWG FA1yg1nfkSGVtYijWgBAKARVYSZdoILhUmkI5gAOk5hNWlA8UFKJLixlyQpc lIgEtKCsA8982XIM2JohENsA0MK8IG8umA4xiQEJuNWHUE+ch/iR0KzN6sLN 1uoxOq9LYQKmsRF9iIxAYSBLrtV3K2sCyDTKlW99hcQMCDAL794VLt4poru2 +07idHDXvAsRAa6Ujrih/fbdfkglizR4H9UnwzwB65lkczsrHiehrBJ2levu Bo0Dzomw6KbM9DXXs2cCah6oci5xKRPR0aE4aWFahsmHw9hIj36sriT0lDI6 dZwNTcdYmiEeEN18O3n7M5qx+kHEcKUNw0WFY3OhQiAlXSChlkYXVkYzy0hS hHgI5HuaXl3RUBcBtXQZd/NKNqEN8Len3RbGWG6lDHj0yjSndrpyEgKsyg8h ymg31W8ctAbwNCURlNiiR69WxuPnjK4QaSklIJIREqcLKtRQ2XTbvKC5Vlnl KMhLZoPNuBdi1QgMeO9Skr0NinlEdgHnYxiawREZ6lFEySbew8SSGCEnpwqj VrGS0LGVv5qi3XxaPZ9aDrEMBcNQ6WLIccwhc5Vw+ABEzMjcUneHqvf1QCbg DT3mhUFgogX56HUHVTehEMZpe1kIcKaB4xEcVaOjxFJs1yCWSRsmMYJBDAwO Nmid8gX37RZYGEJtJop7CgRNZqzTQT2TGbk5FEZBJCLAu/0Ad3uW0s6X8DcU vsE7bNwNTR6ExS6PpdV47sc5YTQQ9J05hcYPuiEiD0vCDeZol4BDEPHBV2Ww XxQp/jdDBkRNmMiEsdn0RRNJ6JOPae6nMMCmzouFmEfPijw6q7w5Ghx3mTuW gi6kIJTESI+OVjZzUbyvPdVNSE3kdOyEilYOyuXn86ISAfHAWlIsorPirjwh qDITznbACDARC+AZju00aBBT8gcqFA5JLMlG6ebUpgckFAgiQiT7O+zY+OZw A4wUNTXj5ec9ptfsuYQjdqHx51aGzr7yNkGIIS79N9tDok7QDTLJKoB2khFI DrgFA0WJw65whIl/H1uORETPXs8IG4NDag78QACUu6owYCtm47MOPOSduKAn ZHhC9LD0NGgevunRoLsbTEUpUPFoMBZDI4gHBkNeaxffxpOw3EpcCR5l/Ybl UtYvlz7vNWRu1lFETp1CRkUh6wYFQUKkUrty8H0nKFPT6uf5Eh6ehyE8Wc0V cLGJAYIIyIiJVh22gq1aUj54xOFKVP1TX1qm32QEShNODVJmUBqOphKbtUUj lFqgKW2FCLMhYgFksKBy+bfad0N0FUFCEhIwDB9kBfDeo461A6epOK/THJld 9TthzEpEtqBlGN5PuEbi8zRHc/dUL65kZ/TOaNmkaBuaiPVFMiOf4szSpjL2 EqmE4VKREKKqkEaDCLbKsFkRmq2xIwUNho3U+Vtoms9yDSq7u0vdiQKkLWjZ b0Td/TUmhzSrpnk7rwocSFg5vTmmkGIKUmB4Q3QcAQxUIXa6BWGNmyB6XPMd j+q9WNDW6gfoorPwI7tgVFiqYlRiAjQQyyilLvr9m0XU7bme0p5w1WddOLUG J+bOAg7BkEGa20NhFTi/e3BAwVAOemg3Rd5bsf2pigD5EpKlllz4fApShTD2 DPqNpsTM8PMiAn6fTPMmozVO4NrMQw4exH7V/CYHQ++drmFMgTwO/Z0KonsB JfQLA1jlGO2eNzNDJxSHEVUlQYw9Z7XtnnS+qFXcu2Bsq5AZpNAht+plZlal rqnjF3eNRdyub3cgWHDa/d67Gd/Pf2CJazJzO5daoitw7jap3alI1pI6OENl DHQolSjWrLHWI9SYFrdi54gyqZHx5zQQXjYIQeVOHqDKQ+fCnmYRBLOqGhs0 SGjontNuLlYsnfBSXbIVKhOBmOLNKZBzIYnz6hrz6K4Ab019wHlhpIgkN0D4 xFDE4KixNwdpnGWBRmwYAhKEkDu8/T6vOWrW8PZNNm05u0TIJURQMWsLC1kc Q9xYawwNa0SGkkZJ/QO5qAIkgsIrEDicyEChElDmrYR58rFOwKBD75JIPhGA GMMy4mJoG0igKsFQiMAVAZD7RlzXL74z6MkXfnS1Bc2UpIQsFbNOosbyxq9j Th6+4leDKG0NmrSO5NfENNDUdtqMohAMDsKHizUlLCEWEDaIYBgyHuyUlZhE JeJli0TngRgOLolNNLS4I0FIzHJOR0z9ppmpAnl7Rh94Vh3pFkKnLLDlI5RH 352j24tyjcSzGzBJ3AQWFwQgCQAiI9nfzwkvtzOMA3hdd9gNwiOs9wgmfqKe KtA+YvQmZyocaG5A2GYNokURXkMhzSTzTuoGJsbCJtOB2AI41XtuVAR73ruw stg3EFYz9bCKxTAZHUkUZUeJI+u8KYgZ6Y1SPd+XtNanLCCSVHxEs8GqK2nt +9NU+400WVTheMzwY23zWyUSCm38TPHJ7M2QdvAQit+YKi7+OHYkDNdvQf5O h+K1y9O9Zj4B9rIG/Z7+9HzUGnwHl595JMzWfwYTgwWQQ0Gp8x9IccOvrPoT q/Ym0yHgPUWiKtrK0xIRebiNMDsiUMSSQaJQ5cHiNXjDa0lwhhAqFsawGoEr 9st/XjqW470By1AK7oDQouJnu3R9b0+bMlsWSQdWrMyKGq0SyvxFUKEPmIFB GEIRSer5qTV1psALOiWNLq5CEBq0ez2mplo9BegBCDGpq1csYcGGrrj5rWvP IGBiQ/Wl+Dp1SpWFYtps5jyzMoawzBZo6qVnTixYUwpREjgEaHBG2m0ycZBo BWG4UpmGQEFMIzLXQ1Eh9cN+0ue6ajT6rZSyxkCYGgxpwKZTH5LZWuayA7BT EPk6gWekQ9c+EOxGMb0ebXIyFZDXqN7nnxIpmUEsfRXQ/jIMC5bCxEOqk8Cy G3JS8fsjez6+PbDqrgLXqMi2ZZov0SMG3UfN4iKnUclph7arxoqC0BUTGNgD ExEheVgQldpMOpB+zr4rXBv8FRZbAQgdCHZ0JOwwICdlk0YREjUsiYCSLlEu SjQBVLa2VTzXh4ApZl1ImxENB+jLS3pgqMtqkSehz+vaT31g8fUryZjDIbU+ w4S8UeYUlbqwmK1DoKjLsrKlCrqom6sMkaKthFKpQoRDiFfCyGnQJNEFwXLM Id8OIKBWYmFS2pTWhj13xjOc3JefkOqBhQp82pAy8QEaTaabAbBaEksunVRQ L0hIaoSXyNB5BNQmPUWKCnuqdDZrsCImJ3Ug3DJSFRWve0aBc8Clw43q5IaY QrmSm1mnc8M3n4sMYgGHBqG2u5z7QkRkkR/n6tWNuERahsOUthJkEXq3lBoo EZIZjKRx5/kq1TQO03jIMmYrsh55BPRAdVCuglLwDL9zT5On05rhXDOtkAxY ZIu2SeEBLlMcxCbvuQeKIwnl4CpI9PIlM66XZBSRA4HrSeYMhyycDvoVPjZo Qg2pErH0D84Uo9o4qK4K2rSpSliN15Jv+XvoHHfpIS7IotV48OI2wqOhDEZQ khowDC5CgBxChMZ+lTsxuUUAOeVF5sqEmSL8jrkdIMo1o+aeZ70he8ZoTSH6 8xSVTYgmtaB+NiyqQwyNmHgMSwQkgmYpizi5CdClKWNfVFymlXKSrSVC05BR QO1YU1gA3DsnM6bFrWgJsF6kyCIIMdl40RMIalJYIYpegAUMcpeKxqWMlcig zWI1mI+vLI+0C+uZRIwBvyiZRChPmMvxShkY2dKsGXHCsQREKCAaLWqHCwVR 5CcLn5iYS9WhJZLWdMBr6gqXA06tFgpqp7huo73wwd3IYWUUYCSrpXkU0FVG rgRkMCYIEA+aklFKKXIIfa6blQ0YmYRCEZsBMED9ljbZ7X3JVQGobCIiiKDO +kjZaw7oxZcJRTmsXeclXGmOSoBlyG1m1KUqBviVCAVhQEm9N3Okzpluszlv hnrEDfzQgoFS+l8bDbiwCdTi01dskQirRTd88wxUYml9szXppJuN+aSRHWul cgsZInYwsHppmwZ8Qe8pnPwnL2+rp0rJ8XYbEiIKjJFgdpfxggTMWSyx4Sa1 0wiaMQckMQ2whkAZIHyvPY3IjhyRbcXtupv0RkhJDQlkG7f3G5bktqXKBhYj SbFZEiIdgK9+hNKi63EuAoIDYF499tum9IVxDuikp2blT1xFbB2woUeXwnr+ jUcoKnSi7dcrcfpGBpanOHBZ7GKAPsZvcg0D3xhrELAbzwYsOPy+wgzoMNBj bpoJsQ/bbu6kb5n3TgYIsITKEgVVEGhoiV+kpZGZDcCFEeQ7/Lc6vKXDuOw4 SQ8dR3wHrMaHdAH4ZI0jlzlVAdlVkWXfvpvr190TyaPW659VFRSzttZdqqco AHfE+I5joN5ixJnXbUqI1OUfbmJCKlgNTl5M2nB1cbdbm3drCTQlJvBJPxfX W3kwuXMwy22IVAkS0brRqePQ8k+eGWeqLn0tBWcaVLSwFBRpoZw7IFTRXkOp ELmML5ePD2nhvROfxVT+4csvIazXGLJcN0EXrvcg9dUXyyBsxRDPM06DKS3J SIRqdTICeggosTKcYUEMb6TpF6nCaRvEBMyeIajbiDSIMXnc2ZRmYgdZIEF6 oHXCQjIRaZCP8iMRkPwsmAhsqAWoAxhXaJGVthaP4kcRH/8XckU4UJC3gv1C --8323328-2125670340-1002120707=:7342-- - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ From owner-netdev@oss.sgi.com Wed Oct 3 08:17:53 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93FHrL16585 for netdev-outgoing; Wed, 3 Oct 2001 08:17:53 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93FHnD16581 for ; Wed, 3 Oct 2001 08:17:50 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id LAA05557; Wed, 3 Oct 2001 11:14:41 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Wed, 3 Oct 2001 11:14:41 -0400 (EDT) From: jamal To: Ingo Molnar cc: , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Linus Torvalds , Alan Cox , Simon Kirby Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, Ingo Molnar wrote: Robert has a driver extension (part of Alexey's iputils) that cna generate in the 140Kpps (for 100Mbps) and about 900Kpps for the e1000, but i'll take a look at Simons stuff if it is available. Marc Boucher has something that is an in-kernel client/server as well. > 10.0.3.4 is running vanilla 2.4.11-pre2 UP, a 466 MHz PII box with enough > RAM, using eepro100. The system effectively locks up - even in the full > knowledge of what is happening, i can hardly switch consoles, let alone do > anything like ifconfig eth0 down to fix the lockup. Until this kind of > load is present the only option is to power-cycle the box. SysRq does not > work. use the netif_rx() return code and hardware flowcontrol to fix it. > and frankly, this has been well-known for a long time - it's just since > Simon sent me this testcode that i realized how trivial it is. Alexey told > me about Linux routers effectively locking up if put under 100 mbit IRQ > load more than a year ago, when i first tried to fix softirq latencies. I > think if you are doing networking patches then you should be aware of it > as well. > I am fully aware of it. We have progessed extensively since then. Look at NAPI. > your refusal to accept this problem as an existing and real problem is > really puzzling me. > I must have miscommunicated. I am not saying there is no problem otherwise i wouldnt be working on this to begin with. I am just against your shotgun approach. cheers, jamal From owner-netdev@oss.sgi.com Wed Oct 3 08:19:38 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93FJcE16688 for netdev-outgoing; Wed, 3 Oct 2001 08:19:38 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93FJYD16681 for ; Wed, 3 Oct 2001 08:19:35 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id LAA05585; Wed, 3 Oct 2001 11:16:33 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Wed, 3 Oct 2001 11:16:33 -0400 (EDT) From: jamal To: Ingo Molnar cc: Linus Torvalds , , Alan Cox , Arjan van de Ven , Alexey Kuznetsov , Subject: Re: [patch] auto-limiting IRQ load take #2, irq-rewrite-2.4.11-F4 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Your approach is still wrong. Please do not accept this patch. cheers, jamal On Wed, 3 Oct 2001, Ingo Molnar wrote: > > the attached patch contains a cleaned up version of IRQ auto-mitigation. > > - i've removed the max_rate limit and have streamlined the impact of the > load-estimator on do_IRQ() to this piece of code: > > desc->total_contexts++; > if (unlikely(in_interrupt())) > goto mitigate_irqload; > > i dont think we can get much cheaper than this. (We could perhaps avoid > the total_contexts counter by saving a 'snapshot' of the existing > kstat.irqs array of counters in every timer tick and comparing the > snapshot to the current kstat.irqs values. That looked pretty unclean > though.) > > - the per-cpu irq counting in -D9 was incorrect as it collapsed all irq > handlers into a single counter. > > - i've removed the net-polling hacks - they are unrelated to this problem. > > the patch is against 2.4.11-pre2. (the eepro100.c fixes from the -ac tree > are already included in -pre2, i only included them in this patch to make > patching & testing against 2.4.10 easier.). > > (i'd like to stress the point again that the goal of this approach is > *not* to be nice. This is an airbag mechanizm, it can and will hurt > performance. But my box does not lock up anymore.) > > Ingo > From owner-netdev@oss.sgi.com Wed Oct 3 08:30:37 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93FUbq16900 for netdev-outgoing; Wed, 3 Oct 2001 08:30:37 -0700 Received: from chiara.elte.hu (chiara.elte.hu [157.181.150.200]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93FUWD16897 for ; Wed, 3 Oct 2001 08:30:33 -0700 Received: by chiara.elte.hu (Postfix, from userid 17806) id 1EEF11FCE; Wed, 3 Oct 2001 17:30:30 +0200 (CEST) Date: Wed, 3 Oct 2001 17:28:08 +0200 (CEST) From: Ingo Molnar Reply-To: To: jamal Cc: , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, jamal wrote: > No. NAPI is for any type of network activities not just for routers or > sniffers. It works just fine with servers. What do you see in there > that will make it not work with servers? eg. such solutions in tulip-NAPI-010910: /* For now we do this to avoid getting into IRQ mode too quickly */ if( jiffies - dev->last_rx == 0 ) goto not_done; [...] not_done: [...] return 1; combined with this code in the net_rx_action softirq handler: + while (!list_empty(&queue->poll_list)) { + struct net_device *dev; [...] + if (dev->quota <= 0 || dev->poll(dev, &budget)) { + local_irq_disable(); + list_del(&dev->poll_list); + list_add_tail(&dev->poll_list, &queue->poll_list); while the stated goal of NAPI is to do 'intelligent, feedback based polling', apparently the code is not trusting its own metrics, and is forcing the interface into polling mode if we are still within the same 10 msec period of time, or if we have looped 300 times (default netdev_max_backlog value). Not very intelligent IMO. In a generic computing environment i want to spend cycles doing useful work, not polling. Even the quick kpolld hack [which i dropped, so please dont regard it as a 'competitor' patch] i consider superior to this, as i can renice kpolld to reduce polling. (plus kpolld sucks up available idle cycles as well.) Unless i royally misunderstand it, i cannot stop the above code from wasting my cycles, and if that is true i do not want to see it in the kernel proper in this form. if the only thing done by a system is processing network packets, then polling is a very nice solution for high loads. So do not take my comments as an attack against polling. *if* you can make polling a success in ~90% of the time we enter tulip_poll() under non-specific server load (ie. not routing), then i think you have really good metrics. Ingo From owner-netdev@oss.sgi.com Wed Oct 3 08:45:41 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93Fjf017120 for netdev-outgoing; Wed, 3 Oct 2001 08:45:41 -0700 Received: from dea.linux-mips.net (a1as06-p202.stg.tli.de [195.252.187.202]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93FjND17115 for ; Wed, 3 Oct 2001 08:45:23 -0700 Received: (from ralf@localhost) by dea.linux-mips.net (8.11.1/8.11.1) id f93FjJ427819 for netdev@oss.sgi.com; Wed, 3 Oct 2001 17:45:19 +0200 Received: from grok.yi.org (IDENT:root@cx97923-a.phnx3.az.home.com [24.9.112.194]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93FhjD17092 for ; Wed, 3 Oct 2001 08:43:45 -0700 Received: from candelatech.com (IDENT:greear@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.2/8.11.2) with ESMTP id f93FgjT17599; Wed, 3 Oct 2001 08:42:45 -0700 Message-ID: <3BBB31F4.C223E12E@candelatech.com> Date: Wed, 03 Oct 2001 08:42:44 -0700 From: Ben Greear Organization: Candela Technologies X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.4.3-12 i686) X-Accept-Language: en MIME-Version: 1.0 To: jamal CC: Ingo Molnar , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , netdev@oss.sgi.com, Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk jamal wrote: > No. NAPI is for any type of network activities not just for routers or > sniffers. It works just fine with servers. What do you see in there that > will make it not work with servers? Will NAPI patch, as it sits today, fix all IRQ lockup problems for all drivers (as Ingo's patch claims to do), or will it just fix drivers (eepro, tulip) that have been integrated with it? -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From owner-netdev@oss.sgi.com Wed Oct 3 08:59:31 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93FxVb17425 for netdev-outgoing; Wed, 3 Oct 2001 08:59:31 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93FxOD17420 for ; Wed, 3 Oct 2001 08:59:24 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id LAA05853; Wed, 3 Oct 2001 11:56:25 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Wed, 3 Oct 2001 11:56:25 -0400 (EDT) From: jamal To: Ingo Molnar cc: , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, Ingo Molnar wrote: > > On Wed, 3 Oct 2001, jamal wrote: > > > No. NAPI is for any type of network activities not just for routers or > > sniffers. It works just fine with servers. What do you see in there > > that will make it not work with servers? > > eg. such solutions in tulip-NAPI-010910: > > /* For now we do this to avoid getting into > IRQ mode too quickly */ > > if( jiffies - dev->last_rx == 0 ) goto not_done; > [...] > not_done: > [...] > return 1; this code was added by Robert to check something; cant remember the details on that specific date. The goal is to test variuos workloads and conditions before reaching conclusions; so it might have been valid on that day only Take it out and things should work just fine. > combined with this code in the net_rx_action softirq handler: > > + while (!list_empty(&queue->poll_list)) { > + struct net_device *dev; > [...] > + if (dev->quota <= 0 || dev->poll(dev, &budget)) { > + local_irq_disable(); > + list_del(&dev->poll_list); > + list_add_tail(&dev->poll_list, &queue->poll_list); > > while the stated goal of NAPI is to do 'intelligent, feedback based > polling', apparently the code is not trusting its own metrics, and is > forcing the interface into polling mode if we are still within the same 10 > msec period of time, or if we have looped 300 times (default > netdev_max_backlog value). Not very intelligent IMO. > You misunderstood. This is to enforce fairness. Read the paper. When you have one device sending 100Kpps and another sending 1pps to the stack, you wanna make sure that the 1pps doesnt get starved -- thats what the purpose of the above code is (hence the Round robin scheduling and the quota per device). > In a generic computing environment i want to spend cycles doing useful > work, not polling. Even the quick kpolld hack [which i dropped, so please > dont regard it as a 'competitor' patch] i consider superior to this, as i > can renice kpolld to reduce polling. (plus kpolld sucks up available idle > cycles as well.) Unless i royally misunderstand it, i cannot stop the > above code from wasting my cycles, and if that is true i do not want to > see it in the kernel proper in this form. > Again, you misunderstood. Please spend a few more minutes reading the code and i should insist you read the paper ;-> The interupt just flags "i, netdev, have work to do"; the the poll thread grabs packets off it when the softirq gets scheduled. So we dont do unecessary polling; we only poll when there is work to be done. In the low-load case this solution reduces to the same as interupt driven system and scales to the system/CPU capacity. > if the only thing done by a system is processing network packets, then > polling is a very nice solution for high loads. So do not take my comments > as an attack against polling. > The poll thread is run as softirq, just as the other half of networking is today. And as should be because networking is extremely important as a subsyestem. > *if* you can make polling a success in ~90% of the time we enter > tulip_poll() under non-specific server load (ie. not routing), then i > think you have really good metrics. we can make it 100% successful; i mentioned that we only do work, if there is work to be done. cheers, jamal From owner-netdev@oss.sgi.com Wed Oct 3 09:01:54 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93G1sT17588 for netdev-outgoing; Wed, 3 Oct 2001 09:01:54 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93G1qD17585 for ; Wed, 3 Oct 2001 09:01:52 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id LAA05871; Wed, 3 Oct 2001 11:58:57 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Wed, 3 Oct 2001 11:58:57 -0400 (EDT) From: jamal To: Ben Greear cc: Ingo Molnar , , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <3BBB31F4.C223E12E@candelatech.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, Ben Greear wrote: > jamal wrote: > > > No. NAPI is for any type of network activities not just for routers or > > sniffers. It works just fine with servers. What do you see in there that > > will make it not work with servers? > > Will NAPI patch, as it sits today, fix all IRQ lockup problems for > all drivers (as Ingo's patch claims to do), or will it just fix > drivers (eepro, tulip) that have been integrated with it? Unfortunately amongst the three of us tulip seemed to be the most common. Robert has a gige intel. So patches appear only for those two drivers. I could write up a document on how to change drivers. cheers, jamal From owner-netdev@oss.sgi.com Wed Oct 3 09:17:01 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93GH1G17861 for netdev-outgoing; Wed, 3 Oct 2001 09:17:01 -0700 Received: from chiara.elte.hu (chiara.elte.hu [157.181.150.200]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93GGwD17858 for ; Wed, 3 Oct 2001 09:16:58 -0700 Received: by chiara.elte.hu (Postfix, from userid 17806) id 9316A1FCE; Wed, 3 Oct 2001 18:16:53 +0200 (CEST) Date: Wed, 3 Oct 2001 18:14:32 +0200 (CEST) From: Ingo Molnar Reply-To: To: Ben Greear Cc: jamal , , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <3BBB3845.DAE47D27@candelatech.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, Ben Greear wrote: > So, couldn't your NAPI patch be used by drivers that are updated, and > let Ingo's patch be a catch-all for un-fixed drivers? As we move > foward, more and more drivers support your version, and Ingo's patch > becomes less utilized. So long as the patches are tuned such that > yours keeps Ingo's from being triggered on devices you support, there > should be no real conflict, eh? exactly. auto-mitigation will not hurt NAPI-enabled devices the least. Also, auto-mitigation is device-independent. perhaps Jamal misunderstood the nature of my patch, so i'd like to state it again: auto-mitigation is a feature that is not triggered normally. I did a quick hack yesterday to include kpolld - that was a mistake, i was wrong, and i've removed it. kpolld was mostly an experiment to prove that TCP network connections can be fully functional during extreme overload situations as well. Also, auto-mitigation will be a nice mechanizm to make people more aware of the NAPI patch: if they ever notice 'Possible IRQ overload:' messages then they can be told to try the NAPI patches. Ingo From owner-netdev@oss.sgi.com Wed Oct 3 09:21:20 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93GLKJ17998 for netdev-outgoing; Wed, 3 Oct 2001 09:21:20 -0700 Received: from mandrakesoft.mandrakesoft.com (nsd.mandrakesoft.com [216.71.84.35] (may be forged)) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93GLGD17994 for ; Wed, 3 Oct 2001 09:21:16 -0700 Received: from localhost (jgarzik@localhost) by mandrakesoft.mandrakesoft.com (8.8.5/8.8.5) with SMTP id LAA32402; Wed, 3 Oct 2001 11:20:38 -0500 Date: Wed, 3 Oct 2001 11:20:38 -0500 (CDT) From: Jeff Garzik To: Ben Greear cc: jamal , Ingo Molnar , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , netdev@oss.sgi.com, Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <3BBB3845.DAE47D27@candelatech.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, Ben Greear wrote: > jamal wrote: > > On Wed, 3 Oct 2001, Ben Greear wrote: > > > jamal wrote: > > > > No. NAPI is for any type of network activities not just for routers or > > > > sniffers. It works just fine with servers. What do you see in there that > > > > will make it not work with servers? > > > > > > Will NAPI patch, as it sits today, fix all IRQ lockup problems for > > > all drivers (as Ingo's patch claims to do), or will it just fix > > > drivers (eepro, tulip) that have been integrated with it? > > > > Unfortunately amongst the three of us tulip seemed to be the most common. > > Robert has a gige intel. So patches appear only for those two drivers. I > > could write up a document on how to change drivers. > > So, couldn't your NAPI patch be used by drivers that are updated, and > let Ingo's patch be a catch-all for un-fixed drivers? As we move foward, > more and more drivers support your version, and Ingo's patch becomes less > utilized. So long as the patches are tuned such that yours keeps Ingo's > from being triggered on devices you support, there should be no real > conflict, eh? The main thing for me is that jamal/robert/ANK's work has been undergoing research and refinement for a while now, with very promising results combined with minimal impact on network drivers. Any of Ingo's solutions need to be tested in a variety of situations before we can jump on it with any confidence. For example, although Ingo dismisses shared-irq situations as an uninteresting case, we need to take that case into account as well, because starvation can definitely occur. I'm all for trying out ideas and test patches, but something as core as hard IRQ handling needs a lot of testing and research in many different real world situations before we use it. So far I do not agree that there is a magic bullet... Jeff From owner-netdev@oss.sgi.com Wed Oct 3 09:34:02 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93GY2U18381 for netdev-outgoing; Wed, 3 Oct 2001 09:34:02 -0700 Received: from neon-gw.transmeta.com (neon-gw-l3.transmeta.com [63.209.4.196]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93GXwD18378 for ; Wed, 3 Oct 2001 09:33:58 -0700 Received: (from root@localhost) by neon-gw.transmeta.com (8.9.3/8.9.3) id JAA27498; Wed, 3 Oct 2001 09:33:44 -0700 Received: from mailhost.transmeta.com(10.1.1.15) by neon-gw.transmeta.com via smap (V2.1) id xma027466; Wed, 3 Oct 01 09:33:16 -0700 Received: from penguin.transmeta.com (penguin.transmeta.com [10.10.27.78]) by deepthought.transmeta.com (8.9.3/8.9.3) with ESMTP id JAA20076; Wed, 3 Oct 2001 09:33:18 -0700 (PDT) Received: from localhost (torvalds@localhost) by penguin.transmeta.com (8.11.2/8.7.3) with ESMTP id f93GXCi09484; Wed, 3 Oct 2001 09:33:12 -0700 X-Authentication-Warning: penguin.transmeta.com: torvalds owned process doing -bs Date: Wed, 3 Oct 2001 09:33:12 -0700 (PDT) From: Linus Torvalds To: Ben Greear cc: jamal , Ingo Molnar , , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <3BBB31F4.C223E12E@candelatech.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, Ben Greear wrote: > > Will NAPI patch, as it sits today, fix all IRQ lockup problems for > all drivers (as Ingo's patch claims to do), or will it just fix > drivers (eepro, tulip) that have been integrated with it? Note that the big question here is WHO CARES? There are two issues, and they are independent: (a) handling of network packet flooding nicely (b) handling screaming devices nicely. First off, some comments: (a) is not a major security issue. If you allow untrusted users full 100/1000Mbps access to your internal network, you have _other_ security issues, like packet sniffing etc that are much much MUCH worse. So the packet flooding thing is very much a corner case, and claiming that we have a big problem is silly. HOWEVER, (a) _can_ be a performance issue under benchmark load. Benchmarks (unlike real life) are almost always set up to have full network bandwidth access, and can show this issue. (b) is to a large degree due to a stupid driver interface. I've wanted to change the IRQ handler functions to return a flag mask for about three years, but with hundreds of drivers it's always been a bit too painful. Why do we want to return a flag mask? Because we want the _driver_ to be able to say "shut me up" (if the driver cannot shut itself up and wants to throttle), and we want the _driver_ to be able to say "Hmm, that interrupt was not for me", so that the higher levels can quickly figure out if we have the case of us having two drivers but three devices, and the third device screaming its head off. Ingo tries to fix both of these with a sledgehammer. I'd rather use a bit more finesse, and as I do not actually agree with the people who seem to think that this is a major problem TODAY, I'll be more than happy to have people think about it. The NAPI people have thought about it - but it has obviously not been descussed _nearly_ widely enough. I personally am very nervous about Ingo's approach. I do not believe that it will work well over a wide range of machines, and I suspect that the "tunables" have been tuned for one load and one machine. I would not be surprised if Ingo finds that trying to put the machine under heavy disk load with multiple disk controllers might also cause interrupt mitigation, which would be unacceptably BAD. Linus From owner-netdev@oss.sgi.com Wed Oct 3 09:51:57 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93Gpvw18845 for netdev-outgoing; Wed, 3 Oct 2001 09:51:57 -0700 Received: from perninha.conectiva.com.br (perninha.conectiva.com.br [200.250.58.156]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93GpjD18840 for ; Wed, 3 Oct 2001 09:51:46 -0700 Received: from burns.conectiva (burns.conectiva [10.0.0.4]) by perninha.conectiva.com.br (Postfix) with SMTP id 6D12339147 for ; Wed, 3 Oct 2001 13:51:35 -0300 (EST) Received: (qmail 2771 invoked by uid 0); 3 Oct 2001 16:49:28 -0000 Received: from duckman.distro.conectiva (root@10.0.17.2) by burns.conectiva with SMTP; 3 Oct 2001 16:49:28 -0000 Received: (from localhost user: 'riel', uid#500) by duckman.distro.conectiva with ESMTP id ; Wed, 3 Oct 2001 13:51:24 -0300 Date: Wed, 3 Oct 2001 13:51:24 -0300 (BRST) From: Rik van Riel X-X-Sender: To: jamal Cc: Ingo Molnar , Linus Torvalds , , Alan Cox , Arjan van de Ven , Alexey Kuznetsov , Subject: Re: [patch] auto-limiting IRQ load take #2, irq-rewrite-2.4.11-F4 In-Reply-To: Message-ID: X-supervisor: aardvark@nl.linux.org MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, jamal wrote: > Your approach is still wrong. Please do not accept this patch. I rather like the fact that Ingo's approach will keep the system alive regardless of what driver is used. Rik -- DMCA, SSSCA, W3C? Who cares? http://thefreeworld.net/ (volunteers needed) http://www.surriel.com/ http://distro.conectiva.com/ From owner-netdev@oss.sgi.com Wed Oct 3 09:54:23 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93GsN218992 for netdev-outgoing; Wed, 3 Oct 2001 09:54:23 -0700 Received: from ms2.inr.ac.ru (minus.inr.ac.ru [193.233.7.97]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93GsKD18987 for ; Wed, 3 Oct 2001 09:54:20 -0700 Received: (from kuznet@localhost) by ms2.inr.ac.ru (8.6.13/ANK) id UAA13938; Wed, 3 Oct 2001 20:53:58 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200110031653.UAA13938@ms2.inr.ac.ru> Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 To: mingo@elte.hu Date: Wed, 3 Oct 2001 20:53:58 +0400 (MSK DST) Cc: hadi@cyberus.ca, linux-kernel@vger.kernel.org, Robert.Olsson@data.slu.se, bcrl@redhat.com, netdev@oss.sgi.com, torvalds@transmeta.com, alan@lxorguk.ukuu.org.uk In-Reply-To: from "Ingo Molnar" at Oct 3, 1 05:28:08 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Sender: owner-netdev@oss.sgi.com Precedence: bulk Hello! > In a generic computing environment i want to spend cycles doing useful > work, not polling. Ingo, "polling" is wrong name. It does not poll. :-) Actually, this misnomer is the worst thing whic I worried about. Citing my old explanation: >"Polling" is not a real polling in fact, it just accepts irqs as >events waking rx softirq with blocking subsequent irqs. >Actual receive happens at softirq. > >Seems, this approach solves the worst half of livelock problem completely: >irqs are throttled and tuned to load automatically. >Well, and drivers become cleaner. Alexey From owner-netdev@oss.sgi.com Wed Oct 3 09:54:25 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93GsPT19014 for netdev-outgoing; Wed, 3 Oct 2001 09:54:25 -0700 Received: from chiara.elte.hu (chiara.elte.hu [157.181.150.200]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93GsJD18986 for ; Wed, 3 Oct 2001 09:54:20 -0700 Received: by chiara.elte.hu (Postfix, from userid 17806) id DBC6E1FCE; Wed, 3 Oct 2001 18:54:16 +0200 (CEST) Date: Wed, 3 Oct 2001 18:51:55 +0200 (CEST) From: Ingo Molnar Reply-To: To: jamal Cc: , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, jamal wrote: > this code was added by Robert to check something; cant remember the > details on that specific date. [...] ok. > > + while (!list_empty(&queue->poll_list)) { > > + struct net_device *dev; > > [...] > > + if (dev->quota <= 0 || dev->poll(dev, &budget)) { > > + local_irq_disable(); > > + list_del(&dev->poll_list); > > + list_add_tail(&dev->poll_list, &queue->poll_list); > You misunderstood. This is to enforce fairness. [...] (i did not criticize the list_add/list_del in any way, it's obviously correct to cycle the polled devices. I highlited that code only to show that the current patch as-is polls too agressively for generic server load.) > Read the paper. (i prefer source code. Can i assume the 'authorative' patch to be the one with the "goto not_done;" line removed, correct?) > > In a generic computing environment i want to spend cycles doing useful > > work, not polling. Even the quick kpolld hack [which i dropped, so please > > dont regard it as a 'competitor' patch] i consider superior to this, as i > > can renice kpolld to reduce polling. (plus kpolld sucks up available idle > > cycles as well.) Unless i royally misunderstand it, i cannot stop the > > above code from wasting my cycles, and if that is true i do not want to > > see it in the kernel proper in this form. > The interupt just flags "i, netdev, have work to do"; [...] (and the only thing i pointed out was that the patch as-is did not limit the amount of polling done.) > > *if* you can make polling a success in ~90% of the time we enter > > tulip_poll() under non-specific server load (ie. not routing), then i > > think you have really good metrics. > > we can make it 100% successful; i mentioned that we only do work, if > there is work to be done. can you really make it 100% successful for rx? Ie. do you only ever call the ->poll() function if there is a new packet waiting? How do you know with a 100% probability that someone on the network just sent a new packet waiting? (without receiving an interrupt to begin with that is.) Ingo From owner-netdev@oss.sgi.com Wed Oct 3 10:08:53 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93H8rQ19422 for netdev-outgoing; Wed, 3 Oct 2001 10:08:53 -0700 Received: from chiara.elte.hu (chiara.elte.hu [157.181.150.200]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93H8oD19419 for ; Wed, 3 Oct 2001 10:08:50 -0700 Received: by chiara.elte.hu (Postfix, from userid 17806) id AA28B1FCF; Wed, 3 Oct 2001 19:08:47 +0200 (CEST) Date: Wed, 3 Oct 2001 19:06:26 +0200 (CEST) From: Ingo Molnar Reply-To: To: Alexey Kuznetsov Cc: , , , , , Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <200110031653.UAA13938@ms2.inr.ac.ru> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001 kuznet@ms2.inr.ac.ru wrote: > Ingo, "polling" is wrong name. It does not poll. :-) ok. i was also mislead by a quick hack in the source code :) > Actually, this misnomer is the worst thing whic I worried about. i think something like: 'offloading hardirq work into softirqs' covers the concept better, right? > Citing my old explanation: > > > "Polling" is not a real polling in fact, it just accepts irqs as > > events waking rx softirq with blocking subsequent irqs. > > Actual receive happens at softirq. > > > > Seems, this approach solves the worst half of livelock problem > > completely: irqs are throttled and tuned to load automatically. > > Well, and drivers become cleaner. i like this approach very much, and indeed this is not polling in any way. i'm worried by the dev->quota variable a bit. As visible now in the 2.4.10-poll.pat and tulip-NAPI-010910.tar.gz code, it keeps calling the ->poll() function until dev->quota is gone. I think it should only keep calling the function until the rx ring is fully processed - and it should re-enable the receiver afterwards, when exiting net_rx_action. Ingo From owner-netdev@oss.sgi.com Wed Oct 3 10:27:44 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93HRiv19704 for netdev-outgoing; Wed, 3 Oct 2001 10:27:44 -0700 Received: from chiara.elte.hu (chiara.elte.hu [157.181.150.200]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93HRfD19700 for ; Wed, 3 Oct 2001 10:27:42 -0700 Received: by chiara.elte.hu (Postfix, from userid 17806) id C00D41FCE; Wed, 3 Oct 2001 19:27:38 +0200 (CEST) Date: Wed, 3 Oct 2001 19:25:18 +0200 (CEST) From: Ingo Molnar Reply-To: To: Linus Torvalds Cc: Ben Greear , jamal , , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, Linus Torvalds wrote: > [...] I would not be surprised if Ingo finds that trying to put the > machine under heavy disk load with multiple disk controllers might > also cause interrupt mitigation, which would be unacceptably BAD. well, just tested my RAID testsystem as well. I have not tested heavy IO-related IRQ load with the patch before (so it was not tuned for that test in any way), but did so now: an IO test running on 12 disks, (5 IO interfaces: 3 SCSI cards and 2 IDE interfaces) producing 150 MB/sec block IO load and a fair number of SCSI and IDE interrupts, did not trigger the overload code. I started the network overload utility during this test, and the code detected overload on the network interrupt (and only on the network interrupt). IO load is still high (down to 130 MB/sec), while a fair amount of networking load is handled as well. (While there certainly are higher IO loads on some Linux boxes, mine should be above the average IO traffic.) Ingo From owner-netdev@oss.sgi.com Wed Oct 3 10:56:20 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93HuKD20268 for netdev-outgoing; Wed, 3 Oct 2001 10:56:20 -0700 Received: from dea.linux-mips.net (a1as01-p32.stg.tli.de [195.252.185.32]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93HuCD20265 for ; Wed, 3 Oct 2001 10:56:13 -0700 Received: (from ralf@localhost) by dea.linux-mips.net (8.11.1/8.11.1) id f93Hu9d28360 for netdev@oss.sgi.com; Wed, 3 Oct 2001 19:56:09 +0200 Received: from grok.yi.org (IDENT:root@cx97923-a.phnx3.az.home.com [24.9.112.194]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93GALD17784 for ; Wed, 3 Oct 2001 09:10:21 -0700 Received: from candelatech.com (IDENT:greear@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.2/8.11.2) with ESMTP id f93G9fT17800; Wed, 3 Oct 2001 09:09:41 -0700 Message-ID: <3BBB3845.DAE47D27@candelatech.com> Date: Wed, 03 Oct 2001 09:09:41 -0700 From: Ben Greear Organization: Candela Technologies X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.4.3-12 i686) X-Accept-Language: en MIME-Version: 1.0 To: jamal CC: Ingo Molnar , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , netdev@oss.sgi.com, Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk jamal wrote: > > On Wed, 3 Oct 2001, Ben Greear wrote: > > > jamal wrote: > > > > > No. NAPI is for any type of network activities not just for routers or > > > sniffers. It works just fine with servers. What do you see in there that > > > will make it not work with servers? > > > > Will NAPI patch, as it sits today, fix all IRQ lockup problems for > > all drivers (as Ingo's patch claims to do), or will it just fix > > drivers (eepro, tulip) that have been integrated with it? > > Unfortunately amongst the three of us tulip seemed to be the most common. > Robert has a gige intel. So patches appear only for those two drivers. I > could write up a document on how to change drivers. > So, couldn't your NAPI patch be used by drivers that are updated, and let Ingo's patch be a catch-all for un-fixed drivers? As we move foward, more and more drivers support your version, and Ingo's patch becomes less utilized. So long as the patches are tuned such that yours keeps Ingo's from being triggered on devices you support, there should be no real conflict, eh? Ben > cheers, > jamal -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From owner-netdev@oss.sgi.com Wed Oct 3 11:10:11 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93IABh20652 for netdev-outgoing; Wed, 3 Oct 2001 11:10:11 -0700 Received: from chiara.elte.hu (chiara.elte.hu [157.181.150.200]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93IA8D20649 for ; Wed, 3 Oct 2001 11:10:09 -0700 Received: by chiara.elte.hu (Postfix, from userid 17806) id B7C711FCE; Wed, 3 Oct 2001 19:30:30 +0200 (CEST) Date: Wed, 3 Oct 2001 19:28:07 +0200 (CEST) From: Ingo Molnar Reply-To: To: jamal Cc: , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Linus Torvalds , Alan Cox , Simon Kirby Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, jamal wrote: > use the netif_rx() return code and hardware flowcontrol to fix it. i'm using hardware flowcontrol in the patch, but at a different, higher level. This part of the do_IRQ() code disables the offending IRQ source: [...] desc->status |= IRQ_MITIGATED|IRQ_PENDING; __disable_irq(desc, irq); which in turn stops that device as well sooner or later. Optionally, in the future, this can be made more finegrained for chipsets that support device-independent IRQ mitigation features, like the USB 2.0 EHCI feature mentioned by David Brownell. i'd prefer it if all subsystems and drivers in the kernel behaved properly and limited their IRQ load - but this does not always happen and users are hit by irq overload situations. Your NAPI patch, or any driver/subsystem that does flowcontrol accurately should never be affected by it in any way. No overhead, no performance hit. Ingo From owner-netdev@oss.sgi.com Wed Oct 3 12:04:11 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93J4B421605 for netdev-outgoing; Wed, 3 Oct 2001 12:04:11 -0700 Received: from lacrosse.corp.redhat.com (host154.207-175-42.redhat.com [207.175.42.154]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93J48D21602 for ; Wed, 3 Oct 2001 12:04:08 -0700 Received: from toomuch.toronto.redhat.com (IDENT:Rx/+TcCcBCjBDjzohOmvJ1KIRzKyd6N4@toomuch.toronto.redhat.com [172.16.14.22]) by lacrosse.corp.redhat.com (8.9.3/8.9.3) with ESMTP id PAA05602; Wed, 3 Oct 2001 15:04:01 -0400 Received: (from bcrl@localhost) by toomuch.toronto.redhat.com (8.11.6/8.11.2) id f93J3tO04005; Wed, 3 Oct 2001 15:03:55 -0400 Date: Wed, 3 Oct 2001 15:03:55 -0400 From: Benjamin LaHaise To: kuznet@ms2.inr.ac.ru Cc: mingo@elte.hu, hadi@cyberus.ca, linux-kernel@vger.kernel.org, Robert.Olsson@data.slu.se, netdev@oss.sgi.com, torvalds@transmeta.com, alan@lxorguk.ukuu.org.uk Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: <20011003150355.A3780@redhat.com> References: <200110031653.UAA13938@ms2.inr.ac.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <200110031653.UAA13938@ms2.inr.ac.ru>; from kuznet@ms2.inr.ac.ru on Wed, Oct 03, 2001 at 08:53:58PM +0400 Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, Oct 03, 2001 at 08:53:58PM +0400, kuznet@ms2.inr.ac.ru wrote: > Citing my old explanation: > > >"Polling" is not a real polling in fact, it just accepts irqs as > >events waking rx softirq with blocking subsequent irqs. > >Actual receive happens at softirq. > > > >Seems, this approach solves the worst half of livelock problem completely: > >irqs are throttled and tuned to load automatically. > >Well, and drivers become cleaner. Well, this sounds like a 2.5 patch. When do we get to merge it? -ben From owner-netdev@oss.sgi.com Wed Oct 3 13:03:17 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93K3Hj22779 for netdev-outgoing; Wed, 3 Oct 2001 13:03:17 -0700 Received: from oof.localnet (h24-78-175-24.vn.shawcable.net [24.78.175.24]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93K3ED22776 for ; Wed, 3 Oct 2001 13:03:14 -0700 Received: from sim by oof.localnet with local (Exim 3.32 #1 (Debian)) id 15osDU-0000bf-00; Wed, 03 Oct 2001 13:02:04 -0700 Date: Wed, 3 Oct 2001 13:02:04 -0700 From: Simon Kirby To: Linus Torvalds Cc: Ben Greear , jamal , Ingo Molnar , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , netdev@oss.sgi.com, Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: <20011003130203.A2315@netnation.com> References: <3BBB31F4.C223E12E@candelatech.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.3.22i Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, Oct 03, 2001 at 09:33:12AM -0700, Linus Torvalds wrote: > Note that the big question here is WHO CARES? > > There are two issues, and they are independent: > (a) handling of network packet flooding nicely > (b) handling screaming devices nicely. > > First off, some comments: > (a) is not a major security issue. If you allow untrusted users full > 100/1000Mbps access to your internal network, you have _other_ > security issues, like packet sniffing etc that are much much MUCH > worse. So the packet flooding thing is very much a corner case, and > claiming that we have a big problem is silly. > > HOWEVER, (a) _can_ be a performance issue under benchmark load. > Benchmarks (unlike real life) are almost always set up to have full > network bandwidth access, and can show this issue. Actually, the way I first started looking at this problem is the result of a few attacks that have happened on our network. It's not just a while(1) sendto(); UDP spamming program that triggers it -- TCP SYN floods show the problem as well, and _there is no way_ to protect against this without using syncookies or some similar method that can only be done on the receiving TCP stack only. At one point, one of our webservers received 30-40Mbit/sec of SYN packets sustained for almost 24 hours. Needless to say, the machine was not happy. Simon- [ Stormix Technologies Inc. ][ NetNation Communications Inc. ] [ sim@stormix.com ][ sim@netnation.com ] [ Opinions expressed are not necessarily those of my employers. ] From owner-netdev@oss.sgi.com Wed Oct 3 14:05:59 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93L5xo24641 for netdev-outgoing; Wed, 3 Oct 2001 14:05:59 -0700 Received: from robur.slu.se (robur.slu.se [130.238.98.12]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93L5pD24638 for ; Wed, 3 Oct 2001 14:05:51 -0700 Received: (from robert@localhost) by robur.slu.se (8.8.7/8.8.7) id XAA10053; Wed, 3 Oct 2001 23:08:07 +0200 From: Robert Olsson MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15291.32311.499838.886628@robur.slu.se> Date: Wed, 3 Oct 2001 23:08:07 +0200 To: Cc: jamal , , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: References: X-Mailer: VM 6.92 under Emacs 19.34.1 Sender: owner-netdev@oss.sgi.com Precedence: bulk Ingo Molnar writes: > (i did not criticize the list_add/list_del in any way, it's obviously > correct to cycle the polled devices. I highlited that code only to show > that the current patch as-is polls too agressively for generic server > load.) Yes I think we need some data here... > can you really make it 100% successful for rx? Ie. do you only ever call > the ->poll() function if there is a new packet waiting? How do you know > with a 100% probability that someone on the network just sent a new packet > waiting? (without receiving an interrupt to begin with that is.) Well we need RX-interrupts not to spin away the CPU or exhaust the the PCI- bus. The NAPI scheme is simple, turn off RX-interrupts when the first packet comes and have the kernel to pull packets from the RX-ring. I tried have pure polling... it easy do just have your driver return "not_done" all the time. Not a good idea. :-) Maybe as sofirq test. If the device has more packets to deliver than is "allowed" we put this back on list and the polling process can give the next device its share and so on. So we handle screaming network devices and packet flooding nicely a deliver a decent performance even under those circumstances. As you seen from some code fragments we have played with some mechanisms to delay the transition from polling to irq-enable. I think I accepted a not_done polls for jiffies in some of the tests. Agree other variants are possible and hopefully better. SMP is another area, robustness and performance of course but in case of SMP we also have to deal with packet reordering which is something we really like to minimize. Even here I think the NAPI polling scheme is interesting. During consecutive polls the device is bound to the same CPU and no packet reordering should occur. And from data we have now we can see packet load is even distributed among different CPU's and should follow the smp_affinity. Cheers. --ro From owner-netdev@oss.sgi.com Wed Oct 3 15:04:36 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93M4ap26101 for netdev-outgoing; Wed, 3 Oct 2001 15:04:36 -0700 Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93M49D26091 for ; Wed, 3 Oct 2001 15:04:09 -0700 Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210]) by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id QAA213526; Wed, 3 Oct 2001 16:59:55 -0500 Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2]) by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v4.97.1) with ESMTP id f93M2GC213134; Wed, 3 Oct 2001 18:02:16 -0400 Received: from crg8.beaverton.ibm.com (crg8.beaverton.ibm.com [138.95.19.9]) by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id f93M2Ct02760; Wed, 3 Oct 2001 15:02:12 -0700 Received: from dyn9-47-8-184.rhe.beaverton.ibm.com (dyn9-47-8-184.rhe.beaverton.ibm.com [9.47.8.184]) by crg8.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) with ESMTP id f93M2Cn04613; Wed, 3 Oct 2001 15:02:12 -0700 (PDT) Date: Wed, 3 Oct 2001 15:02:05 -0700 (PDT) From: Mark Price X-X-Sender: To: cc: , Subject: patch - SNMP Kernel Counters Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Hi Folks, This is my first patch suggestion post, so I hope I've copied the correct people. Please let me know if I've not followed the process correctly. Included below is patch file against the 2.4.10 tree which makes some changes to the SNMP related kernel counters, mainly to try and make the statistics more accurate. I've included a brief summary below. Any thoughts, comments ? Cheers, Mark. o Added support for TcpRtoMin,TcpRtoMax,TcpRtoAlgorithm and TcpMaxConn variables, including a rather ugly cludge to proc.c for TCPMaxConn. o Modified the way IpInDelivers is incremented. Rather than incrementing this counter in each of the other protocols recieve routines (ie. tcp_v4_rcv() raw_rcv() etc.) the counter is incremented prior to the protocol handler being called. Also removed code which decrements IpInDelivers in the protocol handlers, higher level protocol handlers shouldn't be modifying other layers protocol counters. o Removed IpInDiscards increments in other other layers protocol receive routines. o Added IpInDiscards increments in IP routines where frame is discarded due to memory allocation failures. o Increment IcmpOutErrors on AddrMask Requests, which are not supported by the kernel. o Increment IpInAddrErrors during Martian handling. diff -urN linux-2.4.10/include/net/tcp.h linux/include/net/tcp.h --- linux-2.4.10/include/net/tcp.h Sun Sep 23 10:31:58 2001 +++ linux/include/net/tcp.h Tue Oct 2 11:26:24 2001 @@ -331,6 +331,7 @@ #define TCP_DELACK_MIN 4 #define TCP_ATO_MIN 4 #endif +#define TCP_RTO_JACOBSON 4 /* The Van Jacobson retransmission timeout algorithm */ #define TCP_RTO_MAX (120*HZ) #define TCP_RTO_MIN (HZ/5) #define TCP_TIMEOUT_INIT (3*HZ) /* RFC 1122 initial RTO value */ diff -urN linux-2.4.10/net/ipv4/icmp.c linux/net/ipv4/icmp.c --- linux-2.4.10/net/ipv4/icmp.c Fri Sep 7 11:01:21 2001 +++ linux/net/ipv4/icmp.c Tue Oct 2 11:24:44 2001 @@ -826,6 +826,10 @@ if (net_ratelimit()) printk(KERN_DEBUG "a guy asks for address mask. Who is it?\n"); #endif + /* + * Just say that there is an out-error in SNMP for now. + */ + ICMP_INC_STATS(IcmpOutErrors); } /* diff -urN linux-2.4.10/net/ipv4/ip_input.c linux/net/ipv4/ip_input.c --- linux-2.4.10/net/ipv4/ip_input.c Thu Apr 12 12:11:39 2001 +++ linux/net/ipv4/ip_input.c Wed Oct 3 09:35:26 2001 @@ -219,6 +219,7 @@ static inline int ip_local_deliver_finish(struct sk_buff *skb) { int ihl = skb->nh.iph->ihl*4; + int raw_flg = 0; #ifdef CONFIG_NETFILTER_DEBUG nf_debug_ip_local_deliver(skb); @@ -245,13 +246,15 @@ int hash = protocol & (MAX_INET_PROTOS - 1); struct sock *raw_sk = raw_v4_htable[hash]; struct inet_protocol *ipprot; - int flag; + int aflg,flag; /* If there maybe a raw socket we must check - if not we * don't care less */ - if(raw_sk != NULL) + if(raw_sk != NULL) { + raw_flg = 1; raw_sk = raw_v4_input(skb, skb->nh.iph, hash); + } ipprot = (struct inet_protocol *) inet_protos[hash]; flag = 0; @@ -261,11 +264,18 @@ ipprot->protocol == protocol) { int ret; + /* Increment IpInDelivers prior to calling the Fast Path + * protocol handler. + */ + + IP_INC_STATS_BH(IpInDelivers); + /* Fast path... */ ret = ipprot->handler(skb); return ret; } else { + aflg = 1; flag = ip_run_ipprot(skb, skb->nh.iph, ipprot, (raw_sk != NULL)); } } @@ -280,9 +290,16 @@ sock_put(raw_sk); } else if (!flag) { /* Free and report errors */ icmp_send(skb, ICMP_DEST_UNREACH, ICMP_PROT_UNREACH, 0); + /* + * Increment unknown-protocol counter for SNMP. + */ + IP_INC_STATS_BH(IpInUnknownProtos); out: kfree_skb(skb); } + + if (flag || raw_flg || aflg) + IP_INC_STATS_BH(IpInDelivers); } return 0; @@ -343,8 +360,10 @@ --ANK (980813) */ - if (skb_cow(skb, skb_headroom(skb))) + if (skb_cow(skb, skb_headroom(skb))) { + IP_INC_STATS_BH(IpInDiscards); goto drop; + } iph = skb->nh.iph; skb->ip_summed = 0; @@ -393,8 +412,10 @@ IP_INC_STATS_BH(IpInReceives); - if ((skb = skb_share_check(skb, GFP_ATOMIC)) == NULL) + if ((skb = skb_share_check(skb, GFP_ATOMIC)) == NULL) { + IP_INC_STATS_BH(IpInDiscards); goto out; + } if (!pskb_may_pull(skb, sizeof(struct iphdr))) goto inhdr_error; diff -urN linux-2.4.10/net/ipv4/proc.c linux/net/ipv4/proc.c --- linux-2.4.10/net/ipv4/proc.c Wed May 16 10:21:45 2001 +++ linux/net/ipv4/proc.c Tue Oct 2 11:23:46 2001 @@ -135,7 +135,14 @@ "\nTcp: RtoAlgorithm RtoMin RtoMax MaxConn ActiveOpens PassiveOpens AttemptFails EstabResets CurrEstab InSegs OutSegs RetransSegs InErrs OutRsts\n" "Tcp:"); for (i=0; iname); #endif + IP_INC_STATS_BH(IpInAddrErrors); e_inval: err = -EINVAL; goto done; diff -urN linux-2.4.10/net/ipv4/tcp.c linux/net/ipv4/tcp.c --- linux-2.4.10/net/ipv4/tcp.c Thu Sep 20 14:12:56 2001 +++ linux/net/ipv4/tcp.c Tue Oct 2 11:16:06 2001 @@ -2555,4 +2555,15 @@ printk("TCP: Hash tables configured (established %d bind %d)\n", tcp_ehash_size<<1, tcp_bhash_size); + + /* Setup the SNMP Stats for the TCP RTO values. + * TcpMaxConn = -1 as the maximum number of connections is + * dynamic. + */ + + tcp_statistics[0].TcpRtoAlgorithm=TCP_RTO_JACOBSON; + tcp_statistics[0].TcpRtoMin=TCP_RTO_MIN*1000/HZ; + tcp_statistics[0].TcpRtoMax=TCP_RTO_MAX*1000/HZ; + tcp_statistics[0].TcpMaxConn=-1; + } diff -urN linux-2.4.10/net/ipv4/tcp_ipv4.c linux/net/ipv4/tcp_ipv4.c --- linux-2.4.10/net/ipv4/tcp_ipv4.c Fri Sep 7 11:01:21 2001 +++ linux/net/ipv4/tcp_ipv4.c Tue Oct 2 11:25:34 2001 @@ -1538,8 +1538,6 @@ goto discard; #endif /* CONFIG_FILTER */ - IP_INC_STATS_BH(IpInDelivers); - if (sk->state == TCP_ESTABLISHED) { /* Fast path */ TCP_CHECK_TIMER(sk); if (tcp_rcv_established(sk, skb, skb->h.th, skb->len)) diff -urN linux-2.4.10/net/ipv4/udp.c linux/net/ipv4/udp.c --- linux-2.4.10/net/ipv4/udp.c Fri Sep 7 11:01:21 2001 +++ linux/net/ipv4/udp.c Tue Oct 2 11:25:12 2001 @@ -785,8 +785,6 @@ if (sk->filter && skb->ip_summed != CHECKSUM_UNNECESSARY) { if (__udp_checksum_complete(skb)) { UDP_INC_STATS_BH(UdpInErrors); - IP_INC_STATS_BH(IpInDiscards); - ip_statistics[smp_processor_id()*2].IpInDelivers--; kfree_skb(skb); return -1; } @@ -796,8 +794,6 @@ if (sock_queue_rcv_skb(sk,skb)<0) { UDP_INC_STATS_BH(UdpInErrors); - IP_INC_STATS_BH(IpInDiscards); - ip_statistics[smp_processor_id()*2].IpInDelivers--; kfree_skb(skb); return -1; } @@ -836,8 +832,10 @@ udp_queue_rcv_skb(sk, skb1); sk = sknext; } while(sknext); - } else + } else { + UDP_INC_STATS_BH(UdpNoPorts); kfree_skb(skb); + } read_unlock(&udp_hash_lock); return 0; } @@ -880,8 +878,6 @@ u32 saddr = skb->nh.iph->saddr; u32 daddr = skb->nh.iph->daddr; int len = skb->len; - - IP_INC_STATS_BH(IpInDelivers); /* * Validate the packet and the UDP length. From owner-netdev@oss.sgi.com Wed Oct 3 15:25:02 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f93MP2726585 for netdev-outgoing; Wed, 3 Oct 2001 15:25:02 -0700 Received: from webber.adilger.int (h24-64-71-161.cg.shawcable.net [24.64.71.161]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f93MOuD26582 for ; Wed, 3 Oct 2001 15:24:57 -0700 Received: from webber.adilger.int (adilger@localhost [127.0.0.1]) by localhost (8.12.0.Beta19/8.12.0.Beta19/Debian 8.12.0.Beta19) with ESMTP id f93MMPL7028895; Wed, 3 Oct 2001 16:22:25 -0600 Received: (from adilger@localhost) by webber.adilger.int (8.12.0.Beta19/8.12.0.Beta19/Debian 8.12.0.Beta19) id f93MMAeB028893; Wed, 3 Oct 2001 16:22:10 -0600 From: Andreas Dilger Date: Wed, 3 Oct 2001 16:22:10 -0600 To: Robert Olsson Cc: mingo@elte.hu, jamal , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Benjamin LaHaise , netdev@oss.sgi.com, Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: <20011003162210.L8954@turbolinux.com> Mail-Followup-To: Robert Olsson , mingo@elte.hu, jamal , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Benjamin LaHaise , netdev@oss.sgi.com, Linus Torvalds , Alan Cox References: <15291.32311.499838.886628@robur.slu.se> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <15291.32311.499838.886628@robur.slu.se> User-Agent: Mutt/1.3.22i Sender: owner-netdev@oss.sgi.com Precedence: bulk On Oct 03, 2001 23:08 +0200, Robert Olsson wrote: > Ingo Molnar writes: > > (i did not criticize the list_add/list_del in any way, it's obviously > > correct to cycle the polled devices. I highlited that code only to show > > that the current patch as-is polls too agressively for generic server > > load.) > > Yes I think we need some data here... > > > can you really make it 100% successful for rx? Ie. do you only ever call > > the ->poll() function if there is a new packet waiting? How do you know > > with a 100% probability that someone on the network just sent a new packet > > waiting? (without receiving an interrupt to begin with that is.) > > Well we need RX-interrupts not to spin away the CPU or exhaust the the PCI- > bus. The NAPI scheme is simple, turn off RX-interrupts when the first packet > comes and have the kernel to pull packets from the RX-ring. > > I tried have pure polling... it easy do just have your driver return > "not_done" all the time. Not a good idea. :-) Maybe as sofirq test. I think it is rather easy to make this self-regulating (I may be wrong). If you get to the stage where you are turning off IRQs and going to a polling mode, then don't turn IRQs back on until you have a poll (or two or whatever) that there is no work to be done. This will at worst give you 50% polling success, but in practise you wouldn't start polling until there is lots of work to be done, so the real success rate will be much higher. At this point (no work to be done when polling) there are clearly no interrupts would be generated (because no packets have arrived), so it should be reasonable to turn interrupts back on and stop polling (assuming non-broken hardware). You now go back to interrupt-driven work until the rate increases again. This means you limit IRQ rates when needed, but only do one or two excess polls before going back to IRQ-driven work. Granted, I don't know what the overhead of turning the IRQs on and off is, but since we do it all the time already (for each ISR) it can't be that bad. If you are always having work to do when polling, then interrupts will never be turned on again, but who cares at that point because the work is getting done? Similarly, if you have IRQs disabled, but are sharing IRQs there is nothing wrong in polling all devices sharing that IRQ (at least conceptually). I don't know much about IRQ handlers, but I assume that this is already what happens if you are sharing an IRQ - you don't know which of many sources it comes from, so you poll all of them to see if they have any work to be done. If you are polling some of the shared-IRQ devices too frequently (i.e. they never have work to do), you could have some sort of progressive backoff, so you skip polling those for a growing number of polls (this could also be set by the driver if it knows that it could only generate real work every X ms, so we skip about X/poll_rate polls). Cheers, Andreas -- Andreas Dilger \ "If a man ate a pound of pasta and a pound of antipasto, \ would they cancel out, leaving him still hungry?" http://www-mddsp.enel.ucalgary.ca/People/adilger/ -- Dogbert From owner-netdev@oss.sgi.com Wed Oct 3 17:47:29 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f940lTD29282 for netdev-outgoing; Wed, 3 Oct 2001 17:47:29 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f940lQD29279 for ; Wed, 3 Oct 2001 17:47:26 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id UAA08042; Wed, 3 Oct 2001 20:44:30 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Wed, 3 Oct 2001 20:44:30 -0400 (EDT) From: jamal To: Ingo Molnar cc: Alexey Kuznetsov , , Robert Olsson , , , Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, Ingo Molnar wrote: > i like this approach very much, and indeed this is not polling in any way. > > i'm worried by the dev->quota variable a bit. As visible now in the > 2.4.10-poll.pat and tulip-NAPI-010910.tar.gz code, it keeps calling the > ->poll() function until dev->quota is gone. I think it should only keep > calling the function until the rx ring is fully processed - and it should > re-enable the receiver afterwards, when exiting net_rx_action. This would result in an unfairness. Think of one device which receives packets really fast that it takes most of the CPU capacity just processing it. cheers, jamal From owner-netdev@oss.sgi.com Wed Oct 3 17:49:42 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f940ngc29395 for netdev-outgoing; Wed, 3 Oct 2001 17:49:42 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f940ndD29392 for ; Wed, 3 Oct 2001 17:49:39 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id UAA08049; Wed, 3 Oct 2001 20:46:45 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Wed, 3 Oct 2001 20:46:45 -0400 (EDT) From: jamal To: Ingo Molnar cc: , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; CHARSET=US-ASCII Content-ID: Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, Ingo Molnar wrote: > > On Wed, 3 Oct 2001, jamal wrote: > > (and the only thing i pointed out was that the patch as-is did not limit > the amount of polling done.) you mean in the softirq or the one line in the driver? > > > > *if* you can make polling a success in ~90% of the time we enter > > > tulip_poll() under non-specific server load (ie. not routing), then i > > > think you have really good metrics. > > > > we can make it 100% successful; i mentioned that we only do work, if > > there is work to be done. > > can you really make it 100% successful for rx? Ie. do you only ever call > the ->poll() function if there is a new packet waiting? How do you know > with a 100% probability that someone on the network just sent a new packet > waiting? (without receiving an interrupt to begin with that is.) > Take a look at what i think is the NAPI state machine pending a nod from Alexey and Robert: http://www.cyberus.ca/~hadi/NAPI-SM.ps.gz cheers, jamal From owner-netdev@oss.sgi.com Wed Oct 3 17:56:26 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f940uQq29585 for netdev-outgoing; Wed, 3 Oct 2001 17:56:26 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f940uLD29582 for ; Wed, 3 Oct 2001 17:56:22 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id UAA08063; Wed, 3 Oct 2001 20:53:28 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Wed, 3 Oct 2001 20:53:28 -0400 (EDT) From: jamal To: Ingo Molnar cc: , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Linus Torvalds , Alan Cox , Simon Kirby Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, Ingo Molnar wrote: > > On Wed, 3 Oct 2001, jamal wrote: > > > use the netif_rx() return code and hardware flowcontrol to fix it. > > i'm using hardware flowcontrol in the patch, but at a different, higher > level. This part of the do_IRQ() code disables the offending IRQ source: > > [...] > desc->status |= IRQ_MITIGATED|IRQ_PENDING; > __disable_irq(desc, irq); > > which in turn stops that device as well sooner or later. Optionally, in > the future, this can be made more finegrained for chipsets that support > device-independent IRQ mitigation features, like the USB 2.0 EHCI feature > mentioned by David Brownell. > I think each subsytem should be in charge of its own fate. USB applies in whatever subsystem it belongs to. Cooperating subsystems doing what os best for the system. > i'd prefer it if all subsystems and drivers in the kernel behaved properly > and limited their IRQ load - but this does not always happen and users are > hit by irq overload situations. > Your patch with Linus' idea of "flag mask" would be more acceptable as a last resort. All subsytems should be cooperative and we resort to this to send misbehaving kids to their room. > Your NAPI patch, or any driver/subsystem that does flowcontrol accurately > should never be affected by it in any way. No overhead, no performance > hit. so far your appraoch is that of a shotgun i.e "let me fire in that crowd and i'll hit my target but dont care if i take down a few more"; regardless of how noble the reasoning is, it's as Linus described it -- a sledge hammer. cheers, jamal From owner-netdev@oss.sgi.com Wed Oct 3 18:07:27 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9417RE29879 for netdev-outgoing; Wed, 3 Oct 2001 18:07:27 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9417OD29875 for ; Wed, 3 Oct 2001 18:07:24 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id VAA08094; Wed, 3 Oct 2001 21:04:22 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Wed, 3 Oct 2001 21:04:22 -0400 (EDT) From: jamal To: Simon Kirby cc: Linus Torvalds , Ben Greear , Ingo Molnar , , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <20011003130203.A2315@netnation.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, Simon Kirby wrote: > On Wed, Oct 03, 2001 at 09:33:12AM -0700, Linus Torvalds wrote: > > Actually, the way I first started looking at this problem is the result > of a few attacks that have happened on our network. It's not just a > while(1) sendto(); UDP spamming program that triggers it -- TCP SYN > floods show the problem as well, and _there is no way_ to protect against > this without using syncookies or some similar method that can only be > done on the receiving TCP stack only. > > At one point, one of our webservers received 30-40Mbit/sec of SYN packets > sustained for almost 24 hours. Needless to say, the machine was not > happy. > I think you can save yourself a lot of pain today by going to a "better driver"/hardware. Switch to a tulip based board; in particular one which is based on the 21143 chipset. Compile in hardware traffic control and save yourself some pain. The interface was published but so far only the tulip conforms to it. It can sustain upto about 90% of the wire rate before it starts dropping. And at those rates you still have plenty of CPU available. The ingress policer in the traffic control code might also be able to help, however CPU cycles are already wasted by the time that code is hit; with NAPI you should be able to push the filtering much lower in the stack. cheers, jamal From owner-netdev@oss.sgi.com Wed Oct 3 18:13:07 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f941D7N30041 for netdev-outgoing; Wed, 3 Oct 2001 18:13:07 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f941D4D30038 for ; Wed, 3 Oct 2001 18:13:04 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id VAA08116; Wed, 3 Oct 2001 21:10:10 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Wed, 3 Oct 2001 21:10:10 -0400 (EDT) From: jamal To: Benjamin LaHaise cc: , , , , , , Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <20011003150355.A3780@redhat.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, Benjamin LaHaise wrote: > On Wed, Oct 03, 2001 at 08:53:58PM +0400, kuznet@ms2.inr.ac.ru wrote: > > Citing my old explanation: > > > > >"Polling" is not a real polling in fact, it just accepts irqs as > > >events waking rx softirq with blocking subsequent irqs. > > >Actual receive happens at softirq. > > > > > >Seems, this approach solves the worst half of livelock problem completely: > > >irqs are throttled and tuned to load automatically. > > >Well, and drivers become cleaner. > > Well, this sounds like a 2.5 patch. When do we get to merge it? It is backward compatible to 2.4 netif_rx() which means it can go in now. The problem is netdrivers that want to use the interface have to be morphed. As a general disclaimer, i really dont mean to put down Ingo's efforts i just think the irq mitigation idea as is now is wrong for both 2.4 and 2.5 cheers, jamal From owner-netdev@oss.sgi.com Wed Oct 3 18:18:15 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f941IF830207 for netdev-outgoing; Wed, 3 Oct 2001 18:18:15 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f941I8D30202 for ; Wed, 3 Oct 2001 18:18:08 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id VAA08133; Wed, 3 Oct 2001 21:15:20 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Wed, 3 Oct 2001 21:15:20 -0400 (EDT) From: jamal To: cc: Subject: Some NAPI results Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Since we are discussing this, heres more data ;-> ---------- Forwarded message ---------- Date: Wed, 3 Oct 2001 20:42:43 +0200 From: Robert Olsson Hello! Anyway. Uppsala University (35.000 students, 4000 emploees) has now SMP/NAPI in production. But reniceed ksoftirqd to -10. Kernel 2.4.10poll SMP. USER PID %CPU %MEM SIZE RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 1224 512 ? S 18:08 0:03 init [5] root 2 0.0 0.0 0 0 ? SW 18:08 0:00 (keventd) root 3 74.2 0.0 0 0 ? RW< 18:08 100:02 (ksoftirqd_CPU0) root 4 73.9 0.0 0 0 ? SW< 18:08 99:39 (ksoftirqd_CPU1) root 5 0.0 0.0 0 0 ? SW 18:08 0:00 (kswapd) root 6 0.0 0.0 0 0 ? SW 18:08 0:00 (bdflush) root 7 0.0 0.0 0 0 ? SW 18:08 0:00 (kupdated) root 52 0.0 0.0 796 192 ? S 18:08 0:00 voff root 570 0.3 7.4 38696 38184 ? S 18:08 0:28 /sbin/gated . With real data packet load seems to be well distributed. Plus some CPU TX collisions. :-) L-uu1:/# cat /proc/net/softnet_stat 076b152d 00002382 076893e2 00000000 00000000 00000000 00000000 00000000 00060511 07678c10 0000247c 076a7e5c 00000000 00000000 00000000 00000000 00000000 00061b88 L-uu1:/# ipchains -L -n | wc -l 465 L-uu1:/# cat /proc/net/drivers/tulip_eth0 eth0: UP Locked MII Full DuplexLink UP Admin up 2 hour(s) 17 min 6 sec Last input NOW Last output NOW 5min RX bit/s 94.5 M 5min TX bit/s 69.2 M 5min RX pkts/s 14981 5min TX pkts/s 13971 5min TX errors 0 5min RX errors 0 5min RX dropped 0 5min TX dropped 0 5min collisions 0 5min RX missed errors 0 NormalIntr 24221027 AbnormalIntr 1 RxIntr 20983232 RxNoBuf 280 RxDied 0 RxJabber 0 TxIntr 15696981 TxDied 0 TxNoBuf 8512291 TimerIntr 0 TotalIntr 23459016 packets-per-int 5 TX-frees in IRQ/xmit 56077416/66612498 TX-queue stop 3731551 Fb: no 0 low 0 mod 0 high 0 drp 0 Poll start start/pkts 1962176135/128376006 Poll exit done/not_done/oom 701671/1961474461/0 Memory allocation failure (small skb) 0 L-uu1:/# cat /proc/interrupts CPU0 CPU1 0: 412299 413160 IO-APIC-edge timer 1: 80 96 IO-APIC-edge keyboard 2: 0 0 XT-PIC cascade 8: 0 0 IO-APIC-edge rtc 14: 10791 10371 IO-APIC-edge ide0 15: 18 106 IO-APIC-edge ide1 16: 8568624 8569699 IO-APIC-level eth1 17: 11830639 11830125 IO-APIC-level eth0, eth2 Cheers. --ro From owner-netdev@oss.sgi.com Wed Oct 3 18:30:20 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f941UKh30437 for netdev-outgoing; Wed, 3 Oct 2001 18:30:20 -0700 Received: from lacrosse.corp.redhat.com (host154.207-175-42.redhat.com [207.175.42.154]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f941UID30434 for ; Wed, 3 Oct 2001 18:30:18 -0700 Received: from toomuch.toronto.redhat.com (IDENT:Qwb+92No0LFxOYxkGSgFaceFlgUib28G@toomuch.toronto.redhat.com [172.16.14.22]) by lacrosse.corp.redhat.com (8.9.3/8.9.3) with ESMTP id VAA13020; Wed, 3 Oct 2001 21:30:11 -0400 Received: (from bcrl@localhost) by toomuch.toronto.redhat.com (8.11.6/8.11.2) id f941UBN12422; Wed, 3 Oct 2001 21:30:11 -0400 Date: Wed, 3 Oct 2001 21:30:11 -0400 From: Benjamin LaHaise To: jamal Cc: kuznet@ms2.inr.ac.ru, mingo@elte.hu, linux-kernel@vger.kernel.org, Robert.Olsson@data.slu.se, netdev@oss.sgi.com, torvalds@transmeta.com, alan@lxorguk.ukuu.org.uk Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: <20011003213010.F3780@redhat.com> References: <20011003150355.A3780@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: ; from hadi@cyberus.ca on Wed, Oct 03, 2001 at 09:10:10PM -0400 Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, Oct 03, 2001 at 09:10:10PM -0400, jamal wrote: > > Well, this sounds like a 2.5 patch. When do we get to merge it? > > > It is backward compatible to 2.4 netif_rx() which means it can go in now. > The problem is netdrivers that want to use the interface have to be > morphed. I'm alluding to the fact that we need a place to put in-development patches. > As a general disclaimer, i really dont mean to put down Ingo's efforts i > just think the irq mitigation idea as is now is wrong for both 2.4 and 2.5 What is your solution to the problem? Leaving it up to the driver authors doesn't work as they're not perfect. Yes, drivers should attempt to do a good job at irq mitigation, but sometimes a safety net is needed. -ben From owner-netdev@oss.sgi.com Wed Oct 3 18:30:26 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f941UQF30462 for netdev-outgoing; Wed, 3 Oct 2001 18:30:26 -0700 Received: from moonstone (zaqd37ca49c.zaq.ne.jp [211.124.164.156]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f941ULD30447 for ; Wed, 3 Oct 2001 18:30:22 -0700 Received: from localhost (localhost [127.0.0.1]) by moonstone (Postfix) with ESMTP id A3A2DEFA; Thu, 4 Oct 2001 10:30:17 +0900 (JST) Received: by moonstone (Postfix, from userid 1000) id 6A6C5527E; Thu, 4 Oct 2001 10:30:14 +0900 (JST) Delivered-To: rei@oucc.org Received: from orochi.oucc.org [210.199.215.5] by localhost with POP3 (fetchmail-5.9.3) for rei@localhost (single-drop); Thu, 04 Oct 2001 10:30:12 +0900 (JST) Received: (qmail 20071 invoked from network); 4 Oct 2001 01:20:58 -0000 Received: from vger.kernel.org (199.183.24.194) by orochi.oucc.org with SMTP; 4 Oct 2001 01:20:58 -0000 Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id ; Wed, 3 Oct 2001 21:18:00 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id ; Wed, 3 Oct 2001 21:17:50 -0400 Received: from shell.cyberus.ca ([209.195.95.7]:63160 "EHLO shell.cyberus.ca") by vger.kernel.org with ESMTP id ; Wed, 3 Oct 2001 21:17:38 -0400 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id VAA08133; Wed, 3 Oct 2001 21:15:20 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Wed, 3 Oct 2001 21:15:20 -0400 (EDT) Old-From: jamal Old-To: Cc: Subject: Some NAPI results Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Mailing-List: linux-kernel@vger.kernel.org X-Loop: takahashi@future-s.com From: takahashi@future-s.com To: rei@rei.sh X-Virus-Scanned: by AMaViS perl-11 Sender: owner-netdev@oss.sgi.com Precedence: bulk Since we are discussing this, heres more data ;-> ---------- Forwarded message ---------- Date: Wed, 3 Oct 2001 20:42:43 +0200 From: Robert Olsson Hello! Anyway. Uppsala University (35.000 students, 4000 emploees) has now SMP/NAPI in production. But reniceed ksoftirqd to -10. Kernel 2.4.10poll SMP. USER PID %CPU %MEM SIZE RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 1224 512 ? S 18:08 0:03 init [5] root 2 0.0 0.0 0 0 ? SW 18:08 0:00 (keventd) root 3 74.2 0.0 0 0 ? RW< 18:08 100:02 (ksoftirqd_CPU0) root 4 73.9 0.0 0 0 ? SW< 18:08 99:39 (ksoftirqd_CPU1) root 5 0.0 0.0 0 0 ? SW 18:08 0:00 (kswapd) root 6 0.0 0.0 0 0 ? SW 18:08 0:00 (bdflush) root 7 0.0 0.0 0 0 ? SW 18:08 0:00 (kupdated) root 52 0.0 0.0 796 192 ? S 18:08 0:00 voff root 570 0.3 7.4 38696 38184 ? S 18:08 0:28 /sbin/gated From owner-netdev@oss.sgi.com Wed Oct 3 18:42:51 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f941gpE30782 for netdev-outgoing; Wed, 3 Oct 2001 18:42:51 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f941gmD30779 for ; Wed, 3 Oct 2001 18:42:48 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id VAA08197; Wed, 3 Oct 2001 21:39:50 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Wed, 3 Oct 2001 21:39:50 -0400 (EDT) From: jamal To: Benjamin LaHaise cc: , , , , , , Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <20011003213010.F3780@redhat.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, Benjamin LaHaise wrote: > On Wed, Oct 03, 2001 at 09:10:10PM -0400, jamal wrote: > > > Well, this sounds like a 2.5 patch. When do we get to merge it? > > > > > > It is backward compatible to 2.4 netif_rx() which means it can go in now. > > The problem is netdrivers that want to use the interface have to be > > morphed. > > I'm alluding to the fact that we need a place to put in-development patches. > Sorry ;-> Yes, where is is 2.5 again? ;-> > > As a general disclaimer, i really dont mean to put down Ingo's efforts i > > just think the irq mitigation idea as is now is wrong for both 2.4 and 2.5 > > What is your solution to the problem? Leaving it up to the driver authors > doesn't work as they're not perfect. Yes, drivers should attempt to do a > good job at irq mitigation, but sometimes a safety net is needed. > To be honest i am getting a little nervous with what i saw in something that seems to be a stable kernel. I was nervous when i saw ksoftirq, but its already in there. I think we can use the ksoftirq replacement pending testing to show if latency is improved. I have time this weekend, if that patch can be isolated it can be tested with NAPI etc. As for the irq mitigation, in its current form it is insufficient; but would be OK to go into 2.5 with plans to go and implement the isolation feature. I would put NAPI into this same category. We can then backport both back to 2.4. With current 2.4, i say yes, we leave it to the drivers (and infact claim we have a sustainable solution if conformed to) cheers, jamal From owner-netdev@oss.sgi.com Wed Oct 3 23:41:11 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f946fBD02937 for netdev-outgoing; Wed, 3 Oct 2001 23:41:11 -0700 Received: from chiara.elte.hu (chiara.elte.hu [157.181.150.200]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f946f8D02934 for ; Wed, 3 Oct 2001 23:41:08 -0700 Received: by chiara.elte.hu (Postfix, from userid 17806) id 094491FCF; Thu, 4 Oct 2001 08:37:21 +0200 (CEST) Date: Thu, 4 Oct 2001 08:35:02 +0200 (CEST) From: Ingo Molnar Reply-To: To: jamal Cc: Alexey Kuznetsov , , Robert Olsson , , , Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, jamal wrote: > > i'm worried by the dev->quota variable a bit. As visible now in the > > 2.4.10-poll.pat and tulip-NAPI-010910.tar.gz code, it keeps calling the > > ->poll() function until dev->quota is gone. I think it should only keep > > calling the function until the rx ring is fully processed - and it should > > re-enable the receiver afterwards, when exiting net_rx_action. > > This would result in an unfairness. Think of one device which receives > packets really fast that it takes most of the CPU capacity just > processing it. no, i asked something else. i'm asking the following thing. dev->quota, as i read the patch now, can cause extra calls to ->poll() even though the RX ring of that particular device is empty and the driver has indicated it's done processing RX packets. (i'm now assuming that the extra-polling-for-a-jiffy line in the current patch is removed - that one is a showstopper to begin with.) Is this claim of mine correct? Ingo From owner-netdev@oss.sgi.com Wed Oct 3 23:48:09 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f946m9203134 for netdev-outgoing; Wed, 3 Oct 2001 23:48:09 -0700 Received: from grok.yi.org (IDENT:root@cx97923-a.phnx3.az.home.com [24.9.112.194]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f946m6D03131 for ; Wed, 3 Oct 2001 23:48:06 -0700 Received: from candelatech.com (IDENT:greear@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.2/8.11.2) with ESMTP id f946l8T21919; Wed, 3 Oct 2001 23:47:08 -0700 Message-ID: <3BBC05EC.AA9BFB4F@candelatech.com> Date: Wed, 03 Oct 2001 23:47:08 -0700 From: Ben Greear Organization: Candela Technologies X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.4.3-12 i686) X-Accept-Language: en MIME-Version: 1.0 To: jamal CC: Simon Kirby , Ingo Molnar , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , netdev@oss.sgi.com, Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk jamal wrote: > > On Wed, 3 Oct 2001, Simon Kirby wrote: > > > On Wed, Oct 03, 2001 at 09:33:12AM -0700, Linus Torvalds wrote: > > > > Actually, the way I first started looking at this problem is the result > > of a few attacks that have happened on our network. It's not just a > > while(1) sendto(); UDP spamming program that triggers it -- TCP SYN > > floods show the problem as well, and _there is no way_ to protect against > > this without using syncookies or some similar method that can only be > > done on the receiving TCP stack only. > > > > At one point, one of our webservers received 30-40Mbit/sec of SYN packets > > sustained for almost 24 hours. Needless to say, the machine was not > > happy. > > > > I think you can save yourself a lot of pain today by going to a "better > driver"/hardware. Switch to a tulip based board; in particular one which > is based on the 21143 chipset. Compile in hardware traffic control and > save yourself some pain. The tulip driver only started working for my DLINK 4-port NIC after about 2.4.8, and last I checked the ZYNX 4-port still refuses to work, so I wouldn't consider it a paradigm of stability and grace quite yet. Regardless of that, it is often impossible to trade NICS (think built-in 1U servers), and claiming to only work correctly on certain hardware (and potentially lock up hard on other hardware) is a pretty sorry state of affairs... Ben -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From owner-netdev@oss.sgi.com Wed Oct 3 23:51:04 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f946p4v03259 for netdev-outgoing; Wed, 3 Oct 2001 23:51:04 -0700 Received: from grok.yi.org (IDENT:root@cx97923-a.phnx3.az.home.com [24.9.112.194]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f946p1D03256 for ; Wed, 3 Oct 2001 23:51:01 -0700 Received: from candelatech.com (IDENT:greear@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.2/8.11.2) with ESMTP id f946o9T21958; Wed, 3 Oct 2001 23:50:09 -0700 Message-ID: <3BBC06A1.1818E45C@candelatech.com> Date: Wed, 03 Oct 2001 23:50:09 -0700 From: Ben Greear Organization: Candela Technologies X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.4.3-12 i686) X-Accept-Language: en MIME-Version: 1.0 To: jamal CC: Ingo Molnar , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , netdev@oss.sgi.com, Linus Torvalds , Alan Cox , Simon Kirby Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk jamal wrote: > Your patch with Linus' idea of "flag mask" would be more acceptable as a > last resort. All subsytems should be cooperative and we resort to this to > send misbehaving kids to their room. That requires re-writing all the drivers, right? Seems a very bad thing to do in 2.4 > > > Your NAPI patch, or any driver/subsystem that does flowcontrol accurately > > should never be affected by it in any way. No overhead, no performance > > hit. > > so far your appraoch is that of a shotgun i.e "let me fire in > that crowd and i'll hit my target but dont care if i take down a few > more"; regardless of how noble the reasoning is, it's as Linus described > it -- a sledge hammer. Aye, but by shooting this target and getting a few bystanders, you save everyone else... (And it's only a flesh wound!!) Ben -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From owner-netdev@oss.sgi.com Wed Oct 3 23:56:24 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f946uOc03403 for netdev-outgoing; Wed, 3 Oct 2001 23:56:24 -0700 Received: from mandrakesoft.mandrakesoft.com (nsd.netnomics.com [216.71.84.35] (may be forged)) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f946uND03400 for ; Wed, 3 Oct 2001 23:56:23 -0700 Received: from localhost (jgarzik@localhost) by mandrakesoft.mandrakesoft.com (8.8.5/8.8.5) with SMTP id BAA24874; Thu, 4 Oct 2001 01:55:23 -0500 Date: Thu, 4 Oct 2001 01:55:23 -0500 (CDT) From: Jeff Garzik To: Ben Greear cc: jamal , Ingo Molnar , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , netdev@oss.sgi.com, Linus Torvalds , Alan Cox , Simon Kirby Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <3BBC06A1.1818E45C@candelatech.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, Ben Greear wrote: > That requires re-writing all the drivers, right? NAPI? no. You move some existing code into a separate function, mainly Jeff From owner-netdev@oss.sgi.com Thu Oct 4 01:45:33 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f948jXC05269 for netdev-outgoing; Thu, 4 Oct 2001 01:45:33 -0700 Received: from peace.netnation.com (mail@peace.netnation.com [204.174.223.2]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f948jUD05265 for ; Thu, 4 Oct 2001 01:45:30 -0700 Received: from sim by peace.netnation.com with local (Exim 3.13 #5) id 15p48C-00017q-00; Thu, 04 Oct 2001 01:45:24 -0700 Date: Thu, 4 Oct 2001 01:45:24 -0700 From: Simon Kirby To: jamal Cc: Ben Greear , Ingo Molnar , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , netdev@oss.sgi.com, Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: <20011004014524.A1496@netnation.com> References: <20011003130203.A2315@netnation.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0i In-Reply-To: ; from hadi@cyberus.ca on Wed, Oct 03, 2001 at 09:04:22PM -0400 Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, Oct 03, 2001 at 09:04:22PM -0400, jamal wrote: > I think you can save yourself a lot of pain today by going to a "better > driver"/hardware. Switch to a tulip based board; in particular one which > is based on the 21143 chipset. Compile in hardware traffic control and > save yourself some pain. Or an Acenic-based card, but that's more expensive. The problem we had with Tulip-based cards is that it's hard to find a good model (variant) that is supported with different kernel versions and stock drivers, doesn't change internally with time, and is easily distinguishable by our hardware suppliers. "Intel EtherExpress PRO100+" is difficult to get wrong, and there are generally less issues with driver compatibility because there are many fewer (no) clones, just a few different board revisions. The same goes with 3COM 905/980s, etc. I'm not saying Tulips aren't better (they probably are, competition is good), but eepro100s are quite simple (and have been reliable for our servers much more than 3com 905s and other cards have been in the past). Simon- [ Stormix Technologies Inc. ][ NetNation Communications Inc. ] [ sim@stormix.com ][ sim@netnation.com ] [ Opinions expressed are not necessarily those of my employers. ] From owner-netdev@oss.sgi.com Thu Oct 4 02:49:03 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f949n3O06699 for netdev-outgoing; Thu, 4 Oct 2001 02:49:03 -0700 Received: from colin.muc.de (root@colin.muc.de [193.149.48.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f949n0D06696 for ; Thu, 4 Oct 2001 02:49:00 -0700 Received: by colin.muc.de id <140552-3>; Thu, 4 Oct 2001 11:49:26 +0200 Message-ID: <20011004114917.57615@colin.muc.de> Date: Thu, 4 Oct 2001 11:49:17 +0200 From: Andi Kleen To: Mark Price Cc: netdev@oss.sgi.com, davem@redhat.com, kuznet@ms2.inr.ac.ru Subject: Re: patch - SNMP Kernel Counters References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 0.88e In-Reply-To: ; from Mark Price on Thu, Oct 04, 2001 at 12:02:05AM +0200 Sender: owner-netdev@oss.sgi.com Precedence: bulk On Thu, Oct 04, 2001 at 12:02:05AM +0200, Mark Price wrote: > + > + /* Setup the SNMP Stats for the TCP RTO values. > + * TcpMaxConn = -1 as the maximum number of connections is > + * dynamic. > + */ > + > + tcp_statistics[0].TcpRtoAlgorithm=TCP_RTO_JACOBSON; That's not quite correct. 2.4 doesn't use the Van Jacobson algorithm anymore (see ftp.inr.ac.ru:/ip-routing/README.rto). I don't know what the right value should be however. -Andi From owner-netdev@oss.sgi.com Thu Oct 4 04:38:47 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94Bclf08465 for netdev-outgoing; Thu, 4 Oct 2001 04:38:47 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94BcZD08461 for ; Thu, 4 Oct 2001 04:38:35 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id HAA09480; Thu, 4 Oct 2001 07:34:56 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Thu, 4 Oct 2001 07:34:56 -0400 (EDT) From: jamal To: Ingo Molnar cc: , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Linus Torvalds , Alan Cox , Simon Kirby Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Thu, 4 Oct 2001, Ingo Molnar wrote: > > On Wed, 3 Oct 2001, jamal wrote: > > > > which in turn stops that device as well sooner or later. Optionally, > > > in the future, this can be made more finegrained for chipsets that > > > support device-independent IRQ mitigation features, like the USB 2.0 > > > EHCI feature mentioned by David Brownell. > > > I think each subsytem should be in charge of its own fate. USB applies > > in whatever subsystem it belongs to. Cooperating subsystems doing what > > os best for the system. > > this is a claim that is nearly perverse and shows a fundamental > misunderstanding of how Linux handles error situations. Perhaps we should > never check NULL pointer dereference in the networking code? Should the > NMI oopser not debug networking related lockups? Should we never print a > warning message on a double enable_irq() in a bad networking driver? > > *of course* if a chipset supports IRQ mitigation then the generic IRQ code > can be enabled to use it. We can have networking devices over USB as well. > USB is a bus protocol that provides access to devices, not just a > 'subsystem'. And *of course*, the IRQ code is completely right to do > various sanity checks - as it does today. > > Linux has various safety nets in various places - always had. It's always > the history of problems in a certain area, the seriousness and impact of > the problem, and the intrusiveness of the safety approach that decides > whether some safety net is added or not, whether it's put under > CONFIG_KERNEL_DEBUG or not. While everybody is free to disagree about the > importance of this particular safety net, just saying 'do not mess with > *our* interrupts' sounds rather childish. Especially considering that > tools are available to trigger lockups via broadband access. Especially > considering that just a few mails earlier you claimed that such lockups do > not even exist. To quote that paragraph of yours: > Your scheme is definetely a safety net, no doubt. But is incomplete. Whatever subsytem/softirq/process in charge of the USB devices is the next level of delegation. And of course the driver knows best what is good for the goose. We delegate at each level of the hierachy and and the ultimate authority is your code, when it is done right. But i think we are deviating. We started this with network drivers, which is where the real proven issue is. > # Date: Wed, 3 Oct 2001 08:49:51 -0400 (EDT) > # From: jamal > > [...] > # You dont need the patch for 2.4 to work against any lockups. And > # infact i am suprised that you observe _any_ lockups on a PIII which > # are not observed on my PII. Linux as is, without any tuneups can > # handle upto about 40000 packets/sec input before you start observing > # user space startvations. This is about 30Mbps at 64 byte packets; its > # about 60Mbps at 128 byte packets and comfortable at 100Mbps with byte > # size of 256. We really dont have a problem at 100Mbps. > > so you should never see any lockups. > For your P3 is what i meant since i see none on my P2 at the 100Mbps with 256 bytes. But you probably meant user-space starvation under maybe twice that rate to which i agree and apologize for misunderstanding. > > Your patch with Linus' idea of "flag mask" would be more acceptable as > > a last resort. All subsytems should be cooperative and we resort to > > this to send misbehaving kids to their room. > > i have nothing against it in 2.5, of course. Until then => my patch adds > an irq.c daddy that sends the bad kids to their room. until then change the eepro to use at least hardware flow control. If it has mitigation, use the return codes from netif_rx(). Lets see if that doesnt help you; yes, its a pain, but avoids a lot of unknowns which your patch introduces. > > > Your NAPI patch, or any driver/subsystem that does flowcontrol accurately > > > should never be affected by it in any way. No overhead, no performance > > > hit. > > > > so far your appraoch is that of a shotgun [...] > > i'm not sure what this has to do with your NAPI patch. You should never > see the code trigger. It's an unused sledgehammer (or shotgun) put into > the garage, as far as NAPI is concerned. And besides, there are lots of > people on your continent that believe in spare shotguns ;) > > i'd rather compare this approach to an airbag, or perhaps shackles. > Interrupt auto-limiting, despite your absurd and misleading analogy, does > not 'destroy' or 'kill' anything. It merely limits an IRQ source for up to > 10 msecs (if HZ is 1000 then it's only 1 msec), if that IRQ source has > been detected to be critically misbehaving. Well, i meant two things: 1) you shut down shared interupts; take a look at this posting by Marcus Sundberg --------------- 0: 7602983 XT-PIC timer 1: 10575 XT-PIC keyboard 2: 0 XT-PIC cascade 8: 1 XT-PIC rtc 11: 1626004 XT-PIC Toshiba America Info Systems ToPIC95 PCI \ to Cardbus Bridge with ZV Support, Toshiba America Info Systems ToPIC95 PCI \ to Cardbus Bridge with ZV Support (#2), usb-uhci, eth0, BreezeCom Card, \ Intel 440MX, irda0 12: 1342 XT-PIC PS/2 Mouse 14: 23605 XT-PIC ide0 ----------------------------- Now you go and shut down IRQ 11 and punish all devices there. If you can avoid that, it is acceptable as a temporary replacement to be upgraded to a better scheme. 2) By not being granular enough and shutting down sources of noise, you are actually not being effective in increasing system utilization. weve beat this to death. cheers, jamal From owner-netdev@oss.sgi.com Thu Oct 4 04:44:48 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94BimE08614 for netdev-outgoing; Thu, 4 Oct 2001 04:44:48 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94BikD08610 for ; Thu, 4 Oct 2001 04:44:46 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id HAA09531; Thu, 4 Oct 2001 07:41:40 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Thu, 4 Oct 2001 07:41:39 -0400 (EDT) From: jamal To: Ingo Molnar cc: Alexey Kuznetsov , , Robert Olsson , , , Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Thu, 4 Oct 2001, Ingo Molnar wrote: > i'm asking the following thing. dev->quota, as i read the patch now, can > cause extra calls to ->poll() even though the RX ring of that particular > device is empty and the driver has indicated it's done processing RX > packets. (i'm now assuming that the extra-polling-for-a-jiffy line in the > current patch is removed - that one is a showstopper to begin with.) Is > this claim of mine correct? There should be no extra calls to ->poll() and if there are we should fix them. Take a look at the state machine i posted earlier. The one liner is removed. cheers, jamal From owner-netdev@oss.sgi.com Thu Oct 4 04:50:30 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94BoUc08811 for netdev-outgoing; Thu, 4 Oct 2001 04:50:30 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94BoRD08808 for ; Thu, 4 Oct 2001 04:50:27 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id HAA09544; Thu, 4 Oct 2001 07:47:26 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Thu, 4 Oct 2001 07:47:26 -0400 (EDT) From: jamal To: Ben Greear cc: Simon Kirby , Ingo Molnar , , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <3BBC05EC.AA9BFB4F@candelatech.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, Ben Greear wrote: > The tulip driver only started working for my DLINK 4-port NIC after > about 2.4.8, and last I checked the ZYNX 4-port still refuses to work, > so I wouldn't consider it a paradigm of stability and grace quite yet. The tests in www.cyberus.ca/~hadi/247-res/ were done with 4-port znyx cards using 2.4.7. What kind of problems are you having? Maybe i can help. > Regardless of that, it is often impossible to trade NICS (think > built-in 1U servers), and claiming to only work correctly on certain > hardware (and potentially lock up hard on other hardware) is a pretty > sorry state of affairs... My point is that the API exists. Driver owners could use it; this discussion seems to have at least helped to point in the existence of the API. Alexey had the hardware flow control in there since 2.1.x .., us that at least. In my opinion, Ingos patch is radical enough to be allowed in when we are approaching stability. And it is a lazy way of solving the problem cheers, jamal From owner-netdev@oss.sgi.com Thu Oct 4 04:52:45 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94Bqjk08907 for netdev-outgoing; Thu, 4 Oct 2001 04:52:45 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94BqhD08904 for ; Thu, 4 Oct 2001 04:52:43 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id HAA09549; Thu, 4 Oct 2001 07:49:46 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Thu, 4 Oct 2001 07:49:46 -0400 (EDT) From: jamal To: Ingo Molnar cc: Simon Kirby , Linus Torvalds , Ben Greear , , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Thu, 4 Oct 2001, Ingo Molnar wrote: > > On Wed, 3 Oct 2001, jamal wrote: > > > I think you can save yourself a lot of pain today by going to a > > "better driver"/hardware. Switch to a tulip based board; [...] > > This is not an option in many cases. (eg. where a company standardizes on > something non-tulip, or due to simple financial/organizational reasons.) > What you say is the approach i see in the FreeBSD camp frequently: "use > these [limited set of] wonderful cards and drivers, the rest sucks > hardware-design-wise and we dont really care about them", which elitist > attitude i strongly disagree with. > It is not elitist. Maybe we can force people to use the API now. it exists. And hardware flow control does not require special hardware features. As well NAPI kills the requirement for mitigation in the future. cheers, jamal From owner-netdev@oss.sgi.com Thu Oct 4 04:53:33 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94BrXf08996 for netdev-outgoing; Thu, 4 Oct 2001 04:53:33 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94BrUD08993 for ; Thu, 4 Oct 2001 04:53:30 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id HAA09555; Thu, 4 Oct 2001 07:50:32 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Thu, 4 Oct 2001 07:50:32 -0400 (EDT) From: jamal To: Ingo Molnar cc: Ben Greear , , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Linus Torvalds , Alan Cox , Simon Kirby Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Thu, 4 Oct 2001, Ingo Molnar wrote: > > On Wed, 3 Oct 2001, Ben Greear wrote: > > > > so far your appraoch is that of a shotgun i.e "let me fire in > > > that crowd and i'll hit my target but dont care if i take down a few > > > more"; regardless of how noble the reasoning is, it's as Linus described > > > it -- a sledge hammer. > > > > Aye, but by shooting this target and getting a few bystanders, you save > > everyone else... (And it's only a flesh wound!!) > > especially considering that the current code nukes the whole city ;) > Ingo, cut down on the bad mushrooms ;-> cheers, jamal From owner-netdev@oss.sgi.com Thu Oct 4 04:57:22 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94BvMm09227 for netdev-outgoing; Thu, 4 Oct 2001 04:57:22 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94BvJD09224 for ; Thu, 4 Oct 2001 04:57:19 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id HAA09563; Thu, 4 Oct 2001 07:54:19 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Thu, 4 Oct 2001 07:54:19 -0400 (EDT) From: jamal To: Simon Kirby cc: Ben Greear , Ingo Molnar , , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <20011004014524.A1496@netnation.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Thu, 4 Oct 2001, Simon Kirby wrote: > On Wed, Oct 03, 2001 at 09:04:22PM -0400, jamal wrote: > > > I think you can save yourself a lot of pain today by going to a "better > > driver"/hardware. Switch to a tulip based board; in particular one which > > is based on the 21143 chipset. Compile in hardware traffic control and > > save yourself some pain. > > Or an Acenic-based card, but that's more expensive. > > The problem we had with Tulip-based cards is that it's hard to find a > good model (variant) that is supported with different kernel versions and > stock drivers, doesn't change internally with time, and is easily > distinguishable by our hardware suppliers. "Intel EtherExpress PRO100+" > is difficult to get wrong, and there are generally less issues with > driver compatibility because there are many fewer (no) clones, just a few > different board revisions. The same goes with 3COM 905/980s, etc. > > I'm not saying Tulips aren't better (they probably are, competition is > good), but eepro100s are quite simple (and have been reliable for our > servers much more than 3com 905s and other cards have been in the past). > Has nothing to do with specific hardware although i see your point. send me an eepro and i'll at least add hardware flow control for you. The API is simple, its up to the driver maintainers to use. This discussion is good to make people aware of those drivers. cheers, jamal From owner-netdev@oss.sgi.com Thu Oct 4 05:13:30 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94CDU109592 for netdev-outgoing; Thu, 4 Oct 2001 05:13:30 -0700 Received: from chiara.elte.hu (chiara.elte.hu [157.181.150.200]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94CDRD09589 for ; Thu, 4 Oct 2001 05:13:28 -0700 Received: by chiara.elte.hu (Postfix, from userid 17806) id E13A51FD3; Thu, 4 Oct 2001 08:58:58 +0200 (CEST) Date: Thu, 4 Oct 2001 08:56:36 +0200 (CEST) From: Ingo Molnar Reply-To: To: Jeff Garzik Cc: Ben Greear , jamal , , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Linus Torvalds , Alan Cox , Simon Kirby Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Thu, 4 Oct 2001, Jeff Garzik wrote: > On Wed, 3 Oct 2001, Ben Greear wrote: > > That requires re-writing all the drivers, right? > > NAPI? [...] Ben is talking about the long-planned "irq_action->handler() returns a code that indicates progress" approach Linus talked about. *that* needs the changing of every driver, since every IRQ handler prototype that is 'void' now needs to be changed to return 'int'. (the change is trivial, but intrusive.) Ingo From owner-netdev@oss.sgi.com Thu Oct 4 05:45:02 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94Cj2v10045 for netdev-outgoing; Thu, 4 Oct 2001 05:45:02 -0700 Received: from netcore.fi (netcore.fi [193.94.160.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94CiwD10032 for ; Thu, 4 Oct 2001 05:44:58 -0700 Received: from localhost (pekkas@localhost) by netcore.fi (8.11.6/8.11.6) with ESMTP id f94Cipg27219 for ; Thu, 4 Oct 2001 15:44:52 +0300 Date: Thu, 4 Oct 2001 15:44:51 +0300 (EEST) From: Pekka Savola To: Subject: Should IP addresses on interfaces not UP respond to ping? Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Hi, With 2.2.18 I noticed something that looked interesting: # /sbin/ip a l dev eth4 7: eth4: mtu 1500 qdisc pfifo_fast qlen 100 link/ether 00:80:c8:c9:b8:14 brd ff:ff:ff:ff:ff:ff inet x.y.7.252/24 brd x.y.7.255 scope global eth4 Note that the interface is not UP. Whether it's promisc or not does not affect this. However, the address is still pingable from outside, through eth0! Also noticed the same behaviour in 2.4.10. Is this the intended behaviour, probably? One could argue that if interface isn't UP, it shouldn't be able to send or receive packets at all. I wonder what changing this would break.. -- Pekka Savola "Tell me of difficulties surmounted, Netcore Oy not those you stumble over and fall" Systems. Networks. Security. -- Robert Jordan: A Crown of Swords From owner-netdev@oss.sgi.com Thu Oct 4 05:48:21 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94CmL410168 for netdev-outgoing; Thu, 4 Oct 2001 05:48:21 -0700 Received: from robur.slu.se (robur.slu.se [130.238.98.12]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94CmGD10164 for ; Thu, 4 Oct 2001 05:48:16 -0700 Received: (from robert@localhost) by robur.slu.se (8.8.7/8.8.7) id OAA23956; Thu, 4 Oct 2001 14:50:24 +0200 From: Robert Olsson MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15292.23312.641442.909160@robur.slu.se> Date: Thu, 4 Oct 2001 14:50:24 +0200 To: takahashi@future-s.com Cc: rei@rei.sh, Subject: Some NAPI results In-Reply-To: References: X-Mailer: VM 6.92 under Emacs 19.34.1 Sender: owner-netdev@oss.sgi.com Precedence: bulk Correct! We were interested to see the the packet distribution on the different CPU's with this scheme. Reality had data we couldn't get in the lab. Seems there is nothing to complain about. Yet. I'd better say. :-) cat /proc/net/softnet_stat 2b85c320 0000d374 6524ce48 00000000 00000000 00000000 00000000 00000000 0007119d 2b8b5e29 0000d615 653eba32 00000000 00000000 00000000 00000000 00000000 00072f44 First col: pkts Second col: drops Third col: time_squeeze (by the polling loop) Last col: cpu_collsions (TX) eth0: UP Locked MII Full DuplexLink UP Admin up 20 hour(s) 23 min 44 sec Last input NOW Last output NOW 5min RX bit/s 91.5 M 5min TX bit/s 57.4 M 5min RX pkts/s 14261 5min TX pkts/s 12742 Cheers. --ro takahashi@future-s.com writes: > > > Since we are discussing this, heres more data ;-> > > > ---------- Forwarded message ---------- > Date: Wed, 3 Oct 2001 20:42:43 +0200 > From: Robert Olsson > > Hello! > > Anyway. Uppsala University (35.000 students, 4000 emploees) > has now SMP/NAPI in production. > But reniceed ksoftirqd to -10. Kernel 2.4.10poll SMP. > > > USER PID %CPU %MEM SIZE RSS TTY STAT START TIME COMMAND > root 1 0.0 0.0 1224 512 ? S 18:08 0:03 init [5] > root 2 0.0 0.0 0 0 ? SW 18:08 0:00 (keventd) > root 3 74.2 0.0 0 0 ? RW< 18:08 100:02 (ksoftirqd_CPU0) > root 4 73.9 0.0 0 0 ? SW< 18:08 99:39 (ksoftirqd_CPU1) > root 5 0.0 0.0 0 0 ? SW 18:08 0:00 (kswapd) > root 6 0.0 0.0 0 0 ? SW 18:08 0:00 (bdflush) > root 7 0.0 0.0 0 0 ? SW 18:08 0:00 (kupdated) > root 52 0.0 0.0 796 192 ? S 18:08 0:00 voff > root 570 0.3 7.4 38696 38184 ? S 18:08 0:28 /sbin/gated From owner-netdev@oss.sgi.com Thu Oct 4 06:02:52 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94D2qr10420 for netdev-outgoing; Thu, 4 Oct 2001 06:02:52 -0700 Received: from robur.slu.se (robur.slu.se [130.238.98.12]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94D2mD10412 for ; Thu, 4 Oct 2001 06:02:48 -0700 Received: (from robert@localhost) by robur.slu.se (8.8.7/8.8.7) id PAA24126; Thu, 4 Oct 2001 15:05:05 +0200 From: Robert Olsson MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15292.24193.308549.459631@robur.slu.se> Date: Thu, 4 Oct 2001 15:05:05 +0200 To: Cc: jamal , Alexey Kuznetsov , , Robert Olsson , , , Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: References: X-Mailer: VM 6.92 under Emacs 19.34.1 Sender: owner-netdev@oss.sgi.com Precedence: bulk Ingo Molnar writes: > > i'm asking the following thing. dev->quota, as i read the patch now, can > cause extra calls to ->poll() even though the RX ring of that particular > device is empty and the driver has indicated it's done processing RX > packets. (i'm now assuming that the extra-polling-for-a-jiffy line in the > current patch is removed - that one is a showstopper to begin with.) Is > this claim of mine correct? Hello! Well I'm the one to blame... :-) This comes from my experiments to delay to polling before going into RX-irq-enable mode. This is one of the areas to be addressed further with NAPI. And this code was not in any of the files that I announced I think..? Cheers. --ro From owner-netdev@oss.sgi.com Thu Oct 4 06:07:55 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94D7t710530 for netdev-outgoing; Thu, 4 Oct 2001 06:07:55 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94D7oD10526 for ; Thu, 4 Oct 2001 06:07:50 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id JAA09919; Thu, 4 Oct 2001 09:05:00 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Thu, 4 Oct 2001 09:04:59 -0400 (EDT) From: jamal To: Robert Olsson cc: Subject: Re: Some NAPI results In-Reply-To: <15292.23312.641442.909160@robur.slu.se> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Now if we can cut that second column to all 0s we are set ;-> cheers, jamal On Thu, 4 Oct 2001, Robert Olsson wrote: > > Correct! > > We were interested to see the the packet distribution on the different CPU's > with this scheme. Reality had data we couldn't get in the lab. Seems there > is nothing to complain about. Yet. I'd better say. :-) > > cat /proc/net/softnet_stat > 2b85c320 0000d374 6524ce48 00000000 00000000 00000000 00000000 00000000 0007119d > 2b8b5e29 0000d615 653eba32 00000000 00000000 00000000 00000000 00000000 00072f44 > > First col: pkts > Second col: drops > Third col: time_squeeze (by the polling loop) > Last col: cpu_collsions (TX) > > eth0: UP Locked MII Full DuplexLink UP > Admin up 20 hour(s) 23 min 44 sec > Last input NOW > Last output NOW > 5min RX bit/s 91.5 M > 5min TX bit/s 57.4 M > 5min RX pkts/s 14261 > 5min TX pkts/s 12742 > > Cheers. > --ro > > > takahashi@future-s.com writes: > > > > > > Since we are discussing this, heres more data ;-> > > > > > > ---------- Forwarded message ---------- > > Date: Wed, 3 Oct 2001 20:42:43 +0200 > > From: Robert Olsson > > > > Hello! > > > > Anyway. Uppsala University (35.000 students, 4000 emploees) > > has now SMP/NAPI in production. > > But reniceed ksoftirqd to -10. Kernel 2.4.10poll SMP. > > > > > > USER PID %CPU %MEM SIZE RSS TTY STAT START TIME COMMAND > > root 1 0.0 0.0 1224 512 ? S 18:08 0:03 init [5] > > root 2 0.0 0.0 0 0 ? SW 18:08 0:00 (keventd) > > root 3 74.2 0.0 0 0 ? RW< 18:08 100:02 (ksoftirqd_CPU0) > > root 4 73.9 0.0 0 0 ? SW< 18:08 99:39 (ksoftirqd_CPU1) > > root 5 0.0 0.0 0 0 ? SW 18:08 0:00 (kswapd) > > root 6 0.0 0.0 0 0 ? SW 18:08 0:00 (bdflush) > > root 7 0.0 0.0 0 0 ? SW 18:08 0:00 (kupdated) > > root 52 0.0 0.0 796 192 ? S 18:08 0:00 voff > > root 570 0.3 7.4 38696 38184 ? S 18:08 0:28 /sbin/gated > From owner-netdev@oss.sgi.com Thu Oct 4 07:27:01 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94ER1W12023 for netdev-outgoing; Thu, 4 Oct 2001 07:27:01 -0700 Received: from chiara.elte.hu (chiara.elte.hu [157.181.150.200]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94EQqD12017 for ; Thu, 4 Oct 2001 07:26:52 -0700 Received: by chiara.elte.hu (Postfix, from userid 17806) id DFA371FCE; Thu, 4 Oct 2001 08:31:00 +0200 (CEST) Date: Thu, 4 Oct 2001 08:28:40 +0200 (CEST) From: Ingo Molnar Reply-To: To: jamal Cc: , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Linus Torvalds , Alan Cox , Simon Kirby Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, jamal wrote: > > which in turn stops that device as well sooner or later. Optionally, > > in the future, this can be made more finegrained for chipsets that > > support device-independent IRQ mitigation features, like the USB 2.0 > > EHCI feature mentioned by David Brownell. > I think each subsytem should be in charge of its own fate. USB applies > in whatever subsystem it belongs to. Cooperating subsystems doing what > os best for the system. this is a claim that is nearly perverse and shows a fundamental misunderstanding of how Linux handles error situations. Perhaps we should never check NULL pointer dereference in the networking code? Should the NMI oopser not debug networking related lockups? Should we never print a warning message on a double enable_irq() in a bad networking driver? *of course* if a chipset supports IRQ mitigation then the generic IRQ code can be enabled to use it. We can have networking devices over USB as well. USB is a bus protocol that provides access to devices, not just a 'subsystem'. And *of course*, the IRQ code is completely right to do various sanity checks - as it does today. Linux has various safety nets in various places - always had. It's always the history of problems in a certain area, the seriousness and impact of the problem, and the intrusiveness of the safety approach that decides whether some safety net is added or not, whether it's put under CONFIG_KERNEL_DEBUG or not. While everybody is free to disagree about the importance of this particular safety net, just saying 'do not mess with *our* interrupts' sounds rather childish. Especially considering that tools are available to trigger lockups via broadband access. Especially considering that just a few mails earlier you claimed that such lockups do not even exist. To quote that paragraph of yours: # Date: Wed, 3 Oct 2001 08:49:51 -0400 (EDT) # From: jamal [...] # You dont need the patch for 2.4 to work against any lockups. And # infact i am suprised that you observe _any_ lockups on a PIII which # are not observed on my PII. Linux as is, without any tuneups can # handle upto about 40000 packets/sec input before you start observing # user space startvations. This is about 30Mbps at 64 byte packets; its # about 60Mbps at 128 byte packets and comfortable at 100Mbps with byte # size of 256. We really dont have a problem at 100Mbps. so you should never see any lockups. > Your patch with Linus' idea of "flag mask" would be more acceptable as > a last resort. All subsytems should be cooperative and we resort to > this to send misbehaving kids to their room. i have nothing against it in 2.5, of course. Until then => my patch adds an irq.c daddy that sends the bad kids to their room. > > Your NAPI patch, or any driver/subsystem that does flowcontrol accurately > > should never be affected by it in any way. No overhead, no performance > > hit. > > so far your appraoch is that of a shotgun [...] i'm not sure what this has to do with your NAPI patch. You should never see the code trigger. It's an unused sledgehammer (or shotgun) put into the garage, as far as NAPI is concerned. And besides, there are lots of people on your continent that believe in spare shotguns ;) i'd rather compare this approach to an airbag, or perhaps shackles. Interrupt auto-limiting, despite your absurd and misleading analogy, does not 'destroy' or 'kill' anything. It merely limits an IRQ source for up to 10 msecs (if HZ is 1000 then it's only 1 msec), if that IRQ source has been detected to be critically misbehaving. Ingo From owner-netdev@oss.sgi.com Thu Oct 4 07:32:49 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94EWnE12353 for netdev-outgoing; Thu, 4 Oct 2001 07:32:49 -0700 Received: from smp.paktronix.com (pak200.pakuni.net [207.91.34.200]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94EWiD12350 for ; Thu, 4 Oct 2001 07:32:44 -0700 Received: from netmonster.pakint.net (netmonster [192.168.3.13]) by smp.paktronix.com (8.9.3/8.9.3) with ESMTP id JAA21295; Thu, 4 Oct 2001 09:11:43 -0500 Date: Thu, 4 Oct 2001 09:43:07 -0500 (CDT) From: "Matthew G. Marsh" X-X-Sender: To: Pekka Savola cc: Subject: Re: Should IP addresses on interfaces not UP respond to ping? In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Thu, 4 Oct 2001, Pekka Savola wrote: > Hi, > > With 2.2.18 I noticed something that looked interesting: > > # /sbin/ip a l dev eth4 > 7: eth4: mtu 1500 qdisc pfifo_fast qlen 100 > link/ether 00:80:c8:c9:b8:14 brd ff:ff:ff:ff:ff:ff > inet x.y.7.252/24 brd x.y.7.255 scope global eth4 > > Note that the interface is not UP. Whether it's promisc or not does not > affect this. True. Any address on the system will be responded to by any interface. I use this all the time to assign addresses to dummy that are not advertised but are available. > However, the address is still pingable from outside, through eth0! > > Also noticed the same behaviour in 2.4.10. > > Is this the intended behaviour, probably? > > One could argue that if interface isn't UP, it shouldn't be able to send > or receive packets at all. I wonder what changing this would break.. > > -- > Pekka Savola "Tell me of difficulties surmounted, > Netcore Oy not those you stumble over and fall" > Systems. Networks. Security. -- Robert Jordan: A Crown of Swords > > -------------------------------------------------- Matthew G. Marsh, President Paktronix Systems LLC 1506 North 59th Street Omaha NE 68104 Phone: (402) 932-7250 x101 Email: mgm@paktronix.com WWW: http://www.paktronix.com -------------------------------------------------- From owner-netdev@oss.sgi.com Thu Oct 4 07:48:26 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94EmQA12859 for netdev-outgoing; Thu, 4 Oct 2001 07:48:26 -0700 Received: from netcore.fi (netcore.fi [193.94.160.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94EmKD12856 for ; Thu, 4 Oct 2001 07:48:20 -0700 Received: from localhost (pekkas@localhost) by netcore.fi (8.11.6/8.11.6) with ESMTP id f94EmB128258; Thu, 4 Oct 2001 17:48:11 +0300 Date: Thu, 4 Oct 2001 17:48:11 +0300 (EEST) From: Pekka Savola To: "Matthew G. Marsh" cc: Subject: Re: Should IP addresses on interfaces not UP respond to ping? In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Thu, 4 Oct 2001, Matthew G. Marsh wrote: > On Thu, 4 Oct 2001, Pekka Savola wrote: > > > Hi, > > > > With 2.2.18 I noticed something that looked interesting: > > > > # /sbin/ip a l dev eth4 > > 7: eth4: mtu 1500 qdisc pfifo_fast qlen 100 > > link/ether 00:80:c8:c9:b8:14 brd ff:ff:ff:ff:ff:ff > > inet x.y.7.252/24 brd x.y.7.255 scope global eth4 > > > > Note that the interface is not UP. Whether it's promisc or not does not > > affect this. > > True. Any address on the system will be responded to by any interface. I > use this all the time to assign addresses to dummy that are not advertised > but are available. Sorry, I should have articulated this better. There are two issues here: 1) interface eth4 answers from eth0 (known weak host behaviour, nothing new here) 2) interface address answers to ping etc. even if the interface has been brought down. I should have focused more on 2) in my statement. I believe you answered to 1). > > > However, the address is still pingable from outside, through eth0! > > > > Also noticed the same behaviour in 2.4.10. > > > > Is this the intended behaviour, probably? > > > > One could argue that if interface isn't UP, it shouldn't be able to send > > or receive packets at all. I wonder what changing this would break.. > > > > -- > > Pekka Savola "Tell me of difficulties surmounted, > > Netcore Oy not those you stumble over and fall" > > Systems. Networks. Security. -- Robert Jordan: A Crown of Swords > > > > > > -------------------------------------------------- > Matthew G. Marsh, President > Paktronix Systems LLC > 1506 North 59th Street > Omaha NE 68104 > Phone: (402) 932-7250 x101 > Email: mgm@paktronix.com > WWW: http://www.paktronix.com > -------------------------------------------------- > -- Pekka Savola "Tell me of difficulties surmounted, Netcore Oy not those you stumble over and fall" Systems. Networks. Security. -- Robert Jordan: A Crown of Swords From owner-netdev@oss.sgi.com Thu Oct 4 08:23:15 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94FNFX14099 for netdev-outgoing; Thu, 4 Oct 2001 08:23:15 -0700 Received: from www.hockin.org (adsl-64-166-241-227.dsl.snfc21.pacbell.net [64.166.241.227]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94FNDD14096 for ; Thu, 4 Oct 2001 08:23:14 -0700 Received: (from thockin@localhost) by www.hockin.org (8.10.2/8.10.2) id f94F3RK12908; Thu, 4 Oct 2001 08:03:27 -0700 From: Tim Hockin Message-Id: <200110041503.f94F3RK12908@www.hockin.org> Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 To: hadi@cyberus.ca (jamal) Date: Thu, 4 Oct 2001 08:03:27 -0700 (PDT) Cc: sim@netnation.com (Simon Kirby), greearb@candelatech.com (Ben Greear), mingo@elte.hu (Ingo Molnar), linux-kernel@vger.kernel.org, kuznet@ms2.inr.ac.ru (Alexey Kuznetsov), Robert.Olsson@data.slu.se (Robert Olsson), bcrl@redhat.com (Benjamin LaHaise), netdev@oss.sgi.com, alan@lxorguk.ukuu.org.uk (Alan Cox) In-Reply-To: from "jamal" at Oct 04, 2001 07:54:19 AM X-Mailer: ELM [version 2.5 PL3] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk > Has nothing to do with specific hardware although i see your point. > send me an eepro and i'll at least add hardware flow control for you. > The API is simple, its up to the driver maintainers to use. This > discussion is good to make people aware of those drivers. is there a place where this is explained? I'd be happy to make drivers on which I work support this. It's like ethtool - easy to do, but no one has done it because they didn't know. From owner-netdev@oss.sgi.com Thu Oct 4 08:24:50 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94FOo514210 for netdev-outgoing; Thu, 4 Oct 2001 08:24:50 -0700 Received: from smp.paktronix.com (pak200.pakuni.net [207.91.34.200]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94FOhD14206 for ; Thu, 4 Oct 2001 08:24:43 -0700 Received: from netmonster.pakint.net (netmonster [192.168.3.13]) by smp.paktronix.com (8.9.3/8.9.3) with ESMTP id KAA21489; Thu, 4 Oct 2001 10:03:41 -0500 Date: Thu, 4 Oct 2001 10:35:05 -0500 (CDT) From: "Matthew G. Marsh" X-X-Sender: To: Pekka Savola cc: Subject: Re: Should IP addresses on interfaces not UP respond to ping? In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Thu, 4 Oct 2001, Pekka Savola wrote: > On Thu, 4 Oct 2001, Matthew G. Marsh wrote: > > On Thu, 4 Oct 2001, Pekka Savola wrote: > > > > > Hi, > > > > > > With 2.2.18 I noticed something that looked interesting: > > > > > > # /sbin/ip a l dev eth4 > > > 7: eth4: mtu 1500 qdisc pfifo_fast qlen 100 > > > link/ether 00:80:c8:c9:b8:14 brd ff:ff:ff:ff:ff:ff > > > inet x.y.7.252/24 brd x.y.7.255 scope global eth4 > > > > > > Note that the interface is not UP. Whether it's promisc or not does not > > > affect this. > > Sorry, I should have articulated this better. There are two issues here: > > 1) interface eth4 answers from eth0 (known weak host behaviour, nothing > new here) > > 2) interface address answers to ping etc. even if the interface has been > brought down. > > I should have focused more on 2) in my statement. I believe you answered > to 1). Yes and also to 2) - The behaviour you see with 2) is actually what I (perhaps mistakenly) rely on. IE: ip addr add 10.1.1.1/24 dev dummy where dev dummy is used for binding a tunnel or security daemon etcetc. and the regular interfaces on the system are all 192.168.x.x style Then on the remote host I add: ip ro ad 10.1.1.1/32 via 192.168.x.x where 192.168.x.x is the interface accessable from the other device. In this way I can use control statements on the mechanism listening on 10.1.1.1 that do not interfere with the regular interfaces of the system. IE I can use statements such as: ip rule add from 192.168.y.y dev ${192.168.x.x} prohibit > > > However, the address is still pingable from outside, through eth0! And this is the type of behavior that I do use. Where the address is available through the "real" physical interfaces of the system even if it is DOWN or bound to dummy etc. > > > Also noticed the same behaviour in 2.4.10. > > > > > > Is this the intended behaviour, probably? I cannot speak to intention but I do know that this behaviour has worked since 2.1.38 or so. > > > One could argue that if interface isn't UP, it shouldn't be able to send > > > or receive packets at all. I wonder what changing this would break.. I suspect it would break some internal RPDB code... But I really do not know... Alexey? > > > -- > > > Pekka Savola "Tell me of difficulties surmounted, > > > Netcore Oy not those you stumble over and fall" > > > Systems. Networks. Security. -- Robert Jordan: A Crown of Swords -------------------------------------------------- Matthew G. Marsh, President Paktronix Systems LLC 1506 North 59th Street Omaha NE 68104 Phone: (402) 932-7250 x101 Email: mgm@paktronix.com WWW: http://www.paktronix.com -------------------------------------------------- From owner-netdev@oss.sgi.com Thu Oct 4 08:56:59 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94Fux614941 for netdev-outgoing; Thu, 4 Oct 2001 08:56:59 -0700 Received: from grok.yi.org (IDENT:root@cx97923-a.phnx3.az.home.com [24.9.112.194]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94FusD14935 for ; Thu, 4 Oct 2001 08:56:54 -0700 Received: from candelatech.com (IDENT:greear@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.2/8.11.2) with ESMTP id f94Fu2T24771; Thu, 4 Oct 2001 08:56:02 -0700 Message-ID: <3BBC8692.9F48DA85@candelatech.com> Date: Thu, 04 Oct 2001 08:56:02 -0700 From: Ben Greear Organization: Candela Technologies X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.4.3-12 i686) X-Accept-Language: en MIME-Version: 1.0 To: jamal CC: Simon Kirby , Ingo Molnar , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , netdev@oss.sgi.com, Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk jamal wrote: > > On Wed, 3 Oct 2001, Ben Greear wrote: > > > The tulip driver only started working for my DLINK 4-port NIC after > > about 2.4.8, and last I checked the ZYNX 4-port still refuses to work, > > so I wouldn't consider it a paradigm of stability and grace quite yet. > > The tests in www.cyberus.ca/~hadi/247-res/ were done with 4-port znyx > cards using 2.4.7. > What kind of problems are you having? Maybe i can help. Mostly problems with auto-negotiation it seems. Earlier 2.4 kernels just would never go 100bt/FD. Later (broken) versions would claim to be 100bt/FD, but they still showed lots of collisions and frame errors. I'll try the ZYNX on the latest kernel in the next few days and let you know what I find... > My point is that the API exists. Driver owners could use it; this > discussion seems to have at least helped to point in the existence of the > API. Alexey had the hardware flow control in there since 2.1.x .., us > that at least. In my opinion, Ingos patch is radical enough to be allowed > in when we are approaching stability. And it is a lazy way of solving the > problem The API has been there since 2.1.x, and yet few drivers support it? I can see why Ingo decided to fix the problem generically. I think it would be great if his code printed a log message upon trigger that basically said: "You should get yourself a NAPI enabled driver that does flow-control if possible." That may give the appropriate visibility to the issue and let the driver writers improve their drivers accordingly... Ben > > cheers, > jamal > > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From owner-netdev@oss.sgi.com Thu Oct 4 09:07:57 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94G7v115343 for netdev-outgoing; Thu, 4 Oct 2001 09:07:57 -0700 Received: from l.himel.bg (IDENT:root@unamed.infotel.bg [212.39.68.18] (may be forged)) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94G7mD15334 for ; Thu, 4 Oct 2001 09:07:49 -0700 Received: from linux.himel.bg (IDENT:ja@linux.himel.bg [127.0.0.1]) by l.himel.bg (8.9.3/8.9.3) with ESMTP id TAA31795; Thu, 4 Oct 2001 19:07:37 +0300 Date: Thu, 4 Oct 2001 19:07:37 +0300 (EEST) From: Julian Anastasov X-X-Sender: To: Pekka Savola cc: Subject: Re: Should IP addresses on interfaces not UP respond to ping? In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Hello, On Thu, 4 Oct 2001, Pekka Savola wrote: > With 2.2.18 I noticed something that looked interesting: > > # /sbin/ip a l dev eth4 > 7: eth4: mtu 1500 qdisc pfifo_fast qlen 100 > link/ether 00:80:c8:c9:b8:14 brd ff:ff:ff:ff:ff:ff > inet x.y.7.252/24 brd x.y.7.255 scope global eth4 > > Note that the interface is not UP. Whether it's promisc or not does not > affect this. > > However, the address is still pingable from outside, through eth0! You can avoid this by using rp_filter protection. If not, the kernel should not stop your traffic. Note that rp_filter is used not only for security reasons. By changing the device status you disable only the link communications, i.e. the link routes disappear. > Also noticed the same behaviour in 2.4.10. > > Is this the intended behaviour, probably? > > One could argue that if interface isn't UP, it shouldn't be able to send > or receive packets at all. I wonder what changing this would break.. This is true - no traffic on down-ed interface. You can think for it as the route is bound to device but the local IP addresses are not. The sockets can be bound and not bound to devices. May be if one device fails the connection can continue to use another device, there are many variants :) At least, there is a way to control all of them and to achieve the required behaviour. Regards -- Julian Anastasov From owner-netdev@oss.sgi.com Thu Oct 4 09:27:23 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94GRN415746 for netdev-outgoing; Thu, 4 Oct 2001 09:27:23 -0700 Received: from grok.yi.org (IDENT:root@cx97923-a.phnx3.az.home.com [24.9.112.194]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94GRKD15743 for ; Thu, 4 Oct 2001 09:27:20 -0700 Received: from candelatech.com (IDENT:greear@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.2/8.11.2) with ESMTP id f94GRDT24976; Thu, 4 Oct 2001 09:27:13 -0700 Message-ID: <3BBC8DE1.5AAFE716@candelatech.com> Date: Thu, 04 Oct 2001 09:27:13 -0700 From: Ben Greear Organization: Candela Technologies X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.4.3-12 i686) X-Accept-Language: en MIME-Version: 1.0 To: jamal CC: Robert Olsson , netdev@oss.sgi.com Subject: Re: Some NAPI results References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk jamal wrote: > > Now if we can cut that second column to all 0s we are set ;-> > > cheers, > jamal > > On Thu, 4 Oct 2001, Robert Olsson wrote: > > > > > Correct! > > > > We were interested to see the the packet distribution on the different CPU's > > with this scheme. Reality had data we couldn't get in the lab. Seems there > > is nothing to complain about. Yet. I'd better say. :-) > > > > cat /proc/net/softnet_stat > > 2b85c320 0000d374 6524ce48 00000000 00000000 00000000 00000000 00000000 0007119d > > 2b8b5e29 0000d615 653eba32 00000000 00000000 00000000 00000000 00000000 00072f44 > > > > First col: pkts > > Second col: drops > > Third col: time_squeeze (by the polling loop) > > Last col: cpu_collsions (TX) So you're priting out counters in HEX?? This seems one place where a nice base-10 number would be appropriate :) Ben -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From owner-netdev@oss.sgi.com Thu Oct 4 10:11:44 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94HBiC16457 for netdev-outgoing; Thu, 4 Oct 2001 10:11:44 -0700 Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94HBfD16454 for ; Thu, 4 Oct 2001 10:11:41 -0700 Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id NAA21000; Thu, 4 Oct 2001 13:09:08 -0400 Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2]) by westrelay03.boulder.ibm.com (8.11.1m3/NCO v4.97.1) with ESMTP id f94HBQe21586; Thu, 4 Oct 2001 11:11:26 -0600 Received: from crg8.beaverton.ibm.com (crg8.beaverton.ibm.com [138.95.19.9]) by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id f94HBQt01864; Thu, 4 Oct 2001 10:11:26 -0700 Received: from dyn9-47-8-184.rhe.beaverton.ibm.com (dyn9-47-8-184.rhe.beaverton.ibm.com [9.47.8.184]) by crg8.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) with ESMTP id f94HBPn15176; Thu, 4 Oct 2001 10:11:25 -0700 (PDT) Date: Thu, 4 Oct 2001 10:11:16 -0700 (PDT) From: Mark Price X-X-Sender: To: Andi Kleen cc: , , Subject: Re: patch - SNMP Kernel Counters In-Reply-To: <20011004114917.57615@colin.muc.de> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Thanks for the info Andi, in terms of the patch this is easy to resolve the MIB definition of tcpRtoAlgorithm is: tcpRtoAlgorithm OBJECT-TYPE SYNTAX INTEGER { other(1), -- none of the following constant(2), -- a constant rto rsre(3), -- MIL-STD-1778, Appendix B vanj(4) -- Van Jacobson's algorithm [10] } I'll recode it as other(1), and resubmit the patch. Cheers, Mark. > That's not quite correct. 2.4 doesn't use the Van Jacobson algorithm anymore > (see ftp.inr.ac.ru:/ip-routing/README.rto). I don't know what the right > value should be however. > > -Andi > > From owner-netdev@oss.sgi.com Thu Oct 4 10:27:27 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94HRRG17813 for netdev-outgoing; Thu, 4 Oct 2001 10:27:27 -0700 Received: from xmailserver.org ([208.129.208.52]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94HRND17810 for ; Thu, 4 Oct 2001 10:27:23 -0700 Received: from blue1.dev.mcafeelabs.com (161.69.79.199) by i750raq.dev.mcafeelabs.com. (161.69.86.244) with [XMail 1.1 (Linux/Ix86) ESMTP Server] id for from ; Thu, 04 Oct 2001 11:43:04 -0700 Date: Thu, 4 Oct 2001 10:32:10 -0700 (PDT) From: Davide Libenzi X-X-Sender: davide@blue1.dev.mcafeelabs.com To: Andreas Dilger cc: Robert Olsson , , jamal , , Alexey Kuznetsov , Benjamin LaHaise , , Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <20011003162210.L8954@turbolinux.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, Andreas Dilger wrote: > If you get to the stage where you are turning off IRQs and going to a > polling mode, then don't turn IRQs back on until you have a poll (or > two or whatever) that there is no work to be done. This will at worst > give you 50% polling success, but in practise you wouldn't start polling > until there is lots of work to be done, so the real success rate will > be much higher. > > At this point (no work to be done when polling) there are clearly no > interrupts would be generated (because no packets have arrived), so it > should be reasonable to turn interrupts back on and stop polling (assuming > non-broken hardware). You now go back to interrupt-driven work until > the rate increases again. This means you limit IRQ rates when needed, > but only do one or two excess polls before going back to IRQ-driven work. > > Granted, I don't know what the overhead of turning the IRQs on and off > is, but since we do it all the time already (for each ISR) it can't be > that bad. > > If you are always having work to do when polling, then interrupts will > never be turned on again, but who cares at that point because the work > is getting done? Similarly, if you have IRQs disabled, but are sharing > IRQs there is nothing wrong in polling all devices sharing that IRQ > (at least conceptually). > > I don't know much about IRQ handlers, but I assume that this is already > what happens if you are sharing an IRQ - you don't know which of many > sources it comes from, so you poll all of them to see if they have any > work to be done. If you are polling some of the shared-IRQ devices too > frequently (i.e. they never have work to do), you could have some sort > of progressive backoff, so you skip polling those for a growing number > of polls (this could also be set by the driver if it knows that it could > only generate real work every X ms, so we skip about X/poll_rate polls). This seems a pretty nice solution that achieve 1) to limit the irq frequency 2) avoid the huge shared irqs latency given by the irq masking. By having a per irq # poll callbacks could give the opportunity to poll "time to time" sharing devices during the offending device poll loop. - Davide From owner-netdev@oss.sgi.com Thu Oct 4 10:41:32 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94HfW818198 for netdev-outgoing; Thu, 4 Oct 2001 10:41:32 -0700 Received: from webber.adilger.int (h24-64-71-161.cg.shawcable.net [24.64.71.161]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94HfRD18192 for ; Thu, 4 Oct 2001 10:41:27 -0700 Received: from webber.adilger.int (adilger@localhost [127.0.0.1]) by localhost (8.12.0.Beta19/8.12.0.Beta19/Debian 8.12.0.Beta19) with ESMTP id f94HeML7031454; Thu, 4 Oct 2001 11:40:22 -0600 Received: (from adilger@localhost) by webber.adilger.int (8.12.0.Beta19/8.12.0.Beta19/Debian 8.12.0.Beta19) id f94HeLQ2031452; Thu, 4 Oct 2001 11:40:21 -0600 From: Andreas Dilger Date: Thu, 4 Oct 2001 11:40:21 -0600 To: jamal Cc: Ingo Molnar , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , netdev@oss.sgi.com, Simon Kirby Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: <20011004114021.F31061@turbolinux.com> Mail-Followup-To: jamal , Ingo Molnar , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , netdev@oss.sgi.com, Simon Kirby References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.3.22i Sender: owner-netdev@oss.sgi.com Precedence: bulk On Oct 04, 2001 07:34 -0400, jamal wrote: > 1) you shut down shared interupts; take a look at this posting by Marcus > Sundberg > > --------------- > > 0: 7602983 XT-PIC timer > 1: 10575 XT-PIC keyboard > 2: 0 XT-PIC cascade > 8: 1 XT-PIC rtc > 11: 1626004 XT-PIC Toshiba America Info Systems ToPIC95 PCI > \ > to Cardbus Bridge with ZV Support, Toshiba America Info Systems ToPIC95 > PCI \ > to Cardbus Bridge with ZV Support (#2), usb-uhci, eth0, BreezeCom Card, \ > Intel 440MX, irda0 12: 1342 XT-PIC PS/2 Mouse > 14: 23605 XT-PIC ide0 > > ----------------------------- > > Now you go and shut down IRQ 11 and punish all devices there. If you can > avoid that, it is acceptable as a temporary replacement to be upgraded to > a better scheme. Well, if we fall back to polling devices if the IRQ is disabled, then the shared interrupt case can be handled as well. However, there were complaints about the patch when Ingo had device polling included, as opposed to just IRQ mitigation. > 2) By not being granular enough and shutting down sources of noise, you > are actually not being effective in increasing system utilization. Well, since the IRQ itself uses system resources, if it is disabled it will allow those resources to actually do something (i.e. polling instead, when we know there is a lot of work to do). Even if it does not have polling in the patch, the choice is to turn off the IRQ, or have the system hang because it can not make any progress because of the high number of interrupts. If your patch ensures that the network IRQ load is kept down, then Ingo's will never be activated. Cheers, Andreas -- Andreas Dilger \ "If a man ate a pound of pasta and a pound of antipasto, \ would they cancel out, leaving him still hungry?" http://www-mddsp.enel.ucalgary.ca/People/adilger/ -- Dogbert From owner-netdev@oss.sgi.com Thu Oct 4 11:13:32 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94IDW719186 for netdev-outgoing; Thu, 4 Oct 2001 11:13:32 -0700 Received: from the-village.bc.nu (lightning.swansea.linux.org.uk [194.168.151.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94IDTD19183 for ; Thu, 4 Oct 2001 11:13:29 -0700 Received: from alan by the-village.bc.nu with local (Exim 3.22 #1) id 15pD2u-0003ae-00; Thu, 04 Oct 2001 19:16:32 +0100 Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 To: torvalds@transmeta.com (Linus Torvalds) Date: Thu, 4 Oct 2001 19:16:32 +0100 (BST) Cc: greearb@candelatech.com (Ben Greear), hadi@cyberus.ca (jamal), mingo@elte.hu (Ingo Molnar), linux-kernel@vger.kernel.org, kuznet@ms2.inr.ac.ru (Alexey Kuznetsov), Robert.Olsson@data.slu.se (Robert Olsson), bcrl@redhat.com (Benjamin LaHaise), netdev@oss.sgi.com, alan@lxorguk.ukuu.org.uk (Alan Cox) In-Reply-To: from "Linus Torvalds" at Oct 03, 2001 09:33:12 AM X-Mailer: ELM [version 2.5 PL6] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: From: Alan Cox Sender: owner-netdev@oss.sgi.com Precedence: bulk > (a) is not a major security issue. If you allow untrusted users full > 100/1000Mbps access to your internal network, you have _other_ > security issues, like packet sniffing etc that are much much MUCH > worse. So the packet flooding thing is very much a corner case, and > claiming that we have a big problem is silly. Not nowdays. 100Mbit pipes to the backbone are routine for web serving in the real world - at least the paying end (aka porn). Alan From owner-netdev@oss.sgi.com Thu Oct 4 11:26:16 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94IQGr19465 for netdev-outgoing; Thu, 4 Oct 2001 11:26:16 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94IQBD19461 for ; Thu, 4 Oct 2001 11:26:11 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id OAA11688; Thu, 4 Oct 2001 14:23:05 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Thu, 4 Oct 2001 14:23:04 -0400 (EDT) From: jamal To: Ben Greear cc: Simon Kirby , Ingo Molnar , , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <3BBC8692.9F48DA85@candelatech.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Thu, 4 Oct 2001, Ben Greear wrote: > jamal wrote: > > > > On Wed, 3 Oct 2001, Ben Greear wrote: > > > > > The tulip driver only started working for my DLINK 4-port NIC after > > > about 2.4.8, and last I checked the ZYNX 4-port still refuses to work, > > > so I wouldn't consider it a paradigm of stability and grace quite yet. > > > > The tests in www.cyberus.ca/~hadi/247-res/ were done with 4-port znyx > > cards using 2.4.7. > > What kind of problems are you having? Maybe i can help. > > Mostly problems with auto-negotiation it seems. Earlier 2.4 kernels > just would never go 100bt/FD. Later (broken) versions would claim to > be 100bt/FD, but they still showed lots of collisions and frame errors. > > I'll try the ZYNX on the latest kernel in the next few days and let you > know what I find... Please do. > > > My point is that the API exists. Driver owners could use it; this > > discussion seems to have at least helped to point in the existence of the > > API. Alexey had the hardware flow control in there since 2.1.x .., us > > that at least. In my opinion, Ingos patch is radical enough to be allowed > > in when we are approaching stability. And it is a lazy way of solving the > > problem > > The API has been there since 2.1.x, and yet few drivers support it? I > can see why Ingo decided to fix the problem generically. That logic is convoluted. > > > cat /proc/net/softnet_stat > > > 2b85c320 0000d374 6524ce48 00000000 00000000 00000000 00000000 00000000 0$ > > > 2b8b5e29 0000d615 653eba32 00000000 00000000 00000000 00000000 00000000 0$ > > So you're priting out counters in HEX?? This seems one place where a nice > base-10 number would be appropriate :) Its mostly for formating reasons: 2b85c320 is 730186528 (and wont fit in one line) cheers, jamal From owner-netdev@oss.sgi.com Thu Oct 4 11:36:08 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94Ia8Z19777 for netdev-outgoing; Thu, 4 Oct 2001 11:36:08 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94Ia4D19773 for ; Thu, 4 Oct 2001 11:36:04 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id OAA11756; Thu, 4 Oct 2001 14:33:00 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Thu, 4 Oct 2001 14:33:00 -0400 (EDT) From: jamal To: Andreas Dilger cc: Ingo Molnar , , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Simon Kirby Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <20011004114021.F31061@turbolinux.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Thu, 4 Oct 2001, Andreas Dilger wrote: > On Oct 04, 2001 07:34 -0400, jamal wrote: > > 1) you shut down shared interupts; take a look at this posting by Marcus > > Sundberg > > > > --------------- > > > > 0: 7602983 XT-PIC timer > > 1: 10575 XT-PIC keyboard > > 2: 0 XT-PIC cascade > > 8: 1 XT-PIC rtc > > 11: 1626004 XT-PIC Toshiba America Info Systems ToPIC95 PCI > > \ > > to Cardbus Bridge with ZV Support, Toshiba America Info Systems ToPIC95 > > PCI \ > > to Cardbus Bridge with ZV Support (#2), usb-uhci, eth0, BreezeCom Card, \ > > Intel 440MX, irda0 12: 1342 XT-PIC PS/2 Mouse > > 14: 23605 XT-PIC ide0 > > > > ----------------------------- > > > > Now you go and shut down IRQ 11 and punish all devices there. If you can > > avoid that, it is acceptable as a temporary replacement to be upgraded to > > a better scheme. > > Well, if we fall back to polling devices if the IRQ is disabled, then the > shared interrupt case can be handled as well. However, there were complaints > about the patch when Ingo had device polling included, as opposed to just > IRQ mitigation. > I dont think youve followed the discussions too well and normally i wouldnt respond but you addressed me. Ingos netdevice polling is not the right approach, please look at NAPI and read the paper. NAPI does all what youve been suggesting. We are not even discussing that at this point. We are discussing the sledgehammer effect and how you could break a finger or two trying to kill that fly with it. The example above illustrates it. cheers, jamal From owner-netdev@oss.sgi.com Thu Oct 4 14:23:10 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94LNAY22946 for netdev-outgoing; Thu, 4 Oct 2001 14:23:10 -0700 Received: from shed.alex.org.uk (shed.alex.org.uk [195.224.53.219]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94LN6D22943 for ; Thu, 4 Oct 2001 14:23:07 -0700 Received: from [195.224.237.69] (localhost [127.0.0.1]) by shed.alex.org.uk (Postfix) with ESMTP id 249C0A4CB; Thu, 4 Oct 2001 22:23:04 +0100 (BST) Date: Thu, 04 Oct 2001 22:22:56 +0100 From: Alex Bligh - linux-kernel Reply-To: Alex Bligh - linux-kernel To: Robert Olsson , jamal Cc: Ingo Molnar , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Benjamin LaHaise , netdev@oss.sgi.com, Linus Torvalds , Alan Cox , Alex Bligh - linux-kernel Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: <302417729.1002234175@[195.224.237.69]> In-Reply-To: <15291.5314.595897.458571@robur.slu.se> References: <15291.5314.595897.458571@robur.slu.se> X-Mailer: Mulberry/2.1.0 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Sender: owner-netdev@oss.sgi.com Precedence: bulk > > The paper is at: http://www.cyberus.ca/~hadi/usenix-paper.tgz > > Robert can point you to the latest patches. > > > Current code... there are still some parts we like to better. > > Available via ftp from robur.slu.se:/pub/Linux/net-development/NAPI/ > 2.4.10-poll.pat I seem to remember jamal saying the NAPI stuff was available since 2.(early). Is there a stable 2.2.20 patch? -- Alex Bligh From owner-netdev@oss.sgi.com Thu Oct 4 14:28:24 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94LSOZ23081 for netdev-outgoing; Thu, 4 Oct 2001 14:28:24 -0700 Received: from shed.alex.org.uk (shed.alex.org.uk [195.224.53.219]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94LSMD23078 for ; Thu, 4 Oct 2001 14:28:22 -0700 Received: from [195.224.237.69] (localhost [127.0.0.1]) by shed.alex.org.uk (Postfix) with ESMTP id D2F3BA4CB; Thu, 4 Oct 2001 22:28:19 +0100 (BST) Date: Thu, 04 Oct 2001 22:28:17 +0100 From: Alex Bligh - linux-kernel Reply-To: Alex Bligh - linux-kernel To: mingo@elte.hu, jamal Cc: linux-kernel@vger.kernel.org, Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , netdev@oss.sgi.com, Linus Torvalds , Alan Cox , Simon Kirby , Alex Bligh - linux-kernel Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: <302737894.1002234496@[195.224.237.69]> In-Reply-To: References: X-Mailer: Mulberry/2.1.0 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Sender: owner-netdev@oss.sgi.com Precedence: bulk --On Wednesday, 03 October, 2001 4:51 PM +0200 Ingo Molnar wrote: > your refusal to accept this problem as an existing and real problem is > really puzzling me. In at least one environment known to me (router), I'd rather it kept accepting packets, and f/w'ing them, and didn't switch VTs etc. By dropping down performance, you've made the DoS attack even more successful than it would otherwise have been (the kiddie looks at effect on the host at the end). -- Alex Bligh From owner-netdev@oss.sgi.com Thu Oct 4 14:49:56 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94Lnu423501 for netdev-outgoing; Thu, 4 Oct 2001 14:49:56 -0700 Received: from lacrosse.corp.redhat.com (host154.207-175-42.redhat.com [207.175.42.154]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94LnsD23498 for ; Thu, 4 Oct 2001 14:49:54 -0700 Received: from toomuch.toronto.redhat.com (IDENT:9Kws4WDQCcIKAs81TZgYk2WveiAdNAkg@toomuch.toronto.redhat.com [172.16.14.22]) by lacrosse.corp.redhat.com (8.9.3/8.9.3) with ESMTP id RAA22777; Thu, 4 Oct 2001 17:49:45 -0400 Received: (from bcrl@localhost) by toomuch.toronto.redhat.com (8.11.6/8.11.2) id f94LnjZ20090; Thu, 4 Oct 2001 17:49:45 -0400 Date: Thu, 4 Oct 2001 17:49:45 -0400 From: Benjamin LaHaise To: Alex Bligh - linux-kernel Cc: mingo@elte.hu, jamal , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Robert Olsson , netdev@oss.sgi.com, Linus Torvalds , Alan Cox , Simon Kirby Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: <20011004174945.B18528@redhat.com> References: <302737894.1002234496@[195.224.237.69]> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <302737894.1002234496@[195.224.237.69]>; from linux-kernel@alex.org.uk on Thu, Oct 04, 2001 at 10:28:17PM +0100 Sender: owner-netdev@oss.sgi.com Precedence: bulk On Thu, Oct 04, 2001 at 10:28:17PM +0100, Alex Bligh - linux-kernel wrote: > In at least one environment known to me (router), I'd rather it > kept accepting packets, and f/w'ing them, and didn't switch VTs etc. > By dropping down performance, you've made the DoS attack even > more successful than it would otherwise have been (the kiddie > looks at effect on the host at the end). Then bug the driver author of your ethernet cards or turn the hammer off. You're the sysadmin, you know that your system is unusual. Deal with it. -ben From owner-netdev@oss.sgi.com Thu Oct 4 15:06:42 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94M6gX23793 for netdev-outgoing; Thu, 4 Oct 2001 15:06:42 -0700 Received: from the-village.bc.nu (lightning.swansea.linux.org.uk [194.168.151.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94M6dD23790 for ; Thu, 4 Oct 2001 15:06:40 -0700 Received: from alan by the-village.bc.nu with local (Exim 3.22 #1) id 15pGhY-0004Qz-00; Thu, 04 Oct 2001 23:10:44 +0100 Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 To: linux-kernel@alex.org.uk Date: Thu, 4 Oct 2001 23:10:44 +0100 (BST) Cc: mingo@elte.hu, hadi@cyberus.ca (jamal), linux-kernel@vger.kernel.org, kuznet@ms2.inr.ac.ru (Alexey Kuznetsov), Robert.Olsson@data.slu.se (Robert Olsson), bcrl@redhat.com (Benjamin LaHaise), netdev@oss.sgi.com, torvalds@transmeta.com (Linus Torvalds), alan@lxorguk.ukuu.org.uk (Alan Cox), sim@netnation.com (Simon Kirby) In-Reply-To: <302737894.1002234496@[195.224.237.69]> from "Alex Bligh - linux-kernel" at Oct 04, 2001 10:28:17 PM X-Mailer: ELM [version 2.5 PL6] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: From: Alan Cox Sender: owner-netdev@oss.sgi.com Precedence: bulk > In at least one environment known to me (router), I'd rather it > kept accepting packets, and f/w'ing them, and didn't switch VTs etc. > By dropping down performance, you've made the DoS attack even > more successful than it would otherwise have been (the kiddie > looks at effect on the host at the end). You only think that. After a few minutes the kiddie pulls down your routing because your route daemons execute no code. Also during the attack your sshd wont run so you cant log in to find out what is up From owner-netdev@oss.sgi.com Thu Oct 4 15:41:16 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94MfGi24208 for netdev-outgoing; Thu, 4 Oct 2001 15:41:16 -0700 Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94MevD24204 for ; Thu, 4 Oct 2001 15:40:57 -0700 Received: from westrelay03.boulder.ibm.com (westrelay03.boulder.ibm.com [9.99.140.24]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id SAA96140; Thu, 4 Oct 2001 18:38:15 -0400 Received: from gateway.beaverton.ibm.com (gateway.beaverton.ibm.com [138.95.180.1]) by westrelay03.boulder.ibm.com (8.11.1m3/NCO v4.97.1) with ESMTP id f94Mece56768; Thu, 4 Oct 2001 16:40:38 -0600 Received: from crg8.beaverton.ibm.com (crg8.beaverton.ibm.com [138.95.19.9]) by gateway.beaverton.ibm.com (8.10.0.Beta10/8.8.5) with ESMTP id f94MeZ526547; Thu, 4 Oct 2001 15:40:35 -0700 (PDT) Received: from dyn9-47-8-184.rhe.beaverton.ibm.com (dyn9-47-8-184.rhe.beaverton.ibm.com [9.47.8.184]) by crg8.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) with ESMTP id f94MeYn23141; Thu, 4 Oct 2001 15:40:34 -0700 (PDT) Date: Thu, 4 Oct 2001 15:40:24 -0700 (PDT) From: Mark Price X-X-Sender: To: Andi Kleen cc: , , , Subject: Re: patch - SNMP Kernel Counters In-Reply-To: <20011004114917.57615@colin.muc.de> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Thu, 4 Oct 2001, Andi Kleen wrote: > That's not quite correct. 2.4 doesn't use the Van Jacobson algorithm anymore > (see ftp.inr.ac.ru:/ip-routing/README.rto). I don't know what the right > value should be however. > > -Andi Modified my patch to use other(1) as the Algorithm. Full patch included again below. Cheers, Mark. diff -urN linux-2.4.10/include/net/tcp.h linux/include/net/tcp.h --- linux-2.4.10/include/net/tcp.h Sun Sep 23 10:31:58 2001 +++ linux/include/net/tcp.h Thu Oct 4 10:22:33 2001 @@ -331,6 +331,7 @@ #define TCP_DELACK_MIN 4 #define TCP_ATO_MIN 4 #endif +#define TCP_RTO_OTHER 1 /* RTO Algorithm used by linux */ #define TCP_RTO_MAX (120*HZ) #define TCP_RTO_MIN (HZ/5) #define TCP_TIMEOUT_INIT (3*HZ) /* RFC 1122 initial RTO value */ diff -urN linux-2.4.10/net/ipv4/icmp.c linux/net/ipv4/icmp.c --- linux-2.4.10/net/ipv4/icmp.c Fri Sep 7 11:01:21 2001 +++ linux/net/ipv4/icmp.c Tue Oct 2 11:24:44 2001 @@ -826,6 +826,10 @@ if (net_ratelimit()) printk(KERN_DEBUG "a guy asks for address mask. Who is it?\n"); #endif + /* + * Just say that there is an out-error in SNMP for now. + */ + ICMP_INC_STATS(IcmpOutErrors); } /* diff -urN linux-2.4.10/net/ipv4/ip_input.c linux/net/ipv4/ip_input.c --- linux-2.4.10/net/ipv4/ip_input.c Thu Apr 12 12:11:39 2001 +++ linux/net/ipv4/ip_input.c Wed Oct 3 09:35:26 2001 @@ -219,6 +219,7 @@ static inline int ip_local_deliver_finish(struct sk_buff *skb) { int ihl = skb->nh.iph->ihl*4; + int raw_flg = 0; #ifdef CONFIG_NETFILTER_DEBUG nf_debug_ip_local_deliver(skb); @@ -245,13 +246,15 @@ int hash = protocol & (MAX_INET_PROTOS - 1); struct sock *raw_sk = raw_v4_htable[hash]; struct inet_protocol *ipprot; - int flag; + int aflg,flag; /* If there maybe a raw socket we must check - if not we * don't care less */ - if(raw_sk != NULL) + if(raw_sk != NULL) { + raw_flg = 1; raw_sk = raw_v4_input(skb, skb->nh.iph, hash); + } ipprot = (struct inet_protocol *) inet_protos[hash]; flag = 0; @@ -261,11 +264,18 @@ ipprot->protocol == protocol) { int ret; + /* Increment IpInDelivers prior to calling the Fast Path + * protocol handler. + */ + + IP_INC_STATS_BH(IpInDelivers); + /* Fast path... */ ret = ipprot->handler(skb); return ret; } else { + aflg = 1; flag = ip_run_ipprot(skb, skb->nh.iph, ipprot, (raw_sk != NULL)); } } @@ -280,9 +290,16 @@ sock_put(raw_sk); } else if (!flag) { /* Free and report errors */ icmp_send(skb, ICMP_DEST_UNREACH, ICMP_PROT_UNREACH, 0); + /* + * Increment unknown-protocol counter for SNMP. + */ + IP_INC_STATS_BH(IpInUnknownProtos); out: kfree_skb(skb); } + + if (flag || raw_flg || aflg) + IP_INC_STATS_BH(IpInDelivers); } return 0; @@ -343,8 +360,10 @@ --ANK (980813) */ - if (skb_cow(skb, skb_headroom(skb))) + if (skb_cow(skb, skb_headroom(skb))) { + IP_INC_STATS_BH(IpInDiscards); goto drop; + } iph = skb->nh.iph; skb->ip_summed = 0; @@ -393,8 +412,10 @@ IP_INC_STATS_BH(IpInReceives); - if ((skb = skb_share_check(skb, GFP_ATOMIC)) == NULL) + if ((skb = skb_share_check(skb, GFP_ATOMIC)) == NULL) { + IP_INC_STATS_BH(IpInDiscards); goto out; + } if (!pskb_may_pull(skb, sizeof(struct iphdr))) goto inhdr_error; diff -urN linux-2.4.10/net/ipv4/proc.c linux/net/ipv4/proc.c --- linux-2.4.10/net/ipv4/proc.c Wed May 16 10:21:45 2001 +++ linux/net/ipv4/proc.c Tue Oct 2 11:23:46 2001 @@ -135,7 +135,14 @@ "\nTcp: RtoAlgorithm RtoMin RtoMax MaxConn ActiveOpens PassiveOpens AttemptFails EstabResets CurrEstab InSegs OutSegs RetransSegs InErrs OutRsts\n" "Tcp:"); for (i=0; iname); #endif + IP_INC_STATS_BH(IpInAddrErrors); e_inval: err = -EINVAL; goto done; diff -urN linux-2.4.10/net/ipv4/tcp.c linux/net/ipv4/tcp.c --- linux-2.4.10/net/ipv4/tcp.c Thu Sep 20 14:12:56 2001 +++ linux/net/ipv4/tcp.c Thu Oct 4 10:23:07 2001 @@ -2555,4 +2555,15 @@ printk("TCP: Hash tables configured (established %d bind %d)\n", tcp_ehash_size<<1, tcp_bhash_size); + + /* Setup the SNMP Stats for the TCP RTO values. + * TcpMaxConn = -1 as the maximum number of connections is + * dynamic. + */ + + tcp_statistics[0].TcpRtoAlgorithm=TCP_RTO_OTHER; + tcp_statistics[0].TcpRtoMin=TCP_RTO_MIN*1000/HZ; + tcp_statistics[0].TcpRtoMax=TCP_RTO_MAX*1000/HZ; + tcp_statistics[0].TcpMaxConn=-1; + } diff -urN linux-2.4.10/net/ipv4/tcp_ipv4.c linux/net/ipv4/tcp_ipv4.c --- linux-2.4.10/net/ipv4/tcp_ipv4.c Fri Sep 7 11:01:21 2001 +++ linux/net/ipv4/tcp_ipv4.c Tue Oct 2 11:25:34 2001 @@ -1538,8 +1538,6 @@ goto discard; #endif /* CONFIG_FILTER */ - IP_INC_STATS_BH(IpInDelivers); - if (sk->state == TCP_ESTABLISHED) { /* Fast path */ TCP_CHECK_TIMER(sk); if (tcp_rcv_established(sk, skb, skb->h.th, skb->len)) diff -urN linux-2.4.10/net/ipv4/udp.c linux/net/ipv4/udp.c --- linux-2.4.10/net/ipv4/udp.c Fri Sep 7 11:01:21 2001 +++ linux/net/ipv4/udp.c Tue Oct 2 11:25:12 2001 @@ -785,8 +785,6 @@ if (sk->filter && skb->ip_summed != CHECKSUM_UNNECESSARY) { if (__udp_checksum_complete(skb)) { UDP_INC_STATS_BH(UdpInErrors); - IP_INC_STATS_BH(IpInDiscards); - ip_statistics[smp_processor_id()*2].IpInDelivers--; kfree_skb(skb); return -1; } @@ -796,8 +794,6 @@ if (sock_queue_rcv_skb(sk,skb)<0) { UDP_INC_STATS_BH(UdpInErrors); - IP_INC_STATS_BH(IpInDiscards); - ip_statistics[smp_processor_id()*2].IpInDelivers--; kfree_skb(skb); return -1; } @@ -836,8 +832,10 @@ udp_queue_rcv_skb(sk, skb1); sk = sknext; } while(sknext); - } else + } else { + UDP_INC_STATS_BH(UdpNoPorts); kfree_skb(skb); + } read_unlock(&udp_hash_lock); return 0; } @@ -880,8 +878,6 @@ u32 saddr = skb->nh.iph->saddr; u32 daddr = skb->nh.iph->daddr; int len = skb->len; - - IP_INC_STATS_BH(IpInDelivers); /* * Validate the packet and the UDP length. From owner-netdev@oss.sgi.com Thu Oct 4 16:20:43 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94NKhQ24754 for netdev-outgoing; Thu, 4 Oct 2001 16:20:43 -0700 Received: from shed.alex.org.uk (shed.alex.org.uk [195.224.53.219]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94NKdD24751 for ; Thu, 4 Oct 2001 16:20:39 -0700 Received: from [195.224.237.69] (localhost [127.0.0.1]) by shed.alex.org.uk (Postfix) with ESMTP id 6881AA4CB; Fri, 5 Oct 2001 00:20:37 +0100 (BST) Date: Fri, 05 Oct 2001 00:20:34 +0100 From: Alex Bligh - linux-kernel Reply-To: Alex Bligh - linux-kernel To: Benjamin LaHaise , Alex Bligh - linux-kernel Cc: mingo@elte.hu, jamal , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Robert Olsson , netdev@oss.sgi.com, Linus Torvalds , Alan Cox , Simon Kirby , Alex Bligh - linux-kernel Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: <309455016.1002241234@[195.224.237.69]> In-Reply-To: <20011004174945.B18528@redhat.com> References: <20011004174945.B18528@redhat.com> X-Mailer: Mulberry/2.1.0 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Sender: owner-netdev@oss.sgi.com Precedence: bulk --On Thursday, 04 October, 2001 5:49 PM -0400 Benjamin LaHaise wrote: >> In at least one environment known to me (router), I'd rather it >> kept accepting packets, and f/w'ing them, and didn't switch VTs etc. >> By dropping down performance, you've made the DoS attack even >> more successful than it would otherwise have been (the kiddie >> looks at effect on the host at the end). > > Then bug the driver author of your ethernet cards or turn the hammer off. > You're the sysadmin, you know that your system is unusual. Deal with it. The hammer has an average age of 13yrs and is difficult to turn off, unfortunately. Rather than bugging the author of the driver card, we've actually been trying to fix it, down to rewriting the firmware. So for this purpose I/we am/are the driver maintainer thanks. However, there are limitations like bus speed which mean that in practice if we receive a large enough number of small packets each second, the box will saturate. My point was merely that some applications (and using a linux box as a router is not that 'unusual') want to deprioritize different things under resource starvation. Changing the default, in an unconfigurable way, isn't a great idea. Sure dealing with external resource exhaustions for hosts is indeed a good idea. I was just suggesting that it wasn't always what you wanted to do. Not sure this required jumping down my throat. -- Alex Bligh From owner-netdev@oss.sgi.com Thu Oct 4 16:26:55 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94NQtq24923 for netdev-outgoing; Thu, 4 Oct 2001 16:26:55 -0700 Received: from lacrosse.corp.redhat.com (host154.207-175-42.redhat.com [207.175.42.154]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94NQrD24918 for ; Thu, 4 Oct 2001 16:26:53 -0700 Received: from toomuch.toronto.redhat.com (IDENT:K+yE6xrMUy6olPuSex+eCaNMqlMEDTWN@toomuch.toronto.redhat.com [172.16.14.22]) by lacrosse.corp.redhat.com (8.9.3/8.9.3) with ESMTP id TAA17902; Thu, 4 Oct 2001 19:26:45 -0400 Received: (from bcrl@localhost) by toomuch.toronto.redhat.com (8.11.6/8.11.2) id f94NQjZ20400; Thu, 4 Oct 2001 19:26:45 -0400 Date: Thu, 4 Oct 2001 19:26:45 -0400 From: Benjamin LaHaise To: Alex Bligh - linux-kernel Cc: mingo@elte.hu, jamal , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Robert Olsson , netdev@oss.sgi.com, Linus Torvalds , Alan Cox , Simon Kirby Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: <20011004192645.A20389@redhat.com> References: <20011004174945.B18528@redhat.com> <309455016.1002241234@[195.224.237.69]> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <309455016.1002241234@[195.224.237.69]>; from linux-kernel@alex.org.uk on Fri, Oct 05, 2001 at 12:20:34AM +0100 Sender: owner-netdev@oss.sgi.com Precedence: bulk On Fri, Oct 05, 2001 at 12:20:34AM +0100, Alex Bligh - linux-kernel wrote: > Rather than bugging the author of the driver card, we've actually > been trying to fix it, down to rewriting the firmware. So for > this purpose I/we am/are the driver maintainer thanks. However, > there are limitations like bus speed which mean that in practice > if we receive a large enough number of small packets each second, > the box will saturate. Not if the driver has a decent irq mitigation schema and uses the hw flow control + NAPI bits. > Not sure this required jumping down my throat. Frankly I'm sick of this entire discussion where people claim that no form of interrupt throttling is ever needed. It's an emergency measure that is needed under some circumstances as very few drivers properly protect against this kind of DoS. Drivers that do things correctly will never trigger the hammer. Plus it's configurable. If you'd bothered to read and understand the rest of this thread you wouldn't have posted. -ben From owner-netdev@oss.sgi.com Thu Oct 4 16:28:51 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94NSp425028 for netdev-outgoing; Thu, 4 Oct 2001 16:28:51 -0700 Received: from shed.alex.org.uk (shed.alex.org.uk [195.224.53.219]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94NSnD25019 for ; Thu, 4 Oct 2001 16:28:49 -0700 Received: from [195.224.237.69] (localhost [127.0.0.1]) by shed.alex.org.uk (Postfix) with ESMTP id 2ECBBA4CB; Fri, 5 Oct 2001 00:28:47 +0100 (BST) Date: Fri, 05 Oct 2001 00:28:44 +0100 From: Alex Bligh - linux-kernel Reply-To: Alex Bligh - linux-kernel To: Alan Cox , linux-kernel@alex.org.uk Cc: mingo@elte.hu, jamal , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , netdev@oss.sgi.com, Linus Torvalds , Simon Kirby , Alex Bligh - linux-kernel Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: <309943687.1002241724@[195.224.237.69]> In-Reply-To: References: X-Mailer: Mulberry/2.1.0 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Sender: owner-netdev@oss.sgi.com Precedence: bulk --On Thursday, 04 October, 2001 11:10 PM +0100 Alan Cox wrote: > You only think that. After a few minutes the kiddie pulls down your > routing because your route daemons execute no code. Also during the > attack your sshd wont run so you cant log in to find out what is up There is truth in this. Which is why doing things like a crude WRED on the card, in the firmware, (i.e. before it sends the data into user space) is something we looked at but never got round to. -- Alex Bligh From owner-netdev@oss.sgi.com Thu Oct 4 16:36:27 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94NaRU25300 for netdev-outgoing; Thu, 4 Oct 2001 16:36:27 -0700 Received: from loquat.bbn.com (crodrigues.bbn.com [128.89.72.49]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94NaMD25297 for ; Thu, 4 Oct 2001 16:36:23 -0700 Received: (from crodrigu@localhost) by loquat.bbn.com (8.11.2/8.11.2) id f94NaOM13983 for netdev@oss.sgi.com; Thu, 4 Oct 2001 19:36:24 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by loquat.bbn.com (8.11.2/8.11.2) with ESMTP id f930V3f05266 for ; Tue, 2 Oct 2001 20:31:03 -0400 Received: from po1.bbn.com [192.1.50.38] by localhost with POP3 (fetchmail-5.7.4) for crodrigu@localhost (single-drop); Tue, 02 Oct 2001 20:31:03 -0400 (EDT) Received: from gto-mailer1.bbn.com (gto-mailer1.bbn.com [128.33.0.36]) by po1.bbn.com (8.9.1/8.9.1) with ESMTP id UAA14724 for ; Tue, 2 Oct 2001 20:29:26 -0400 (EDT) Received: (from root@localhost) by gto-mailer1.bbn.com (8.9.3+Sun/8.9.3) id UAA03610 for crodrigu@po1.bbn.com@gto-mailer1.bbn.com; Tue, 2 Oct 2001 20:32:07 -0400 (EDT) Received: (from root@localhost) by gto-mailer1.bbn.com (8.9.3+Sun/8.9.3) id UAA03603 for crodrigu@bbn.com@gto-mailer1.bbn.com; Tue, 2 Oct 2001 20:32:07 -0400 (EDT) Received: from cam-smtp1.bbn.com (cam-smtp1.bbn.com [192.1.120.100]) by gto-mailer1.bbn.com (8.9.3+Sun/8.9.3) with ESMTP id UAA03595 for ; Tue, 2 Oct 2001 20:32:06 -0400 (EDT) Received: by cam-smtp1.bbn.com (8.9.1/8.9.1) id UAA22792 for crodrigu@bbn.com@cam-smtp1.bbn.com; Tue, 2 Oct 2001 20:31:24 -0400 (EDT) Received: from usw-sf-list1.sourceforge.net (usw-sf-fw2.sourceforge.net [216.136.171.252]) by cam-smtp1.bbn.com (8.9.1/8.9.1) with ESMTP id UAA22786 for ; Tue, 2 Oct 2001 20:31:16 -0400 (EDT) X-Recipient: Received: from localhost ([127.0.0.1] helo=usw-sf-list1.sourceforge.net) by usw-sf-list1.sourceforge.net with esmtp (Exim 3.31-VA-mm2 #1 (Debian)) id 15oZuJ-0007oA-00; Tue, 02 Oct 2001 17:29:03 -0700 Received: from crodrigues.bbn.com ([128.89.72.49] helo=loquat.bbn.com) by usw-sf-list1.sourceforge.net with esmtp (Exim 3.31-VA-mm2 #1 (Debian)) id 15oZtw-0007dC-00 for ; Tue, 02 Oct 2001 17:28:40 -0700 Received: (from crodrigu@localhost) by loquat.bbn.com (8.11.2/8.11.2) id f92I28w04496 for diffserv-general@lists.sourceforge.net; Tue, 2 Oct 2001 14:02:08 -0400 From: Craig Rodrigues To: diffserv-general@lists.sourceforge.net Message-ID: <20011002140208.A4220@bbn.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i Subject: [Linux Diffserv] Need to be root to setsockopt() for EF? X-BeenThere: diffserv-general@lists.sourceforge.net X-Mailman-Version: 2.0.5 List-Help: List-Post: List-Subscribe: , List-Id: General discussion of Differentiated Services on Linux List-Unsubscribe: , List-Archive: X-Original-Date: Tue, 2 Oct 2001 14:02:08 -0400 Date: Tue, 2 Oct 2001 14:02:08 -0400 Sender: owner-netdev@oss.sgi.com Precedence: bulk Hi, I am using Linux kernel 2.4.2 on a Redhat 7.1 system. I wrote a test program which sends out UDP packets and toggles the IP Diffserv byte using the setsockopt() call. In my test, I found that if I used setsockopt(IPPROTO_IP, IPTOS, ......) with the following DSCP values: Expedited Forwarding, DSCP = 0x2E CS5, DSCP = 0x28 CS6, DSCP = 0x30 CS7, DSCP = 0x38 setsockopt() would return -1, and errno would be set to "Operation not permitted". If I ran the program again as root, this did not happen. I ran the same program under FreeBSD, and did not get this error. Can someone shed some light as to why I got this error under Linux? Is it configuration problem, or is there some sort of policy decision in the kernel that requires the process to be run as root when setting those DSCP values? Thanks. -- Craig Rodrigues Distributed Systems and Logistics, Office 6/304 crodrigu@bbn.com BBN Technologies Cambridge, MA _______________________________________________ Diffserv-general mailing list Diffserv-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/diffserv-general From owner-netdev@oss.sgi.com Thu Oct 4 16:47:29 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94NlTM25572 for netdev-outgoing; Thu, 4 Oct 2001 16:47:29 -0700 Received: from tisch.mail.mindspring.net (tisch.mail.mindspring.net [207.69.200.157]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94NlRD25569 for ; Thu, 4 Oct 2001 16:47:27 -0700 Received: from phantasy.awol.org (user-37katno.dialup.mindspring.com [207.69.118.248]) by tisch.mail.mindspring.net (8.9.3/8.8.5) with ESMTP id TAA18065; Thu, 4 Oct 2001 19:47:02 -0400 (EDT) Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 From: Robert Love To: Benjamin LaHaise Cc: Alex Bligh - linux-kernel , mingo@elte.hu, jamal , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Robert Olsson , netdev@oss.sgi.com, Linus Torvalds , Alan Cox , Simon Kirby In-Reply-To: <20011004192645.A20389@redhat.com> References: <20011004174945.B18528@redhat.com> <309455016.1002241234@[195.224.237.69]> <20011004192645.A20389@redhat.com> Content-Type: text/plain Content-Transfer-Encoding: 7bit X-Mailer: Evolution/0.15.99+cvs.2001.10.03.20.06 (Preview Release) Date: 04 Oct 2001 19:47:10 -0400 Message-Id: <1002239236.872.8.camel@phantasy> Mime-Version: 1.0 Sender: owner-netdev@oss.sgi.com Precedence: bulk On Thu, 2001-10-04 at 19:26, Benjamin LaHaise wrote: > Frankly I'm sick of this entire discussion where people claim that no > form of interrupt throttling is ever needed. It's an emergency measure > that is needed under some circumstances as very few drivers properly > protect against this kind of DoS. Drivers that do things correctly will > never trigger the hammer. Plus it's configurable. If you'd bothered to > read and understand the rest of this thread you wouldn't have posted. Agreed. I am actually amazed that the opposite of what is happening does not happen -- that more people aren't clamoring for this solution. Six months ago I was testing some TCP application and by accident placed a sendto() in an infinite loop. The destination of the packets (on my LAN) locked up completely! And this was a powerful Pentium III with a 3c905 NIC. Not acceptable. Robert Love From owner-netdev@oss.sgi.com Thu Oct 4 16:51:52 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f94Npqn25684 for netdev-outgoing; Thu, 4 Oct 2001 16:51:52 -0700 Received: from neon-gw.transmeta.com (neon-gw-l3.transmeta.com [63.209.4.196]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f94NppD25681 for ; Thu, 4 Oct 2001 16:51:51 -0700 Received: (from root@localhost) by neon-gw.transmeta.com (8.9.3/8.9.3) id QAA30914; Thu, 4 Oct 2001 16:51:35 -0700 Received: from mailhost.transmeta.com(10.1.1.15) by neon-gw.transmeta.com via smap (V2.1) id xma030880; Thu, 4 Oct 01 16:51:27 -0700 Received: from penguin.transmeta.com (penguin.transmeta.com [10.10.27.78]) by deepthought.transmeta.com (8.9.3/8.9.3) with ESMTP id QAA09925; Thu, 4 Oct 2001 16:51:31 -0700 (PDT) Received: from localhost (torvalds@localhost) by penguin.transmeta.com (8.11.2/8.7.3) with ESMTP id f94NpCE01035; Thu, 4 Oct 2001 16:51:12 -0700 X-Authentication-Warning: penguin.transmeta.com: torvalds owned process doing -bs Date: Thu, 4 Oct 2001 16:51:12 -0700 (PDT) From: Linus Torvalds To: Robert Love cc: Benjamin LaHaise , Alex Bligh - linux-kernel , , jamal , , Alexey Kuznetsov , Robert Olsson , , Alan Cox , Simon Kirby Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <1002239236.872.8.camel@phantasy> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On 4 Oct 2001, Robert Love wrote: > > Agreed. I am actually amazed that the opposite of what is happening > does not happen -- that more people aren't clamoring for this solution. Ehh.. I think that most people who are against Ingo's patches are so mainly because there _is_ an alternative that looks nicer. Linus From owner-netdev@oss.sgi.com Thu Oct 4 17:01:35 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9501Zt25885 for netdev-outgoing; Thu, 4 Oct 2001 17:01:35 -0700 Received: from grok.yi.org (IDENT:root@cx97923-a.phnx3.az.home.com [24.9.112.194]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9501WD25882 for ; Thu, 4 Oct 2001 17:01:32 -0700 Received: from candelatech.com (ip-216-73-180-118.hqglobal.net [216.73.180.118]) by grok.yi.org (8.11.2/8.11.2) with ESMTP id f95007T27169; Thu, 4 Oct 2001 17:00:07 -0700 Message-ID: <3BBCF802.1B650B20@candelatech.com> Date: Thu, 04 Oct 2001 17:00:02 -0700 From: Ben Greear Organization: Candela Technologies Inc X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.4.7 i586) X-Accept-Language: en MIME-Version: 1.0 To: Linus Torvalds CC: Robert Love , Benjamin LaHaise , Alex Bligh - linux-kernel , mingo@elte.hu, jamal , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Robert Olsson , netdev@oss.sgi.com, Alan Cox , Simon Kirby Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk Linus Torvalds wrote: > > On 4 Oct 2001, Robert Love wrote: > > > > Agreed. I am actually amazed that the opposite of what is happening > > does not happen -- that more people aren't clamoring for this solution. > > Ehh.. I think that most people who are against Ingo's patches are so > mainly because there _is_ an alternative that looks nicer. > > Linus The alternative (NAPI) only works with Tulip and Intel NICs, it seems. When the alternative works for every driver known (including 3rd party ones, like the e100), then it will truly be an alternative. Untill then, it will be a great feature for those who can use it, and the rest of the poor folks will need a big generic hammer. >From personal experince, I imagine the problem is also that it was not invented here, where here is where each of sit. And I include myself in that bias! Ben -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From owner-netdev@oss.sgi.com Thu Oct 4 17:16:08 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f950G8w26100 for netdev-outgoing; Thu, 4 Oct 2001 17:16:08 -0700 Received: from xmailserver.org ([208.129.208.52]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f950G3D26097 for ; Thu, 4 Oct 2001 17:16:05 -0700 Received: from blue1.dev.mcafeelabs.com (161.69.79.199) by i750raq.dev.mcafeelabs.com. (161.69.86.244) with [XMail 1.1 (Linux/Ix86) ESMTP Server] id for from ; Thu, 04 Oct 2001 18:29:38 -0700 Date: Thu, 4 Oct 2001 17:18:42 -0700 (PDT) From: Davide Libenzi X-X-Sender: davide@blue1.dev.mcafeelabs.com To: Ben Greear cc: Linus Torvalds , Robert Love , Benjamin LaHaise , Alex Bligh - linux-kernel , , jamal , , Alexey Kuznetsov , Robert Olsson , , Alan Cox , Simon Kirby Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <3BBCF802.1B650B20@candelatech.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Thu, 4 Oct 2001, Ben Greear wrote: > Linus Torvalds wrote: > > > > On 4 Oct 2001, Robert Love wrote: > > > > > > Agreed. I am actually amazed that the opposite of what is happening > > > does not happen -- that more people aren't clamoring for this solution. > > > > Ehh.. I think that most people who are against Ingo's patches are so > > mainly because there _is_ an alternative that looks nicer. > > > > Linus > > The alternative (NAPI) only works with Tulip and Intel NICs, it seems. > When the alternative works for every driver known (including 3rd party > ones, like the e100), then it will truly be an alternative. Untill > then, it will be a great feature for those who can use it, and the > rest of the poor folks will need a big generic hammer. NAPI needs aware drivers and introduces changes to the queue processing ( packets left in DMA ring ) and it'll be at least 2.5.x It's clearly a nicer solution that does not suffer of drawbacks that Ingo's code have. Ingo's patch is more hack-ish but addresses the problem with minimal changes. - Davide From owner-netdev@oss.sgi.com Thu Oct 4 19:05:10 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9525Ao27493 for netdev-outgoing; Thu, 4 Oct 2001 19:05:10 -0700 Received: from shell.cyberus.ca (shell.cyberus.ca [209.195.95.7]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f95255D27490 for ; Thu, 4 Oct 2001 19:05:05 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id WAA13511; Thu, 4 Oct 2001 22:01:57 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Thu, 4 Oct 2001 22:01:56 -0400 (EDT) From: jamal To: Ben Greear cc: Linus Torvalds , Robert Love , Benjamin LaHaise , Alex Bligh - linux-kernel , , , Alexey Kuznetsov , Robert Olsson , , Alan Cox , Simon Kirby Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <3BBCF802.1B650B20@candelatech.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Thu, 4 Oct 2001, Ben Greear wrote: > Linus Torvalds wrote: > > > > On 4 Oct 2001, Robert Love wrote: > > > > > > Agreed. I am actually amazed that the opposite of what is happening > > > does not happen -- that more people aren't clamoring for this solution. > > > > Ehh.. I think that most people who are against Ingo's patches are so > > mainly because there _is_ an alternative that looks nicer. > > > > Linus > > The alternative (NAPI) only works with Tulip and Intel NICs, it seems. > When the alternative works for every driver known (including 3rd party > ones, like the e100), then it will truly be an alternative. Untill > then, it will be a great feature for those who can use it, and the > rest of the poor folks will need a big generic hammer. > Ben, Lets put some reality check and history just for entertainment value: It took ten years of Linux existence (and i am just using you as an example no pun intended) to realize your life was actually an emergency that depended on Ingos patch. Maybe i am being cruel, so lets backtrack only over the last 4 years when Alexey first had the HFC in there; i am willing to bet a large amount of money that you didnt once use it or even care to post a query if such a thing existed. Ok, so lets assume you didnt know it existed ... over a year back i posted widely on it with conjunction to the return code to the netif_rx() extension ... and still you didnt care that much although you seem to be a user of one of the converted drivers -- the tulip and in particular the znyx hardware which was used in the testing. IIRC, you actually said something on that post .. Then one bright early morn, Eastern time zone, Ingo appears, not in the form of atoms rather electrons masquareding as bits ... cheers, jamal PS:- I am going to try and mitigate myself from this thread now; my email-sending rate will be drastically reduced. From owner-netdev@oss.sgi.com Thu Oct 4 22:18:24 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f955IO129146 for netdev-outgoing; Thu, 4 Oct 2001 22:18:24 -0700 Received: from netcore.fi (netcore.fi [193.94.160.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f955ILD29143 for ; Thu, 4 Oct 2001 22:18:21 -0700 Received: from localhost (pekkas@localhost) by netcore.fi (8.11.6/8.11.6) with ESMTP id f955I6v00533; Fri, 5 Oct 2001 08:18:06 +0300 Date: Fri, 5 Oct 2001 08:18:06 +0300 (EEST) From: Pekka Savola To: Craig Rodrigues cc: , Subject: Re: [Linux Diffserv] Need to be root to setsockopt() for EF? In-Reply-To: <20011002140208.A4220@bbn.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Tue, 2 Oct 2001, Craig Rodrigues wrote: > Can someone shed some light as to why I got this > error under Linux? Is it configuration problem, > or is there some sort of policy decision in the kernel that > requires the process to be run as root when setting > those DSCP values? A part of DSCP field was previously Precedence. Linux has required that in order to use 'Critical' or higher Precedence, one must have CAP_NET_ADMIN capability, in most cases, root. I'm not one to say whether this restriction should be removed. Perhaps. -- Pekka Savola "Tell me of difficulties surmounted, Netcore Oy not those you stumble over and fall" Systems. Networks. Security. -- Robert Jordan: A Crown of Swords From owner-netdev@oss.sgi.com Fri Oct 5 07:30:43 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f95EUhx06278 for netdev-outgoing; Fri, 5 Oct 2001 07:30:43 -0700 Received: from robur.slu.se (robur.slu.se [130.238.98.12]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f95EUdD06275 for ; Fri, 5 Oct 2001 07:30:39 -0700 Received: (from robert@localhost) by robur.slu.se (8.8.7/8.8.7) id QAA21638; Fri, 5 Oct 2001 16:32:49 +0200 From: Robert Olsson MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15293.50321.681223.620902@robur.slu.se> Date: Fri, 5 Oct 2001 16:32:49 +0200 To: Alex Bligh - linux-kernel Cc: Robert Olsson , jamal , Ingo Molnar , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Benjamin LaHaise , netdev@oss.sgi.com, Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <302417729.1002234175@[195.224.237.69]> References: <15291.5314.595897.458571@robur.slu.se> <302417729.1002234175@[195.224.237.69]> X-Mailer: VM 6.92 under Emacs 19.34.1 Sender: owner-netdev@oss.sgi.com Precedence: bulk Alex Bligh - linux-kernel writes: > I seem to remember jamal saying the NAPI stuff was available > since 2.(early). Is there a stable 2.2.20 patch? Hello! Current NAPI incarnation came first for 2.4.3 and holds ANK trademark. Jamal had pre-NAPI patches long before and we have been testing/profiling polling and flow control versions of popular network drivers in the lab and on on highly loaded Internet sites for a long time. I consider the NAPI work to initiated by Jamal at OLS two years ago. No I don't know of any usable code for 2.2.* Cheers. --ro From owner-netdev@oss.sgi.com Fri Oct 5 07:50:10 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f95EoAl06685 for netdev-outgoing; Fri, 5 Oct 2001 07:50:10 -0700 Received: from robur.slu.se (robur.slu.se [130.238.98.12]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f95Eo6D06681 for ; Fri, 5 Oct 2001 07:50:06 -0700 Received: (from robert@localhost) by robur.slu.se (8.8.7/8.8.7) id QAA21905; Fri, 5 Oct 2001 16:52:16 +0200 From: Robert Olsson MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15293.51488.280808.469262@robur.slu.se> Date: Fri, 5 Oct 2001 16:52:16 +0200 To: Andreas Dilger Cc: Robert Olsson , mingo@elte.hu, jamal , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Benjamin LaHaise , netdev@oss.sgi.com, Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <20011003162210.L8954@turbolinux.com> References: <15291.32311.499838.886628@robur.slu.se> <20011003162210.L8954@turbolinux.com> X-Mailer: VM 6.92 under Emacs 19.34.1 Sender: owner-netdev@oss.sgi.com Precedence: bulk Andreas Dilger writes: > If you get to the stage where you are turning off IRQs and going to a > polling mode, then don't turn IRQs back on until you have a poll (or > two or whatever) that there is no work to be done. This will at worst > give you 50% polling success, but in practise you wouldn't start polling > until there is lots of work to be done, so the real success rate will > be much higher. > > At this point (no work to be done when polling) there are clearly no > interrupts would be generated (because no packets have arrived), so it > should be reasonable to turn interrupts back on and stop polling (assuming > non-broken hardware). You now go back to interrupt-driven work until > the rate increases again. This means you limit IRQ rates when needed, > but only do one or two excess polls before going back to IRQ-driven work. Hello! Yes this has been considered and actually I think Jamal did this in one of the pre NAPI patch. I tried something similar... but instead of using a number of excess polls I was doing excess polls for a short time (a jiffie). This was the showstopper mentioned the previous mails. :-) Anyway it up to driver to decide this policy. If the driver returns "not_done" it is simply polled again. So low-rate network drivers can have a different policy compared to an OC-48 driver. Even continues polling is therefore possible and even showstoppers. :-) There are protection for polling livelocks. Cheers. --ro From owner-netdev@oss.sgi.com Fri Oct 5 08:07:17 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f95F7Hw07278 for netdev-outgoing; Fri, 5 Oct 2001 08:07:17 -0700 Received: from posta2.elte.hu (posta2.elte.hu [157.181.151.9]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f95F7FD07275 for ; Fri, 5 Oct 2001 08:07:15 -0700 Received: from chiara.elte.hu (chiara.elte.hu [157.181.150.200]) by posta2.elte.hu (Postfix) with ESMTP id A591348002 for ; Fri, 5 Oct 2001 17:07:05 +0200 (CEST) Received: by chiara.elte.hu (Postfix, from userid 17806) id 071D51FCF; Thu, 4 Oct 2001 08:52:50 +0200 (CEST) Date: Thu, 4 Oct 2001 08:50:30 +0200 (CEST) From: Ingo Molnar Reply-To: To: jamal Cc: Simon Kirby , Linus Torvalds , Ben Greear , , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Wed, 3 Oct 2001, jamal wrote: > I think you can save yourself a lot of pain today by going to a > "better driver"/hardware. Switch to a tulip based board; [...] This is not an option in many cases. (eg. where a company standardizes on something non-tulip, or due to simple financial/organizational reasons.) What you say is the approach i see in the FreeBSD camp frequently: "use these [limited set of] wonderful cards and drivers, the rest sucks hardware-design-wise and we dont really care about them", which elitist attitude i strongly disagree with. Ingo From owner-netdev@oss.sgi.com Fri Oct 5 08:20:52 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f95FKqr08158 for netdev-outgoing; Fri, 5 Oct 2001 08:20:52 -0700 Received: from robur.slu.se (robur.slu.se [130.238.98.12]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f95FKmD08147 for ; Fri, 5 Oct 2001 08:20:48 -0700 Received: (from robert@localhost) by robur.slu.se (8.8.7/8.8.7) id RAA22428; Fri, 5 Oct 2001 17:22:56 +0200 From: Robert Olsson MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15293.53328.435292.971669@robur.slu.se> Date: Fri, 5 Oct 2001 17:22:56 +0200 To: Alan Cox Cc: linux-kernel@alex.org.uk, mingo@elte.hu, hadi@cyberus.ca (jamal), linux-kernel@vger.kernel.org, kuznet@ms2.inr.ac.ru (Alexey Kuznetsov), Robert.Olsson@data.slu.se (Robert Olsson), bcrl@redhat.com (Benjamin LaHaise), netdev@oss.sgi.com, torvalds@transmeta.com (Linus Torvalds), sim@netnation.com (Simon Kirby) Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: References: <302737894.1002234496@[195.224.237.69]> X-Mailer: VM 6.92 under Emacs 19.34.1 Sender: owner-netdev@oss.sgi.com Precedence: bulk Alan Cox writes: > You only think that. After a few minutes the kiddie pulls down your routing > because your route daemons execute no code. Also during the attack your sshd > wont run so you cant log in to find out what is up Indeed. I have a real example from a university core router with BGP and full Internet routing. I managed to get in via ssh during the DoS attack. We see that the 5 min dropping rate is about the same as the input rate. The duration of this attack was more half an hour and BGP survied and the box was pretty manageable. This was with a hacked tulip driver switching to RX-polling at high loads. eth2: UP Locked MII Full DuplexLink UP Admin up 6 day(s) 13 hour(s) 47 min 51 sec Last input NOW Last output NOW 5min RX bit/s 23.9 M 5min TX bit/s 1.1 M 5min RX pkts/s 46439 5min TX pkts/s 759 5min TX errors 0 5min RX errors 0 5min RX dropped 47038 5min TX dropped 0 5min collisions 0 Well, this was a router but I think we very soon have the same demands for most Internet servers. Cheers. --ro From owner-netdev@oss.sgi.com Fri Oct 5 08:25:21 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f95FPLU08395 for netdev-outgoing; Fri, 5 Oct 2001 08:25:21 -0700 Received: from ms2.inr.ac.ru (minus.inr.ac.ru [193.233.7.97]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f95FPID08392 for ; Fri, 5 Oct 2001 08:25:18 -0700 Received: (from kuznet@localhost) by ms2.inr.ac.ru (8.6.13/ANK) id TAA20980; Fri, 5 Oct 2001 19:25:06 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200110051525.TAA20980@ms2.inr.ac.ru> Subject: Re: patch - SNMP Kernel Counters To: mkprice@us.ibm.com (Mark Price) Date: Fri, 5 Oct 2001 19:25:06 +0400 (MSK DST) Cc: netdev@oss.sgi.com, davem@redhat.com In-Reply-To: from "Mark Price" at Oct 3, 1 03:02:05 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Sender: owner-netdev@oss.sgi.com Precedence: bulk Hello! Great. > o Added support for TcpRtoMin,TcpRtoMax,TcpRtoAlgorithm and TcpMaxConn > variables, including a rather ugly cludge to proc.c for TCPMaxConn. OK. > o Modified the way IpInDelivers is incremented. OK. Technically wrong though. Please try to invent some way to eliminate additional variable "aflg" in some way yet... Also: > + IP_INC_STATS_BH(IpInDiscards); > goto drop; It is pretty evident that increment must follow the label "drop". No? Well, this happens in several places. Alexey From owner-netdev@oss.sgi.com Fri Oct 5 08:33:32 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f95FXWH08623 for netdev-outgoing; Fri, 5 Oct 2001 08:33:32 -0700 Received: from ms2.inr.ac.ru (minus.inr.ac.ru [193.233.7.97]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f95FXTD08619 for ; Fri, 5 Oct 2001 08:33:30 -0700 Received: (from kuznet@localhost) by ms2.inr.ac.ru (8.6.13/ANK) id TAA21087; Fri, 5 Oct 2001 19:33:01 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200110051533.TAA21087@ms2.inr.ac.ru> Subject: Re: patch - SNMP Kernel Counters To: ak@muc.de (Andi Kleen) Date: Fri, 5 Oct 2001 19:33:01 +0400 (MSK DST) Cc: mkprice@us.ibm.com, netdev@oss.sgi.com, davem@redhat.com In-Reply-To: <20011004114917.57615@colin.muc.de> from "Andi Kleen" at Oct 4, 1 11:49:17 am X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Sender: owner-netdev@oss.sgi.com Precedence: bulk Hello! > That's not quite correct. 2.4 doesn't use the Van Jacobson algorithm anymore Anymore? Funny note. Look into formerly used algo. :-) At least, the differences are less than differences between original algo and its implementaion in BSD, and then between linux-2.2 and BSD. Actually, I do not know any implemenattion closer to VJ's proposal. :-) No matter, the sense is to distinuguish RFC algorithm which does not use rttvar. Alexey From owner-netdev@oss.sgi.com Fri Oct 5 08:41:45 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f95Ffj508841 for netdev-outgoing; Fri, 5 Oct 2001 08:41:45 -0700 Received: from coruscant.gnumonks.org (mail@coruscant.franken.de [193.174.159.226]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f95FfED08809 for ; Fri, 5 Oct 2001 08:41:15 -0700 Received: from uucp by coruscant.gnumonks.org with local-bsmtp (Exim 3.33 #1) id 15pX68-0001AE-00 for netdev@oss.sgi.com; Fri, 05 Oct 2001 17:41:12 +0200 Received: from laforge by naboo.gnumonks.org with local (Exim 3.22 #1) id 15pX9y-0000Cc-00; Fri, 05 Oct 2001 17:45:10 +0200 Date: Fri, 5 Oct 2001 17:45:10 +0200 From: Harald Welte To: David Miller Cc: netdev@oss.sgi.com, Netfilter Development Mailinglist Subject: [PATCH] fix bug in mac address matching code [bugtraq-advisory] Message-ID: <20011005174510.A739@naboo.gnumonks.org> Mail-Followup-To: Harald Welte , David Miller , netdev@oss.sgi.com, Netfilter Development Mailinglist Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="61jdw2sOBCFtR2d/" Content-Disposition: inline User-Agent: Mutt/1.3.17i X-Operating-System: Linux naboo.gnumonks.org 2.4.9 X-Date: Today is Pungenday, the 59th day of Bureaucracy in the YOLD 3167 Sender: owner-netdev@oss.sgi.com Precedence: bulk --61jdw2sOBCFtR2d/ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi Dave! Attached is a (proposed) patch to fix the problem described in the attached (should be on bugtraq by now) advisory. I'm not entirely sure if my idea behind this patch is correct, or if there are any more sophisticated checks you recommend for finding out if this skb has a valid mac address at the skb->mac.raw pointer. Please have a look and apply or comment. Thanks in advance, Harald. --- linux-2.4.9/net/ipv4/netfilter/ipt_mac.c Tue Oct 2 18:50:56 2001 +++ linux-2.4.9-ipt_mac-fix/net/ipv4/netfilter/ipt_mac.c Tue Oct 2 19:32:20 2001 @@ -20,7 +20,7 @@ /* Is mac pointer valid? */ return (skb->mac.raw >= skb->head - && skb->mac.raw < skb->head + skb->len - ETH_HLEN + && (skb->mac.raw + ETH_HLEN) <= skb->data /* If so, compare... */ && ((memcmp(skb->mac.ethernet->h_source, info->srcaddr, ETH_ALEN) == 0) ^ info->invert)); -- Live long and prosper - Harald Welte / laforge@gnumonks.org http://www.gnumonks.org/ ============================================================================ GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M- V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*) --61jdw2sOBCFtR2d/ Content-Type: message/rfc822 Content-Disposition: inline Return-Path:X-Sieve: cmu-sieve 2.0 Received: from samba.sourceforge.net ([198.186.203.85] helo=lists.samba.org) by coruscant.gnumonks.org with esmtp (Exim 3.33 #1) id 15lwdM-0003rg-00 for laforge@gnumonks.org; Tue, 25 Sep 2001 20:08:41 +0200 Received: from va.samba.org (localhost [127.0.0.1]) by lists.samba.org (Postfix) with ESMTP id 26AD24B14; Tue, 25 Sep 2001 11:02:08 -0700 (PDT) Delivered-To: netfilter@lists.samba.org Received: from mail1.netservers.co.uk (mail1.netservers.co.uk [193.115.249.2]) by lists.samba.org (Postfix) with ESMTP id B27924C2F for ; Tue, 25 Sep 2001 11:00:52 -0700 (PDT) Received: from server (firewall1.cammail.net [193.115.249.1]) by mail1.netservers.co.uk (8.12.0/8.12.0) with ESMTP id f8PI3aV9011365; Tue, 25 Sep 2001 19:03:36 +0100 From: Chris Wilson X-X-Sender: To: Cc: Subject: Bug in Linux 2.4 / iptables MAC match module Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Virus-Scanned: by Netservers.co.uk (http://netservers.co.uk/) Sender: netfilter-admin@lists.samba.org Errors-To: netfilter-admin@lists.samba.org X-BeenThere: netfilter@lists.samba.org X-Mailman-Version: 2.0.6 Precedence: bulk List-Help: List-Post: List-Subscribe: , List-Id: netfilter user discussion list List-Unsubscribe: , List-Archive: Date: Tue, 25 Sep 2001 19:03:45 +0100 (BST) Dear Netfilter Team, Thank you very much for your hard work in providing a world-leading firewall solution for free on Linux systems. Unfortunately we think we have discovered a bug in the Netfilter/iptables MAC address matching module. Please could you let us know as soon as you have some information regarding this bug. We very much hope to hear from you before Wednesday 3rd October 2001. If not then we shall be forced, reluctantly, to publish this advisory. Thank you again for your help and hard work. Yours sincerely, Chris Wilson, NetServers lead developer. ------------------------------------------------------------------------- __ _ _ ___ --[ | \| |__| |/ _/ __ __ _ _ __ __ __ ]-- --[ | \ \ /o_| _\_ \/o_| _| | |/o_| _|_ \ ]-- --[ |_|\__\__|__|__/\__|_| \_/ \__|_|___/ ]-- --[ netservers security advisory 01-09-26 ]-- SUBJECT : Bug in Linux 2.4 / iptables MAC match module SUMMARY : MAC match module does not match small packets EFFECTS : Malicious users may bypass MAC-based DROP rules pcAnywhere does not function correctly if allowed by MAC address SUMMARY ------- The Linux 2.4 kernel includes a new and very powerful firewalling, NAT and packet mangling architecture called Netfilter. The main component of Netfilter is iptables, a generic structure for allowing firewall rules to perform NAT, mangle packets, and access custom extensions for packet matching and mangling. One of the extensions supplied by default is the MAC module, which matches packets travelling through the firewall on the basis of their MAC (ethernet hardware) address. This module offers administrators some protection against malicious internal (or directly connected) users who spoof or change their IP address. The MAC module does not correctly match very small packets. For example, small ping packets can be generated by the Unix command 'ping somehost -s 4', or similarly under Windows with 'ping somehost -l 4'. Netcat with the -u option can generate small UDP packets which exhibit the same problem. REPRODUCTION ------------ To reproduce the problem, you will need 2 machines: - Victim, which runs iptables. - Attacker, which can generate small ICMP or UDP packets. We have used the DNS names 'Victim' and 'Attacker' to represent the IP addresses of these machines, and AT:TA:CK:ER:00:00 as the MAC address of the attacker. Please substitute real values if attempting to reproduce this problem. On Victim, at a root prompt: victim# iptables -P INPUT ACCEPT victim# iptables -F INPUT victim# iptables -I INPUT -p icmp -m mac --mac-source AT:TA:CK:ER:00:00 -j DROP victim# iptables -L INPUT -v Chain INPUT (policy ACCEPT xxxx packets, xxxxxxx bytes) pkts bytes target prot opt in out source destination 0 0 DROP icmp -- any any anywhere anywhere MAC AT:TA:CK:ER:00:00 [note that the packet and byte counters are zero] On Attacker (assuming Attacker runs Linux or similar) attacker# ping -s 8 -c 4 Victim PING Victim (xx.xx.xx.xx) from xx.xx.xx.xx : 8(36) bytes of data. --- xx.xx.xx.xx ping statistics --- 4 packets transmitted, 0 packets received, 100% packet loss [the ping packets were dropped, correctly] On Victim again: victim# iptables -L INPUT -v Chain INPUT (policy ACCEPT 231 packets, 39475 bytes) pkts bytes target prot opt in out source destination 4 144 DROP icmp -- any any anywhere anywhere MAC 00:03:47:87:BA:C5 [note that the packet and byte counters have increased, the packet counter is showing 4 packets which is correct] Now back to Attacker: attacker# ping -s 4 -c 4 Victim PING Victim (xx.xx.xx.xx) from xx.xx.xx.xx : 4(32) bytes of data. 12 bytes from xx.xx.xx.xx: icmp_seq=0 ttl=255 12 bytes from xx.xx.xx.xx: icmp_seq=1 ttl=255 12 bytes from xx.xx.xx.xx: icmp_seq=2 ttl=255 12 bytes from xx.xx.xx.xx: icmp_seq=3 ttl=255 --- xx.xx.xx.xx ping statistics --- 4 packets transmitted, 4 packets received, 0% packet loss [note that this time, all packets were replied to, whereas before they were dropped by the rule] And finally, back to Victim: victim# iptables -L INPUT -v Chain INPUT (policy ACCEPT 231 packets, 39475 bytes) pkts bytes target prot opt in out source destination 4 144 DROP icmp -- any any anywhere anywhere MAC AT:TA:CK:ER:00:00 [note that the packet counters have not increased, the packets did not match the drop rule] IMPLICATIONS ------------ >From a security point of view: - Malicious internal users may evade restrictions placed on their MAC address in some cases. For example, they might ping internal or external hosts to determine whether they are running, even though firewall rules disallow this. - They may also use small ICMP or UDP packets to leak information through the firewall, if no other rule prevents them from doing so. - Several applications use small ICMP or UDP packets, for example ping, netcat, and Symantec pcAnywhere. Administrators will not be able to restrict the use of these programs to certain known MAC addresses, because of this bug. This may result in lower overall security (especially as we know no complete workaround as yet). WORKAROUND ---------- The simplest, but least secure, workaround is to avoid matching by MAC address, but only match on IP address. This is common practice, but less secure than matching by MAC address. Another workaround is to use the latest version of iptables (1.2.3) from http://netfilter.samba.org. This includes a modules called "length" which can be used to match small packets. Some administrators might like to allow ICMP and/or UDP packets below a certain size with a command like this (UNTESTED): iptables -I INPUT -p icmp -m length --length 0:4 -j ACCEPT Note that using such a command will reduce the security of your iptables-protected hosts. SOLUTION -------- Wait for a patch from the Netfilter developers. POLITICS -------- This advisory is RFpolicy compliant (http://www.wiretrip.net/rfp/policy.html). If we do not receive any acknowledgement from the Netfilter team within 5 working days (by 9am GMT on Wednesday 3rd October) we will assume that they are not interested and publish this advisory on Bugtraq without a proper solution or workaround. We really hope that they will contact us and such measures will not be necessary. COPYRIGHT --------- This advisory is copyright (C) 2001 Netservers.co.uk. Redistribution is permitted provided the contents and text of the advisory are not modified in any way. Ciao, Chris. -- ___ __ _ / __// / ,__(_)_ | Chris Wilson -- UNIX Firewall Lead Developer | / (_ / ,\/ _/ /_ \ | NetServers.co.uk http://www.netservers.co.uk | \ _//_/_/_//_/___/ | 21 Signet Court, Cambridge, UK. 01223 576516 | --61jdw2sOBCFtR2d/ Content-Type: text/plain; charset=us-ascii Content-Description: Mail describing reasons behind patch Content-Disposition: attachment; filename=foo From laforge@gnumonks.org Tue Oct 2 19:57:28 2001 Date: Tue, 2 Oct 2001 19:57:28 +0200 From: Harald Welte To: Chris Wilson Cc: Rusty Russell , Marc Boucher , James Morris , Netfilter Development Mailinglist Subject: Re: Bug in Linux 2.4 / iptables MAC match module Message-ID: <20011002195727.H3206@naboo.gnumonks.org> Mail-Followup-To: Harald Welte , Chris Wilson , Rusty Russell , Marc Boucher , James Morris , Netfilter Development Mailinglist References: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="7DO5AaGCk89r4vaK" Content-Disposition: inline User-Agent: Mutt/1.3.17i In-Reply-To: ; from chris@netservers.co.uk on Tue, Sep 25, 2001 at 07:03:45PM +0100 X-Operating-System: Linux naboo.gnumonks.org 2.4.9 X-Date: Today is Setting Orange, the 56th day of Bureaucracy in the YOLD 3167 Status: RO Content-Length: 2931 Lines: 83 --7DO5AaGCk89r4vaK Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Sep 25, 2001 at 07:03:45PM +0100, Chris Wilson wrote: > Please could you let us know as soon as you have some information > regarding this bug. We very much hope to hear from you before Wednesday > 3rd October 2001. If not then we shall be forced, reluctantly, to publish > this advisory. ok. There is news about this now.=20 As far as I understood the problem, the problematic piece of code was: /* Is mac pointer valid? */ return (skb->mac.raw >=3D skb->head && skb->mac.raw < skb->head + skb->len - ETH_HLEN /* If so, compare... */ && ((memcmp(skb->mac.ethernet->h_source, info->srcaddr, ETH_ALEN) =3D=3D 0) ^ info->invert)); skb->head points to first byte of the layer 2 packet skb->mac.raw points to first byte of destination mac address skb->data points to first byte of layer 3 packet (=3D=3D ip header) skb->len length of layer three packet (layer 2 payload) in bytes ETH_HLEN length of layer 2 header (14 bytes, 2*6byte mac + 2byte l3prot) The first check checks, if the pointer to the beginning of the mac address is greater or equal than the skb->head. That's ok. The second check, however does something strange. It checks if the pointer= to the beginning of the mac address (first byte of destination mac) is smaller than skb->head + skb->len - ETH_HLEN. This doesn't seem to make sense to m= e. I guess the intention was to check if the whole mac address fits within the skb's valid data area. But this is not what was done. skb->head + skb->len does not point to the end of the packet, skb->data + skb->len would do. The original calculation "head + len" leads= to the assumption the packet is shorter than it really is (by "skb->data - skb= ->head" bytes short). I think it's better to check if skb->mac.raw + ETH_HLEN <=3D skb->data. Thi= s is what attached patch does. Could you please verify that your problem is gone with attached patch? Thanks. > Chris Wilson, NetServers lead developer. --=20 Live long and prosper - Harald Welte / laforge@gnumonks.org http://www.gnumonks.org/ =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M-= =20 V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*) --7DO5AaGCk89r4vaK Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE7ugAHXaXGVTD0i/8RAsxgAJ4knuljKePvMqz5f/SI5yc1G4a0TQCgmgBr MAj9vLeWqv/5zxvH78AjaC0= =LmeI -----END PGP SIGNATURE----- --7DO5AaGCk89r4vaK-- --61jdw2sOBCFtR2d/-- From owner-netdev@oss.sgi.com Fri Oct 5 09:42:50 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f95Ggo610009 for netdev-outgoing; Fri, 5 Oct 2001 09:42:50 -0700 Received: from ms2.inr.ac.ru (minus.inr.ac.ru [193.233.7.97]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f95GgjD10005 for ; Fri, 5 Oct 2001 09:42:45 -0700 Received: (from kuznet@localhost) by ms2.inr.ac.ru (8.6.13/ANK) id UAA21823; Fri, 5 Oct 2001 20:42:12 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200110051642.UAA21823@ms2.inr.ac.ru> Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 To: mingo@elte.hu Date: Fri, 5 Oct 2001 20:42:12 +0400 (MSK DST) Cc: hadi@cyberus.ca, linux-kernel@vger.kernel.org, Robert.Olsson@data.slu.se, bcrl@redhat.com, netdev@oss.sgi.com, torvalds@transmeta.com, alan@lxorguk.ukuu.org.uk In-Reply-To: from "Ingo Molnar" at Oct 4, 1 08:35:02 am X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Sender: owner-netdev@oss.sgi.com Precedence: bulk Hello! > i'm asking the following thing. dev->quota, as i read the patch now, can > cause extra calls to ->poll() even though the RX ring of that particular > device is empty and the driver has indicated it's done processing RX > packets. (i'm now assuming that the extra-polling-for-a-jiffy line in the > current patch is removed - that one is a showstopper to begin with.) Is > this claim of mine correct? No. If ring is empty, device is removed from poll list and dev->poll is not called more. dev->quota is to preempt service when ring does not want to clear. In this case work remains for the next round after all the rest of interfaces are served. Well, it is to allow to give user control on distribution cpu time between interfaces, when cpu is 100% utilized and we have to drop something. Devices with lower weights will get less service. > packets. (i'm now assuming that the extra-polling-for-a-jiffy line in the It is not so bogus with current kernel with working ksoftirqd. The goal was to check what happens really when we enforce polling even when machine is generally happy. For me it is not evident apriori: more cpu is eaten uselessly or less due to absent irqs. Note, that on dedicated router it is pretty normal to spin in context of ksoftirqd switching to control tasks when it is required. And, actually, it is amazing feature of the scheme, that it is so easy to add such option. Anyway, to all that I remember, the question remained unanswered. :-) Robert even observed that only 9% of cpu is eaten, which is surely cannot be true. :-) Alexey From owner-netdev@oss.sgi.com Fri Oct 5 11:15:20 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f95IFKY12558 for netdev-outgoing; Fri, 5 Oct 2001 11:15:20 -0700 Received: from sgi.com (sgi.SGI.COM [192.48.153.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f95IF8D12554 for ; Fri, 5 Oct 2001 11:15:08 -0700 Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227]) by sgi.com (980327.SGI.8.8.8-aspam/980304.SGI-aspam: SGI does not authorize the use of its proprietary systems or networks for unsolicited or bulk email from the Internet.) via ESMTP id LAA01027 for ; Fri, 5 Oct 2001 11:15:26 -0700 (PDT) mail_from (mkprice@us.ibm.com) Received: from southrelay03.raleigh.ibm.com (southrelay03.raleigh.ibm.com [9.37.3.210]) by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id NAA236892; Fri, 5 Oct 2001 13:02:55 -0500 Received: from gateway1.beaverton.ibm.com (gateway1.beaverton.ibm.com [138.95.180.2]) by southrelay03.raleigh.ibm.com (8.11.1m3/NCO v4.97.1) with ESMTP id f95I5H959074; Fri, 5 Oct 2001 14:05:18 -0400 Received: from crg8.beaverton.ibm.com (crg8.beaverton.ibm.com [138.95.19.9]) by gateway1.beaverton.ibm.com (8.11.2/8.11.2) with ESMTP id f95I5HU09968; Fri, 5 Oct 2001 11:05:17 -0700 Received: from dyn9-47-8-184.rhe.beaverton.ibm.com (dyn9-47-8-184.rhe.beaverton.ibm.com [9.47.8.184]) by crg8.beaverton.ibm.com (8.10.0.Beta10/8.8.5/token.aware-1.2) with ESMTP id f95I5Fn07552; Fri, 5 Oct 2001 11:05:16 -0700 (PDT) Date: Fri, 5 Oct 2001 11:05:06 -0700 (PDT) From: Mark Price X-X-Sender: To: cc: , , Subject: Re: patch - SNMP Kernel Counters In-Reply-To: <200110051525.TAA20980@ms2.inr.ac.ru> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Fri, 5 Oct 2001 kuznet@ms2.inr.ac.ru wrote: > > o Modified the way IpInDelivers is incremented. > > OK. > > Technically wrong though. Please try to invent some way to eliminate > additional variable "aflg" in some way yet... I'll rework it. > Also: > > > + IP_INC_STATS_BH(IpInDiscards); > > goto drop; > > It is pretty evident that increment must follow the label "drop". No? > Well, this happens in several places. The reason I did not increment this counter in the "drop:" label is that the counter should not be incremented on all drops. The description of IpInDiscards is: DESCRIPTION "The number of input IP datagrams for which no problems were encountered to prevent their continued processing, but which were discarded (e.g., for lack of buffer space). Note that this counter does not include any datagrams discarded while awaiting re-assembly." So I only increment in two places, once when skb_cow() fails, and once when skb_share_check() fails. Cheers, Mark. From owner-netdev@oss.sgi.com Fri Oct 5 11:37:15 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f95IbFF13226 for netdev-outgoing; Fri, 5 Oct 2001 11:37:15 -0700 Received: from ms2.inr.ac.ru (minus.inr.ac.ru [193.233.7.97]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f95IbBD13223 for ; Fri, 5 Oct 2001 11:37:12 -0700 Received: (from kuznet@localhost) by ms2.inr.ac.ru (8.6.13/ANK) id WAA22806; Fri, 5 Oct 2001 22:37:03 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200110051837.WAA22806@ms2.inr.ac.ru> Subject: Re: patch - SNMP Kernel Counters To: mkprice@us.ibm.com (Mark Price) Date: Fri, 5 Oct 2001 22:37:03 +0400 (MSK DST) Cc: mkprice@us.ibm.com, netdev@oss.sgi.com, davem@redhat.com In-Reply-To: from "Mark Price" at Oct 5, 1 11:05:06 am X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Sender: owner-netdev@oss.sgi.com Precedence: bulk Hello! > The reason I did not increment this counter in the "drop:" label is that > the counter should not be incremented on all drops. The description of > IpInDiscards is: I see. Indeed. > So I only increment in two places, once when skb_cow() fails, and once > when skb_share_check() fails. Well, make another label sort of "drop_and_account", "drop_with_acct" :-) Invent good name. :-) Alexey From owner-netdev@oss.sgi.com Fri Oct 5 11:49:50 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f95InoO13677 for netdev-outgoing; Fri, 5 Oct 2001 11:49:50 -0700 Received: from webber.adilger.int (h24-64-71-161.cg.shawcable.net [24.64.71.161]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f95InhD13674 for ; Fri, 5 Oct 2001 11:49:43 -0700 Received: from webber.adilger.int (adilger@localhost [127.0.0.1]) by localhost (8.12.0.Beta19/8.12.0.Beta19/Debian 8.12.0.Beta19) with ESMTP id f95ImRL7000613; Fri, 5 Oct 2001 12:48:27 -0600 Received: (from adilger@localhost) by webber.adilger.int (8.12.0.Beta19/8.12.0.Beta19/Debian 8.12.0.Beta19) id f95ImO85000611; Fri, 5 Oct 2001 12:48:24 -0600 Date: Fri, 5 Oct 2001 12:48:24 -0600 From: Andreas Dilger To: Robert Olsson Cc: mingo@elte.hu, jamal , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Benjamin LaHaise , netdev@oss.sgi.com, Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: <20011005124824.F315@turbolinux.com> Mail-Followup-To: Robert Olsson , mingo@elte.hu, jamal , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Benjamin LaHaise , netdev@oss.sgi.com, Linus Torvalds , Alan Cox References: <15291.32311.499838.886628@robur.slu.se> <20011003162210.L8954@turbolinux.com> <15293.51488.280808.469262@robur.slu.se> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <15293.51488.280808.469262@robur.slu.se> User-Agent: Mutt/1.3.22i "X-GPG-Key: 1024D/0D35BED6" "X-GPG-Fingerprint: 7A37 5D79 BF1B CECA D44F 8A29 A488 39F5 0D35 BED6" Sender: owner-netdev@oss.sgi.com Precedence: bulk On Oct 05, 2001 16:52 +0200, Robert Olsson wrote: > > If you get to the stage where you are turning off IRQs and going to a > > polling mode, then don't turn IRQs back on until you have a poll (or > > two or whatever) that there is no work to be done. This will at worst > > give you 50% polling success, but in practise you wouldn't start polling > > until there is lots of work to be done, so the real success rate will > > be much higher. > > > > At this point (no work to be done when polling) there are clearly no > > interrupts would be generated (because no packets have arrived), so it > > should be reasonable to turn interrupts back on and stop polling (assuming > > non-broken hardware). You now go back to interrupt-driven work until > > the rate increases again. This means you limit IRQ rates when needed, > > but only do one or two excess polls before going back to IRQ-driven work. > > Yes this has been considered and actually I think Jamal did this in one of > the pre NAPI patch. I tried something similar... but instead of using a > number of excess polls I was doing excess polls for a short time (a > jiffie). This was the showstopper mentioned the previous mails. :-) (Note that I hadn't read the NAPI paper until after I posted the above, and it appears that I was describing pretty much what NAPI already does ;-). In light of that, I wholly agree that NAPI is a superior solution for handling IRQ overload from a network device. > Anyway it up to driver to decide this policy. If the driver returns > "not_done" it is simply polled again. So low-rate network drivers can have > a different policy compared to an OC-48 driver. Even continues polling is > therefore possible and even showstoppers. :-) There are protection for > polling livelocks. One question which I have is why would you ever want to continue polling if there is no work to be done? Is it a tradeoff between the amount of time to handle an IRQ vs. the time to do a poll? An assumption that if there was previous network traffic there is likely to be more the next time the interface is checked (assuming you have other work to do between the time you last polled the device and the next poll)? Is enabling/disabling of the RX interrupts on the network card an issue in the sense of "you need to wait X us after writing to this register for it to take effect" or other issue which makes it preferrable to have some "hysteresis" between changing state from IRQ-driven to polling? Cheers, Andreas -- Andreas Dilger \ "If a man ate a pound of pasta and a pound of antipasto, \ would they cancel out, leaving him still hungry?" http://www-mddsp.enel.ucalgary.ca/People/adilger/ -- Dogbert From owner-netdev@oss.sgi.com Fri Oct 5 12:02:23 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f95J2NH14125 for netdev-outgoing; Fri, 5 Oct 2001 12:02:23 -0700 Received: from xmailserver.org ([208.129.208.52]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f95J2JD14121 for ; Fri, 5 Oct 2001 12:02:19 -0700 Received: from blue1.dev.mcafeelabs.com (161.69.79.199) by i750raq.dev.mcafeelabs.com. (161.69.86.244) with [XMail 1.1 (Linux/Ix86) ESMTP Server] id for from ; Fri, 05 Oct 2001 13:18:01 -0700 Date: Fri, 5 Oct 2001 12:07:08 -0700 (PDT) From: Davide Libenzi X-X-Sender: davide@blue1.dev.mcafeelabs.com To: Andreas Dilger cc: Robert Olsson , , jamal , , Alexey Kuznetsov , Benjamin LaHaise , , Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <20011005124824.F315@turbolinux.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk On Fri, 5 Oct 2001, Andreas Dilger wrote: > On Oct 05, 2001 16:52 +0200, Robert Olsson wrote: > > > If you get to the stage where you are turning off IRQs and going to a > > > polling mode, then don't turn IRQs back on until you have a poll (or > > > two or whatever) that there is no work to be done. This will at worst > > > give you 50% polling success, but in practise you wouldn't start polling > > > until there is lots of work to be done, so the real success rate will > > > be much higher. > > > > > > At this point (no work to be done when polling) there are clearly no > > > interrupts would be generated (because no packets have arrived), so it > > > should be reasonable to turn interrupts back on and stop polling (assuming > > > non-broken hardware). You now go back to interrupt-driven work until > > > the rate increases again. This means you limit IRQ rates when needed, > > > but only do one or two excess polls before going back to IRQ-driven work. > > > > Yes this has been considered and actually I think Jamal did this in one of > > the pre NAPI patch. I tried something similar... but instead of using a > > number of excess polls I was doing excess polls for a short time (a > > jiffie). This was the showstopper mentioned the previous mails. :-) > > (Note that I hadn't read the NAPI paper until after I posted the above, and > it appears that I was describing pretty much what NAPI already does ;-). In > light of that, I wholly agree that NAPI is a superior solution for handling > IRQ overload from a network device. > > > Anyway it up to driver to decide this policy. If the driver returns > > "not_done" it is simply polled again. So low-rate network drivers can have > > a different policy compared to an OC-48 driver. Even continues polling is > > therefore possible and even showstoppers. :-) There are protection for > > polling livelocks. > > One question which I have is why would you ever want to continue polling > if there is no work to be done? According to the doc the poll is stopped when 1) there're no more packets to be fetched from dma ring 2) quota is reached. - Davide From owner-netdev@oss.sgi.com Fri Oct 5 12:17:52 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f95JHqc14853 for netdev-outgoing; Fri, 5 Oct 2001 12:17:52 -0700 Received: from ms2.inr.ac.ru (minus.inr.ac.ru [193.233.7.97]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f95JHmD14849 for ; Fri, 5 Oct 2001 12:17:49 -0700 Received: (from kuznet@localhost) by ms2.inr.ac.ru (8.6.13/ANK) id XAA23007; Fri, 5 Oct 2001 23:17:22 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200110051917.XAA23007@ms2.inr.ac.ru> Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 To: adilger@turbolabs.com (Andreas Dilger) Date: Fri, 5 Oct 2001 23:17:22 +0400 (MSK DST) Cc: Robert.Olsson@data.slu.se, mingo@elte.hu, hadi@cyberus.ca, linux-kernel@vger.kernel.org, bcrl@redhat.com, netdev@oss.sgi.com, torvalds@transmeta.com, alan@lxorguk.ukuu.org.uk In-Reply-To: <20011005124824.F315@turbolinux.com> from "Andreas Dilger" at Oct 5, 1 12:48:24 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Sender: owner-netdev@oss.sgi.com Precedence: bulk Hello! > One question which I have is why would you ever want to continue polling > if there is no work to be done? Is it a tradeoff between the amount of > time to handle an IRQ vs. the time to do a poll? Yes. IRQ even taken alone eat non-trivial amount of resources. Actually, I remember Jamal worked with machine, which had no io-apic and only irq ack/mask/unmask eated >15% of cpu there. :-) > An assumption that if > there was previous network traffic there is likely to be more the next > time the interface is checked (assuming you have other work to do between > the time you last polled the device and the next poll)? Exactly. Note also that the testing of "goto not_done" was made in pure environment: dedicated router. Continuous polling is an evident advantage in this situation, only power is eaten. I would not enable this on a notebook. :-) > Is enabling/disabling of the RX interrupts on the network card an issue > in the sense of "you need to wait X us after writing to this register > for it to take effect" or other issue which makes it preferrable to have > some "hysteresis" between changing state from IRQ-driven to polling? "some hysteresis" is right word. This loop is an experiment with still unknown result yet. Originally, Jamal proposed to spin several times. I killed this. Robert proposed to check inifinite loop yet. (Note, jiffies check is just a way to get rid of completely idle devices, one jiffie is enough lonf time to be considered infinite). Alexey From owner-netdev@oss.sgi.com Fri Oct 5 16:49:05 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f95Nn5a25564 for netdev-outgoing; Fri, 5 Oct 2001 16:49:05 -0700 Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f95Nn3D25560 for ; Fri, 5 Oct 2001 16:49:03 -0700 Received: from localhost (IDENT:davem@pizda.ninka.net [127.0.0.1]) by pizda.ninka.net (8.9.3/8.9.3) with ESMTP id QAA30676; Fri, 5 Oct 2001 16:48:54 -0700 Date: Fri, 05 Oct 2001 16:48:54 -0700 (PDT) Message-Id: <20011005.164854.45178205.davem@redhat.com> To: laforge@gnumonks.org Cc: netdev@oss.sgi.com, netfilter-devel@lists.samba.org Subject: Re: [PATCH] fix bug in mac address matching code [bugtraq-advisory] From: "David S. Miller" In-Reply-To: <20011005174510.A739@naboo.gnumonks.org> References: <20011005174510.A739@naboo.gnumonks.org> X-Mailer: Mew version 2.0 on Emacs 21.0 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk From: Harald Welte Date: Fri, 5 Oct 2001 17:45:10 +0200 Please have a look and apply or comment. Applied, I'll push this to Linus today. Franks a lot, David S. Miller davem@redhat.com From owner-netdev@oss.sgi.com Sat Oct 6 23:09:48 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9769mW08127 for netdev-outgoing; Sat, 6 Oct 2001 23:09:48 -0700 Received: from robur.slu.se (robur.slu.se [130.238.98.12]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9769iD08123 for ; Sat, 6 Oct 2001 23:09:44 -0700 Received: (from robert@localhost) by robur.slu.se (8.8.7/8.8.7) id IAA22404; Sun, 7 Oct 2001 08:11:28 +0200 From: Robert Olsson MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15295.61967.701804.309307@robur.slu.se> Date: Sun, 7 Oct 2001 08:11:27 +0200 To: kuznet@ms2.inr.ac.ru Cc: adilger@turbolabs.com (Andreas Dilger), Robert.Olsson@data.slu.se, mingo@elte.hu, hadi@cyberus.ca, linux-kernel@vger.kernel.org, bcrl@redhat.com, netdev@oss.sgi.com, torvalds@transmeta.com, alan@lxorguk.ukuu.org.uk Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <200110051917.XAA23007@ms2.inr.ac.ru> References: <20011005124824.F315@turbolinux.com> <200110051917.XAA23007@ms2.inr.ac.ru> X-Mailer: VM 6.92 under Emacs 19.34.1 Sender: owner-netdev@oss.sgi.com Precedence: bulk kuznet@ms2.inr.ac.ru writes: > "some hysteresis" is right word. This loop is an experiment with still > unknown result yet. Originally, Jamal proposed to spin several times. > I killed this. Robert proposed to check inifinite loop yet. (Note, > jiffies check is just a way to get rid of completely idle devices, > one jiffie is enough lonf time to be considered infinite). > And from our discussion about packet-reordering we get even more motivation for the "extra-polls" not only to save IRQ's We may expand this to others too... As polling-lists are per CPU and consecutive polls stays within the same CPU the device becomes bound to one CPU. We are protected against packet reordering as long there are consecutive polls. I've consulted some CS people who has worked with this issues and I have understood packet reordering is non-trivial problem at least with a general approach. So to me it seems we do very well with a very simple scheme and as I understand all SMP networking will benefit from this. Our "field-test" indicates that the packet load is still well distributed among the CPU's. So maybe the showstopper comes out as a showwinner. :-) Cheers. --ro From owner-netdev@oss.sgi.com Sun Oct 7 17:33:20 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f980XKc31063 for netdev-outgoing; Sun, 7 Oct 2001 17:33:20 -0700 Received: from athlon.random ([195.223.140.107]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f980XCD31058 for ; Sun, 7 Oct 2001 17:33:13 -0700 Received: (from andrea@localhost) by athlon.random (8.11.2/8.10.2/SuSE Linux 8.10.0-0.3) id f980VIr24603; Mon, 8 Oct 2001 02:31:18 +0200 Date: Mon, 8 Oct 2001 02:31:18 +0200 From: Andrea Arcangeli To: Ingo Molnar Cc: jamal , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , netdev@oss.sgi.com, Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: <20011008023118.L726@athlon.random> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: ; from mingo@elte.hu on Wed, Oct 03, 2001 at 06:51:55PM +0200 X-GnuPG-Key-URL: http://e-mind.com/~andrea/aa.gnupg.asc X-PGP-Key-URL: http://e-mind.com/~andrea/aa.asc Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1953 Lines: 37 [ I hope not to reiterate the obvious, I didn't read every single email of this thread ] > > > In a generic computing environment i want to spend cycles doing useful > > > work, not polling. Even the quick kpolld hack [which i dropped, so please > > > dont regard it as a 'competitor' patch] i consider superior to this, as i > > > can renice kpolld to reduce polling. (plus kpolld sucks up available idle > > > cycles as well.) Unless i royally misunderstand it, i cannot stop the > > > above code from wasting my cycles, and if that is true i do not want to > > > see it in the kernel proper in this form. > > On Wed, 3 Oct 2001, jamal wrote: > > The interupt just flags "i, netdev, have work to do"; [...] On Wed, Oct 03, 2001 at 06:51:55PM +0200, Ingo Molnar wrote: > (and the only thing i pointed out was that the patch as-is did not limit > the amount of polling done.) You're perfectly right that it's not ok for a generic computing environment to spend lots of cpu in polling, but it is clear that in a dedicated router/firewall we can just shutdown the NIC interrupt forever via disable_irq (no matter if the nic supports hw flow control or not, and in turn no matter if the kid tries to spam the machine with small packets) and dedicate 1 cpu to the polling-work with ksoftirqd polling forever the NIC to deliver maximal routing performance or something like that. ksoftirqd will ensure fairness with the userspace load as well. You probably wouldn't get a benefit with tux because you would potentially lose way too much cpu with true polling and you're traffic is mostly going from the server to the clients not the othet way around (plus the clients uses delayed acks etc..), but the world isn't just tux. Of course we agree that such a "polling router/firewall" behaviour must not be the default but it must be enabled on demand by the admin via sysctl or whatever else userspace API. And I don't see any problem with that. Andrea From owner-netdev@oss.sgi.com Sun Oct 7 17:56:27 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f980uRt31505 for netdev-outgoing; Sun, 7 Oct 2001 17:56:27 -0700 Received: from sgi.com (sgi.SGI.COM [192.48.153.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f980uMD31501 for ; Sun, 7 Oct 2001 17:56:22 -0700 Received: from athlon.random ([195.223.140.107]) by sgi.com (980327.SGI.8.8.8-aspam/980304.SGI-aspam: SGI does not authorize the use of its proprietary systems or networks for unsolicited or bulk email from the Internet.) via ESMTP id RAA04302 for ; Sun, 7 Oct 2001 17:55:09 -0700 (PDT) mail_from (andrea@suse.de) Received: (from andrea@localhost) by athlon.random (8.11.2/8.10.2/SuSE Linux 8.10.0-0.3) id f980VIr24603; Mon, 8 Oct 2001 02:31:18 +0200 Date: Mon, 8 Oct 2001 02:31:18 +0200 From: Andrea Arcangeli To: Ingo Molnar Cc: jamal , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , netdev@oss.sgi.com, Linus Torvalds , Alan Cox Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: <20011008023118.L726@athlon.random> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: ; from mingo@elte.hu on Wed, Oct 03, 2001 at 06:51:55PM +0200 X-GnuPG-Key-URL: http://e-mind.com/~andrea/aa.gnupg.asc X-PGP-Key-URL: http://e-mind.com/~andrea/aa.asc Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1953 Lines: 37 [ I hope not to reiterate the obvious, I didn't read every single email of this thread ] > > > In a generic computing environment i want to spend cycles doing useful > > > work, not polling. Even the quick kpolld hack [which i dropped, so please > > > dont regard it as a 'competitor' patch] i consider superior to this, as i > > > can renice kpolld to reduce polling. (plus kpolld sucks up available idle > > > cycles as well.) Unless i royally misunderstand it, i cannot stop the > > > above code from wasting my cycles, and if that is true i do not want to > > > see it in the kernel proper in this form. > > On Wed, 3 Oct 2001, jamal wrote: > > The interupt just flags "i, netdev, have work to do"; [...] On Wed, Oct 03, 2001 at 06:51:55PM +0200, Ingo Molnar wrote: > (and the only thing i pointed out was that the patch as-is did not limit > the amount of polling done.) You're perfectly right that it's not ok for a generic computing environment to spend lots of cpu in polling, but it is clear that in a dedicated router/firewall we can just shutdown the NIC interrupt forever via disable_irq (no matter if the nic supports hw flow control or not, and in turn no matter if the kid tries to spam the machine with small packets) and dedicate 1 cpu to the polling-work with ksoftirqd polling forever the NIC to deliver maximal routing performance or something like that. ksoftirqd will ensure fairness with the userspace load as well. You probably wouldn't get a benefit with tux because you would potentially lose way too much cpu with true polling and you're traffic is mostly going from the server to the clients not the othet way around (plus the clients uses delayed acks etc..), but the world isn't just tux. Of course we agree that such a "polling router/firewall" behaviour must not be the default but it must be enabled on demand by the admin via sysctl or whatever else userspace API. And I don't see any problem with that. Andrea From owner-netdev@oss.sgi.com Sun Oct 7 20:37:06 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f983b6S02817 for netdev-outgoing; Sun, 7 Oct 2001 20:37:06 -0700 Received: from sgi.com (sgi.SGI.COM [192.48.153.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f983avD02813 for ; Sun, 7 Oct 2001 20:36:57 -0700 Received: from tux.rsn.bth.se (tux.rsn.bth.se [194.47.143.135]) by sgi.com (980327.SGI.8.8.8-aspam/980304.SGI-aspam: SGI does not authorize the use of its proprietary systems or networks for unsolicited or bulk email from the Internet.) via ESMTP id UAA07863 for ; Sun, 7 Oct 2001 20:35:43 -0700 (PDT) mail_from (gandalf@wlug.westbo.se) Received: from localhost (gandalf@localhost [127.0.0.1]) by localhost (8.12.0.Beta19/8.12.0.Beta19/Debian 8.12.0.Beta19) with ESMTP id f983Ge9X010492 for ; Mon, 8 Oct 2001 05:16:40 +0200 Date: Mon, 8 Oct 2001 05:16:40 +0200 (CEST) From: Martin Josefsson X-Sender: gandalf@tux.rsn.bth.se To: netdev@oss.sgi.com Subject: 2.2 performance on high network load much much better than 2.4 (fwd) Message-ID: X-message-flag: Get yourself a real mail client! http://www.washington.edu/pine/ MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 3550 Lines: 107 I forwrd this mail from lkml because I know some networking people aren't on lkml. /Martin Never argue with an idiot. They drag you down to their level, then beat you with experience. ---------- Forwarded message ---------- Date: Sun, 7 Oct 2001 18:54:27 +0200 From: Santiago Garcia Mantinan To: linux-kernel@vger.kernel.org Subject: 2.2 performance on high network load much much better than 2.4 Hi! I was very concerned about the problems when machines are spammed, so I started some tests comparing the different kernels and how they reacted to this, I was astonished to see that 2.2.19 can cope very very well with those spams while 2.4 is absolutely frozen under the same circunstances, so I was wondering... how can this be, what has changed so much that now the interrupts on a 2.4 kernel are so so so much slower than before like if they were doing much more job? :-???? Well, this are some numbers I have noted down when comparing several kernels... The test consisted on doing a "time bunzip2 -t linux-2.4.9.bz2" noting the time and the received network packets and overruns, during this time, in all cases counter for errors, dropped or frame showed 0. The test takes 31 seconds if done without network load. Spammed host: CPU PIII at 868MHz RAM 256MB Chipset Via Apollo NIC 3Com 905C 3c59x standard kernel driver Spammers: P200MMX, and P166 with rtl8139 NICs using Simon Kirby's udpspam. Network: 8 ports SVEC FD821 10/100Mb FD switch based on realtek chips. Behaviour under 2.4.9ac18 kernel: Time: 42 minutes 48 seconds RX packets (aprox): 109 million overruns: 664 Interrupts as mesured by vmstat 1: from 6400 to 7700 (6500 average) Behaviour under 2.4.10 kernel: Time: 17 minutes 51 seconds RX packets (aprox): 53.6 million overruns: 657 Interrupts as mesured by vmstat 1: from 6400 to 7700 (6500 average) Behaviour under 2.2.19 kernel: Time: 1 minute 8 seconds RX packets (aprox): 3 million overruns: 81000 Interrupts as mesured by vmstat 1: from 35000 to 40000 (38500 average) Note on the overruns on 2.2.19, they grow a lot when the first test is carried, but if I continue to make tests they are much lower, in fact I even added another spammer machine to the 2.2.19 test (dual P133) and I got this results where you can see that the overruns are much much lower even though the load was increased: Time: 1 minute 17 seconds RX packets (aprox): 4.3 million overruns: 476 Interrupts as mesured by vmstat 1: from 40000 to 44000 (43500 average) One other thing that was seen on 2.2.19 and that did not appear on 2.4 was that 2.2.19 said from time to time... Too much work in interrupt, status e401. I tried to spam 2.4.9ac18 with the same three machines I had used for the last 2.2.19 test but the machine was almost totally frozen, after 6 hours of test I stopped it. So... is this a problem with 2.4 kernels that I'm the only one experimenting? Is there any explanation why 2.4 kernels are so bad on handling this spamming while 2.2.19 does quite well? If you need more data or want me to run any other tests, please tell me, but send me expecific notes on what you want me to do. Hope this helps finding what is happening with 2.4 kernels regarding to this bad network behaviour. Regards... -- Manty/BestiaTester -> http://manty.net - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ From owner-netdev@oss.sgi.com Mon Oct 8 07:05:31 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f98E5VP15172 for netdev-outgoing; Mon, 8 Oct 2001 07:05:31 -0700 Received: from shell.cyberus.ca ([216.191.240.114]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f98E5RD15169 for ; Mon, 8 Oct 2001 07:05:27 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id JAA05584; Mon, 8 Oct 2001 09:58:22 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Mon, 8 Oct 2001 09:58:22 -0400 (EDT) From: jamal To: cc: Andreas Dilger , , , , , , , Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <200110051917.XAA23007@ms2.inr.ac.ru> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1340 Lines: 41 On Fri, 5 Oct 2001 kuznet@ms2.inr.ac.ru wrote: > Hello! > > > One question which I have is why would you ever want to continue polling > > if there is no work to be done? Is it a tradeoff between the amount of > > time to handle an IRQ vs. the time to do a poll? > > Yes. IRQ even taken alone eat non-trivial amount of resources. > > Actually, I remember Jamal worked with machine, which had > no io-apic and only irq ack/mask/unmask eated >15% of cpu there. :-) > This was Robert actually; conclusion was Interupts are very expensive. If we can get rid of as many of them as possible, we are getting a side benefit. I cant find the old data, but Robert has some data over here: http://robur.slu.se/Linux/net-development/experiments/010301 > "some hysteresis" is right word. This loop is an experiment with still > unknown result yet. Originally, Jamal proposed to spin several times. > I killed this. It was a good idea you killed it, now that i think in retrospect, The solution is much cleaner without it. > Robert proposed to check inifinite loop yet. (Note, > jiffies check is just a way to get rid of completely idle devices, > one jiffie is enough lonf time to be considered infinite). > In my opinion we really dont need this. I did some quick testing, with and without it and i dont see any differences. cheers, jamal From owner-netdev@oss.sgi.com Mon Oct 8 07:23:29 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f98ENTG16039 for netdev-outgoing; Mon, 8 Oct 2001 07:23:29 -0700 Received: from shell.cyberus.ca ([216.191.240.114]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f98ENHD16031 for ; Mon, 8 Oct 2001 07:23:18 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id KAA05627; Mon, 8 Oct 2001 10:20:24 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Mon, 8 Oct 2001 10:20:24 -0400 (EDT) From: jamal To: Martin Josefsson cc: , Subject: Re: 2.2 performance on high network load much much better than 2.4 (fwd) In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 4306 Lines: 133 On Mon, 8 Oct 2001, Martin Josefsson wrote: > I forwrd this mail from lkml because I know some networking people aren't > on lkml. > Thanks for doing this ;-> I wish people would post networking related issues to netdev (or cc netdev at least); maybe it should be on the FAQ > > Never argue with an idiot. They drag you down to their level, then beat you with experience. > > ---------- Forwarded message ---------- > Date: Sun, 7 Oct 2001 18:54:27 +0200 > From: Santiago Garcia Mantinan > To: linux-kernel@vger.kernel.org > Subject: 2.2 performance on high network load much much better than 2.4 > > Hi! > > I was very concerned about the problems when machines are spammed, so I > started some tests comparing the different kernels and how they reacted to > this, I was astonished to see that 2.2.19 can cope very very well with those > spams while 2.4 is absolutely frozen under the same circunstances, so I was > wondering... how can this be, what has changed so much that now the > interrupts on a 2.4 kernel are so so so much slower than before like if they > were doing much more job? :-???? > > Well, this are some numbers I have noted down when comparing several > kernels... > > The test consisted on doing a "time bunzip2 -t linux-2.4.9.bz2" noting the > time and the received network packets and overruns, during this time, in all > cases counter for errors, dropped or frame showed 0. The test takes 31 > seconds if done without network load. > > Spammed host: > CPU PIII at 868MHz > RAM 256MB > Chipset Via Apollo > NIC 3Com 905C 3c59x standard kernel driver > > Spammers: P200MMX, and P166 with rtl8139 NICs using Simon Kirby's udpspam. > Network: 8 ports SVEC FD821 10/100Mb FD switch based on realtek chips. > > > Behaviour under 2.4.9ac18 kernel: > > Time: 42 minutes 48 seconds > RX packets (aprox): 109 million overruns: 664 > > Interrupts as mesured by vmstat 1: from 6400 to 7700 (6500 average) > > > Behaviour under 2.4.10 kernel: > > Time: 17 minutes 51 seconds > RX packets (aprox): 53.6 million overruns: 657 > > Interrupts as mesured by vmstat 1: from 6400 to 7700 (6500 average) That doesnt appear to be too many interupts/sec > > > Behaviour under 2.2.19 kernel: > > Time: 1 minute 8 seconds > RX packets (aprox): 3 million overruns: 81000 > > Interrupts as mesured by vmstat 1: from 35000 to 40000 (38500 average) > Big difference from 6500 ;-> This is over 6 times more. > Note on the overruns on 2.2.19, they grow a lot when the first test is > carried, but if I continue to make tests they are much lower, in fact I even > added another spammer machine to the 2.2.19 test (dual P133) and I got this > results where you can see that the overruns are much much lower even though > the load was increased: > > Time: 1 minute 17 seconds > RX packets (aprox): 4.3 million overruns: 476 > > Interrupts as mesured by vmstat 1: from 40000 to 44000 (43500 average) > > One other thing that was seen on 2.2.19 and that did not appear on 2.4 was > that 2.2.19 said from time to time... > > Too much work in interrupt, status e401. > Can you post how many packets were dropped by the hardware? just post the output of ifconfig. > > I tried to spam 2.4.9ac18 with the same three machines I had used for the > last 2.2.19 test but the machine was almost totally frozen, after 6 hours of > test I stopped it. > I am confused. Is this a different test? You say above taht for that kernel "Time: 42 minutes 48 seconds" for "RX packets (aprox): 109 million overruns: 664" > > So... is this a problem with 2.4 kernels that I'm the only one > experimenting? > > Is there any explanation why 2.4 kernels are so bad on handling this > spamming while 2.2.19 does quite well? > And from you results above it is inconclusive that is the case. It seems to me the time is proportional to the amount of packets received. I guess i am confused a little about your results. Also what would help is to describe your packet sizes, send and receive rates etc. > If you need more data or want me to run any other tests, please tell me, but > send me expecific notes on what you want me to do. > One thing would help is to say what the packet sizes are etc. And the switch in between might cause issues; can you try one powerful machine directly connected? cheers, jamal From owner-netdev@oss.sgi.com Mon Oct 8 07:47:57 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f98Elv216846 for netdev-outgoing; Mon, 8 Oct 2001 07:47:57 -0700 Received: from shell.cyberus.ca ([216.191.240.114]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f98ElrD16843 for ; Mon, 8 Oct 2001 07:47:53 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id KAA05649; Mon, 8 Oct 2001 10:45:02 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Mon, 8 Oct 2001 10:45:02 -0400 (EDT) From: jamal To: , cc: Bernd Eckenfels Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 2107 Lines: 46 >Yes, have a look at the work of the Click Modular Router PPL from MIT, >having a Polling Router Module Implementatin which outperforms Linux >Kernel Routing by far (according to their paper :) I have read the click paper; i also just looked at the code and it seems the tulip driver they use has the same roots as us (based on Alexey's initial HFC driver) Several things to note/observe: - They use some very specialized piece of hardware (with two PCI buses). - Roberts results on a single PCI bus hardware was showing ~360Kpps routing vs clicks 435Kpps. This is not "far off" given the differences in hardware. What would be really interesting is to have the click folks post their latency results. I am curious as to what a purely polling scheme they have would achieve (as opposed to NAPI which is a mixture of interupts and polls). - Linux is already "very modular" as a router with both the traffic control framework and netfilter. I like their language specification etc; ours is a little more primitive in comparison. - Click seems to only run on a system that is designated as a router (as you seem to point out). Linux has a few other perks, but the above were to compare the two. > You can find the Link to Click somewhere on my Page: > http://www.freefire.org/tools/index.en.php3in the Operating System > section (i think) Nice web page and collection, btw. The right web page seems to be: http://www.freefire.org/tools/index.en.php3 I looked at the latest click paper on SMP. It would help if they were aware of whats happening on Linux (since it seems to be their primary OS). softnet does what they are asking for sans the scheduling (which in Linux proper is done via the IRQ scheduling). They also have a way for the admin to specify the scheduling scheme; which is nice, but i am not sure to be very valuable; I'll read the paper again to avoid hasty judgement. It would be nice to work with the click people (at least to avoid redundant work and maybe to get Linux mentioned in their paper -- they even mention ALTQ but forget Linux, which is more advanced ;->). cheers, jamal From owner-netdev@oss.sgi.com Mon Oct 8 07:56:33 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f98EuX917420 for netdev-outgoing; Mon, 8 Oct 2001 07:56:33 -0700 Received: from the-village.bc.nu (lightning.swansea.linux.org.uk [194.168.151.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f98EuUD17416 for ; Mon, 8 Oct 2001 07:56:30 -0700 Received: from alan by the-village.bc.nu with local (Exim 3.22 #1) id 15qbtV-0000hd-00; Mon, 08 Oct 2001 16:00:37 +0100 Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 To: andrea@suse.de (Andrea Arcangeli) Date: Mon, 8 Oct 2001 16:00:36 +0100 (BST) Cc: mingo@elte.hu (Ingo Molnar), hadi@cyberus.ca (jamal), linux-kernel@vger.kernel.org, kuznet@ms2.inr.ac.ru (Alexey Kuznetsov), Robert.Olsson@data.slu.se (Robert Olsson), bcrl@redhat.com (Benjamin LaHaise), netdev@oss.sgi.com, torvalds@transmeta.com (Linus Torvalds), alan@lxorguk.ukuu.org.uk (Alan Cox) In-Reply-To: <20011008023118.L726@athlon.random> from "Andrea Arcangeli" at Oct 08, 2001 02:31:18 AM X-Mailer: ELM [version 2.5 PL6] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: From: Alan Cox Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 352 Lines: 10 > Of course we agree that such a "polling router/firewall" behaviour must > not be the default but it must be enabled on demand by the admin via > sysctl or whatever else userspace API. And I don't see any problem with > that. No I don't agree. "Stop random end users crashing my machine at will" is not a magic sysctl option - its a default. Alan From owner-netdev@oss.sgi.com Mon Oct 8 08:04:31 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f98F4VR17873 for netdev-outgoing; Mon, 8 Oct 2001 08:04:31 -0700 Received: from mandrakesoft.mandrakesoft.com (mandrakesoft.mandrakesoft.com [216.71.84.35]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f98F4TD17870 for ; Mon, 8 Oct 2001 08:04:29 -0700 Received: from localhost (jgarzik@localhost) by mandrakesoft.mandrakesoft.com (8.8.5/8.8.5) with SMTP id KAA26574; Mon, 8 Oct 2001 10:03:57 -0500 Date: Mon, 8 Oct 2001 10:03:57 -0500 (CDT) From: Jeff Garzik To: Alan Cox cc: Andrea Arcangeli , Ingo Molnar , jamal , Linux-Kernel , netdev@oss.sgi.com, Linus Torvalds Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 640 Lines: 20 On Mon, 8 Oct 2001, Alan Cox wrote: > > Of course we agree that such a "polling router/firewall" behaviour must > > not be the default but it must be enabled on demand by the admin via > > sysctl or whatever else userspace API. And I don't see any problem with > > that. > > No I don't agree. "Stop random end users crashing my machine at will" is not > a magic sysctl option - its a default. I think (Ingo's?) analogy of an airbag was appropriate, if that's indeed how the code winds up functioning. Having a mechanism that prevents what would otherwise be a lockup is useful. NAPI is useful. Having both would be nice :) Jeff From owner-netdev@oss.sgi.com Mon Oct 8 08:07:49 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f98F7n918147 for netdev-outgoing; Mon, 8 Oct 2001 08:07:49 -0700 Received: from the-village.bc.nu (lightning.swansea.linux.org.uk [194.168.151.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f98F7lD18126 for ; Mon, 8 Oct 2001 08:07:47 -0700 Received: from alan by the-village.bc.nu with local (Exim 3.22 #1) id 15qc5N-0000p5-00; Mon, 08 Oct 2001 16:12:53 +0100 Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 To: jgarzik@mandrakesoft.com (Jeff Garzik) Date: Mon, 8 Oct 2001 16:12:53 +0100 (BST) Cc: alan@lxorguk.ukuu.org.uk (Alan Cox), andrea@suse.de (Andrea Arcangeli), mingo@elte.hu (Ingo Molnar), hadi@cyberus.ca (jamal), linux-kernel@vger.kernel.org (Linux-Kernel), netdev@oss.sgi.com, torvalds@transmeta.com (Linus Torvalds) In-Reply-To: from "Jeff Garzik" at Oct 08, 2001 10:03:57 AM X-Mailer: ELM [version 2.5 PL6] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: From: Alan Cox Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 563 Lines: 14 > I think (Ingo's?) analogy of an airbag was appropriate, if that's indeed > how the code winds up functioning. Very much so "Driver killed because the air bag enable is off by default and only mentioned on page 87 of the handbook in a footnote" > Having a mechanism that prevents what would otherwise be a lockup is > useful. NAPI is useful. Having both would be nice :) NAPI is important - the irq disable tactic is a last resort. If the right hardware is irq flood aware it should only ever trigger to save us from irq routing errors (eg cardbus hangs) From owner-netdev@oss.sgi.com Mon Oct 8 08:13:21 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f98FDLx18388 for netdev-outgoing; Mon, 8 Oct 2001 08:13:21 -0700 Received: from shell.cyberus.ca ([216.191.240.114]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f98FDID18385 for ; Mon, 8 Oct 2001 08:13:18 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id LAA05689; Mon, 8 Oct 2001 11:09:57 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Mon, 8 Oct 2001 11:09:57 -0400 (EDT) From: jamal To: Alan Cox cc: Jeff Garzik , Andrea Arcangeli , Ingo Molnar , Linux-Kernel , , Linus Torvalds Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 930 Lines: 32 On Mon, 8 Oct 2001, Alan Cox wrote: > NAPI is important - the irq disable tactic is a last resort. If the right > hardware is irq flood aware it should only ever trigger to save us from > irq routing errors (eg cardbus hangs) Agreed. As long as the IRQ flood protector can do proper isolation. Here's hat i see on my dell latitude laptop with a built in ethernet (not cardbus related ;->) ------------------------------- [root@jzny /root]# cat /proc/interrupts CPU0 0: 29408219 XT-PIC timer 1: 332192 XT-PIC keyboard 2: 0 XT-PIC cascade 10: 643040 XT-PIC Texas Instruments PCI1410 PC card Cardbus Controller, eth0 11: 17 XT-PIC usb-uhci 12: 2207062 XT-PIC PS/2 Mouse 14: 307504 XT-PIC ide0 NMI: 0 LOC: 0 ERR: 0 MIS: 0 ----------------------------- cheers, jamal From owner-netdev@oss.sgi.com Mon Oct 8 08:17:17 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f98FHH718547 for netdev-outgoing; Mon, 8 Oct 2001 08:17:17 -0700 Received: from the-village.bc.nu (lightning.swansea.linux.org.uk [194.168.151.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f98FHED18543 for ; Mon, 8 Oct 2001 08:17:15 -0700 Received: from alan by the-village.bc.nu with local (Exim 3.22 #1) id 15qcES-0000rh-00; Mon, 08 Oct 2001 16:22:16 +0100 Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 To: hadi@cyberus.ca (jamal) Date: Mon, 8 Oct 2001 16:22:16 +0100 (BST) Cc: alan@lxorguk.ukuu.org.uk (Alan Cox), jgarzik@mandrakesoft.com (Jeff Garzik), andrea@suse.de (Andrea Arcangeli), mingo@elte.hu (Ingo Molnar), linux-kernel@vger.kernel.org (Linux-Kernel), netdev@oss.sgi.com, torvalds@transmeta.com (Linus Torvalds) In-Reply-To: from "jamal" at Oct 08, 2001 11:09:57 AM X-Mailer: ELM [version 2.5 PL6] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: From: Alan Cox Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 631 Lines: 13 > On Mon, 8 Oct 2001, Alan Cox wrote: > > > NAPI is important - the irq disable tactic is a last resort. If the right > > hardware is irq flood aware it should only ever trigger to save us from > > irq routing errors (eg cardbus hangs) > > Agreed. As long as the IRQ flood protector can do proper isolation. > Here's hat i see on my dell latitude laptop with a built in ethernet (not > cardbus related ;->) It doesnt save you from horrible performance. NAPI is there to do that, it saves you from a dead box. You can at least rmmod the cardbus controller with protection in place (or go looking for the problem with a debugger) From owner-netdev@oss.sgi.com Mon Oct 8 08:21:35 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f98FLZu18740 for netdev-outgoing; Mon, 8 Oct 2001 08:21:35 -0700 Received: from penguin.e-mind.com (penguin.e-mind.com [195.223.140.120]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f98FLWD18733 for ; Mon, 8 Oct 2001 08:21:32 -0700 Received: from black.random ([195.223.140.107]) by penguin.e-mind.com (8.9.1a/8.9.1/Debian/GNU) with ESMTP id RAA00942; Mon, 8 Oct 2001 17:27:15 +0200 Received: from athlon.random (athlon.random [192.168.1.7]) by black.random (8.11.0/8.10.2/SuSE Linux 8.10.0-0.3) with ESMTP id f98FMWZ15680; Mon, 8 Oct 2001 17:22:32 +0200 Received: (from andrea@localhost) by athlon.random (8.11.2/8.10.2/SuSE Linux 8.10.0-0.3) id f98FJqZ07422; Mon, 8 Oct 2001 17:19:52 +0200 Date: Mon, 8 Oct 2001 17:19:52 +0200 From: Andrea Arcangeli To: Alan Cox Cc: Ingo Molnar , jamal , linux-kernel@vger.kernel.org, Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , netdev@oss.sgi.com, Linus Torvalds Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: <20011008171952.Z726@athlon.random> References: <20011008023118.L726@athlon.random> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: ; from alan@lxorguk.ukuu.org.uk on Mon, Oct 08, 2001 at 04:00:36PM +0100 X-GnuPG-Key-URL: http://e-mind.com/~andrea/aa.gnupg.asc X-PGP-Key-URL: http://e-mind.com/~andrea/aa.asc Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1298 Lines: 27 On Mon, Oct 08, 2001 at 04:00:36PM +0100, Alan Cox wrote: > > Of course we agree that such a "polling router/firewall" behaviour must > > not be the default but it must be enabled on demand by the admin via > > sysctl or whatever else userspace API. And I don't see any problem with > > that. > > No I don't agree. "Stop random end users crashing my machine at will" is not > a magic sysctl option - its a default. The "random user hanging my machine" has nothing to do with "it is ok in a router to dedicate one cpu to polling". The whole email was about "in a router is ok to poll" I'm not saying "to solve the food problem you should be forced to turn on polling". I also said that if you turn on polling you also solve the DoS, yes, but that was just a side note. My only implicit thought about the side note was that most machines sensible to the DoS are routers where people wants the max performance and where they can dedicate one cpu (also in UP) to polling. So the only argument I can make is that the amount of userbase concerned about the "current" hardirq DoS would decrease significantly if polling method would becomes available in linux. I'm certainly not saying that the "stop random user crashing my machine at will" should be a sysctl option and not the default. Andrea From owner-netdev@oss.sgi.com Mon Oct 8 08:25:45 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f98FPj918913 for netdev-outgoing; Mon, 8 Oct 2001 08:25:45 -0700 Received: from sgi.com (sgi.SGI.COM [192.48.153.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f98FPiD18910 for ; Mon, 8 Oct 2001 08:25:44 -0700 Received: from shell.cyberus.ca ([216.191.240.114]) by sgi.com (980327.SGI.8.8.8-aspam/980304.SGI-aspam: SGI does not authorize the use of its proprietary systems or networks for unsolicited or bulk email from the Internet.) via ESMTP id IAA08612 for ; Mon, 8 Oct 2001 08:25:43 -0700 (PDT) mail_from (hadi@cyberus.ca) Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id LAA05723; Mon, 8 Oct 2001 11:20:32 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Mon, 8 Oct 2001 11:20:32 -0400 (EDT) From: jamal To: Alan Cox cc: Jeff Garzik , Andrea Arcangeli , Ingo Molnar , Linux-Kernel , , Linus Torvalds Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 528 Lines: 16 On Mon, 8 Oct 2001, Alan Cox wrote: > It doesnt save you from horrible performance. NAPI is there to do that, it > saves you from a dead box. You can at least rmmod the cardbus controller > with protection in place (or go looking for the problem with a debugger) I hear you, but I think isolation is important; If i am telneted (literal example here) onto that machine (note eth0 is not cardbus based) and cardbus is causing the loops then iam screwed. [The same applies to everything that shares interupts] cheers, jamal From owner-netdev@oss.sgi.com Mon Oct 8 08:25:55 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f98FPtp18954 for netdev-outgoing; Mon, 8 Oct 2001 08:25:55 -0700 Received: from penguin.e-mind.com (penguin.e-mind.com [195.223.140.120]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f98FPrD18951 for ; Mon, 8 Oct 2001 08:25:53 -0700 Received: from black.random ([195.223.140.107]) by penguin.e-mind.com (8.9.1a/8.9.1/Debian/GNU) with ESMTP id RAA01108; Mon, 8 Oct 2001 17:32:04 +0200 Received: from athlon.random (athlon.random [192.168.1.7]) by black.random (8.11.0/8.10.2/SuSE Linux 8.10.0-0.3) with ESMTP id f98FRLZ15684; Mon, 8 Oct 2001 17:27:21 +0200 Received: (from andrea@localhost) by athlon.random (8.11.2/8.10.2/SuSE Linux 8.10.0-0.3) id f98FOoN07799; Mon, 8 Oct 2001 17:24:50 +0200 Date: Mon, 8 Oct 2001 17:24:50 +0200 From: Andrea Arcangeli To: Alan Cox Cc: Jeff Garzik , Ingo Molnar , jamal , Linux-Kernel , netdev@oss.sgi.com, Linus Torvalds Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: <20011008172450.A726@athlon.random> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: ; from alan@lxorguk.ukuu.org.uk on Mon, Oct 08, 2001 at 04:12:53PM +0100 X-GnuPG-Key-URL: http://e-mind.com/~andrea/aa.gnupg.asc X-PGP-Key-URL: http://e-mind.com/~andrea/aa.asc Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 679 Lines: 16 On Mon, Oct 08, 2001 at 04:12:53PM +0100, Alan Cox wrote: > "Driver killed because the air bag enable is off by default and only > mentioned on page 87 of the handbook in a footnote" Nobody suggested to not add "an airbag" by default. Infact the polling isn't an airbag at all, when you poll you're flying so you never need an airbag at all, only when you're on the ground you may need the airbag. Another thing I said recently is that the hardirq airbag have nothing to do with softirqs, and that's right. Patch messing the softirq logic in function of the hardirq airbag are just totally broken or at least confusing because incidentally merged together by mistake. Andrea From owner-netdev@oss.sgi.com Mon Oct 8 08:30:19 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f98FUJG19169 for netdev-outgoing; Mon, 8 Oct 2001 08:30:19 -0700 Received: from the-village.bc.nu (lightning.swansea.linux.org.uk [194.168.151.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f98FUFD19166 for ; Mon, 8 Oct 2001 08:30:16 -0700 Received: from alan by the-village.bc.nu with local (Exim 3.22 #1) id 15qcRA-0000uJ-00; Mon, 08 Oct 2001 16:35:24 +0100 Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 To: hadi@cyberus.ca (jamal) Date: Mon, 8 Oct 2001 16:35:24 +0100 (BST) Cc: alan@lxorguk.ukuu.org.uk (Alan Cox), jgarzik@mandrakesoft.com (Jeff Garzik), andrea@suse.de (Andrea Arcangeli), mingo@elte.hu (Ingo Molnar), linux-kernel@vger.kernel.org (Linux-Kernel), netdev@oss.sgi.com, torvalds@transmeta.com (Linus Torvalds) In-Reply-To: from "jamal" at Oct 08, 2001 11:20:32 AM X-Mailer: ELM [version 2.5 PL6] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: From: Alan Cox Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 876 Lines: 20 > I hear you, but I think isolation is important; > If i am telneted (literal example here) onto that machine (note eth0 is > not cardbus based) and cardbus is causing the loops then iam screwed. > [The same applies to everything that shares interupts] Worst case it sucks, but it isnt dead. Once you disable the IRQ and kick over to polling the cardbus and the ethernet both still get regular service. Ok so your pps rate and your latency are unpleasant, but you are not dead. For a shared IRQ we know we can safely switch to a 200Hz poll of shared irq lines marked 'stuck'. The problem ones are non shared ISA devices going mad - there you have to be careful not to fake more irqs than real ones are delivered since some ISA device drivers "know" the IRQ is for them. Even at 200Hz polling a typical cardbus card with say 32 ring buffer slots can process 6000pps. Alan From owner-netdev@oss.sgi.com Mon Oct 8 08:30:56 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f98FUut19217 for netdev-outgoing; Mon, 8 Oct 2001 08:30:56 -0700 Received: from the-village.bc.nu (lightning.swansea.linux.org.uk [194.168.151.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f98FUrD19214 for ; Mon, 8 Oct 2001 08:30:54 -0700 Received: from alan by the-village.bc.nu with local (Exim 3.22 #1) id 15qcRa-0000uS-00; Mon, 08 Oct 2001 16:35:50 +0100 Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 To: andrea@suse.de (Andrea Arcangeli) Date: Mon, 8 Oct 2001 16:35:50 +0100 (BST) Cc: alan@lxorguk.ukuu.org.uk (Alan Cox), jgarzik@mandrakesoft.com (Jeff Garzik), mingo@elte.hu (Ingo Molnar), hadi@cyberus.ca (jamal), linux-kernel@vger.kernel.org (Linux-Kernel), netdev@oss.sgi.com, torvalds@transmeta.com (Linus Torvalds) In-Reply-To: <20011008172450.A726@athlon.random> from "Andrea Arcangeli" at Oct 08, 2001 05:24:50 PM X-Mailer: ELM [version 2.5 PL6] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: From: Alan Cox Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 286 Lines: 6 > Another thing I said recently is that the hardirq airbag have nothing to > do with softirqs, and that's right. Patch messing the softirq logic in > function of the hardirq airbag are just totally broken or at least > confusing because incidentally merged together by mistake. Agreed From owner-netdev@oss.sgi.com Mon Oct 8 09:00:20 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f98G0KQ20040 for netdev-outgoing; Mon, 8 Oct 2001 09:00:20 -0700 Received: from shell.cyberus.ca ([216.191.240.114]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f98G0ED20037 for ; Mon, 8 Oct 2001 09:00:14 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id LAA05759; Mon, 8 Oct 2001 11:57:11 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Mon, 8 Oct 2001 11:57:11 -0400 (EDT) From: jamal To: Alan Cox cc: Jeff Garzik , Andrea Arcangeli , Ingo Molnar , Linux-Kernel , , Linus Torvalds Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 990 Lines: 27 On Mon, 8 Oct 2001, Alan Cox wrote: > > I hear you, but I think isolation is important; > > If i am telneted (literal example here) onto that machine (note eth0 is > > not cardbus based) and cardbus is causing the loops then iam screwed. > > [The same applies to everything that shares interupts] > > Worst case it sucks, but it isnt dead. > > Once you disable the IRQ and kick over to polling the cardbus and the > ethernet both still get regular service. Ok so your pps rate and your > latency are unpleasant, but you are not dead. > Agreed if you add the polling cardbus bit. Note polling cardbus would require more changes than the above. My concern was more of the following: This is a temporary solution. A quickie if you may. The proper solution is to have the isolation part. If we push this in, doesnt it result in procastination of "we'll do it later" Why not do it properly since this was never a show stopper to begin with? [The showstopper was networking] cheers, jamal From owner-netdev@oss.sgi.com Mon Oct 8 09:05:59 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f98G5xK20222 for netdev-outgoing; Mon, 8 Oct 2001 09:05:59 -0700 Received: from the-village.bc.nu (lightning.swansea.linux.org.uk [194.168.151.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f98G5qD20218 for ; Mon, 8 Oct 2001 09:05:53 -0700 Received: from alan by the-village.bc.nu with local (Exim 3.22 #1) id 15qczg-00011N-00; Mon, 08 Oct 2001 17:11:04 +0100 Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 To: hadi@cyberus.ca (jamal) Date: Mon, 8 Oct 2001 17:11:03 +0100 (BST) Cc: alan@lxorguk.ukuu.org.uk (Alan Cox), jgarzik@mandrakesoft.com (Jeff Garzik), andrea@suse.de (Andrea Arcangeli), mingo@elte.hu (Ingo Molnar), linux-kernel@vger.kernel.org (Linux-Kernel), netdev@oss.sgi.com, torvalds@transmeta.com (Linus Torvalds) In-Reply-To: from "jamal" at Oct 08, 2001 11:57:11 AM X-Mailer: ELM [version 2.5 PL6] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: From: Alan Cox Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 508 Lines: 22 > Agreed if you add the polling cardbus bit. > Note polling cardbus would require more changes than the above. I don't think it does. There are two pieces to the problem a) Not dying horribly b) Handling it elegantly b) is driver specific (NAPI etc) and I think well understood to the point its being used already for performance reasons a) is as simple as if(stuck_in_irq(foo) && irq_shared(foo)) { disable_real_irq(foo); timer_fake_irq_foo(); } We know spoofing a shared irq is safe. Alan From owner-netdev@oss.sgi.com Mon Oct 8 09:14:14 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f98GEEV20425 for netdev-outgoing; Mon, 8 Oct 2001 09:14:14 -0700 Received: from shell.cyberus.ca ([216.191.240.114]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f98GEBD20422 for ; Mon, 8 Oct 2001 09:14:12 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id MAA05770; Mon, 8 Oct 2001 12:11:12 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Mon, 8 Oct 2001 12:11:12 -0400 (EDT) From: jamal To: Alan Cox cc: Jeff Garzik , Andrea Arcangeli , Ingo Molnar , Linux-Kernel , , Linus Torvalds Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 655 Lines: 23 On Mon, 8 Oct 2001, Alan Cox wrote: > > Agreed if you add the polling cardbus bit. > > Note polling cardbus would require more changes than the above. > > I don't think it does. I was repsonding to your earlier comment that: > Once you disable the IRQ and kick over to polling the cardbus and the > ethernet both still get regular service. Ok so your pps rate and your > latency are unpleasant, but you are not dead. basically pointing that we'll need more work to be done to get Ingos patch to poll the cardbus and eth0 in the example i gave. those will have to be per driver. Did i miss something? Agree on your other points there cheers, jamal From owner-netdev@oss.sgi.com Mon Oct 8 10:39:59 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f98Hdxk22169 for netdev-outgoing; Mon, 8 Oct 2001 10:39:59 -0700 Received: from robur.slu.se (robur.slu.se [130.238.98.12]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f98HduD22166 for ; Mon, 8 Oct 2001 10:39:56 -0700 Received: (from robert@localhost) by robur.slu.se (8.8.7/8.8.7) id TAA18835; Mon, 8 Oct 2001 19:42:08 +0200 From: Robert Olsson MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15297.58736.505995.577308@robur.slu.se> Date: Mon, 8 Oct 2001 19:42:08 +0200 To: jamal Cc: , Andreas Dilger , , , , , , , Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: References: <200110051917.XAA23007@ms2.inr.ac.ru> X-Mailer: VM 6.92 under Emacs 19.34.1 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 484 Lines: 19 jamal writes: > > This was Robert actually; conclusion was Interupts are very expensive. If > we can get rid of as many of them as possible, we are getting a side > benefit. I cant find the old data, but Robert has some data over here: > http://robur.slu.se/Linux/net-development/experiments/010301 Jamal! I think you ment: http://robur.slu.se/Linux/net-development/experiments/010313 MB with PIC irq controller IO-APIC boards does a lot better. Cheers. --ro From owner-netdev@oss.sgi.com Mon Oct 8 10:42:36 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f98HgaK22252 for netdev-outgoing; Mon, 8 Oct 2001 10:42:36 -0700 Received: from shell.cyberus.ca ([216.191.240.114]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f98HgYD22249 for ; Mon, 8 Oct 2001 10:42:34 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id NAA05855; Mon, 8 Oct 2001 13:39:32 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Mon, 8 Oct 2001 13:39:31 -0400 (EDT) From: jamal To: Robert Olsson cc: , Andreas Dilger , , , , , , Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <15297.58736.505995.577308@robur.slu.se> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 267 Lines: 15 On Mon, 8 Oct 2001, Robert Olsson wrote: > I think you ment: > http://robur.slu.se/Linux/net-development/experiments/010313 > > MB with PIC irq controller IO-APIC boards does a lot better. > Ooops, Yes, i am sorry (12 days difference only ;->) cheers, jamal From owner-netdev@oss.sgi.com Mon Oct 8 17:36:36 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f990aae31330 for netdev-outgoing; Mon, 8 Oct 2001 17:36:36 -0700 Received: from laird.sea.internap.com (laird.sea.internap.com [206.253.215.165]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f990aYD31327 for ; Mon, 8 Oct 2001 17:36:34 -0700 Received: from localhost ([127.0.0.1]) by laird.sea.internap.com with esmtp (v3.30.1) id 15qksf-00069t-00; Mon, 08 Oct 2001 17:36:21 -0700 Date: Mon, 8 Oct 2001 17:36:21 -0700 (PDT) From: Scott Laird X-X-Sender: To: jamal cc: , , Bernd Eckenfels Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1066 Lines: 28 On Mon, 8 Oct 2001, jamal wrote: > > Several things to note/observe: > - They use some very specialized piece of hardware (with two PCI buses). Huh? It was just an L440GX, which was probably the single most common PC server board for a while in 1999-2000. Most of VA Linux's systems used them. I wouldn't call them "very specialized." > - Roberts results on a single PCI bus hardware was showing ~360Kpps > routing vs clicks 435Kpps. This is not "far off" given the differences in > hardware. What would be really interesting is to have the click folks > post their latency results. I am curious as to what a purely polling > scheme they have would achieve (as opposed to NAPI which is a mixture of > interupts and polls). Their 'TOCS00' paper lists a 29us one-way latency on page 22. Click looks interesting, much more so then most academic network projects, but I'm still not sure if it'd really be useful in most "real" environments. It looks too flexible for most people to manage. It'd be an interesting addition to my test lab, though :-). Scott From owner-netdev@oss.sgi.com Mon Oct 8 20:19:59 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f993Jxm02159 for netdev-outgoing; Mon, 8 Oct 2001 20:19:59 -0700 Received: from shell.cyberus.ca ([216.191.240.114]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f993JqD02156 for ; Mon, 8 Oct 2001 20:19:53 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id XAA06767; Mon, 8 Oct 2001 23:17:00 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Mon, 8 Oct 2001 23:17:00 -0400 (EDT) From: jamal To: Scott Laird cc: , , Bernd Eckenfels Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 2542 Lines: 62 On Mon, 8 Oct 2001, Scott Laird wrote: > > > On Mon, 8 Oct 2001, jamal wrote: > > > > Several things to note/observe: > > - They use some very specialized piece of hardware (with two PCI buses). > > Huh? It was just an L440GX, which was probably the single most common PC > server board for a while in 1999-2000. Most of VA Linux's systems used > them. I wouldn't call them "very specialized." > Ok, sorry you are right, not very high end, but not exactly cheap even at the time to have a motherboard with two PCI busses (i for one would have been delighted to have had access to one even today); Nevertheless, impressive numbers still. I could do achieve MLFR of ~200Kpps on an elcheapo PII with 4port znyx cards on an ASUS that has a single PCI bus; and from what Donald Becker was saying we could probably do better with 4 interface cards rather than a single 4-port card due to bus mastership issues. I suppose thats why Robert can pull more packets on only two gige NICs on a single bus. He's more than likely hitting PCI bottlenecks at this point. A second PCI bus with a second set of cards should help (dis)prove this theory. > > - Roberts results on a single PCI bus hardware was showing ~360Kpps > > routing vs clicks 435Kpps. This is not "far off" given the differences in > > hardware. What would be really interesting is to have the click folks > > post their latency results. I am curious as to what a purely polling > > scheme they have would achieve (as opposed to NAPI which is a mixture of > > interupts and polls). > > Their 'TOCS00' paper lists a 29us one-way latency on page 22. > Thats a very good number. I wonder what it means though and at what rates those numbers are extracted. For example some of the tests i run on the znyx card with only two ports generating traffic -- you can observe a rough latency of around 33us upto about the MLFFR and then the latency jumps sharply to hunderds of us. Infact at 147Kpps input, you observe anywhere in the range of upto 800us although we are clearly flat at the MLFFR throughput on the output. These numbers might also be affected by the latency measurement scheme used, > Click looks interesting, much more so then most academic network projects, > but I'm still not sure if it'd really be useful in most "real" agreed, although i think we need to have more research of the type that click is bringing ... > environments. It looks too flexible for most people to manage. It'd be > an interesting addition to my test lab, though :-). indeed. cheers, jamal From owner-netdev@oss.sgi.com Mon Oct 8 21:04:28 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9944SC02854 for netdev-outgoing; Mon, 8 Oct 2001 21:04:28 -0700 Received: from almesberger.net (IDENT:root@lsb-catv-1-p021.vtxnet.ch [212.147.5.21]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9944PD02851 for ; Mon, 8 Oct 2001 21:04:25 -0700 Received: (from almesber@localhost) by almesberger.net (8.9.3/8.9.3) id GAA31776; Tue, 9 Oct 2001 06:04:14 +0200 Date: Tue, 9 Oct 2001 06:04:13 +0200 From: Werner Almesberger To: jamal Cc: linux-kernel@vger.kernel.org, netdev@oss.sgi.com Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: <20011009060413.A29436@almesberger.net> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: ; from hadi@cyberus.ca on Mon, Oct 08, 2001 at 10:45:02AM -0400 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 614 Lines: 16 jamal wrote: > - Linux is already "very modular" as a router with both the traffic > control framework and netfilter. I like their language specification etc; > ours is a little more primitive in comparison. I guess you're talking about iproute2/tc ;-) Things are better with tcng: http://tcng.sourceforge.net/ Click covers more areas than just Traffic Control, though. - Werner -- _________________________________________________________________________ / Werner Almesberger, Lausanne, CH wa@almesberger.net / /_http://icawww.epfl.ch/almesberger/_____________________________________/ From owner-netdev@oss.sgi.com Mon Oct 8 21:49:15 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f994nF203619 for netdev-outgoing; Mon, 8 Oct 2001 21:49:15 -0700 Received: from vasquez.zip.com.au (vasquez.zip.com.au [203.12.97.41]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f994nBD03613 for ; Mon, 8 Oct 2001 21:49:11 -0700 Received: from zip.com.au (root@zipperii.zip.com.au [61.8.0.87]) by vasquez.zip.com.au (8.9.3/8.9.3/Debian 8.9.3-21) with ESMTP id OAA10138; Tue, 9 Oct 2001 14:48:58 +1000 X-Authentication-Warning: vasquez.zip.com.au: Host root@zipperii.zip.com.au [61.8.0.87] claimed to be zip.com.au Message-ID: <3BC281A9.16D038E9@zip.com.au> Date: Mon, 08 Oct 2001 21:48:41 -0700 From: Andrew Morton X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.4.10-ac7 i686) X-Accept-Language: en MIME-Version: 1.0 To: jamal CC: Martin Josefsson , netdev@oss.sgi.com, manty@manty.net Subject: Re: 2.2 performance on high network load much much better than 2.4(fwd) References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1122 Lines: 32 jamal wrote: > > ... > > > > > Behaviour under 2.2.19 kernel: > > > > Time: 1 minute 8 seconds > > RX packets (aprox): 3 million overruns: 81000 > > > > Interrupts as mesured by vmstat 1: from 35000 to 40000 (38500 average) > > > > Big difference from 6500 ;-> This is over 6 times more. > And the userspace throughput was an order of magnitude better. Fascinating. One could envisage one scenario where a less efficient interrupt routine results in more throughput: suppose the ISR takes 9 usecs and frames are arriving at 10 usec intervals. That's one interrupt per frame and userspace starvation. If you slow the ISR down a bit then it will end up handling two packets per interrupt and, assuming that the entry and exit is the majority of the cost, you'll end up actually yielding more time for userspace processing. But the most this can possibly account for is a factor of two, not a factor of ten. The 2.2 and 2.4 drivers aren't significantly different. 2.4 has max_interrupt_work of 32, whereas it's only 20 in 2.2. But that doesn't explain much. Weird stuff. Martin, do you have a URL for udpspam? From owner-netdev@oss.sgi.com Tue Oct 9 01:19:18 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f998JIc08010 for netdev-outgoing; Tue, 9 Oct 2001 01:19:18 -0700 Received: from robur.slu.se (robur.slu.se [130.238.98.12]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f998JCD08007 for ; Tue, 9 Oct 2001 01:19:12 -0700 Received: (from robert@localhost) by robur.slu.se (8.8.7/8.8.7) id KAA31597; Tue, 9 Oct 2001 10:21:24 +0200 From: Robert Olsson MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15298.45956.35137.638849@robur.slu.se> Date: Tue, 9 Oct 2001 10:21:24 +0200 To: Andrew Morton Cc: jamal , Martin Josefsson , netdev@oss.sgi.com, manty@manty.net Subject: Re: 2.2 performance on high network load much much better than 2.4(fwd) In-Reply-To: <3BC281A9.16D038E9@zip.com.au> References: <3BC281A9.16D038E9@zip.com.au> X-Mailer: VM 6.92 under Emacs 19.34.1 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 2065 Lines: 52 Andrew Morton writes: > > > Behaviour under 2.2.19 kernel: > > > > > > Time: 1 minute 8 seconds > > > RX packets (aprox): 3 million overruns: 81000 > > > > > > Interrupts as mesured by vmstat 1: from 35000 to 40000 (38500 average) > > > > > Big difference from 6500 ;-> This is over 6 times more. I did't follow comparision. > And the userspace throughput was an order of magnitude better. Fascinating. > > One could envisage one scenario where a less efficient interrupt > routine results in more throughput: suppose the ISR takes 9 usecs > and frames are arriving at 10 usec intervals. That's one interrupt per > frame and userspace starvation. If you slow the ISR down a bit then > it will end up handling two packets per interrupt and, assuming that > the entry and exit is the majority of the cost, you'll end up actually > yielding more time for userspace processing. But the most this can > possibly account for is a factor of two, not a factor of ten. Yes kind of mitigation but hard to tune. :-) And with the NAPI you can get more RX-irqs compared to an Interrupt Mitigation scheme. This at packet rates before the switch consecutive poll occurs. So some people may be very suprised why RX interrupts did not go away with "polling". The worst case behaviour is very good. But is up to driver choose done/not_done strategy. Also it possible to have RX hardware IM on with NAPI. But it would be nice if reduce complexity and have a very clean interface. My experiences with Interrupt Mitigation isn't too good especially with GIGE. At start I was really blended by amazing few interrupts and pretty good performance. After some months of tuning to improve performance at all packet sizes. I more or less give this up its too fragile at least for me. I prefer more robust and simpler scheme even if it would cost some performance. > Weird stuff. Martin, do you have a URL for udpspam? As an alternative: The packet generator I use is now in Alexeys iputils package. pg3.c. Cheers. --ro From owner-netdev@oss.sgi.com Tue Oct 9 01:52:06 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f998q6B08530 for netdev-outgoing; Tue, 9 Oct 2001 01:52:06 -0700 Received: from manty.net ([213.96.224.204]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f998pxD08526 for ; Tue, 9 Oct 2001 01:51:59 -0700 Received: from man.beta.es (man.beta.es [192.168.1.1]) by manty.net (Postfix) with ESMTP id 7B00315EBE; Tue, 9 Oct 2001 10:51:52 +0200 (CEST) Received: by man.beta.es (Postfix, from userid 1000) id F2A95B921; Tue, 9 Oct 2001 10:51:50 +0200 (CEST) Date: Tue, 9 Oct 2001 10:51:50 +0200 From: Santiago Garcia Mantinan To: jamal Cc: Martin Josefsson , netdev@oss.sgi.com Subject: Re: 2.2 performance on high network load much much better than 2.4 (fwd) Message-ID: <20011009105150.A953@man.beta.es> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.3.22i Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 3002 Lines: 67 > Thanks for doing this ;-> I wish people would post networking related > issues to netdev (or cc netdev at least); maybe it should be on the FAQ Well, sorry for not doing it, you are totally right, I should have posted on netdev or at least cced. And sorry also for taking me so long to answer this. > Can you post how many packets were dropped by the hardware? just post the > output of ifconfig. As I said at the beginning of the message... > > time and the received network packets and overruns, during this time, in > > all cases counter for errors, dropped or frame showed 0. The test takes Wich means the on the output of ifconfig all counters for packages where 0 except the ones that I have noted on my comments Received packages and overruns (leaving aside the transmited packages that is), but If you want the full output I can run tests again. > > I tried to spam 2.4.9ac18 with the same three machines I had used for the > > last 2.2.19 test but the machine was almost totally frozen, after 6 hours of > > test I stopped it. > I am confused. Is this a different test? > You say above taht for that kernel "Time: 42 minutes 48 seconds" for > "RX packets (aprox): 109 million overruns: 664" Well, yes, I added one more machine sendding packages for this test, so the load was even higher on the system and the bunzip2 did get even less cpu time, the console started to give very slow response because of the high load that this extra machine put on it. > And from you results above it is inconclusive that is the case. > It seems to me the time is proportional to the amount of packets received. Yes, time is proportional to the amount of packages as the packages/second was constant, I mean that I had all that time the two (or the three in that special test) machines running this program spamming at full speed, using the hole cpu to spam. > I guess i am confused a little about your results. > Also what would help is to describe your packet sizes, send and > receive rates etc. The packages are 4 bytes udp packages they are sent at the full ratio at which a P200MMX running 2.2.19 and a P166 running 2.4.10 can, I haven't calculated any ratios, but If you want I can try to, one could guess that from the number of interrupts that 2.2.19 was showing, I suppose. If we assume that then it would be 38500 packages per second, but If I calculate that from the time of the test being run and the number of received packages shown by ifconfig I get 42500 wich seems a more exact number to me. > One thing would help is to say what the packet sizes are etc. And the I'm attaching the code of udpspam.c from Simon Kirby as he has said that the license was GPL without any warranty, he also said that he would publish it on his web but I haven't been able to find it. > switch in between might cause issues; can you try one powerful machine > directly connected? I tried that also, but there was no noticeable difference. Regards... -- Manty/BestiaTester -> http://manty.net From owner-netdev@oss.sgi.com Tue Oct 9 02:11:57 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f999BvZ09112 for netdev-outgoing; Tue, 9 Oct 2001 02:11:57 -0700 Received: from manty.net ([213.96.224.204]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f999BiD09107 for ; Tue, 9 Oct 2001 02:11:44 -0700 Received: from man.beta.es (man.beta.es [192.168.1.1]) by manty.net (Postfix) with ESMTP id 478F715EBE; Tue, 9 Oct 2001 11:11:37 +0200 (CEST) Received: by man.beta.es (Postfix, from userid 1000) id E6AF1B921; Tue, 9 Oct 2001 11:11:36 +0200 (CEST) Date: Tue, 9 Oct 2001 11:11:36 +0200 From: Santiago Garcia Mantinan To: Andrew Morton Cc: jamal , Martin Josefsson , netdev@oss.sgi.com Subject: Re: 2.2 performance on high network load much much better than 2.4(fwd) Message-ID: <20011009111136.B953@man.beta.es> References: <3BC281A9.16D038E9@zip.com.au> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="wRRV7LY7NUeQGEoC" Content-Disposition: inline In-Reply-To: <3BC281A9.16D038E9@zip.com.au> User-Agent: Mutt/1.3.22i Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 3719 Lines: 151 --wRRV7LY7NUeQGEoC Content-Type: text/plain; charset=us-ascii Content-Disposition: inline > Weird stuff. Martin, do you have a URL for udpspam? OOps, sorry, I forgot to attach it to the previous message, I'm attaching it now. Regards... -- Manty/BestiaTester -> http://manty.net --wRRV7LY7NUeQGEoC Content-Type: text/x-csrc; charset=us-ascii Content-Disposition: attachment; filename="udpspam.c" /* udpspam.c * Simon Kirby , 2001/09/28 * * Based on code from 1997 originally written to test outgoing link * speed by sending UDP packets of various sizes to a remote closed * port and timing reply latencies. :) * * With a small packet size, this code is will easy choke most OSes, * and even made an OpenBSD box kernel trap (eep). It takes out * most routers and cheap switches as well, but decent hardware is * fine with it. Saturates the CPU, but can usually transmit around * 30-60 Mbit of tiny packets on a Linux 2.4 box with a recent CPU * and a PCI NIC. * * DO NOT RUN REMOTELY unless with a command to stop the process * shortly after if you are unable to abort the process with ^C. * For example: udpspam somehost & sleep 5; killall -9 udpspam * * Compile with: * * gcc -o udpspam udpspam.c -O2 -Wall -funroll-loops */ #include #include #include #include #include #include #include #include #include #include #include #include /* Typedefs */ typedef unsigned char byte; typedef unsigned short word; typedef unsigned long dword; /* Globals */ #define PAYLOAD_SIZE 4 static char payload[PAYLOAD_SIZE]; /* Socket functions */ static int resolve_address(struct sockaddr_in *d,char *s,dword port){ struct sockaddr_in *address; struct hostent *host; address = (struct sockaddr_in *)d; address->sin_family = AF_INET; address->sin_port = htons(port); if (inet_aton(s,&address->sin_addr) == 0){ /* String specified was not an IP */ host = gethostbyname(s); if (host){ memcpy(host->h_addr,(char *)&address->sin_addr,host->h_length); return 1; } return 0; } return 1; } static int get_udp_socket(){ struct sockaddr_in udp; int fd,option; fd = socket(AF_INET,SOCK_DGRAM,0); if (fd == -1){ fprintf(stderr,"Unable to create datagram socket: %s\n", strerror(errno)); return -1; } option = 0; if (setsockopt(fd,SOL_SOCKET,SO_REUSEADDR,(void *)&option,sizeof(option))){ fprintf(stderr,"Unable to setsockopt() on udp listening socket: %s\n", strerror(errno)); return -1; } option = 0; setsockopt(fd,SOL_SOCKET,SO_BROADCAST,(void *)&option,sizeof(option)); udp.sin_addr.s_addr = htonl(INADDR_ANY); udp.sin_port = htons(0); udp.sin_family = AF_INET; if (bind(fd,(struct sockaddr *)&udp,sizeof(udp))){ fprintf(stderr,"Unable to bind() udp socket: %s\n", strerror(errno)); return -1; } if (fcntl(fd,F_SETFL,O_NONBLOCK)){ fprintf(stderr,"Warning: Unable to set non-blocking udp socket: %s", strerror(errno)); } return fd; } static int socket_connect_address(int fd,struct sockaddr_in *d){ if (connect(fd,(struct sockaddr *)d,sizeof(struct sockaddr)) == -1){ fprintf(stderr,"Unable to connect() udp socket: %s\n",strerror(errno)); return 0; } return 1; } int main(int argc,char *argv[]){ int fd; struct sockaddr_in remote; resolve_address(&remote,argv[1],1); remote.sin_port = htons(12345); fd = get_udp_socket(); if (fd == -1) exit(-1); socket_connect_address(fd,&remote); memset(payload,0,PAYLOAD_SIZE); while (1) sendto(fd,payload,PAYLOAD_SIZE,0,(struct sockaddr *)&remote,sizeof(remote)); exit(0); } --wRRV7LY7NUeQGEoC-- From owner-netdev@oss.sgi.com Tue Oct 9 04:28:48 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f99BSmW12747 for netdev-outgoing; Tue, 9 Oct 2001 04:28:48 -0700 Received: from shell.cyberus.ca ([216.191.240.114]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f99BSdD12743 for ; Tue, 9 Oct 2001 04:28:40 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id HAA07549; Tue, 9 Oct 2001 07:25:43 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Tue, 9 Oct 2001 07:25:42 -0400 (EDT) From: jamal To: Santiago Garcia Mantinan cc: Martin Josefsson , Subject: Re: 2.2 performance on high network load much much better than 2.4 (fwd) In-Reply-To: <20011009105150.A953@man.beta.es> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 3625 Lines: 93 On Tue, 9 Oct 2001, Santiago Garcia Mantinan wrote: > > Can you post how many packets were dropped by the hardware? just post the > > output of ifconfig. > > As I said at the beginning of the message... Just include it in your next post even if it is diffs in counters. > > > > I tried to spam 2.4.9ac18 with the same three machines I had used for the > > > last 2.2.19 test but the machine was almost totally frozen, after 6 hours of > > > test I stopped it. > > > I am confused. Is this a different test? > > You say above taht for that kernel "Time: 42 minutes 48 seconds" for > > "RX packets (aprox): 109 million overruns: 664" > > Well, yes, I added one more machine sendding packages for this test, so the > load was even higher on the system and the bunzip2 did get even less cpu > time, the console started to give very slow response because of the high > load that this extra machine put on it. > This is why it's confusing; run the same tests for both 2.2 and 2.4 Also the 2.4 kernels before ksoftirq was introduced; try 2.4.5 vs 2.4.10 (although iam sure those results will be inconclusive since there may be other issues such as VM etc that might be affecting you). > > And from you results above it is inconclusive that is the case. > > It seems to me the time is proportional to the amount of packets received. > > Yes, time is proportional to the amount of packages as the packages/second > was constant, I mean that I had all that time the two (or the three in that > special test) machines running this program spamming at full speed, using > the hole cpu to spam. > Lets be consistent: blast specific amount of traffic at target, run time bzip2 on target and collect stats when complete. > > I guess i am confused a little about your results. > > Also what would help is to describe your packet sizes, send and > > receive rates etc. > > The packages are 4 bytes udp packages they are sent at the full ratio at > which a P200MMX running 2.2.19 and a P166 running 2.4.10 can, I haven't > calculated any ratios, but If you want I can try to, one could guess that > from the number of interrupts that 2.2.19 was showing, I suppose. If we > assume that then it would be 38500 packages per second, but If I calculate > that from the time of the test being run and the number of received packages > shown by ifconfig I get 42500 wich seems a more exact number to me. > I think the number of interupts shown are useful, but too "white box". Lets go "black box" for now. Some of the useful data to collect will be: - ifconfig stats - /proc/net/softnet_stats in 2.4 - user space time allocated to the bzip2 activity - some vm stats on memory: Can someone point to good stats that can be retrieved from /proc? - get the output of /proc/interupts before and after run and calculate the difference during each run. > > One thing would help is to say what the packet sizes are etc. And the > > I'm attaching the code of udpspam.c from Simon Kirby as he has said that the > license was GPL without any warranty, he also said that he would publish it > on his web but I haven't been able to find it. > This is fine; you are probably better off just using pg3 at: ftp://ftp.crc.ca/pub/mirrors/by-site/amber.inr.ac.ru/IProute/iputils-ss011002.tar.gz package p3.c BTW, Robert, this variant seems unidirectional and doesnt have latency measurements > > switch in between might cause issues; can you try one powerful machine > > directly connected? > > I tried that also, but there was no noticeable difference. It is a variable in the equation. The less variables, the easier the computation. cheers, jamal From owner-netdev@oss.sgi.com Tue Oct 9 04:53:45 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f99BrjG13351 for netdev-outgoing; Tue, 9 Oct 2001 04:53:45 -0700 Received: from shell.cyberus.ca ([216.191.240.114]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f99BrgD13347 for ; Tue, 9 Oct 2001 04:53:42 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id HAA07576; Tue, 9 Oct 2001 07:50:46 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Tue, 9 Oct 2001 07:50:46 -0400 (EDT) From: jamal To: Martin Josefsson cc: Santiago Garcia Mantinan , Subject: Re: 2.2 performance on high network load much much better than 2.4 (fwd) In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 821 Lines: 26 On Tue, 9 Oct 2001, Martin Josefsson wrote: > The recieving end was a 2 x pIII 600 server with a D-Link DFE570-TX NIC > (quad tulip) running 2.4.8-ac12 + tulip-ss010402-polling driver. > > The sending machine was my workstation here, a pIII 700 with eepro100 NIC > running 2.4.9-ac5. This machine is attached to eth1 on the server via a > crossover cable. > > The reciever has a few iptables modules loaded, ip_conntrack was one of > them. (the sender doesn't have any iptables modules loaded) Its been proven that contrack does slow down things .. [Good test results deleted] Could you repeat the tests with NAPI? The interesting bit is when you start sending on the other ethernet ports at really high rates output of cat /proc/interupts and /proc/net/softnet_stat also /proc/net/drivers/* output cheers, jamal From owner-netdev@oss.sgi.com Tue Oct 9 05:06:24 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f99C6O013850 for netdev-outgoing; Tue, 9 Oct 2001 05:06:24 -0700 Received: from sgi.com (sgi.SGI.COM [192.48.153.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f99C6ID13838 for ; Tue, 9 Oct 2001 05:06:18 -0700 Received: from tux.rsn.bth.se (tux.rsn.bth.se [194.47.143.135]) by sgi.com (980327.SGI.8.8.8-aspam/980304.SGI-aspam: SGI does not authorize the use of its proprietary systems or networks for unsolicited or bulk email from the Internet.) via ESMTP id EAA07640 for ; Tue, 9 Oct 2001 04:45:46 -0700 (PDT) mail_from (gandalf@wlug.westbo.se) Received: from localhost (gandalf@localhost [127.0.0.1]) by localhost (8.12.0.Beta19/8.12.0.Beta19/Debian 8.12.0.Beta19) with ESMTP id f99BQM9X006356; Tue, 9 Oct 2001 13:26:23 +0200 Date: Tue, 9 Oct 2001 13:26:22 +0200 (CEST) From: Martin Josefsson X-Sender: gandalf@tux.rsn.bth.se To: Santiago Garcia Mantinan cc: jamal , netdev@oss.sgi.com Subject: Re: 2.2 performance on high network load much much better than 2.4 (fwd) In-Reply-To: <20011009105150.A953@man.beta.es> Message-ID: X-message-flag: Get yourself a real mail client! http://www.washington.edu/pine/ MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 3394 Lines: 75 On Tue, 9 Oct 2001, Santiago Garcia Mantinan wrote: [snip] > > I guess i am confused a little about your results. > > Also what would help is to describe your packet sizes, send and > > receive rates etc. > > The packages are 4 bytes udp packages they are sent at the full ratio at > which a P200MMX running 2.2.19 and a P166 running 2.4.10 can, I haven't > calculated any ratios, but If you want I can try to, one could guess that > from the number of interrupts that 2.2.19 was showing, I suppose. If we > assume that then it would be 38500 packages per second, but If I calculate > that from the time of the test being run and the number of received packages > shown by ifconfig I get 42500 wich seems a more exact number to me. I ran this test here. The recieving end was a 2 x pIII 600 server with a D-Link DFE570-TX NIC (quad tulip) running 2.4.8-ac12 + tulip-ss010402-polling driver. The sending machine was my workstation here, a pIII 700 with eepro100 NIC running 2.4.9-ac5. This machine is attached to eth1 on the server via a crossover cable. The reciever has a few iptables modules loaded, ip_conntrack was one of them. (the sender doesn't have any iptables modules loaded) The thing that was limiting the packets per second recieved by the server was the rate my workstation could send them out at. When I left my workstation alone (stopped typing on the keyboard, didn't move the mouse and stopped the mp3's) the server recieved ~100k pps. This was no real problem for the server, it didn't drop any packets or get any overruns, it just happily chewed along. This was an SMP machine but I tested to bind the recieving interface to CPU0 by using smp_affinity and that didn't make any diffrence in the performance of the machine. According to vmstat one of the CPU's was operating at about 90-100% load (giving about 50-55% idle in the machine). As I couldn't get it to start dropping packets with this load of 100k pps I increased the load a little. I attached two more machines, on eth0 and eth3. But I didn't run udpspam on them, they just ran 'nc serverip port < /dev/zero' and the server ran two 'nc -l -p port > /dev/null' This test also involves some pipe activity so it's not a good networktest. But both the new senders managed to send at rate between 95 and 98Mbit to the server, this is in fullsize 1500byte packets. When the server started recieving these two extra streams it did get a little busy and dropping packets on eth1. (I had resetted smp_affinity to ffffffff before starting this test.) Then I put eth1 (the one with the high pps) on CPU0 and eth[03] on CPU1. then everybody was happy. According to vmstat there was about 30% cpu idle in total in the server. This last test probably doesn't help at all but the point is that my SMP pIII 600 had no real problems recieving 100k pps even when binding the NIC to one cpu. And it can probably do even better when it's routing the packets instead of terminating the stream (my other tests have showed this to be true). The tulip-ss010402-polling is as the name says a polling driver. a hybrid that runs with interrupts at low packetrates and switches to polling when there's higher loads. I probably don't have to say this to Jamal or Robert as they are the ones who implemented this stuff :) /Martin Never argue with an idiot. They drag you down to their level, then beat you with experience. From owner-netdev@oss.sgi.com Tue Oct 9 05:07:13 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f99C7Db13872 for netdev-outgoing; Tue, 9 Oct 2001 05:07:13 -0700 Received: from sgi.com (sgi.SGI.COM [192.48.153.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f99C7AD13868 for ; Tue, 9 Oct 2001 05:07:10 -0700 Received: from tux.rsn.bth.se (tux.rsn.bth.se [194.47.143.135]) by sgi.com (980327.SGI.8.8.8-aspam/980304.SGI-aspam: SGI does not authorize the use of its proprietary systems or networks for unsolicited or bulk email from the Internet.) via ESMTP id FAB08472 for ; Tue, 9 Oct 2001 05:07:07 -0700 (PDT) mail_from (gandalf@wlug.westbo.se) Received: from localhost (gandalf@localhost [127.0.0.1]) by localhost (8.12.0.Beta19/8.12.0.Beta19/Debian 8.12.0.Beta19) with ESMTP id f99C759X016148; Tue, 9 Oct 2001 14:07:05 +0200 Date: Tue, 9 Oct 2001 14:07:05 +0200 (CEST) From: Martin Josefsson X-Sender: gandalf@tux.rsn.bth.se To: jamal cc: Santiago Garcia Mantinan , netdev@oss.sgi.com Subject: Re: 2.2 performance on high network load much much better than 2.4 (fwd) In-Reply-To: Message-ID: X-message-flag: Get yourself a real mail client! http://www.washington.edu/pine/ MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1300 Lines: 36 On Tue, 9 Oct 2001, jamal wrote: > On Tue, 9 Oct 2001, Martin Josefsson wrote: > > > The recieving end was a 2 x pIII 600 server with a D-Link DFE570-TX NIC > > (quad tulip) running 2.4.8-ac12 + tulip-ss010402-polling driver. > > > > The sending machine was my workstation here, a pIII 700 with eepro100 NIC > > running 2.4.9-ac5. This machine is attached to eth1 on the server via a > > crossover cable. > > > > The reciever has a few iptables modules loaded, ip_conntrack was one of > > them. (the sender doesn't have any iptables modules loaded) > > > Its been proven that contrack does slow down things .. Yes I know that's why I mentioned it. > [Good test results deleted] > > Could you repeat the tests with NAPI? The interesting bit is when you > start sending on the other ethernet ports at really high rates > output of cat /proc/interupts and /proc/net/softnet_stat also > /proc/net/drivers/* output I'll see if I can rerun the tests again today with the NAPI-patched kernel I have here (2.4.9-ac18 + 2.4.10-poll.pat + tulip-NAPI-ss011004 (mislabeled as 011010 on robur.slu.se) I might have to wait with these tests untill tomorrow, boring school work that needs to be done :( /Martin Never argue with an idiot. They drag you down to their level, then beat you with experience. From owner-netdev@oss.sgi.com Tue Oct 9 12:09:48 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f99J9mx21767 for netdev-outgoing; Tue, 9 Oct 2001 12:09:48 -0700 Received: from tux.rsn.bth.se (tux.rsn.bth.se [194.47.143.135]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f99J9YD21760 for ; Tue, 9 Oct 2001 12:09:34 -0700 Received: from localhost (gandalf@localhost [127.0.0.1]) by localhost (8.12.0.Beta19/8.12.0.Beta19/Debian 8.12.0.Beta19) with ESMTP id f99J9T9X031508; Tue, 9 Oct 2001 21:09:29 +0200 Date: Tue, 9 Oct 2001 21:09:29 +0200 (CEST) From: Martin Josefsson X-Sender: gandalf@tux.rsn.bth.se To: netdev@oss.sgi.com cc: linux-kernel@vger.kernel.org, netfilter-devel@lists.samba.org Subject: NMI Watchdog detected LOCKUP on CPU1 Message-ID: X-message-flag: Get yourself a real mail client! http://www.washington.edu/pine/ MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 6163 Lines: 153 Hi (Sorry for the crosspost but I don't know on which list the people capable of helping are subscribed :) (it's kernel, networkrelated and possibly netfilter related) As you can see (see topic) I have a small problem. I got this while stresstesting an SMP router here. hardware: 2 x pIII 600 440BX chipset motherboard 2 x 32MB pc100 ram 128MB flashdisk D-Link DFE570-TX (quad DECchip 21143) NIC Berkshire PCI watchdogcard. Kernel: 2.4.8-ac12 (got a hang on 2.4.9-ac18 too but didn't have a monitor or serial console attached to it and the watchdogcard rebooted it) The tulipdriver I'm using isn't the "vanilla" one, it's the optimized polling version by Jamal and Robert (ANK was involved here too I think?) I can't rule this driver out but I don't think it's at fault here. I was running a quite heavy stresstest between 3 ports on the NIC. All was going just fine until I heard the watchdog go *beep* as it rebooted the router. And then I found this on the serial-console: "NMI Watchdog detected LOCKUP on CPU1, registers:" and a stackdump (decoded below) loaded modules: ipt_MASQUERADE 1360 1 (autoclean) ip_nat_irc 2768 0 (unused) ip_conntrack_irc 2496 0 [ip_nat_irc] ip_nat_ftp 3184 0 (unused) ip_conntrack_ftp 3280 0 [ip_nat_ftp] iptable_nat 13424 7 [ipt_MASQUERADE ip_nat_irc ip_nat_ftp] ip_conntrack 14224 8 [ipt_MASQUERADE ip_nat_irc ip_conntrack_irc ip_nat_ftp ip_conntrack_ftp iptable_nat] iptable_filter 1728 0 (unused) ip_tables 10592 5 [ipt_MASQUERADE iptable_nat iptable_filter] tulip 42224 3 /proc/version: Linux version 2.4.8-ac12 (root@tux) (gcc version 2.95.4 20010604 (Debian prerelease)) #7 SMP Sun Aug 26 22:02:31 CEST 2001 the router is set up to SNAT packets going out of eth0 and then I have a portforward leading in to a machine on another port in the NIC. I had one machine on each of eth0, eth1 and eth3 generating and recieving traffic. (the traffic was generated by 'nc ip port < /dev/zero' and revieced by 'nc -l -p port > /dev/null' so there's some pipeing going on too.) two streams coming in on eth1 and eth3 where SNAT'd out via eth0 and one stream coming in on eth0 was DNAT'd to the machine on eth1. And then there where two streams between eth1 and eth3, one in each direction. = there where quite a lot of packets. The router had been running this test for over 36 hours when it died. As I wrote earlier, 2.4.9-ac18 died too, within a few hours but I had no chance of getting the output :( Decoded Oops: ksymoops 2.4.1 on i686 2.4.9-ac5. Options used -V (default) -K (specified) -L (specified) -O (specified) -m System.map-2.4.8-ac12 (specified) NMI Watchdog detected LOCKUP on CPU1, registers: CPU: 1 EIP: 0010:[] Using defaults from ksymoops -t elf32-i386 -a i386 EFLAGS: 00000087 eax: c483f2a4 ebx: c1e8c200 ecx: c1e8c29c edx: c1120000 esi: c1e8c2d0 edi: c11b2158 ebp: 5671b6f9 esp: c1121d58 ds: 0018 es: 0018 ss: 0018 Process swapper (pid: 0, stackpage=c1121000) Stack: c483e8b5 c1e8c200 c11b2140 c4816734 c1e8c200 00000000 c022394e c1e8c200 00000000 c11b2158 c48065b1 c1970c80 c1679320 c26a3740 c11b2000 c02482d0 000000c8 c11b2158 00000000 5671b6f9 00000006 00000000 00000282 c11b2140 Call Trace: [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] Code: 81 38 00 00 00 01 75 f8 f0 81 28 00 00 00 01 0f 85 e4 ff ff >>EIP; c0105c5f <__write_lock_failed+7/20> <===== Trace; c483e8b5 Trace; c4816734 Trace; c022394e <__kfree_skb+96/134> Trace; c48065b1 Trace; c02482d0 Trace; c022e71b Trace; c0226e9d Trace; c0248390 Trace; c022d408 Trace; c0245950 Trace; c02482b2 Trace; c02482d0 Trace; c02459aa Trace; c022d408 Trace; c02458e8 Trace; c0245950 Trace; c0244aed Trace; c0244910 Trace; c022d408 Trace; c0244776 Trace; c0244910 Trace; c022771e Trace; c011956f Trace; c010861b Trace; c01051b0 Trace; c01051b0 Trace; c010a714 Trace; c01051b0 Trace; c01051b0 Trace; c01051dd Trace; c0105242 Trace; c011577b <_call_console_drivers+53/58> Code; c0105c5f <__write_lock_failed+7/20> 00000000 <_EIP>: Code; c0105c5f <__write_lock_failed+7/20> <===== 0: 81 38 00 00 00 01 cmpl $0x1000000,(%eax) <===== Code; c0105c65 <__write_lock_failed+d/20> 6: 75 f8 jne 0 <_EIP> Code; c0105c67 <__write_lock_failed+f/20> 8: f0 81 28 00 00 00 01 lock subl $0x1000000,(%eax) Code; c0105c6e <__write_lock_failed+16/20> f: 0f 85 e4 ff ff 00 jne fffff9 <_EIP+0xfffff9> c1105c58 To me it looks like ip_finish_output2 freaked and did something nasty. I hope someone who knows the networking a little better than me can help with this one. My goal is to make linux survive at least a month of stresstesting :) /Martin Never argue with an idiot. They drag you down to their level, then beat you with experience. From owner-netdev@oss.sgi.com Wed Oct 10 09:53:53 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9AGrr924992 for netdev-outgoing; Wed, 10 Oct 2001 09:53:53 -0700 Received: from ms2.inr.ac.ru (minus.inr.ac.ru [193.233.7.97]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9AGroD24989 for ; Wed, 10 Oct 2001 09:53:51 -0700 Received: from mops.inr.ac.ru (mops.inr.ac.ru [193.233.7.60]) by ms2.inr.ac.ru (8.6.13/ANK) with ESMTP id UAA04173; Wed, 10 Oct 2001 20:53:41 +0400 Received: (from kuznet@localhost) by mops.inr.ac.ru (8.9.3/8.9.3) id XAA00770; Sun, 7 Oct 2001 23:49:35 +0400 Message-Id: <200110071949.XAA00770@mops.inr.ac.ru> Subject: Re: [Linux Diffserv] Need to be root to setsockopt() for EF? To: pekkas@netcore.FI (Pekka Savola) Date: Sun, 7 Oct 2001 23:49:35 +0400 (MSD) Cc: netdev@oss.sgi.com In-Reply-To: from "Pekka Savola" at Oct 5, 1 09:45:00 am From: Alexey Kuznetsov X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 596 Lines: 17 Hello! > A part of DSCP field was previously Precedence. > > Linux has required that in order to use 'Critical' or higher Precedence, > one must have CAP_NET_ADMIN capability, in most cases, root. > > I'm not one to say whether this restriction should be removed. Perhaps. Not removed, but made _stronger_. Essentially, allowing user to set an arbitrary DSCP is an evidence of security hole and subject of CAP_NET_RAW or ADMIN. Actually, one of considered variants was to allow to set by default only three values: 0 and those which used to correspong low-delay and high-throghput. Alexey From owner-netdev@oss.sgi.com Wed Oct 10 10:10:26 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9AHAQI25525 for netdev-outgoing; Wed, 10 Oct 2001 10:10:26 -0700 Received: from dibbler.ne.mediaone.net (dibbler.ne.mediaone.net [24.218.56.247]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9AHAMD25519 for ; Wed, 10 Oct 2001 10:10:22 -0700 Received: (from rodrigc@localhost) by dibbler.ne.mediaone.net (8.11.0/8.11.0) id f9AHAG814470 for netdev@oss.sgi.com; Wed, 10 Oct 2001 13:10:16 -0400 Date: Wed, 10 Oct 2001 13:10:16 -0400 From: Craig Rodrigues To: netdev@oss.sgi.com Subject: Re: [Linux Diffserv] Need to be root to setsockopt() for EF? Message-ID: <20011010131016.A14465@mediaone.net> References: <200110071949.XAA00770@mops.inr.ac.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <200110071949.XAA00770@mops.inr.ac.ru>; from kuznet@ms2.inr.ac.ru on Sun, Oct 07, 2001 at 11:49:35PM +0400 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1268 Lines: 36 On Sun, Oct 07, 2001 at 11:49:35PM +0400, Alexey Kuznetsov wrote: > Hello! > > > A part of DSCP field was previously Precedence. > > > > Linux has required that in order to use 'Critical' or higher Precedence, > > one must have CAP_NET_ADMIN capability, in most cases, root. > > > > I'm not one to say whether this restriction should be removed. Perhaps. > > Not removed, but made _stronger_. > > Essentially, allowing user to set an arbitrary DSCP is an evidence of security > hole and subject of CAP_NET_RAW or ADMIN. Actually, one of considered > variants was to allow to set by default only three values: 0 and those > which used to correspong low-delay and high-throghput. Hi, This is very interesting information, since I am trying to develop an application which uses Diffserv, but works on multiple operating systems. Can you point me to a document which explains what these CAP_NET_ADMIN is, and how it is related to setting DSCP values? If there is no formal document, can you direct me to a section of the Linux kernel which I can grep to see how this works? I'm a newbie to Linux kernel networking internals, so some guidance would be appreciated. :) -- Craig Rodrigues http://www.gis.net/~craigr rodrigc@mediaone.net From owner-netdev@oss.sgi.com Wed Oct 10 23:53:59 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9B6rxR10787 for netdev-outgoing; Wed, 10 Oct 2001 23:53:59 -0700 Received: from titan.bieringer.de (mail.bieringer.de [195.226.187.51]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9B6rtD10784 for ; Wed, 10 Oct 2001 23:53:56 -0700 Received: (qmail 16524 invoked from network); 11 Oct 2001 06:53:47 -0000 Received: from pd950f980.dip.t-dialin.net (HELO worker.muc.bieringer.de) (217.80.249.128) by mail.bieringer.de with SMTP; 11 Oct 2001 06:53:47 -0000 Date: Thu, 11 Oct 2001 08:55:21 +0200 From: Peter Bieringer To: Maillist netdev , Maillist USAGI-users Subject: IPv6 & QoS in Linux - current state? Message-ID: <30150000.1002783321@localhost> X-Mailer: Mulberry/2.1.0 (Linux/x86) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 704 Lines: 26 Hi, I got a question regarding the implementation of IPv6 & QoS in Linux. Does anyone know the current state? > I am interested in utilizing the Qos features for Ipv6 > for which I wanted to know whether Linux supports > any QoS ( Flow label and traffic class) for Ipv6. > If yes then in which distribution and also if API's > are available so that the services could be invoked for > any real time application. Results will also be filled into my kernel-IPv6-status page (where QoS is still missing as I've seen). Perhaps also interesting: > Also do you have any information of any other > Ipv6 stack vendor who has implemented > QoS? Thanks for any answers, Peter From owner-netdev@oss.sgi.com Thu Oct 11 04:43:13 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9BBhDg17968 for netdev-outgoing; Thu, 11 Oct 2001 04:43:13 -0700 Received: from castor.erda.se ([195.163.32.186]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9BBh5D17965 for ; Thu, 11 Oct 2001 04:43:06 -0700 Received: by castor.erda.se with Internet Mail Service (5.5.2653.19) id <4J8SGY12>; Thu, 11 Oct 2001 13:45:00 +0200 Message-ID: <499AC99D205FD51197CD0002A574C4F805240A@castor.erda.se> From: =?iso-8859-1?Q?=22Wikstr=F6m=2C_M=E5rten=22?= To: "'linux-net@vger.kernel.org '" , "'netdev@oss.sgi.com '" Subject: SyncPPP vs. ppp_synctty Date: Thu, 11 Oct 2001 13:44:50 +0200 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C1524A.22D5F7B0" Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 2581 Lines: 78 This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C1524A.22D5F7B0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Is the use of syncppp discouraged in favour of ppp_generic? Background: I have an E1/T1 interface from SBE that ships with its own syncppp. = This lacks some vital functions (e.g. IPv6, Multilink PPP), and this seems = also be the case with the syncppp that ships with the kernel. I was thinking about implementing it myself, but then I found some old postings on the = lkml suggesting that there was no use putting effort into it, since you = should use ppp_generic instead. To me the ppp_generic and syncppp seem = completely different and I am very confused right now. Should I update syncppp or = try to hack the card driver to use generic ppp instead? any clarifications appreciated. BTW, which mailing list would have been most appropriate for this = posting? I can't tell them apart. Sorry for crossposting. /M=E5rten ------_=_NextPart_001_01C1524A.22D5F7B0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable SyncPPP vs. ppp_synctty

Is the use of syncppp discouraged in favour of = ppp_generic?

Background:
I have an E1/T1 interface from SBE that ships with = its own syncppp. This lacks some vital functions (e.g. IPv6, Multilink = PPP), and this seems also be the case with the syncppp that ships with = the kernel. I was thinking about implementing it myself, but then I = found some old postings on the lkml suggesting that there was no use = putting effort into it, since you should use ppp_generic instead. To me = the ppp_generic and syncppp seem completely different and I am very = confused right now. Should I update syncppp or try to hack the card = driver to use generic ppp instead?

any clarifications appreciated.

BTW, which mailing list would have been most = appropriate for this posting? I can't tell them apart. Sorry for = crossposting.

/M=E5rten

------_=_NextPart_001_01C1524A.22D5F7B0-- From owner-netdev@oss.sgi.com Thu Oct 11 12:10:15 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9BJAFe29389 for netdev-outgoing; Thu, 11 Oct 2001 12:10:15 -0700 Received: from ms2.inr.ac.ru (minus.inr.ac.ru [193.233.7.97]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9BJACD29386 for ; Thu, 11 Oct 2001 12:10:12 -0700 Received: (from kuznet@localhost) by ms2.inr.ac.ru (8.6.13/ANK) id XAA28172; Thu, 11 Oct 2001 23:09:53 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200110111909.XAA28172@ms2.inr.ac.ru> Subject: Re: IPv6 & QoS in Linux - current state? To: pb@bieringer.DE (Peter Bieringer) Date: Thu, 11 Oct 2001 23:09:53 +0400 (MSK DST) Cc: netdev@oss.sgi.com In-Reply-To: <30150000.1002783321@localhost> from "Peter Bieringer" at Oct 11, 1 11:15:01 am X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 371 Lines: 16 Hello! > Does anyone know the current state? The state is the same as with IP. I.e. "complete". > > any QoS ( Flow label and traffic class) for Ipv6. > > If yes then in which distribution No idea. User level support for diffserv and intserv may be missing in all the distributions. Kernel side is present as soon as distibution distributes kernel. :-) Alexey From owner-netdev@oss.sgi.com Thu Oct 11 12:50:49 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9BJonp30736 for netdev-outgoing; Thu, 11 Oct 2001 12:50:49 -0700 Received: from mandrakesoft.mandrakesoft.com (nsd.mandrakesoft.com [216.71.84.35] (may be forged)) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9BJokD30723 for ; Thu, 11 Oct 2001 12:50:46 -0700 Received: from localhost (jgarzik@localhost) by mandrakesoft.mandrakesoft.com (8.8.5/8.8.5) with SMTP id OAA19941; Thu, 11 Oct 2001 14:50:31 -0500 Date: Thu, 11 Oct 2001 14:50:31 -0500 (CDT) From: Jeff Garzik To: kuznet@ms2.inr.ac.ru cc: Peter Bieringer , netdev@oss.sgi.com Subject: Re: IPv6 & QoS in Linux - current state? In-Reply-To: <200110111909.XAA28172@ms2.inr.ac.ru> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 643 Lines: 28 On Thu, 11 Oct 2001 kuznet@ms2.inr.ac.ru wrote: > Hello! > > > Does anyone know the current state? > > The state is the same as with IP. I.e. "complete". > > > > > any QoS ( Flow label and traffic class) for Ipv6. > > > If yes then in which distribution > > No idea. User level support for diffserv and intserv > may be missing in all the distributions. > > Kernel side is present as soon as distibution distributes kernel. :-) Speaking of QoS, the new RealTek chips support two Tx outgoing queues, one for "normal Tx", and one for "priority Tx". Is there any way to make the priority Tx queue useful to the kernel? Jeff From owner-netdev@oss.sgi.com Thu Oct 11 14:43:30 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9BLhUd01420 for netdev-outgoing; Thu, 11 Oct 2001 14:43:30 -0700 Received: from titan.bieringer.de (mail.bieringer.de [195.226.187.51]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9BLhPD01415 for ; Thu, 11 Oct 2001 14:43:26 -0700 Received: (qmail 23841 invoked from network); 11 Oct 2001 21:43:19 -0000 Received: from pd950f97f.dip.t-dialin.net (HELO worker.muc.bieringer.de) (217.80.249.127) by mail.bieringer.de with SMTP; 11 Oct 2001 21:43:19 -0000 Date: Thu, 11 Oct 2001 23:44:54 +0200 From: Peter Bieringer To: kuznet@ms2.inr.ac.ru cc: netdev@oss.sgi.com Subject: Re: IPv6 & QoS in Linux - current state? Message-ID: <36430000.1002836694@localhost> In-Reply-To: <200110111909.XAA28172@ms2.inr.ac.ru> References: <200110111909.XAA28172@ms2.inr.ac.ru> X-Mailer: Mulberry/2.1.0 (Linux/x86) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 805 Lines: 39 Hi, --On Thursday, October 11, 2001 11:09:53 PM +0400 kuznet@ms2.inr.ac.ru wrote: >> Does anyone know the current state? > > The state is the same as with IP. I.e. "complete". Nice to hear. >> > any QoS ( Flow label and traffic class) for Ipv6. >> > If yes then in which distribution > > No idea. User level support for diffserv and intserv > may be missing in all the distributions. Ok. > Kernel side is present as soon as distibution distributes kernel. BTW: can you give me an update for the current values in my table and perhaps extensions? Currently: Flow label support 2.4.x: working 2.2.x: unknown Flow label specific routing 2.4.x: in progress 2.2.x: unknown Flow label support on application level 2.4.x: working? 2.2.x: working? Anthing also missing? TIA, Peter From owner-netdev@oss.sgi.com Thu Oct 11 20:29:52 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9C3TqH08282 for netdev-outgoing; Thu, 11 Oct 2001 20:29:52 -0700 Received: from dea.linux-mips.net (a1as15-p114.stg.tli.de [195.252.192.114]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9C3T5D08273 for ; Thu, 11 Oct 2001 20:29:06 -0700 Received: (from ralf@localhost) by dea.linux-mips.net (8.11.1/8.11.1) id f9C3ScC30072 for netdev@oss.sgi.com; Fri, 12 Oct 2001 05:28:38 +0200 Received: from grok.yi.org (IDENT:root@cx97923-a.phnx3.az.home.com [24.9.112.194]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9C0RhD04971 for ; Thu, 11 Oct 2001 17:27:43 -0700 Received: from candelatech.com ([66.89.142.2]) by grok.yi.org (8.11.2/8.11.2) with ESMTP id f9C0Qli17856; Thu, 11 Oct 2001 17:26:47 -0700 Message-ID: <3BC62DBB.6871E5F@candelatech.com> Date: Thu, 11 Oct 2001 16:39:39 -0700 From: Ben Greear Organization: Candela Technologies Inc X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.4.7 i686) X-Accept-Language: en MIME-Version: 1.0 To: Jeff Garzik CC: kuznet@ms2.inr.ac.ru, Peter Bieringer , netdev@oss.sgi.com Subject: Re: IPv6 & QoS in Linux - current state? References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1021 Lines: 34 Jeff Garzik wrote: > > On Thu, 11 Oct 2001 kuznet@ms2.inr.ac.ru wrote: > > > Hello! > > > > > Does anyone know the current state? > > > > The state is the same as with IP. I.e. "complete". > > > > > > > > any QoS ( Flow label and traffic class) for Ipv6. > > > > If yes then in which distribution > > > > No idea. User level support for diffserv and intserv > > may be missing in all the distributions. > > > > Kernel side is present as soon as distibution distributes kernel. :-) > > Speaking of QoS, the new RealTek chips support two Tx outgoing queues, > one for "normal Tx", and one for "priority Tx". Is there any way to > make the priority Tx queue useful to the kernel? > You could look at VLAN's priority bits and do good things based on them when VLANs are in use... Ben > Jeff -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From owner-netdev@oss.sgi.com Thu Oct 11 23:10:06 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9C6A6x11479 for netdev-outgoing; Thu, 11 Oct 2001 23:10:06 -0700 Received: from netcore.fi (netcore.fi [193.94.160.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9C6A1D11468 for ; Thu, 11 Oct 2001 23:10:01 -0700 Received: from localhost (pekkas@localhost) by netcore.fi (8.11.6/8.11.6) with ESMTP id f9C699p29037; Fri, 12 Oct 2001 09:09:09 +0300 Date: Fri, 12 Oct 2001 09:09:09 +0300 (EEST) From: Pekka Savola To: cc: Peter Bieringer , Subject: Re: IPv6 & QoS in Linux - current state? In-Reply-To: <200110111909.XAA28172@ms2.inr.ac.ru> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 693 Lines: 19 On Thu, 11 Oct 2001 kuznet@ms2.inr.ac.ru wrote: > > > any QoS ( Flow label and traffic class) for Ipv6. > > > If yes then in which distribution > > No idea. User level support for diffserv and intserv > may be missing in all the distributions. Hmm, I'm not sure if it's glibc, kernel or both, but some updated API/advanced API hooks for setting QoS related bits in the datagrams from the userspace without raw sockets or the like were missing at least some time ago. I believe both. -- Pekka Savola "Tell me of difficulties surmounted, Netcore Oy not those you stumble over and fall" Systems. Networks. Security. -- Robert Jordan: A Crown of Swords From owner-netdev@oss.sgi.com Fri Oct 12 05:01:04 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9CC14D18903 for netdev-outgoing; Fri, 12 Oct 2001 05:01:04 -0700 Received: from shell.cyberus.ca ([216.191.240.114]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9CC12D18899 for ; Fri, 12 Oct 2001 05:01:02 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id HAA17316; Fri, 12 Oct 2001 07:57:57 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Fri, 12 Oct 2001 07:57:57 -0400 (EDT) From: jamal To: Jeff Garzik cc: , Peter Bieringer , Subject: Re: IPv6 & QoS in Linux - current state? In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 319 Lines: 14 On Thu, 11 Oct 2001, Jeff Garzik wrote: > Speaking of QoS, the new RealTek chips support two Tx outgoing queues, > one for "normal Tx", and one for "priority Tx". Is there any way to > make the priority Tx queue useful to the kernel? Jeff, Use the skb->priority to select what queue to send it to. cheers, jamal From owner-netdev@oss.sgi.com Fri Oct 12 05:12:57 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9CCCvj19158 for netdev-outgoing; Fri, 12 Oct 2001 05:12:57 -0700 Received: from shell.cyberus.ca ([216.191.240.114]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9CCCqD19155 for ; Fri, 12 Oct 2001 05:12:52 -0700 Received: from localhost (hadi@localhost) by shell.cyberus.ca (8.9.3/666/Cyberus Online Inc.) with ESMTP id IAA17348; Fri, 12 Oct 2001 08:09:52 -0400 (EDT) X-Authentication-Warning: shell.cyberus.ca: hadi owned process doing -bs Date: Fri, 12 Oct 2001 08:09:52 -0400 (EDT) From: jamal To: Craig Rodrigues cc: Subject: Re: [Linux Diffserv] Need to be root to setsockopt() for EF? In-Reply-To: <20011010131016.A14465@mediaone.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1786 Lines: 58 On Wed, 10 Oct 2001, Craig Rodrigues wrote: > On Sun, Oct 07, 2001 at 11:49:35PM +0400, Alexey Kuznetsov wrote: > > Hello! > > > > > A part of DSCP field was previously Precedence. > > > > > > Linux has required that in order to use 'Critical' or higher Precedence, > > > one must have CAP_NET_ADMIN capability, in most cases, root. > > > > > > I'm not one to say whether this restriction should be removed. Perhaps. > > > > Not removed, but made _stronger_. > > > > Essentially, allowing user to set an arbitrary DSCP is an evidence of security > > hole and subject of CAP_NET_RAW or ADMIN. Actually, one of considered > > variants was to allow to set by default only three values: 0 and those > > which used to correspong low-delay and high-throghput. > > Hi, > > This is very interesting information, since I am trying to develop > an application which uses Diffserv, but works on multiple > operating systems. In the case of diffserv, the DSCP setting is done by the router. Using Linux as a router allows you to do this. Settings even by root are not very valuable since in the network they will be overriden. > Can you point me to a document which explains what these > CAP_NET_ADMIN is, and how it is related to setting DSCP values? > Think about it: if actually the network cared about dscp value, as set by joe_l00s3r, we would have chaos. Everyone would be trying to get the highest qos at the expense of the rest of the world. Hope this helps cheers, jamal > If there is no formal document, can you direct me to a section > of the Linux kernel which I can grep to see how this works? > > I'm a newbie to Linux kernel networking internals, so some > guidance would be appreciated. :) > > > -- > Craig Rodrigues > http://www.gis.net/~craigr > rodrigc@mediaone.net > From owner-netdev@oss.sgi.com Fri Oct 12 08:37:45 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9CFbjc23550 for netdev-outgoing; Fri, 12 Oct 2001 08:37:45 -0700 Received: from hq.pm.waw.pl (hq.pm.waw.pl [195.116.170.10]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9CFbfD23547 for ; Fri, 12 Oct 2001 08:37:41 -0700 Received: (from uucp@localhost) by hq.pm.waw.pl with UUCP id f9CFbRX08749; Fri, 12 Oct 2001 17:37:27 +0200 Received: (from khc@localhost) by defiant.pm.waw.pl (8.11.6/8.11.6) id f9C9YGu06952; Fri, 12 Oct 2001 11:34:16 +0200 To: "=?iso-8859-1?q?Wikstr=F6m,_M=E5rten?=" Cc: "'linux-net@vger.kernel.org '" , "'netdev@oss.sgi.com '" Subject: Re: SyncPPP vs. ppp_synctty References: <499AC99D205FD51197CD0002A574C4F805240A@castor.erda.se> From: Krzysztof Halasa Date: 12 Oct 2001 11:34:15 +0200 In-Reply-To: "=?iso-8859-1?q?Wikstr=F6m,_M=E5rten?="'s message of "Thu, 11 Oct 2001 13:44:50 +0200" Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by oss.sgi.com id f9CFbgD23548 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1042 Lines: 22 "Wikström, Mårten" writes: > Is the use of syncppp discouraged in favour of ppp_generic? > > Background: > I have an E1/T1 interface from SBE that ships with its own syncppp. This > lacks some vital functions (e.g. IPv6, Multilink PPP), and this seems also > be the case with the syncppp that ships with the kernel. I was thinking > about implementing it myself, but then I found some old postings on the lkml > suggesting that there was no use putting effort into it, since you should > use ppp_generic instead. To me the ppp_generic and syncppp seem completely > different and I am very confused right now. Should I update syncppp or try > to hack the card driver to use generic ppp instead? What I think is we need just one PPP driver. Hovewer, ppp_generic requires tty device (either async or sync) and insists on creating network device. That isn't sane behavior if you just want existing net device to talk PPP. We should merge the drivers, AFAIK 2.5 is coming... -- Krzysztof Halasa Network Administrator From owner-netdev@oss.sgi.com Fri Oct 12 10:53:00 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9CHr0g26794 for netdev-outgoing; Fri, 12 Oct 2001 10:53:00 -0700 Received: from ms2.inr.ac.ru (minus.inr.ac.ru [193.233.7.97]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9CHqvD26790 for ; Fri, 12 Oct 2001 10:52:57 -0700 Received: (from kuznet@localhost) by ms2.inr.ac.ru (8.6.13/ANK) id VAA12684; Fri, 12 Oct 2001 21:52:42 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200110121752.VAA12684@ms2.inr.ac.ru> Subject: Re: IPv6 & QoS in Linux - current state? To: jgarzik@mandrakesoft.com (Jeff Garzik) Date: Fri, 12 Oct 2001 21:52:42 +0400 (MSK DST) Cc: pb@bieringer.DE, netdev@oss.sgi.com In-Reply-To: from "Jeff Garzik" at Oct 11, 1 02:50:31 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 735 Lines: 20 Hello! > Speaking of QoS, the new RealTek chips support two Tx outgoing queues, > one for "normal Tx", and one for "priority Tx". Is there any way to > make the priority Tx queue useful to the kernel? Unfortunately, no generic way. It is a design problem to think on. "tbusy" approach is inadequate for this. The only working scheme is used by ATM, which hooks special qdisc "atm" and multiplexes packets to proper VCs. But true solution apparently should be attachable to an arbitrary qdisc, so I think the solution is to split "tbusy"/"hard_start_xmit". Another approach (also working in your case) is to allocate two devices: ethX and ethX.high. It also works, but results in routing asymmetry, which is real desease. Alexey From owner-netdev@oss.sgi.com Fri Oct 12 11:54:02 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9CIs2b28115 for netdev-outgoing; Fri, 12 Oct 2001 11:54:02 -0700 Received: from ms2.inr.ac.ru (minus.inr.ac.ru [193.233.7.97]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9CIrxD28109 for ; Fri, 12 Oct 2001 11:53:59 -0700 Received: (from kuznet@localhost) by ms2.inr.ac.ru (8.6.13/ANK) id WAA13004; Fri, 12 Oct 2001 22:53:46 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200110121853.WAA13004@ms2.inr.ac.ru> Subject: Re: IPv6 & QoS in Linux - current state? To: pekkas@netcore.fi (Pekka Savola) Date: Fri, 12 Oct 2001 22:53:46 +0400 (MSK DST) Cc: pb@bieringer.DE, netdev@oss.sgi.com In-Reply-To: from "Pekka Savola" at Oct 12, 1 09:09:09 am X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 341 Lines: 14 Hello! > Hmm, I'm not sure if it's glibc, Surely, it is not glibc, because people may compile rsvpd (good sample, because it accesses almost all the elements of the engine) using glibc. :-) I mean scripts for setting up diffserv (which is higjly non trivial), rsvpd with its api etc. etc. Well, and documentation, of course. :-( Alexey From owner-netdev@oss.sgi.com Fri Oct 12 14:18:07 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9CLI7W30857 for netdev-outgoing; Fri, 12 Oct 2001 14:18:07 -0700 Received: from e33.bld.us.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9CLI5D30854 for ; Fri, 12 Oct 2001 14:18:06 -0700 Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e33.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id QAA111388 for ; Fri, 12 Oct 2001 16:15:38 -0500 Received: from d03nm035.boulder.ibm.com (d03nm035.boulder.ibm.com [9.99.140.35]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id f9CLGmg219206 for ; Fri, 12 Oct 2001 15:16:48 -0600 Subject: OT: linux-pci mailing list To: netdev@oss.sgi.com X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: "Bruce Allan" Date: Fri, 12 Oct 2001 14:14:05 -0700 X-MIMETrack: Serialize by Router on D03NM035/03/M/IBM(Release 5.0.8 |June 18, 2001) at 10/12/2001 03:16:48 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 356 Lines: 12 Sort of off-topic, but I thought someone on this list may know... Has the linux-pci mailing list moved to a server different than listproc@atrey.karlin.mff.cuni.cz ? I tried subscribing, but got a "user unknown" delivery failure report. Bruce Allan/Beaverton/IBM IBM Linux Technology Center 503-578-4187 IBM Tie-line 775-4187 bruce.allan@us.ibm.com From owner-netdev@oss.sgi.com Fri Oct 12 15:43:49 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9CMhnB32094 for netdev-outgoing; Fri, 12 Oct 2001 15:43:49 -0700 Received: from sgi.com (sgi.SGI.COM [192.48.153.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9CMhhD32089 for ; Fri, 12 Oct 2001 15:43:44 -0700 Received: from posta2.elte.hu (posta2.elte.hu [157.181.151.9]) by sgi.com (980327.SGI.8.8.8-aspam/980304.SGI-aspam: SGI does not authorize the use of its proprietary systems or networks for unsolicited or bulk email from the Internet.) via ESMTP id PAA09795 for ; Fri, 12 Oct 2001 15:43:41 -0700 (PDT) mail_from (mingo@elte.hu) Received: from chiara.elte.hu (chiara.elte.hu [157.181.150.200]) by posta2.elte.hu (Postfix) with ESMTP id 018B648004 for ; Fri, 5 Oct 2001 17:40:41 +0200 (CEST) Received: by chiara.elte.hu (Postfix, from userid 17806) id B7CB31FD2; Thu, 4 Oct 2001 08:55:10 +0200 (CEST) Date: Thu, 4 Oct 2001 08:52:20 +0200 (CEST) From: Ingo Molnar Reply-To: To: Ben Greear Cc: jamal , , Alexey Kuznetsov , Robert Olsson , Benjamin LaHaise , , Linus Torvalds , Alan Cox , Simon Kirby Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 In-Reply-To: <3BBC06A1.1818E45C@candelatech.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 491 Lines: 15 On Wed, 3 Oct 2001, Ben Greear wrote: > > so far your appraoch is that of a shotgun i.e "let me fire in > > that crowd and i'll hit my target but dont care if i take down a few > > more"; regardless of how noble the reasoning is, it's as Linus described > > it -- a sledge hammer. > > Aye, but by shooting this target and getting a few bystanders, you save > everyone else... (And it's only a flesh wound!!) especially considering that the current code nukes the whole city ;) Ingo From owner-netdev@oss.sgi.com Sat Oct 13 00:44:32 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9D7iWx08041 for netdev-outgoing; Sat, 13 Oct 2001 00:44:32 -0700 Received: from s1.uklinux.net (ns1.uklinux.net [212.1.130.11]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9D7iTD08038 for ; Sat, 13 Oct 2001 00:44:29 -0700 Received: from whinlatter (user88063@ppp-1-189.cvx2.telinco.net [212.1.140.189]) by s1.uklinux.net (8.11.6/8.11.6) with ESMTP id f9D7iMH13069 for ; Sat, 13 Oct 2001 08:44:22 +0100 Envelope-To: Received: from roger by whinlatter with local (Exim 3.12 #1 (Debian)) id 15rNxv-0005UM-00 for ; Wed, 10 Oct 2001 19:20:23 +0100 Date: Wed, 10 Oct 2001 19:20:23 +0100 From: Roger Leigh To: netdev@oss.sgi.com Subject: Broken ethernet driver Message-ID: <20011010192023.A21092@whinlatter.uklinux.net> Reply-To: rl117@york.ac.uk Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1080 Lines: 24 Hello, Sorry if this is the wrong list. I get the following when compiling linux 2.4.11: gcc -D__KERNEL__ -I/usr/src/kernel-source-2.4.11/include -Wall -Wstrict-prototypes -Wno-trigraphs -O2 -fomit-frame-pointer -fno-strict-aliasing -fno-common -pipe -mpreferred-stack-boundary=2 -march=i686 -DMODULE -DMODVERSIONS -include /usr/src/kernel-source-2.4.11/include/linux/modversions.h -c -o at1700.o at1700.c at1700.c:475: conflicting types for `read_eeprom' at1700.c:161: previous declaration of `read_eeprom' make[2]: *** [at1700.o] Error 1 The driver is written by Donald Becker, but I didn't think that he maintained it any longer: it's copyrighted 1993-98. I don't use the driver myself, but came across it when compiling a module, so I thought I had better report it. Regards, Roger PS. Please CC any replies as I am not subscribed to the list. Thanks. -- Roger Leigh ** Registration Number: 151826, http://counter.li.org ** Need Epson Stylus Utilities? http://gimp-print.sourceforge.net/ For GPG Public Key: finger rl117@tower.york.ac.uk or see public keyservers. From owner-netdev@oss.sgi.com Sat Oct 13 02:15:17 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9D9FHx09147 for netdev-outgoing; Sat, 13 Oct 2001 02:15:17 -0700 Received: from jabberwock.ucw.cz (root@jabberwock.ucw.cz [212.71.128.53]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9D9FDD09143 for ; Sat, 13 Oct 2001 02:15:14 -0700 Received: from albireo.ucw.cz (mj@albireo.ucw.cz [62.168.0.14]) by jabberwock.ucw.cz (8.9.3/8.9.3/Debian 8.9.3-21) with ESMTP id LAA23268; Sat, 13 Oct 2001 11:15:01 +0200 Received: (from mj@localhost) by albireo.ucw.cz (8.12.0.Beta7/8.12.0.Beta7/Debian 8.12.0.Beta7-1) id f9D9Eq94000718; Sat, 13 Oct 2001 11:14:52 +0200 Date: Sat, 13 Oct 2001 11:14:48 +0200 From: Martin Mares To: Bruce Allan Cc: netdev@oss.sgi.com Subject: Re: OT: linux-pci mailing list Message-ID: <20011013111448.A709@ucw.cz> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.3.20i Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 638 Lines: 16 Hello! > Sort of off-topic, but I thought someone on this list may know... > > Has the linux-pci mailing list moved to a server different than > listproc@atrey.karlin.mff.cuni.cz ? I tried subscribing, but got a "user > unknown" delivery failure report. It has moved to Majordomo on the same server and I apparently didn't update all the references to the list :-) Where is yours? Have a nice fortnight -- Martin `MJ' Mares http://atrey.karlin.mff.cuni.cz/~mj/ Faculty of Math and Physics, Charles University, Prague, Czech Rep., Earth P.C.M.C.I.A. stands for `People Can't Memorize Computer Industry Acronyms' From owner-netdev@oss.sgi.com Sat Oct 13 05:22:50 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9DCMoc11707 for netdev-outgoing; Sat, 13 Oct 2001 05:22:50 -0700 Received: from dea.linux-mips.net (a1as01-p120.stg.tli.de [195.252.185.120]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9DCMaD11700 for ; Sat, 13 Oct 2001 05:22:37 -0700 Received: (from ralf@localhost) by dea.linux-mips.net (8.11.1/8.11.1) id f9DCLwo18947 for netdev@oss.sgi.com; Sat, 13 Oct 2001 14:21:58 +0200 Received: from web20406.mail.yahoo.com (web20406.mail.yahoo.com [216.136.226.125]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9DBO3D10984 for ; Sat, 13 Oct 2001 04:24:03 -0700 Message-ID: <20011013112401.1759.qmail@web20406.mail.yahoo.com> Received: from [63.112.207.238] by web20406.mail.yahoo.com via HTTP; Sat, 13 Oct 2001 04:24:01 PDT Date: Sat, 13 Oct 2001 04:24:01 -0700 (PDT) From: Brad Chapman Subject: Re: Broken ethernet driver To: rl117@york.ac.uk Cc: netdev@oss.sgi.com In-Reply-To: <20011010192023.A21092@whinlatter.uklinux.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1422 Lines: 47 Mr. Leigh, --- Roger Leigh wrote: > Hello, > > Sorry if this is the wrong list. I get the following when compiling > linux 2.4.11: > > gcc -D__KERNEL__ -I/usr/src/kernel-source-2.4.11/include -Wall > -Wstrict-prototypes -Wno-trigraphs -O2 -fomit-frame-pointer > -fno-strict-aliasing -fno-common -pipe -mpreferred-stack-boundary=2 > -march=i686 -DMODULE -DMODVERSIONS -include > /usr/src/kernel-source-2.4.11/include/linux/modversions.h -c -o at1700.o > at1700.c > at1700.c:475: conflicting types for `read_eeprom' > at1700.c:161: previous declaration of `read_eeprom' > make[2]: *** [at1700.o] Error 1 > > The driver is written by Donald Becker, but I didn't think that he > maintained it any longer: it's copyrighted 1993-98. I don't use the > driver myself, but came across it when compiling a module, so I > thought I had better report it. Don't use 2.4.11. There is a very nasty DoS inode bug, plus a bunch of other stupid compile errors. Go back to 2.4.10 or use 2.4.13-pre1. > > Regards, > Roger > > PS. Please CC any replies as I am not subscribed to the list. Thanks. > Brad ===== Brad Chapman Permanent e-mails: kakadu_croc@yahoo.com jabiru_croc@yahoo.com Alternate e-mails: kakadu@adelphia.net kakadu@netscape.net __________________________________________________ Do You Yahoo!? Make a great connection at Yahoo! Personals. http://personals.yahoo.com From owner-netdev@oss.sgi.com Sat Oct 13 12:35:00 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9DJZ0D18100 for netdev-outgoing; Sat, 13 Oct 2001 12:35:00 -0700 Received: from bug.ucw.cz (cisco7500-mainGW.gts.cz [194.213.32.131]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9DJYkD18069 for ; Sat, 13 Oct 2001 12:34:47 -0700 Received: (from root@localhost) by bug.ucw.cz (8.9.3/8.8.5) id VAA05686; Sat, 13 Oct 2001 21:31:53 +0200 Date: Wed, 10 Oct 2001 16:25:11 +0000 From: Pavel Machek To: Alan Cox Cc: jamal , Jeff Garzik , Andrea Arcangeli , Ingo Molnar , Linux-Kernel , netdev@oss.sgi.com, Linus Torvalds Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: <20011010162510.A35@toy.ucw.cz> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0.1i In-Reply-To: ; from alan@lxorguk.ukuu.org.uk on Mon, Oct 08, 2001 at 04:35:24PM +0100 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 615 Lines: 15 Hi! > Even at 200Hz polling a typical cardbus card with say 32 ring buffer slots > can process 6000pps. On my velo, I have pcmcia but don't quite know how to drive it properly. I have not figured interrupts, so I ran ne2000 in polling mode. .5MB/sec is not bad for as slow hardware as velo is (see sig). Next experiment was ATA flash. I had to bump HZ to 1000 for it, and I'm getting spurious unexpected interrupt messaegs, but it works surprisingly well. Pavel -- Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt, details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html. From owner-netdev@oss.sgi.com Sat Oct 13 12:36:25 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9DJaP918223 for netdev-outgoing; Sat, 13 Oct 2001 12:36:25 -0700 Received: from bug.ucw.cz (cisco7500-mainGW.gts.cz [194.213.32.131]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9DJaBD18220 for ; Sat, 13 Oct 2001 12:36:11 -0700 Received: (from root@localhost) by bug.ucw.cz (8.9.3/8.8.5) id VAA05719; Sat, 13 Oct 2001 21:32:02 +0200 Date: Wed, 10 Oct 2001 16:26:52 +0000 From: Pavel Machek To: Alan Cox Cc: jamal , Jeff Garzik , Andrea Arcangeli , Ingo Molnar , Linux-Kernel , netdev@oss.sgi.com, Linus Torvalds Subject: Re: [announce] [patch] limiting IRQ load, irq-rewrite-2.4.11-B5 Message-ID: <20011010162652.B35@toy.ucw.cz> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0.1i In-Reply-To: ; from alan@lxorguk.ukuu.org.uk on Mon, Oct 08, 2001 at 05:11:03PM +0100 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 712 Lines: 27 Hi! > > Agreed if you add the polling cardbus bit. > > Note polling cardbus would require more changes than the above. > > I don't think it does. There are two pieces to the problem > > a) Not dying horribly > b) Handling it elegantly > > b) is driver specific (NAPI etc) and I think well understood to the point > its being used already for performance reasons > > a) is as simple as > > if(stuck_in_irq(foo) && irq_shared(foo)) > { > disable_real_irq(foo); > timer_fake_irq_foo(); > } I'd kill irq_shared() test, and added a printk :-). Pavel -- Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt, details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html. From owner-netdev@oss.sgi.com Sat Oct 13 18:43:15 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9E1hFo24386 for netdev-outgoing; Sat, 13 Oct 2001 18:43:15 -0700 Received: from cu518.adelaide.adsl.on.net (cu518.adelaide.adsl.on.net [150.101.236.10]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9E1hCD24368 for ; Sat, 13 Oct 2001 18:43:12 -0700 Received: from ns.aus.com (laptop.ns.aus.com [10.0.2.6]) by cu518.adelaide.adsl.on.net (8.11.0/8.11.0) with ESMTP id f9E4oHC31662 for ; Sun, 14 Oct 2001 14:20:17 +0930 Message-ID: <3BC8F5DD.2050602@ns.aus.com> Date: Sun, 14 Oct 2001 11:48:05 +0930 From: Richard Sharpe Reply-To: rsharpe@ns.aus.com User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.4) Gecko/20010917 X-Accept-Language: en-us MIME-Version: 1.0 To: netdev@oss.sgi.com Subject: Retrieving in.terface statistics frequently and without pain Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 548 Lines: 19 Hi, What is the quickest way to retrieve interface statistics quickly and without pain? I am interested in seeing bytes in and out as well as packets in and out. I am also interested in seeing collisions. If the interface was in promiscuous mode, would packets and bytes in/out count total traffic on the wire? I notice that there is no SIOCGIFSTATS IOCTL. Would such a thing be desirable? -- Richard Sharpe, rsharpe@ns.aus.com, LPIC-1 www.samba.org, www.ethereal.com, SAMS Teach Yourself Samba in 24 Hours, Special Edition, Using Samba From owner-netdev@oss.sgi.com Sun Oct 14 11:27:37 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9EIRbX14254 for netdev-outgoing; Sun, 14 Oct 2001 11:27:37 -0700 Received: from grok.yi.org (IDENT:root@cx97923-a.phnx3.az.home.com [24.9.112.194]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9EIRYD14251 for ; Sun, 14 Oct 2001 11:27:34 -0700 Received: from candelatech.com (IDENT:greear@localhost.localdomain [127.0.0.1]) by grok.yi.org (8.11.2/8.11.2) with ESMTP id f9EIRVi06997; Sun, 14 Oct 2001 11:27:31 -0700 Message-ID: <3BC9D913.1922D21A@candelatech.com> Date: Sun, 14 Oct 2001 11:27:31 -0700 From: Ben Greear Organization: Candela Technologies X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.4.3-12 i686) X-Accept-Language: en MIME-Version: 1.0 To: rsharpe@ns.aus.com CC: netdev@oss.sgi.com Subject: Re: Retrieving in.terface statistics frequently and without pain References: <3BC8F5DD.2050602@ns.aus.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 933 Lines: 32 Richard Sharpe wrote: > > Hi, > > What is the quickest way to retrieve interface statistics quickly and > without pain? cat /proc/net/dev It has plenty of stats..you just have to parse the ascii file to get the numbers you need... > > I am interested in seeing bytes in and out as well as packets in and > out. I am also interested in seeing collisions. > > If the interface was in promiscuous mode, would packets and bytes in/out > count total traffic on the wire? > > I notice that there is no SIOCGIFSTATS IOCTL. Would such a thing be > desirable? > > -- > Richard Sharpe, rsharpe@ns.aus.com, LPIC-1 > www.samba.org, www.ethereal.com, SAMS Teach Yourself Samba > in 24 Hours, Special Edition, Using Samba -- Ben Greear President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear From owner-netdev@oss.sgi.com Mon Oct 15 09:26:13 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9FGQD909286 for netdev-outgoing; Mon, 15 Oct 2001 09:26:13 -0700 Received: from castor.erda.se ([195.163.32.186]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9FGQAD09283 for ; Mon, 15 Oct 2001 09:26:11 -0700 Received: by castor.erda.se with Internet Mail Service (5.5.2653.19) id <49387BVS>; Mon, 15 Oct 2001 18:28:10 +0200 Message-ID: <499AC99D205FD51197CD0002A574C4F8052418@castor.erda.se> From: =?iso-8859-1?Q?=22Wikstr=F6m=2C_M=E5rten=22?= To: "'netdev@oss.sgi.com'" , "'linux-net@vger.kernel.org'" Subject: Re: SyncPPP vs. ppp_synctty Date: Mon, 15 Oct 2001 18:28:03 +0200 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by oss.sgi.com id f9FGQBD09284 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 522 Lines: 12 >What I think is we need just one PPP driver. Hovewer, ppp_generic >requires tty device (either async or sync) and insists on creating >network device. That isn't sane behavior if you just want existing >net device to talk PPP. >We should merge the drivers, AFAIK 2.5 is coming... But the ppp_generic requires a tty device, i.e. is character-oriented, where as the syncppp is network-oriented. Aren't they fundamentally different. How can they be merged? Wouldn't that require the network card to emulate a tty? /Mårten From owner-netdev@oss.sgi.com Mon Oct 15 11:56:00 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9FIu0Q11750 for netdev-outgoing; Mon, 15 Oct 2001 11:56:00 -0700 Received: from zcars0m9.nortelnetworks.com (zcars0m9.nortelnetworks.com [47.129.242.157]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9FIttD11747 for ; Mon, 15 Oct 2001 11:55:56 -0700 Received: from zcars04f.ca.nortel.com (zcars04f.ca.nortel.com [47.129.242.57]) by zcars0m9.nortelnetworks.com (8.11.0/8.11.0) with ESMTP id f9FItGS23348 for ; Mon, 15 Oct 2001 14:55:16 -0400 (EDT) Received: from zcard00m.ca.nortel.com by zcars04f.ca.nortel.com; Mon, 15 Oct 2001 14:55:28 -0400 Received: from zcard00f.ca.nortel.com ([47.129.30.8]) by zcard00m.ca.nortel.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id 4ZM97FZL; Mon, 15 Oct 2001 14:55:06 -0400 Received: from pcard0ks.ca.nortel.com ([47.129.117.131]) by zcard00f.ca.nortel.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id 41MMWK4L; Mon, 15 Oct 2001 14:55:05 -0400 Received: from nortelnetworks.com (localhost.localdomain [127.0.0.1]) by pcard0ks.ca.nortel.com (Postfix) with ESMTP id 6C8B04ED for ; Mon, 15 Oct 2001 14:57:03 -0400 (EDT) Message-ID: <3BCB317F.2FFAC442@nortelnetworks.com> Date: Mon, 15 Oct 2001 14:57:03 -0400 From: "Christopher Friesen" X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.4.3-custom i686) X-Accept-Language: en MIME-Version: 1.0 To: netdev@oss.sgi.com Subject: listing proxy ARP entries using "ip" command -- Alexey, help? Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Orig: Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1079 Lines: 37 Last month Lutz Pressler posted the following: //quote begins// I am not able to get information about manually set proxy ARP entries with the "ip" tool (iproute2-ss010824), tested on both 2.2.19 and 2.4.8-ac7 kernels. # ip neigh add proxy 172.20.1.1 dev eth0 The "arp" tool then yields (normal entries trimmed) # arp -n Address HWtype HWaddress Flags Mask Iface 192.168.1.254 ether 00:80:C8:F6:47:6F C eth0 172.20.1.1 * * MP eth0 but "ip" only shows # ip neigh show 192.168.1.254 dev eth0 lladdr 00:80:c8:f6:47:6f nud reachable Is this expected behaviour or a (kernel) bug? Alexey, is there any way to do this with "ip", or do we have to keep the "arp" tool around just to show manual proxy entries? Thanks, Chris -- Chris Friesen | MailStop: 043/33/F10 Nortel Networks | work: (613) 765-0557 3500 Carling Avenue | fax: (613) 765-2986 Nepean, ON K2H 8E9 Canada | email: cfriesen@nortelnetworks.com From owner-netdev@oss.sgi.com Mon Oct 15 12:11:54 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9FJBsb12485 for netdev-outgoing; Mon, 15 Oct 2001 12:11:54 -0700 Received: from router.megatech.cx ([213.96.96.147]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9FJBpD12482 for ; Mon, 15 Oct 2001 12:11:51 -0700 Received: (qmail 1703 invoked from network); 3 Sep 2001 12:04:36 -0000 Received: from unknown (HELO w2000.arrakis.es) (192.168.0.2) by router.megatech.cx with SMTP; 3 Sep 2001 12:04:36 -0000 Message-Id: <5.1.0.14.2.20011015205953.03061148@mail.arrakis.es> X-Sender: mrafael@mail.arrakis.es X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Mon, 15 Oct 2001 21:07:36 +0200 To: netdev@oss.sgi.com From: Mario Rafael Subject: Eliminating a skbuff Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 667 Lines: 18 Hi I am in the need to intercept certain skbuffs, the problem is that with dev_add_pack I only reaceive a copy of the skbuff, no previous or next pointer in the skbuff struct to do the pointer mangling thing, I have tryed kfree_skb and several other functions and nothing seems to happen. Is there any way I can eliminate the skbuff?. I have seen that in the dev struct there are pointers to several functions, one of them is particularly interesting, the hard_start_xmit, this function if I am not wrong is the one that receives and sends all the packets that pass through the nic. Any ideas?, could somebody help?... Thanks. e-Mail : mrafael@arrakis.es From owner-netdev@oss.sgi.com Mon Oct 15 13:15:19 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9FKFJf13773 for netdev-outgoing; Mon, 15 Oct 2001 13:15:19 -0700 Received: from navy.csi.cam.ac.uk (exim@navy.csi.cam.ac.uk [131.111.8.49]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9FKFBD13770 for ; Mon, 15 Oct 2001 13:15:14 -0700 Received: from ssb22.joh.cam.ac.uk ([131.111.142.162] ident=8) by navy.csi.cam.ac.uk with esmtp (Exim 3.22 #1) id 15tE8j-0001fI-00; Mon, 15 Oct 2001 21:15:09 +0100 Received: from ssb22 by ssb22.joh.cam.ac.uk with local (Exim 3.32 #1 (Debian)) id 15tE8o-0005Vh-00; Mon, 15 Oct 2001 21:15:14 +0100 From: "Silas S. Brown" MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15307.17362.517977.504430@ssb22.joh.cam.ac.uk> Date: Mon, 15 Oct 2001 21:15:14 +0100 To: netdev@oss.sgi.com CC: "D. J. Bernstein" , Ross.Anderson@cl.cam.ac.uk Subject: Linux networking: Security bug in SYN cookies X-Mailer: VM 6.92 under 21.4 (patch 4) "Artificial Intelligence" XEmacs Lucid X-URL: http://www.cus.cam.ac.uk/~ssb22/ X-Face: )ctKj[z).+~^{+H='}vNa[1VFbz`)oBmc~c}H$AM,q.A\ol ]:hBhx##s7f1LD`\[eZ-m_}JbOcW.[pF)tqX#!:sa>{9rsJ[hmzsUlp1;[6zMZ,F<;': muWf#[sJbSb~1x1w0}5 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1680 Lines: 43 Hi, There is a security bug in linux/net/ipv4/syncookies.c. It can be fixed. The following attack may be used to make connections to ports that are normally blocked by a SYN-blocking firewall, if the victim host runs SYN cookies as implemented in the Linux kernel. First, find a port on the victim host that is open to the world (say a Web server). Begin SYN-flooding that port. Then send millions of random ACK packets to the port that is blocked by the firewall (finger, FTP, telnet, or whatever). A SYN-blocking firewall would not block ACK packets (because they could be part of outgoing TCP connections) and would thus let them in, while also letting in the flood of SYN packets to the public port. So the victim host sees a SYN queue overflow, starts using SYN cookies, and has a reasonably high probability of being fooled by one of the random ACK packets into behaving as if a connection on the firewalled port had been established. The attacker can then continue with the connection as though the SYN-blocking firewall weren't there. This could be used to gain private information about a site (e.g. by querying a finger daemon that is not normally visible to the world), or for compromising a specific target that may have vulnerable daemons that are blocked by the firewall. The solution (as pointed out by D. J. Bernstein in a private communication in response to the above) is to make the variable tcp_lastsynq_overflow local to each listening port, instead of being a global. Best wishes, Silas -- Silas S Brown, St John's College Cambridge UK http://www.cus.cam.ac.uk/~ssb22 "Have a reputation for being reasonable" - Philippians 4:5, Phillip's From owner-netdev@oss.sgi.com Tue Oct 16 03:48:56 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9GAmuw28247 for netdev-outgoing; Tue, 16 Oct 2001 03:48:56 -0700 Received: from hq.pm.waw.pl (hq.pm.waw.pl [195.116.170.10]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9GAmrD28241 for ; Tue, 16 Oct 2001 03:48:53 -0700 Received: (from uucp@localhost) by hq.pm.waw.pl with UUCP id f9GAloG22993; Tue, 16 Oct 2001 12:47:50 +0200 Received: (from uucp@localhost) by intrepid.pm.waw.pl (8.11.6/8.11.6) with UUCP id f9GAjp600787; Tue, 16 Oct 2001 12:45:51 +0200 Received: (from khc@localhost) by defiant.pm.waw.pl (8.11.6/8.11.6) id f9FFOPb05813; Mon, 15 Oct 2001 17:24:25 +0200 To: "=?iso-8859-1?q?Wikstr=F6m,_M=E5rten?=" Cc: "''linux-net@vger.kernel.org ' '" , "''netdev@oss.sgi.com ' '" Subject: Re: SyncPPP vs. ppp_synctty References: <499AC99D205FD51197CD0002A574C4F8052416@castor.erda.se> From: Krzysztof Halasa Date: 15 Oct 2001 17:24:04 +0200 In-Reply-To: "=?iso-8859-1?q?Wikstr=F6m,_M=E5rten?="'s message of "Mon, 15 Oct 2001 10:25:34 +0200" Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by oss.sgi.com id f9GAmsD28242 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 514 Lines: 12 "Wikström, Mårten" writes: > But the ppp_generic requires a tty device, i.e. is character-oriented, where > as the syncppp is network-oriented. Aren't they fundamentally different. How > can they be merged? Wouldn't that require the network card to emulate a tty? PPP as a whole _is_ network-oriented. generic_ppp currently needs a tty device, but I think it just should be changed to provide PPP services to existing network devices as well. -- Krzysztof Halasa Network Administrator From owner-netdev@oss.sgi.com Tue Oct 16 08:07:23 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9GF7No01139 for netdev-outgoing; Tue, 16 Oct 2001 08:07:23 -0700 Received: from mortlach.security.kpnqwest.com (mortlach.security.kpnqwest.com [193.141.42.138]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9GF7JD01136 for ; Tue, 16 Oct 2001 08:07:20 -0700 Received: from spey.security.kpnqwest.com (spey.security.kpnqwest.com [193.141.42.190]) by mortlach.security.kpnqwest.com (8.12.0.Beta19/8.12.0.Beta19/Debian 8.12.0.Beta19) with ESMTP id f9GF7W0r017196; Tue, 16 Oct 2001 17:07:32 +0200 Received: from glenlivet.xsec.xlink.net (glenlivet.xsec.xlink.net [172.23.42.199]) by spey.xsec.xlink.net (8.12.0.Beta19/8.12.0.Beta19/Debian 8.12.0.Beta19) with ESMTP id f9GF7Ckr016603; Tue, 16 Oct 2001 17:07:12 +0200 Received: (from danisch@localhost) by glenlivet.xsec.xlink.net (8.12.0.Beta19/8.12.0.Beta19/Debian 8.12.0.Beta19) id f9GF7CI0012343; Tue, 16 Oct 2001 17:07:12 +0200 From: Hadmut Danisch Date: Tue, 16 Oct 2001 17:07:12 +0200 To: andreasm@uow.edu.au, netdev@oss.sgi.com Subject: Forcing 3c905 into halfduplex? Message-ID: <20011016170712.A12338@xlink.net> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline User-Agent: Mutt/1.3.20i Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 346 Lines: 18 Hi, I just have some trouble with a 3c905 (Dell Optiplex PC with 3c905 in motherboard) and a CenterCom hub (Linux 2.4.12). The hub doesn't stand full duplex, but the 3c905 always goes into full duplex when using 100 MBit. The module parameters allow forcing into full duplex, but is there a way to force into half duplex? regards Hadmut From owner-netdev@oss.sgi.com Wed Oct 17 04:39:25 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9HBdPw26962 for netdev-outgoing; Wed, 17 Oct 2001 04:39:25 -0700 Received: from vaio.greennet (pool-151-196-235-45.balt.east.verizon.net [151.196.235.45]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9HBdLD26952 for ; Wed, 17 Oct 2001 04:39:21 -0700 Received: from localhost (becker@localhost) by vaio.greennet (8.9.3/8.8.7) with ESMTP id HAA02165; Wed, 17 Oct 2001 07:37:41 -0400 Date: Wed, 17 Oct 2001 07:37:41 -0400 (EDT) From: Donald Becker X-Sender: becker@vaio.greennet To: Hadmut Danisch cc: andreasm@uow.edu.au, netdev@oss.sgi.com Subject: Re: Forcing 3c905 into halfduplex? In-Reply-To: <20011016170712.A12338@xlink.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1179 Lines: 35 On Tue, 16 Oct 2001, Hadmut Danisch wrote: > I just have some trouble with a 3c905 > (Dell Optiplex PC with 3c905 in motherboard) > and a CenterCom hub (Linux 2.4.12). > > The hub doesn't stand full duplex, but the > 3c905 always goes into full duplex when > using 100 MBit. How do you know this? The output of 'vortex-diag'? Note that the card's EEPROM may be configured to forced full duplex mode. This would have been programmed explicitly, the EEPROM doesn't get reprogrammed by accident. > The module parameters allow forcing into > full duplex, but is there a way to force > into half duplex? The Ethernet default is half duplex. The only standard-conforming method of doing full duplex over twisted pair is if the duplex is autonegotiated. The option to force full duplex is to handle the obscure/bogus cases, such where you your Cisco administrator set the equipment to forced full duplex because Cisco shipped broken autonegotiation and doesn't want to replace the hardware. Donald Becker becker@scyld.com Scyld Computing Corporation http://www.scyld.com 410 Severn Ave. Suite 210 Second Generation Beowulf Clusters Annapolis MD 21403 410-990-9993 From owner-netdev@oss.sgi.com Wed Oct 17 06:39:59 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9HDdx531477 for netdev-outgoing; Wed, 17 Oct 2001 06:39:59 -0700 Received: from castor.erda.se ([195.163.32.186]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9HDduD31473 for ; Wed, 17 Oct 2001 06:39:56 -0700 Received: from [157.125.19.91] (157.125.19.91 [157.125.19.91]) by castor.erda.se with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id VAHR41WF; Wed, 17 Oct 2001 15:41:57 +0200 Subject: Re: SyncPPP vs. ppp_synctty From: =?ISO-8859-1?Q?M=E5rten_Wikstr=F6m?= To: Krzysztof Halasa Cc: "\"''linux-net@vger.kernel.org \"' '" , "\"''netdev@oss.sgi.com ' \"'" In-Reply-To: References: <499AC99D205FD51197CD0002A574C4F8052416@castor.erda.se> Content-Type: text/plain; charset=ISO-8859-1 X-Mailer: Evolution/0.15 (Preview Release) Date: 17 Oct 2001 15:39:44 +0200 Message-Id: <1003325988.3359.9.camel@upp-w-marwik.erda.se> Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by oss.sgi.com id f9HDdvD31475 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 677 Lines: 16 mån 2001-10-15 klockan 17.24 skrev Krzysztof Halasa: > "Wikström, Mårten" writes: > > > But the ppp_generic requires a tty device, i.e. is character-oriented, where > > as the syncppp is network-oriented. Aren't they fundamentally different. How > > can they be merged? Wouldn't that require the network card to emulate a tty? > > PPP as a whole _is_ network-oriented. generic_ppp currently needs > a tty device, but I think it just should be changed to provide PPP > services to existing network devices as well. Then we are talking about at major change in ppp_generic, since it relies quite heavily on the tty paradigm. Or am I wrong? /Mårten From owner-netdev@oss.sgi.com Wed Oct 17 07:03:04 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9HE34J31967 for netdev-outgoing; Wed, 17 Oct 2001 07:03:04 -0700 Received: from zcars0m9.nortelnetworks.com (zcars0m9.nortelnetworks.com [47.129.242.157]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9HE2xD31963 for ; Wed, 17 Oct 2001 07:02:59 -0700 Received: from zcars04e.ca.nortel.com (zcars04e.ca.nortel.com [47.129.242.56]) by zcars0m9.nortelnetworks.com (8.11.0/8.11.0) with ESMTP id f9HE2IS07832 for ; Wed, 17 Oct 2001 10:02:18 -0400 (EDT) Received: from zcard00m.ca.nortel.com by zcars04e.ca.nortel.com; Wed, 17 Oct 2001 10:02:35 -0400 Received: from zcard00f.ca.nortel.com ([47.129.30.8]) by zcard00m.ca.nortel.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id 40SSC33V; Wed, 17 Oct 2001 10:02:12 -0400 Received: from pcard0ks.ca.nortel.com ([47.129.117.131]) by zcard00f.ca.nortel.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id 41MMWNNK; Wed, 17 Oct 2001 10:02:11 -0400 Received: from nortelnetworks.com (localhost.localdomain [127.0.0.1]) by pcard0ks.ca.nortel.com (Postfix) with ESMTP id 45F5C4ED for ; Wed, 17 Oct 2001 10:04:16 -0400 (EDT) Message-ID: <3BCD8FE0.9CF3908C@nortelnetworks.com> Date: Wed, 17 Oct 2001 10:04:16 -0400 From: "Christopher Friesen" X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.4.3-custom i686) X-Accept-Language: en MIME-Version: 1.0 To: netdev@oss.sgi.com Subject: listing manual proxy ARP entries using "ip" command -- anyone, help? References: <3BCB317F.2FFAC442@nortelnetworks.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Orig: Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1309 Lines: 42 I posted the following a couple days back and got the message myself, but I have received no responses from anyone so I'm wondering if anyone else actually got the message. Can anyone tell me whether this desired behaviour or if not then will it be fixed eventually? Thanks, Chris "Friesen, Christopher [CAR:3R60:EXCH]" wrote: > > Last month Lutz Pressler posted the following: > > //quote begins// > I am not able to get information about manually set proxy ARP entries > with the "ip" tool (iproute2-ss010824), tested on both 2.2.19 and > 2.4.8-ac7 kernels. > > # ip neigh add proxy 172.20.1.1 dev eth0 > > The "arp" tool then yields (normal entries trimmed) > # arp -n > Address HWtype HWaddress Flags Mask Iface > 192.168.1.254 ether 00:80:C8:F6:47:6F C eth0 > 172.20.1.1 * * MP eth0 > > but "ip" only shows > # ip neigh show > 192.168.1.254 dev eth0 lladdr 00:80:c8:f6:47:6f nud reachable > > Is this expected behaviour or a (kernel) bug? -- Chris Friesen | MailStop: 043/33/F10 Nortel Networks | work: (613) 765-0557 3500 Carling Avenue | fax: (613) 765-2986 Nepean, ON K2H 8E9 Canada | email: cfriesen@nortelnetworks.com From owner-netdev@oss.sgi.com Thu Oct 18 11:50:17 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9IIoHW03194 for netdev-outgoing; Thu, 18 Oct 2001 11:50:17 -0700 Received: from ms2.inr.ac.ru (minus.inr.ac.ru [193.233.7.97]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9IIoED03191 for ; Thu, 18 Oct 2001 11:50:15 -0700 Received: (from kuznet@localhost) by ms2.inr.ac.ru (8.6.13/ANK) id WAA04584; Thu, 18 Oct 2001 22:49:59 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200110181849.WAA04584@ms2.inr.ac.ru> Subject: Re: Question about ISN regeneration when Stateless SYN cookies are used To: nallu17@hotmail.COM (gangadharan annapurna) Date: Thu, 18 Oct 2001 22:49:59 +0400 (MSK DST) Cc: netdev@oss.sgi.com In-Reply-To: from "gangadharan annapurna" at Oct 18, 1 11:15:22 am X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 193 Lines: 9 Hello! > Now when the new SYN+ACK arrives and if the new ISN falls > within the Receive window of the client, then the packet > is wrongly accepted. It is not accepted but resetted. Alexey From owner-netdev@oss.sgi.com Thu Oct 18 16:40:05 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9INe5S09116 for netdev-outgoing; Thu, 18 Oct 2001 16:40:05 -0700 Received: from www.vsol.net ([216.18.21.14]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9INdwD09111 for ; Thu, 18 Oct 2001 16:39:58 -0700 Received: from localhost (root@localhost) by www.vsol.net (8.11.6/8.9.3) with ESMTP id f9INduN20961; Thu, 18 Oct 2001 16:39:56 -0700 Date: Thu, 18 Oct 2001 16:39:56 -0700 (PDT) From: To: , , , , Subject: Linux Kernel 2.4.10, arp -s doesn't work? Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 2366 Lines: 65 Hello! I'm having a problem with proxy arp. In short, I can't make it work: # ifconfig eth0 2.2.2.2 # arp -Ds 2.2.2.3 eth0 pub # arp -an ? (2.2.2.3) at * PERM PUP on eth0 # tcpdump -n arp 16:08:29.828977 B arp who-has 2.2.2.3 tell 2.2.2.254 16:08:30.314221 B arp who-has 2.2.2.3 tell 2.2.2.254 16:08:31.837216 B arp who-has 2.2.2.3 tell 2.2.2.254 16:08:32.272723 B arp who-has 2.2.2.3 tell 2.2.2.254 # Ok, can anybody tell me why there are no arp replies? I'm expecting to see: 16:08:29.833205 > arp reply 2.2.2.3 (0:3:2d:0:5:90) is-at 0:3:2d:0:5:90 (0:d0:c0:a9:b0:0) The application: this is a firewall, using NAT. I'm trying to give some folks behind the firewall full access to the world, and vice versa. Yes, this is not good, security-wise, but customers get what customers want. The machines behind the firewall can't use the automatic proxy arp feature in the kernel because their ips aren't real, and wouldn't make much sense on the outside. The solution I want to use: IP3=2.2.2.3 arp -Ds 2.2.2.3 eth0 pub iptables -A PREROUTING -t nat -d $IP3 -j DNAT --to 10.10.10.191 iptables -A POSTROUTING -t nat -s 10.10.10.191 -j SNAT --to-source $IP3 The solution I have to use: IP3=2.2.2.3 ifconfig eth0:3 $IP3 iptables -A PREROUTING -t nat -d $IP3 -j DNAT --to 10.10.10.191 iptables -A POSTROUTING -t nat -s 10.10.10.191 -j SNAT --to-source $IP3 This is the only way I can see of getting arp replies to be sent, and it looks evil. I saw 'Proxy ARP for Linux', http://www.sjdjweis.com/linux/proxyarp/ but it doesn't use the 'arp' command. I saw http://www.uwsg.indiana.edu/hypermail/linux/kernel/0004.0/0909.html but I don't need to set random mac addresses, although that would be neat. I saw http://www.uwsg.indiana.edu/hypermail/linux/kernel/9901.0/0252.html but I don't need netmasks for my arp command, although that would solve a different problem I had once upon a time. I'm using RedHat 6.2, and 'arp --version' says net-tools 1.50 arp 1.85 (1999-01-05) I'm assuming that since 'arp -an' shows the entry, the linux kernel got the information, and the bug is in the kernel. In short, is this a bug? Or am I doing something wrong? -- N Fudd -- nfudd@mellor.bc.ca I heard that if you play the Windows CD backward, you get a satanic message. But that's nothing compared to when you play it forward: It installs Windows. From owner-netdev@oss.sgi.com Fri Oct 19 08:14:15 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9JFEFR30111 for netdev-outgoing; Fri, 19 Oct 2001 08:14:15 -0700 Received: from smp.paktronix.com (pak200.pakuni.net [207.91.34.200]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9JFE8D30104 for ; Fri, 19 Oct 2001 08:14:08 -0700 Received: from netmonster.pakint.net (netmonster [192.168.3.13]) by smp.paktronix.com (8.9.3/8.9.3) with ESMTP id JAA05114; Fri, 19 Oct 2001 09:56:21 -0500 Date: Fri, 19 Oct 2001 10:23:39 -0500 (CDT) From: "Matthew G. Marsh" X-X-Sender: To: cc: Subject: Re: Linux Kernel 2.4.10, arp -s doesn't work? In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 2312 Lines: 70 On Thu, 18 Oct 2001 nfudd@mellor.bc.ca wrote: > Hello! > > I'm having a problem with proxy arp. In short, I can't make it work: [snip] > The application: this is a firewall, using NAT. I'm trying to give some > folks behind the firewall full access to the world, and vice versa. Yes, > this is not good, security-wise, but customers get what customers want. > > The machines behind the firewall can't use the automatic proxy arp > feature in the kernel because their ips aren't real, and wouldn't make > much sense on the outside. Not proxy arp. Proxy arp is arp on behalf of an _existing_ address located elsewhere. You are merely trying to do One-2-One NAT. See below. > The solution I have to use: > IP3=2.2.2.3 > ifconfig eth0:3 $IP3 Do not use coloned interfaces. Deprecated. Should be removed already. Instead use: ip addr add ${IP3}/32 dev eth0 Then arp will work correctly and so will the following NAT. > iptables -A PREROUTING -t nat -d $IP3 -j DNAT --to 10.10.10.191 > iptables -A POSTROUTING -t nat -s 10.10.10.191 -j SNAT --to-source $IP3 > > This is the only way I can see of getting arp replies to be sent, and > it looks evil. Must be so. You are _not_ doing proxy arp. Proxy arp would be if you actually had one of the customers machines assigned the 2.2.2.3 address for real. [snip] > I'm assuming that since 'arp -an' shows the entry, the linux kernel > got the information, and the bug is in the kernel. Yes and no. The entry is made but there is no route to the address because the address does not exist. Make it exist and give a route entry or just assign the address to the interface. > In short, is this a bug? Or am I doing something wrong? Not a bug. ;-} Definitions are exact. Proxy arp is for a machine that exists and has address assigned. 1-2-1 NAT is for case you are doing. > -- > N Fudd -- nfudd@mellor.bc.ca > I heard that if you play the Windows CD backward, you get a satanic message. > But that's nothing compared to when you play it forward: It installs Windows. -------------------------------------------------- Matthew G. Marsh, President Paktronix Systems LLC 1506 North 59th Street Omaha NE 68104 Phone: (402) 932-7250 x101 Email: mgm@paktronix.com WWW: http://www.paktronix.com -------------------------------------------------- From owner-netdev@oss.sgi.com Fri Oct 19 10:39:55 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9JHdtN02193 for netdev-outgoing; Fri, 19 Oct 2001 10:39:55 -0700 Received: from ms2.inr.ac.ru (minus.inr.ac.ru [193.233.7.97]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9JHdqD02190 for ; Fri, 19 Oct 2001 10:39:52 -0700 Received: (from kuznet@localhost) by ms2.inr.ac.ru (8.6.13/ANK) id VAA03704; Fri, 19 Oct 2001 21:39:08 +0400 From: kuznet@ms2.inr.ac.ru Message-Id: <200110191739.VAA03704@ms2.inr.ac.ru> Subject: Re: Linux Kernel 2.4.10, arp -s doesn't work? To: nfudd@mellor.bc.ca Date: Fri, 19 Oct 2001 21:39:08 +0400 (MSK DST) Cc: netdev@oss.sgi.com, davem@redhat.com, ak@muc.de, pekkas@netcore.fi In-Reply-To: from "nfudd@mellor.bc.ca" at Oct 18, 1 04:39:56 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 305 Lines: 17 Hello! > # ifconfig eth0 2.2.2.2 > # arp -Ds 2.2.2.3 eth0 pub I think it is because the query is on eth0 too. > The solution I have to use: > IP3=2.2.2.3 > ifconfig eth0:3 $IP3 I have no idea why it works, to be honest. It must not, if my guess is right. Where do you query for 2.2.2.3? Alexey From owner-netdev@oss.sgi.com Fri Oct 19 11:11:05 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9JIB5U03237 for netdev-outgoing; Fri, 19 Oct 2001 11:11:05 -0700 Received: from tux.rsn.bth.se (root@tux.rsn.bth.se [194.47.143.135]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9JIB2D03233 for ; Fri, 19 Oct 2001 11:11:02 -0700 Received: from localhost (gandalf@localhost [127.0.0.1]) by localhost (8.12.0.Beta19/8.12.0.Beta19/Debian 8.12.0.Beta19) with ESMTP id f9JI9oZd030578; Fri, 19 Oct 2001 20:09:51 +0200 Date: Fri, 19 Oct 2001 20:09:50 +0200 (CEST) From: Martin Josefsson X-Sender: gandalf@tux.rsn.bth.se To: kuznet@ms2.inr.ac.ru cc: nfudd@mellor.bc.ca, netdev@oss.sgi.com, davem@redhat.com, ak@muc.de, pekkas@netcore.fi Subject: Re: Linux Kernel 2.4.10, arp -s doesn't work? In-Reply-To: <200110191739.VAA03704@ms2.inr.ac.ru> Message-ID: X-message-flag: Get yourself a real mail client! http://www.washington.edu/pine/ MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 614 Lines: 25 On Fri, 19 Oct 2001 kuznet@ms2.inr.ac.ru wrote: > Hello! > > > # ifconfig eth0 2.2.2.2 > > # arp -Ds 2.2.2.3 eth0 pub > > I think it is because the query is on eth0 too. > > > > The solution I have to use: > > IP3=2.2.2.3 > > ifconfig eth0:3 $IP3 > > I have no idea why it works, to be honest. It must not, if my guess is right. > > Where do you query for 2.2.2.3? I had a problem where the proxyarps wasn't getting added but I got no errormessage from arp. It started working when I upgraded arp. /Martin Never argue with an idiot. They drag you down to their level, then beat you with experience. From owner-netdev@oss.sgi.com Fri Oct 19 15:13:30 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9JMDUs08820 for netdev-outgoing; Fri, 19 Oct 2001 15:13:30 -0700 Received: from acmay.homeip.net (IDENT:qmailr@24-25-196-177.san.rr.com [24.25.196.177]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9JMDRD08817 for ; Fri, 19 Oct 2001 15:13:27 -0700 Received: (qmail 13593 invoked by uid 500); 19 Oct 2001 22:13:26 -0000 Date: Fri, 19 Oct 2001 15:13:26 -0700 From: andrew may To: kernel list Cc: netdev@oss.sgi.com Subject: 2.4.12 problems with routes Message-ID: <20011019151326.A13503@ecam.san.rr.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0pre3us Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1295 Lines: 32 I have a strange problem in 2.4.12 that a ping from my machine 172.25.4.9 to 192.168.12.1 works but a ping the other way does not. The ping going out goes to the correct route but a ping reply from the other machine goes to the default route. >From my setup, and a dump of 2 packets that show that MAC dst. On the echo req it has the correct dst, but on the echo reply the dst is the default route. The output is from 2 seperate pings and I cut down on the output. Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 172.25.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 192.168.0.0 172.25.4.17 255.255.0.0 UG 0 0 0 eth0 0.0.0.0 172.25.3.1 0.0.0.0 UG 0 0 0 eth0 pie:~# arp -an ? (172.25.3.1) at 00:20:9C:13:08:24 [ether] on eth0 ? (172.25.4.15) at 00:D0:B7:BF:D3:37 [ether] on eth0 ? (172.25.4.17) at 00:60:08:A6:A4:3D [ether] on eth0 pie:~# tcpdump -r bad-mac -n -v -e 14:47:49.275170 0:d0:b7:3f:af:5 0:20:9c:13:8:24 0800 98: 172.25.4.9 > 192.168.12.1: icmp: echo reply (ttl 255, id 12687, len 84) 14:47:51.389840 0:d0:b7:3f:af:5 0:60:8:a6:a4:3d 0800 98: 172.25.4.9 > 192.168.12.1: icmp: echo request (DF) (ttl 64, id 0, len 84) pie:~# route -n From owner-netdev@oss.sgi.com Fri Oct 19 15:36:52 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9JMaqx09445 for netdev-outgoing; Fri, 19 Oct 2001 15:36:52 -0700 Received: from www.vsol.net ([216.18.21.14]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9JMalD09441 for ; Fri, 19 Oct 2001 15:36:47 -0700 Received: from localhost (root@localhost) by www.vsol.net (8.11.6/8.9.3) with ESMTP id f9JMai103440; Fri, 19 Oct 2001 15:36:44 -0700 Date: Fri, 19 Oct 2001 15:36:44 -0700 (PDT) From: To: "Matthew G. Marsh" cc: Subject: Re: Linux Kernel 2.4.10, arp -s doesn't work? In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1346 Lines: 39 On Fri, 19 Oct 2001, Matthew G. Marsh wrote: > Do not use coloned interfaces. Deprecated. Should be removed already. > Instead use: > > ip addr add ${IP3}/32 dev eth0 > > Then arp will work correctly and so will the following NAT. Thank you! I still worry about having an interface with $IP3's number on the firewall. > > iptables -A PREROUTING -t nat -d $IP3 -j DNAT --to 10.10.10.191 > > iptables -A POSTROUTING -t nat -s 10.10.10.191 -j SNAT --to-source $IP3 > > > > This is the only way I can see of getting arp replies to be sent, and > > it looks evil. > > Must be so. You are _not_ doing proxy arp. Proxy arp would be if you > actually had one of the customers machines assigned the 2.2.2.3 address > for real. 'arp -s' doesn't seem to do anything useful anymore, does it? > > In short, is this a bug? Or am I doing something wrong? > > Not a bug. ;-} Definitions are exact. Proxy arp is for a machine that > exists and has address assigned. 1-2-1 NAT is for case you are doing. Where can I find more information on one-to-one NAT? -- Charles Howes -- chowes@vsol.net "The personal computer allows you to make more mistakes faster than any other invention in human history, with the possible exceptions of handguns and tequila." (It's the mistakes made with handguns, computers *and* tequila that are really spectacular!) From owner-netdev@oss.sgi.com Fri Oct 19 20:57:49 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9K3vnQ14593 for netdev-outgoing; Fri, 19 Oct 2001 20:57:49 -0700 Received: from smp.paktronix.com (pak200.pakuni.net [207.91.34.200]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9K3vfD14590 for ; Fri, 19 Oct 2001 20:57:42 -0700 Received: from netmonster.pakint.net (netmonster [192.168.3.13]) by smp.paktronix.com (8.9.3/8.9.3) with ESMTP id WAA07457; Fri, 19 Oct 2001 22:40:07 -0500 Date: Fri, 19 Oct 2001 22:57:58 -0500 (CDT) From: "Matthew G. Marsh" X-X-Sender: To: cc: Subject: Re: Linux Kernel 2.4.10, arp -s doesn't work? In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 3145 Lines: 82 On Fri, 19 Oct 2001 nfudd@mellor.bc.ca wrote: > On Fri, 19 Oct 2001, Matthew G. Marsh wrote: > > > Do not use coloned interfaces. Deprecated. Should be removed already. > > Instead use: > > > > ip addr add ${IP3}/32 dev eth0 > > > > Then arp will work correctly and so will the following NAT. > > Thank you! I still worry about having an interface with $IP3's number > on the firewall. Yes - that is one of the reasons to be careful in what is allowed to that address using NetFilter. > > > iptables -A PREROUTING -t nat -d $IP3 -j DNAT --to 10.10.10.191 > > > iptables -A POSTROUTING -t nat -s 10.10.10.191 -j SNAT --to-source $IP3 > > > > > > This is the only way I can see of getting arp replies to be sent, and > > > it looks evil. > > > > Must be so. You are _not_ doing proxy arp. Proxy arp would be if you > > actually had one of the customers machines assigned the 2.2.2.3 address > > for real. > > 'arp -s' doesn't seem to do anything useful anymore, does it? > > > > In short, is this a bug? Or am I doing something wrong? > > > > Not a bug. ;-} Definitions are exact. Proxy arp is for a machine that > > exists and has address assigned. 1-2-1 NAT is for case you are doing. > > Where can I find more information on one-to-one NAT? Actually 1-2-1 NAT is merely shorthand to distinguish which NAT I was talking about. NAT essentially comes in two flavours: 1-2-1 is where one ip address is uniquely mapped onto another ip address Many-2-1 is where multiple ip addresses are mapped onto one ip address (covers both 1-2-Many and Many-2-1 mappings) 1-2-1 is traditionally thought of as a "routed NAT" where a router performs the unique change of addresses. Many-2-1 is what is thought of as "IP Masquerade" Both functions are available with the same NetFilter commands. Additionally 1-2-1 NAT is done by the FastNAT structures that are part of the RPDB within Linux kernels. However NetFilter conntrack is not compatible with FastNAT and thus if you use NetFilter conntrack then you cannot use FastNAT. For your case you would be better off using NetFilter NAT with conntrack in order to also apply control to the clients passthru. You an use FastNAT with NetFilter filters (as weirdos such as myself are wont to do... ;-} ), but for standard NetFilter usage such as you need, it is far easier (and you can ask people on this list for help) to use the NetFilter 1-2-1 setup. I do think that someone also posted a patch that allows you to do 1-2-1 NAT over a range correctly. > -- > Charles Howes -- chowes@vsol.net > "The personal computer allows you to make more mistakes faster than > any other invention in human history, with the possible exceptions of > handguns and tequila." > (It's the mistakes made with handguns, computers *and* tequila that > are really spectacular!) ROFL! (having seen and/or participated in such mistakes...) -------------------------------------------------- Matthew G. Marsh, President Paktronix Systems LLC 1506 North 59th Street Omaha NE 68104 Phone: (402) 932-7250 x101 Email: mgm@paktronix.com WWW: http://www.paktronix.com -------------------------------------------------- From owner-netdev@oss.sgi.com Mon Oct 22 01:01:27 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9M81RB28185 for netdev-outgoing; Mon, 22 Oct 2001 01:01:27 -0700 Received: from www.vsol.net ([216.18.21.14]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9M81MD28181 for ; Mon, 22 Oct 2001 01:01:22 -0700 Received: from localhost (root@localhost) by www.vsol.net (8.11.6/8.9.3) with ESMTP id f9M81Jt26599; Mon, 22 Oct 2001 01:01:19 -0700 Date: Mon, 22 Oct 2001 01:01:19 -0700 (PDT) From: To: "Matthew G. Marsh" cc: Subject: Re: Linux Kernel 2.4.10, arp -s doesn't work? In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 2355 Lines: 53 On Fri, 19 Oct 2001, Matthew G. Marsh wrote: > > Where can I find more information on one-to-one NAT? > > Actually 1-2-1 NAT is merely shorthand to distinguish which NAT I was > talking about. NAT essentially comes in two flavours: > > 1-2-1 is where one ip address is uniquely mapped onto another ip address > > Many-2-1 is where multiple ip addresses are mapped onto one ip address > (covers both 1-2-Many and Many-2-1 mappings) > > 1-2-1 is traditionally thought of as a "routed NAT" where a router > performs the unique change of addresses. > > Many-2-1 is what is thought of as "IP Masquerade" > > Both functions are available with the same NetFilter commands. > Additionally 1-2-1 NAT is done by the FastNAT structures that are part of > the RPDB within Linux kernels. However NetFilter conntrack is not > compatible with FastNAT and thus if you use NetFilter conntrack then you > cannot use FastNAT. For your case you would be better off using NetFilter > NAT with conntrack in order to also apply control to the clients passthru. > > You an use FastNAT with NetFilter filters (as weirdos such as myself are > wont to do... ;-} ), but for standard NetFilter usage such as you need, it > is far easier (and you can ask people on this list for help) to use the > NetFilter 1-2-1 setup. I do think that someone also posted a patch that > allows you to do 1-2-1 NAT over a range correctly. Pardon me, but my eyes glazed over. Let me get this straight. There is FastNAT, and there is NetFilter. You pick one or the other when compiling the kernel (somehow). You can load them as modules (somehow). Which things are incompatible and should not be used together? What happens if you accidentally use them together? And is there a manual someplace, or maybe a test suite... > > "The personal computer allows you to make more mistakes faster than > > any other invention in human history, with the possible exceptions of > > handguns and tequila." > > (It's the mistakes made with handguns, computers *and* tequila that > > are really spectacular!) > > ROFL! (having seen and/or participated in such mistakes...) Yikes... what hardware got bullet-holed? :-) -- N Fudd -- nfudd@mellor.bc.ca Methuselah lived to be 969 years old. You boys and girls will see more in the next fifty years than Methuselah saw in his whole lifetime. - Mark Twain From owner-netdev@oss.sgi.com Mon Oct 22 09:08:10 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9MG8Ak06743 for netdev-outgoing; Mon, 22 Oct 2001 09:08:10 -0700 Received: from smp.paktronix.com (pak200.pakuni.net [207.91.34.200]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9MG81D06739 for ; Mon, 22 Oct 2001 09:08:01 -0700 Received: from netmonster.pakint.net (netmonster [192.168.3.13]) by smp.paktronix.com (8.9.3/8.9.3) with ESMTP id KAA13038; Mon, 22 Oct 2001 10:50:51 -0500 Date: Mon, 22 Oct 2001 11:08:40 -0500 (CDT) From: "Matthew G. Marsh" X-X-Sender: To: cc: Subject: Re: Linux Kernel 2.4.10, arp -s doesn't work? In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 4733 Lines: 99 On Mon, 22 Oct 2001 nfudd@mellor.bc.ca wrote: > On Fri, 19 Oct 2001, Matthew G. Marsh wrote: > > > > Where can I find more information on one-to-one NAT? > > > > Actually 1-2-1 NAT is merely shorthand to distinguish which NAT I was > > talking about. NAT essentially comes in two flavours: > > > > 1-2-1 is where one ip address is uniquely mapped onto another ip address > > > > Many-2-1 is where multiple ip addresses are mapped onto one ip address > > (covers both 1-2-Many and Many-2-1 mappings) > > > > 1-2-1 is traditionally thought of as a "routed NAT" where a router > > performs the unique change of addresses. > > > > Many-2-1 is what is thought of as "IP Masquerade" > > > > Both functions are available with the same NetFilter commands. > > Additionally 1-2-1 NAT is done by the FastNAT structures that are part of > > the RPDB within Linux kernels. However NetFilter conntrack is not > > compatible with FastNAT and thus if you use NetFilter conntrack then you > > cannot use FastNAT. For your case you would be better off using NetFilter > > NAT with conntrack in order to also apply control to the clients passthru. > > > > You an use FastNAT with NetFilter filters (as weirdos such as myself are > > wont to do... ;-} ), but for standard NetFilter usage such as you need, it > > is far easier (and you can ask people on this list for help) to use the > > NetFilter 1-2-1 setup. I do think that someone also posted a patch that > > allows you to do 1-2-1 NAT over a range correctly. > > Pardon me, but my eyes glazed over. Let me get this straight. There > is FastNAT, and there is NetFilter. You pick one or the other when > compiling the kernel (somehow). You can load them as modules > (somehow). Which things are incompatible and should not be used together? > What happens if you accidentally use them together? And is there a > manual someplace, or maybe a test suite... The only one which can (and should) be modularized is the NetFilter conntrack. Indeed so long as the conntrack module is not loaded you can use the NAT capabilities of the routing code (this code is referred to as FastNAT). The part to remember is that NetFilter is actually a set of hooks build into the kernel which allow various software and related modules to provide "firewall" capabilities to Linux. This is a good thing as the extensibility of such a framework is available to anyone who wishes to program to those intefaces. However do not confuse this framework with the programs that allow you to perform tasks - such as iptables and conntrack. The framework is compatible with and coexists quite well with the routing code within the kernel - this is the RPDB (Routing Protocol DataBase) which provides both simple (the standard route/ifconfig) IP networking as well as (iproute2) advanced IP networking. The one point of friction - which is what I refer to when I speak of incompatibility in the context of this thread - is that NetFilter conntrack, an external module to the kernel, "breaks" the RPDB sections that allow FastNAT. This is not a bad thing due to the fact that conntrack provides stateful inspection by reason of providing the notion of "Connection" as a variable to the packet filtering engine (NetFilter). Indeed I use both types of NAT in different situations depending on the job I wish to get done. Neither one is a panacea for all NAT. > > > "The personal computer allows you to make more mistakes faster than > > > any other invention in human history, with the possible exceptions of > > > handguns and tequila." > > > (It's the mistakes made with handguns, computers *and* tequila that > > > are really spectacular!) > > > > ROFL! (having seen and/or participated in such mistakes...) > > Yikes... what hardware got bullet-holed? :-) Hehehehe - I remember when a freind of mine purchased some "armour-piercing" ammunition and we decided to try them out on an old AT case. Figured it was fairly thick steel (you know - the cases you would use whenever there was no stepladder handy). Afterwards (so I guess it does not really count) we retired to view the videotape while sipping margeuritas(sp). Unfortunately none of us can find the videotape... sigh - probably in an ATF warehouse somewhere in case they "need" us. > -- > N Fudd -- nfudd@mellor.bc.ca > Methuselah lived to be 969 years old. You boys and girls will see more > in the next fifty years than Methuselah saw in his whole lifetime. > - Mark Twain -------------------------------------------------- Matthew G. Marsh, President Paktronix Systems LLC 1506 North 59th Street Omaha NE 68104 Phone: (402) 932-7250 x101 Email: mgm@paktronix.com WWW: http://www.paktronix.com -------------------------------------------------- From owner-netdev@oss.sgi.com Mon Oct 22 15:22:43 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9MMMhh17720 for netdev-outgoing; Mon, 22 Oct 2001 15:22:43 -0700 Received: from www.vsol.net ([216.18.21.14]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9MMMeD17717 for ; Mon, 22 Oct 2001 15:22:40 -0700 Received: from localhost (root@localhost) by www.vsol.net (8.11.6/8.9.3) with ESMTP id f9MMMcg03550; Mon, 22 Oct 2001 15:22:38 -0700 Date: Mon, 22 Oct 2001 15:22:38 -0700 (PDT) From: To: "Matthew G. Marsh" cc: Subject: Re: Linux Kernel 2.4.10, arp -s doesn't work? In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1144 Lines: 25 On Mon, 22 Oct 2001, Matthew G. Marsh wrote: > The one point of friction - which is what I refer to when I speak of > incompatibility in the context of this thread - is that NetFilter > conntrack, an external module to the kernel, "breaks" the RPDB sections > that allow FastNAT. This is not a bad thing due to the fact that conntrack > provides stateful inspection by reason of providing the notion of > "Connection" as a variable to the packet filtering engine (NetFilter). > Indeed I use both types of NAT in different situations depending on the > job I wish to get done. Neither one is a panacea for all NAT. How do you activate FastNAT? How do you activate SlowNAT/conntrack/NetFilter? Is it that 'RELATED' keyword that I've seen in a couple of places? I've been looking at http://netfilter.samba.org/unreliable-guides/ for my iptables needs; is there a better place? -- N Fudd -- nfudd@mellor.bc.ca Laundry instructions on a shirt made by HEET (Korea): For best results: Wash in cold water separately, hang dry and iron with warm iron. For not so good results: Drag behind car through puddles, blow-dry on roofrack. From owner-netdev@oss.sgi.com Mon Oct 22 18:41:16 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9N1fG621485 for netdev-outgoing; Mon, 22 Oct 2001 18:41:16 -0700 Received: from gateway.total-knowledge.com (c1213523-b.smateo1.sfba.home.com [24.1.66.97]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9N1fDD21476 for ; Mon, 22 Oct 2001 18:41:13 -0700 Received: (qmail 26306 invoked by uid 502); 23 Oct 2001 01:41:13 -0000 Message-ID: <20011023014113.26305.qmail@gateway.total-knowledge.com> Content-Type: text/plain; charset="us-ascii" From: Ilya Volynets Reply-To: ilya@theIlya.com Organization: Total Knowledge To: netdev@oss.sgi.com Subject: sk_buff & DMA Date: Mon, 22 Oct 2001 18:41:10 -0700 X-Mailer: KMail [version 1.3.1] MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 675 Lines: 22 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, I have a device that uses 4K aligned buffers for DMA, and I would like to avoid copying data as much as possible, so I want to do DMA dirrectly to sk_buff My questions are: 1. Are there any explicit guarantees about alignment of newly allocated sk_buff data? 2. If 1 is no, is there any way of making new sk_buff with preallocated memory for data 3. If 1 and 2 are no, why? Ilya. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.4 (GNU/Linux) Comment: For info see http://www.gnupg.org iEYEARECAAYFAjvUyrkACgkQtKh84cA8u2kXbgCfeydCWxL6www0e9CiiYyVJzZb A10An1TfFkjFQiIXBJDl1fo7PTtq14Ik =3hrL -----END PGP SIGNATURE----- From owner-netdev@oss.sgi.com Thu Oct 25 05:19:08 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9PCJ8804899 for netdev-outgoing; Thu, 25 Oct 2001 05:19:08 -0700 Received: from samrat.cisco.com (samrat.cisco.com [192.135.241.27]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9PCJ5D04896 for ; Thu, 25 Oct 2001 05:19:05 -0700 Received: from vjacob-pc.cisco.com (root@[192.135.240.108]) by samrat.cisco.com (8.8.8/2.6/Cisco List Logging/8.8.8) with ESMTP id RAA10369 for ; Thu, 25 Oct 2001 17:49:00 +0530 (IST) Received: from localhost (jvt@localhost) by vjacob-pc.cisco.com (8.9.3/8.9.3) with ESMTP id RAA02324 for ; Thu, 25 Oct 2001 17:48:33 GMT X-Authentication-Warning: vjacob-pc.cisco.com: jvt owned process doing -bs Date: Thu, 25 Oct 2001 17:48:33 +0000 (/etc/localtime) From: Vino Thomas X-Sender: jvt@vjacob-pc.cisco.com To: IPV6 Mailing List Subject: 'sit0' tunnel interface Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 150 Lines: 8 Hello, How do I use the sit0 tunnel interface available in the IPV6 linux code? Any pointers for the sample commands/configurations. Regards, Vino From owner-netdev@oss.sgi.com Thu Oct 25 06:23:48 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9PDNmd06346 for netdev-outgoing; Thu, 25 Oct 2001 06:23:48 -0700 Received: from linux.access.co.jp (xdsl052096.211015.metallic.ne.jp [211.15.52.96]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9PDNiD06342 for ; Thu, 25 Oct 2001 06:23:44 -0700 Received: from localhost ([::ffff:127.0.0.1] helo=linux.access.co.jp) by linux.access.co.jp with esmtp (Exim 3.12 #1 (Debian)) id 15wkTw-00026U-00; Thu, 25 Oct 2001 22:23:36 +0900 Date: Thu, 25 Oct 2001 22:23:36 +0900 Message-ID: <87lmhzg8fb.wl@linux.access.co.jp> From: NOGUCHI Kay To: Vino Thomas cc: netdev@oss.sgi.com Subject: Re: 'sit0' tunnel interface In-Reply-To: References: User-Agent: Wanderlust/2.6.0 (Twist And Shout) SEMI/1.14.3 (Ushinoya) FLIM/1.14.3 (Unebigoryomae) APEL/10.2 Emacs/20.7 (i386-debian-linux-gnu) MULE/4.1 (AOI) MIME-Version: 1.0 (generated by SEMI 1.14.3 - "Ushinoya") Content-Type: text/plain; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1017 Lines: 30 Hi Vino, > How do I use the sit0 tunnel interface available in the IPV6 linux > code? Any pointers for the sample commands/configurations. I'm using the sit i/f like below and can communicate even with the kame based *bsd IPv6 stacks like a charm. # ifconfig sit0 tunnel :: # ifconfig sit1 mtu 1280 Note that you should use sit1 i/f, not sit0 i/f, for the communication. ~~~~ # ifconfig |grep sit sit0 Link encap:IPv6-in-IPv4 NOARP MTU:1480 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 sit1 Link encap:IPv6-in-IPv4 inet6 addr: fe80::d30f:3460/10 Scope:Link UP POINTOPOINT RUNNING NOARP MTU:1280 Metric:1 RX packets:2 errors:0 dropped:0 overruns:0 frame:0 TX packets:2 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 Thank you kay From owner-netdev@oss.sgi.com Thu Oct 25 06:38:43 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9PDchB06857 for netdev-outgoing; Thu, 25 Oct 2001 06:38:43 -0700 Received: from netcore.fi (netcore.fi [193.94.160.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9PDccD06852 for ; Thu, 25 Oct 2001 06:38:39 -0700 Received: from localhost (pekkas@localhost) by netcore.fi (8.11.6/8.11.6) with ESMTP id f9PDbDN20343; Thu, 25 Oct 2001 16:37:13 +0300 Date: Thu, 25 Oct 2001 16:37:12 +0300 (EEST) From: Pekka Savola To: NOGUCHI Kay cc: Vino Thomas , Subject: Re: 'sit0' tunnel interface In-Reply-To: <87lmhzg8fb.wl@linux.access.co.jp> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1017 Lines: 27 On Thu, 25 Oct 2001, NOGUCHI Kay wrote: > > How do I use the sit0 tunnel interface available in the IPV6 linux > > code? Any pointers for the sample commands/configurations. > > I'm using the sit i/f like below and can communicate even with > the kame based *bsd IPv6 stacks like a charm. > > # ifconfig sit0 tunnel :: > # ifconfig sit1 mtu 1280 Please note that the preferred, modern way of doing tunneling is using iproute2. This way, one need not enable sit0 device unnecessarily (also enabling automatic tunneling in the process). E.g.: # /sbin/ip tunnel add sit1 mode sit remote a.b.c.d ttl 64 # ifconfig sit1 up Please note that newer distributions provide "user-friedlier" ways of creating tunnels, e.g. RHL72 (see /usr/share/doc/initscripts-*/ and http://www.bieringer.de). -- Pekka Savola "Tell me of difficulties surmounted, Netcore Oy not those you stumble over and fall" Systems. Networks. Security. -- Robert Jordan: A Crown of Swords From owner-netdev@oss.sgi.com Thu Oct 25 17:16:48 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9Q0GmZ09885 for netdev-outgoing; Thu, 25 Oct 2001 17:16:48 -0700 Received: from manty.net ([213.96.224.204]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9Q0Gg009882 for ; Thu, 25 Oct 2001 17:16:42 -0700 Received: from man.beta.es (man.beta.es [192.168.1.1]) by manty.net (Postfix) with ESMTP id 04AF015EC2; Fri, 26 Oct 2001 02:16:34 +0200 (CEST) Received: by man.beta.es (Postfix, from userid 1000) id 7C4DDB91C; Fri, 26 Oct 2001 02:16:34 +0200 (CEST) Date: Fri, 26 Oct 2001 02:16:34 +0200 From: Santiago Garcia Mantinan To: jamal Cc: netdev@oss.sgi.com Subject: 2.4 performance restablished (was: 2.2 performance on high network load much much better than 2.4) Message-ID: <20011026021634.A5502@man.beta.es> References: <20011009105150.A953@man.beta.es> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.3.23i Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1635 Lines: 34 Sorry it took me so long to test and to reply back, I'm very busy, I anounced this a couple of days ago on #kernelnewbies and I've been willing to reply back to you, well, here it is... > This is why it's confusing; run the same tests for both 2.2 and 2.4 > Also the 2.4 kernels before ksoftirq was introduced; try 2.4.5 vs 2.4.10 > (although iam sure those results will be inconclusive since there may be > other issues such as VM etc that might be affecting you). This was a very good clue, I had tested 2.4s from .6 and up but not .5 when I tested it the results were the same as on 2.2 series, as a result, softirq related code was blamed. Then Rik van Riel passed me a patch to the ksoftirq code to let do_softirq loop, I applied that to 2.4.10 and things got much better, so definitely the problem was related to the softirq handling. Some days ago I tested 2.4.13pre6 and I found out that the problem with softirq was gone, the performance was the same as on 2.2 series, even better I'd say, I could ssh to my smp machine, a dual P133, during a udpspam storm from the PIII 868 and do things through ssh, it was slow of course, but worked, that is unthinckable even on 2.2 kernels on that machine. Then I downgraded till I reached 2.2.11 finding that this and newer kernels were ok regarding to this problem (Inicial tests had been done on kernels from 2.4.6 to 2.4.10, exactly the ones that presented this problem). So to sumarice things up: The problem was on kernels 2.4.6 to 2.4.10, I think it was caused by softirq related code. The problem got solved on 2.4.11. Regards... -- Manty/BestiaTester -> http://manty.net From owner-netdev@oss.sgi.com Thu Oct 25 17:56:33 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9Q0uX210793 for netdev-outgoing; Thu, 25 Oct 2001 17:56:33 -0700 Received: from noxmail.sandelman.ottawa.on.ca (cyphermail.sandelman.ottawa.on.ca [192.139.46.78]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9Q0uR010789 for ; Thu, 25 Oct 2001 17:56:27 -0700 Received: from marajade.sandelman.ottawa.on.ca ([2001:410:402:2:204:76ff:fe2d:8c]) by noxmail.sandelman.ottawa.on.ca (8.11.6/8.11.6) with ESMTP id f9Q0tXW00523 (using TLSv1/SSLv3 with cipher EDH-RSA-DES-CBC3-SHA (168 bits) verified OK); Thu, 25 Oct 2001 20:55:43 -0400 (EDT) Received: from marajade.sandelman.ottawa.on.ca (localhost [[UNIX: localhost]]) by marajade.sandelman.ottawa.on.ca (8.11.6/8.11.0) with ESMTP id f9PIfMM08793; Thu, 25 Oct 2001 14:41:24 -0400 (EDT) Message-Id: <200110251841.f9PIfMM08793@marajade.sandelman.ottawa.on.ca> To: design@lists.freeswan.org, netdev@oss.sgi.com Subject: skb->security and friends In-reply-to: Your message of "Thu, 25 Oct 2001 01:09:50 CDT." Mime-Version: 1.0 (generated by tm-edit 7.108) Content-Type: text/plain; charset=US-ASCII Date: Thu, 25 Oct 2001 14:41:22 -0400 From: Michael Richardson Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 2822 Lines: 70 -----BEGIN PGP SIGNED MESSAGE----- {netdev folks. This is a continuation of a thread started until the title: Re: [Design] reg: encryption of MPLS pkts using IPSEC at: http://lists.freeswan.org/pipermail/design/2001-October/001014.html This has prodded me to post this message, which I've been meaning to do.} >>>>> "Sandeep" == Sandeep Subramaniam writes: Sandeep> I actually want the MPLS pkts created by me to be encrypted by IPSEC Sandeep> transport mode . Sandeep> Assume that I have already created the MPLS pkt in a skb structure Aha, RGB! a customer for the skb->{security,ipcb,fwmark} mechanism. Well maybe. The direction that we are going is to have the FreeSWAN xmit code look at a member in the skb that picks the appropriate SA (chain). The issue is which field to use. Criteria are: 1) should be present in all kernels. 2) must not be a "crypto-shaped-hole" (if it is a designated field, it must be multipurpose) 3) must be at least 16-bits in size. 4) must be settable via netfilter and/or advanced routing. There are several fields that might be used: skb->security (16-bit) skb->nfmark (much contention for this field) skb->cb (see linux/ip.h, and net/ip.h's IPCB macro) skb->priority is currently used by QoS mechanisms. We want precisely the same kind of thing. skb->security seems to have been added to support the NRL code. It is copied in one place (ip_output.c) and zeroed in (skbuff.c), which is about it. We would need to add netfilter support for setting it and testing it. And there is linux/ipsec.h. Seems a bit odd for this to be there. Criteria #2 makes us nervous about using skb->security. Re: criteria #3. People who need more than 64K security associations can probably cope with recompiling their kernel. We are concerned about people who have generic kernels with no built-in crypto who want to do basic stuff. skb->cb is relatively easy to extend, and would require no changes to a base kernel. We can load the load/set code into Netfilter, so this is somewhat a neutral decision. We are seeking opinions. ] ON HUMILITY: to err is human. To moo, bovine. | firewalls [ ] Michael Richardson, Sandelman Software Works, Ottawa, ON |net architect[ ] mcr@sandelman.ottawa.on.ca http://www.sandelman.ottawa.on.ca/ |device driver[ ] panic("Just another NetBSD/notebook using, kernel hacking, security guy"); [ -----BEGIN PGP SIGNATURE----- Version: 2.6.3ia Charset: latin1 Comment: Finger me for keys iQCVAwUBO9hc0IqHRg3pndX9AQGoMQP7BNEMBIEsCO0ePQvULc2rDxVKghhtMKQP v516KLWwqhl5/isTnVgLCAWgHSZmeVRatTY408gMEOU3PYENJGgqReZUDMSZieUn a2wyJD81tCfX/RD8CFFKR8Tov1tbCpXNMy/EuKzhB9e/ExqVWtMJw0RqrfQDOkux JVMqmQs2VWY= =z0ez -----END PGP SIGNATURE----- From owner-netdev@oss.sgi.com Fri Oct 26 02:20:54 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9Q9Ksw19922 for netdev-outgoing; Fri, 26 Oct 2001 02:20:54 -0700 Received: from mail.cgn.kopernikus.de (ns01.passionet.de [62.153.93.33]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9Q9Ko019716 for ; Fri, 26 Oct 2001 02:20:51 -0700 Received: by mail.cgn.kopernikus.de (Postfix, from userid 1) id D4CDB2B3B6; Fri, 26 Oct 2001 11:21:18 +0200 (CEST) Received: from f190 (dialin-212-144-144-229.arcor-ip.net [212.144.144.229]) by mail.cgn.kopernikus.de (Postfix.smtp) with ESMTP id 58AA217BA2; Fri, 26 Oct 2001 11:21:13 +0200 (CEST) Date: Fri, 26 Oct 2001 11:20:31 +0200 From: "Manon F. Goo" To: Michael Richardson , design@lists.freeswan.org, netdev@oss.sgi.com Subject: Re: [Design] skb->security and friends Message-ID: <223034286.1004095231@f190> In-Reply-To: <200110251841.f9PIfMM08793@marajade.sandelman.ottawa.on.ca> References: <200110251841.f9PIfMM08793@marajade.sandelman.ottawa.on.ca> X-Mailer: Mulberry/2.1.1 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-AntiVirus: scanned for viruses by AMaViS 0.2.1 (http://amavis.org/) Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 339 Lines: 16 > > Aha, RGB! a customer for the skb->{security,ipcb,fwmark} mechanism. > Well maybe. > skb->security (16-bit) > skb->nfmark (much contention for this field) is it planed to be able to set nfmark value per connecction for later processing with iptables ? Manon > skb->cb (see linux/ip.h, and net/ip.h's IPCB macro) > From owner-netdev@oss.sgi.com Fri Oct 26 07:23:06 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9QEN6S09519 for netdev-outgoing; Fri, 26 Oct 2001 07:23:06 -0700 Received: from nero.doit.wisc.edu (nero.doit.wisc.edu [128.104.17.130]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9QEMt009515 for ; Fri, 26 Oct 2001 07:22:55 -0700 Received: (from jleu@localhost) by nero.doit.wisc.edu (8.8.7/8.8.7) id KAA12011; Fri, 26 Oct 2001 10:14:38 -0500 Date: Fri, 26 Oct 2001 10:14:37 -0500 From: "James R. Leu" To: Michael Richardson Cc: design@lists.freeswan.org, netdev@oss.sgi.com Subject: Re: skb->security and friends Message-ID: <20011026101437.B11973@nero.doit.wisc.edu> Reply-To: jleu@mindspring.com References: <200110251841.f9PIfMM08793@marajade.sandelman.ottawa.on.ca> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2i In-Reply-To: <200110251841.f9PIfMM08793@marajade.sandelman.ottawa.on.ca>; from mcr@sandelman.ottawa.on.ca on Thu, Oct 25, 2001 at 02:41:22PM -0400 Organization: none Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 4488 Lines: 104 Hello, I'm the developer behind the MPLS for Linux project. As part of my development I've added some fields to skbs and fib_node. Maybe the scheme I've used can be extended for this purpose. I've added a field to skbs called aux_protodata. Its an array of void*. Auxiliary protocols (one that don't fit the normal Linux network stack model) can mark interest in a particular protocol and store data here for later use. For example, MPLS uses this by coping data from fib_nodes into this field in the skb, at the same time it creates a dst_entry that redirect the skb to an mpls_output function. There the aux_protodata is used to find the next hop label forwarding entry needed to transmit this packet on a label switched path. The short coming of this model so far is that it should use a netfilter like scheme for redirecting packets at certain points in the network stack. Why not use netfilter? The places that auxiliary protocols need to modify skbs or dst_entries are different then those provided by netfilter. Plus there should be a clear difference between what is being accomplished via netfilter (IPvX packet redirecting/mangling) and aux_protodata (protocol agnostic redirecting/mangling). If people think netfilter can be modified to support the needs of auxiliary protocol, and this fit the original intent for netfilter, I'd be more then happy to extend it to fit the needs of MPLS and IPSEC. Just my $.02 Jim On Thu, Oct 25, 2001 at 02:41:22PM -0400, Michael Richardson wrote: > -----BEGIN PGP SIGNED MESSAGE----- > > > {netdev folks. > This is a continuation of a thread started until the title: > Re: [Design] reg: encryption of MPLS pkts using IPSEC > at: http://lists.freeswan.org/pipermail/design/2001-October/001014.html > > This has prodded me to post this message, which I've been meaning to do.} > > >>>>> "Sandeep" == Sandeep Subramaniam writes: > Sandeep> I actually want the MPLS pkts created by me to be encrypted by IPSEC > Sandeep> transport mode . > > Sandeep> Assume that I have already created the MPLS pkt in a skb structure > > Aha, RGB! a customer for the skb->{security,ipcb,fwmark} mechanism. Well maybe. > > The direction that we are going is to have the FreeSWAN xmit code look at a > member in the skb that picks the appropriate SA (chain). The issue is which field to > use. Criteria are: > > 1) should be present in all kernels. > 2) must not be a "crypto-shaped-hole" (if it is a designated field, it > must be multipurpose) > 3) must be at least 16-bits in size. > 4) must be settable via netfilter and/or advanced routing. > > There are several fields that might be used: > skb->security (16-bit) > skb->nfmark (much contention for this field) > skb->cb (see linux/ip.h, and net/ip.h's IPCB macro) > > skb->priority is currently used by QoS mechanisms. We want precisely the > same kind of thing. > > skb->security seems to have been added to support the NRL code. It is > copied in one place (ip_output.c) and zeroed in (skbuff.c), which is about > it. We would need to add netfilter support for setting it and testing it. > And there is linux/ipsec.h. Seems a bit odd for this to be there. > Criteria #2 makes us nervous about using skb->security. > > Re: criteria #3. People who need more than 64K security associations can > probably cope with recompiling their kernel. We are concerned about people > who have generic kernels with no built-in crypto who want to do basic stuff. > > skb->cb is relatively easy to extend, and would require no changes to a > base kernel. > > We can load the load/set code into Netfilter, so this is somewhat a neutral > decision. > > We are seeking opinions. > > ] ON HUMILITY: to err is human. To moo, bovine. | firewalls [ > ] Michael Richardson, Sandelman Software Works, Ottawa, ON |net architect[ > ] mcr@sandelman.ottawa.on.ca http://www.sandelman.ottawa.on.ca/ |device driver[ > ] panic("Just another NetBSD/notebook using, kernel hacking, security guy"); [ > > -----BEGIN PGP SIGNATURE----- > Version: 2.6.3ia > Charset: latin1 > Comment: Finger me for keys > > iQCVAwUBO9hc0IqHRg3pndX9AQGoMQP7BNEMBIEsCO0ePQvULc2rDxVKghhtMKQP > v516KLWwqhl5/isTnVgLCAWgHSZmeVRatTY408gMEOU3PYENJGgqReZUDMSZieUn > a2wyJD81tCfX/RD8CFFKR8Tov1tbCpXNMy/EuKzhB9e/ExqVWtMJw0RqrfQDOkux > JVMqmQs2VWY= > =z0ez > -----END PGP SIGNATURE----- -- James R. Leu From owner-netdev@oss.sgi.com Fri Oct 26 08:05:01 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9QF51X19719 for netdev-outgoing; Fri, 26 Oct 2001 08:05:01 -0700 Received: from tux.rsn.bth.se (root@tux.rsn.bth.se [194.47.143.135]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9QF4A019702 for ; Fri, 26 Oct 2001 08:04:10 -0700 Received: from localhost (gandalf@localhost [127.0.0.1]) by localhost (8.12.0.Beta19/8.12.0.Beta19/Debian 8.12.0.Beta19) with ESMTP id f9QF2KLf024168; Fri, 26 Oct 2001 17:02:20 +0200 Date: Fri, 26 Oct 2001 17:02:20 +0200 (CEST) From: Martin Josefsson X-Sender: gandalf@tux.rsn.bth.se To: "Manon F. Goo" cc: Michael Richardson , design@lists.freeswan.org, netdev@oss.sgi.com Subject: Re: [Design] skb->security and friends In-Reply-To: <223034286.1004095231@f190> Message-ID: X-message-flag: Get yourself a real mail client! http://www.washington.edu/pine/ MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-1463811832-849060058-1004108540=:22037" Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 35001 Lines: 598 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. Send mail to mime@docserver.cac.washington.edu for more info. ---1463811832-849060058-1004108540=:22037 Content-Type: TEXT/PLAIN; charset=US-ASCII On Fri, 26 Oct 2001, Manon F. Goo wrote: > > > > > Aha, RGB! a customer for the skb->{security,ipcb,fwmark} mechanism. > > Well maybe. > > skb->security (16-bit) > > skb->nfmark (much contention for this field) > > is it planed to be able to set nfmark value per connecction for later > processing with iptables ? There is an iptablesmodule called CONNMARK for this purpose :) you mark the connection with a mark and all packets in that connection inherit that mark. But I don't think CONNMARK is part of the patch-o-matic :( So you'll have to search the netfilter-devel archives I think. Ahh I actually had the patch here... it's a patch against the netfilter CVS, it's probably not up to date so you might have to apply some hunks by hand. And there's a bug in this patch... ++ case IPT_CONNMARK_SAVE: ++ ct->mark = (*pskb)->nfmark; ++ break; that should read ++ case IPT_CONNMARK_SAVE: ++ (*pskb)->nfmark = ct->mark; ++ break; I've never actually used it but people have said that it works :) Good luck! /Martin Never argue with an idiot. They drag you down to their level, then beat you with experience. ---1463811832-849060058-1004108540=:22037 Content-Type: TEXT/PLAIN; charset=iso-8859-1; name="netfilter-CONNMARK.patch" Content-Transfer-Encoding: BASE64 Content-ID: Content-Description: Content-Disposition: attachment; filename="netfilter-CONNMARK.patch" SW5kZXg6IG5ldGZpbHRlci91c2Vyc3BhY2UvaXB0YWJsZXMuOA0KPT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PQ0KUkNTIGZpbGU6IC9jdnNyb290L25ldGZpbHRl ci91c2Vyc3BhY2UvaXB0YWJsZXMuOCx2DQpyZXRyaWV2aW5nIHJldmlzaW9u IDEuMjANCmRpZmYgLXUgLXcgLXIxLjIwIGlwdGFibGVzLjgNCi0tLSBuZXRm aWx0ZXIvdXNlcnNwYWNlL2lwdGFibGVzLjgJMjMgRmViIDIwMDEgMDk6MDg6 MTMgLTAwMDAJMS4yMA0KKysrIG5ldGZpbHRlci91c2Vyc3BhY2UvaXB0YWJs ZXMuOAkxMCBNYXkgMjAwMSAwNjo1NTo0OSAtMDAwMA0KQEAgLTQ1OCw2ICs0 NTgsMTYgQEANCiBNYXRjaGVzIHBhY2tldHMgd2l0aCB0aGUgZ2l2ZW4gdW5z aWduZWQgbWFyayB2YWx1ZSAoaWYgYSBtYXNrIGlzDQogc3BlY2lmaWVkLCB0 aGlzIGlzIGxvZ2ljYWxseSBBTkRlZCB3aXRoIHRoZSBtYXJrIGJlZm9yZSB0 aGUNCiBjb21wYXJpc29uKS4NCisuU1MgY29ubm1hcmsNCitUaGlzIG1vZHVs ZSBtYXRjaGVzIHRoZSBuZXRmaWx0ZXIgbWFyayBmaWVsZCBhc3NvY2lhdGVk IHdpdGggYSBjb25uZWN0aW9uDQorKHdoaWNoIGNhbiBiZSBzZXQgdXNpbmcg dGhlDQorLkIgQ09OTk1BUksNCit0YXJnZXQgYmVsb3cpLg0KKy5UUA0KKy5C SSAiLS1tYXJrICIgInZhbHVlWy9tYXNrXSINCitNYXRjaGVzIHBhY2tldHMg aW4gY29ubmVjdGlvbnMgd2l0aCB0aGUgZ2l2ZW4gdW5zaWduZWQgbWFyayB2 YWx1ZSAoaWYNCithIG1hc2sgaXMgc3BlY2lmaWVkLCB0aGlzIGlzIGxvZ2lj YWxseSBBTkRlZCB3aXRoIHRoZSBtYXJrIGJlZm9yZSB0aGUNCitjb21wYXJp c29uKS4NCiAuU1Mgb3duZXINCiBUaGlzIG1vZHVsZSBhdHRlbXB0cyB0byBt YXRjaCB2YXJpb3VzIGNoYXJhY3RlcmlzdGljcyBvZiB0aGUgcGFja2V0DQog Y3JlYXRvciwgZm9yIGxvY2FsbHktZ2VuZXJhdGVkIHBhY2tldHMuICBJdCBp cyBvbmx5IHZhbGlkIGluIHRoZQ0KQEAgLTU0OCw2ICs1NTgsMjEgQEANCiB0 YWJsZS4NCiAuVFANCiAuQkkgIi0tc2V0LW1hcmsgIiAibWFyayINCisuU1Mg Q09OTk1BUksNCitUaGlzIGlzIHVzZWQgdG8gc2V0IHRoZSBuZXRmaWx0ZXIg bWFyayB2YWx1ZSBhc3NvY2lhdGVkIHdpdGggdGhlDQorY29ubmVjdGlvbg0K Ky5UUA0KKy5CIC0tc2V0LW1hcmsgbWFyaw0KK1NldCBjb25uZWN0aW9uIG1h cmsgDQorLlRQDQorLkIgLS1zYXZlLW1hcmsNCitTZXQgY29ubmVjdGlvbiBt YXJrIHRvIHRoZSBzYW1lIGFzIHRoZSBvbmUgb24gdGhlIHBhY2tldA0KKy5U UA0KKy5CIC0tcmVzdG9yZS1tYXJrDQorU2V0IHRoZSBuZXRmaWx0ZXIgcGFj a2V0IG1hcmsgdmFsdWUgdG8gdGhlIG9uZSBhc3NvY2lhdGVkIHdpdGgNCit0 aGUgY29ubmVjdGlvbi4gVGhpcyBpcyBvbmx5IHZhbGlkIGluIHRoZQ0KKy5C IG1hbmdsZQ0KK3RhYmxlLg0KIC5TUyBSRUpFQ1QNCiBUaGlzIGlzIHVzZWQg dG8gc2VuZCBiYWNrIGFuIGVycm9yIHBhY2tldCBpbiByZXNwb25zZSB0byB0 aGUgbWF0Y2hlZA0KIHBhY2tldDogb3RoZXJ3aXNlIGl0IGlzIGVxdWl2YWxl bnQgdG8gDQpJbmRleDogbmV0ZmlsdGVyL3VzZXJzcGFjZS9leHRlbnNpb25z Ly5DT05OTUFSSy10ZXN0DQo9PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09DQpSQ1Mg ZmlsZTogbmV0ZmlsdGVyL3VzZXJzcGFjZS9leHRlbnNpb25zLy5DT05OTUFS Sy10ZXN0DQpkaWZmIC1OIG5ldGZpbHRlci91c2Vyc3BhY2UvZXh0ZW5zaW9u cy8uQ09OTk1BUkstdGVzdA0KLS0tIC9kZXYvbnVsbAkxIEphbiAxOTcwIDAw OjAwOjAwIC0wMDAwDQorKysgbmV0ZmlsdGVyL3VzZXJzcGFjZS9leHRlbnNp b25zLy5DT05OTUFSSy10ZXN0CTEwIE1heSAyMDAxIDA2OjU1OjQ5IC0wMDAw DQpAQCAtMCwwICsxLDIgQEANCisjISAvYmluL3NoDQorWyAtZiAkS0VSTkVM X0RJUi9uZXQvaXB2NC9uZXRmaWx0ZXIvaXB0X0NPTk5NQVJLLmMgXSAmJiBl Y2hvIENPTk5NQVJLDQpJbmRleDogbmV0ZmlsdGVyL3VzZXJzcGFjZS9leHRl bnNpb25zLy5jb25ubWFyay10ZXN0DQo9PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 DQpSQ1MgZmlsZTogbmV0ZmlsdGVyL3VzZXJzcGFjZS9leHRlbnNpb25zLy5j b25ubWFyay10ZXN0DQpkaWZmIC1OIG5ldGZpbHRlci91c2Vyc3BhY2UvZXh0 ZW5zaW9ucy8uY29ubm1hcmstdGVzdA0KLS0tIC9kZXYvbnVsbAkxIEphbiAx OTcwIDAwOjAwOjAwIC0wMDAwDQorKysgbmV0ZmlsdGVyL3VzZXJzcGFjZS9l eHRlbnNpb25zLy5jb25ubWFyay10ZXN0CTEwIE1heSAyMDAxIDA2OjU1OjQ5 IC0wMDAwDQpAQCAtMCwwICsxLDIgQEANCisjISAvYmluL3NoDQorWyAtZiAk S0VSTkVMX0RJUi9uZXQvaXB2NC9uZXRmaWx0ZXIvaXB0X2Nvbm5tYXJrLmMg XSAmJiBlY2hvIGNvbm5tYXJrDQpJbmRleDogbmV0ZmlsdGVyL3VzZXJzcGFj ZS9leHRlbnNpb25zL2xpYmlwdF9DT05OTUFSSy5jDQo9PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09DQpSQ1MgZmlsZTogbmV0ZmlsdGVyL3VzZXJzcGFjZS9leHRl bnNpb25zL2xpYmlwdF9DT05OTUFSSy5jDQpkaWZmIC1OIG5ldGZpbHRlci91 c2Vyc3BhY2UvZXh0ZW5zaW9ucy9saWJpcHRfQ09OTk1BUksuYw0KLS0tIC9k ZXYvbnVsbAkxIEphbiAxOTcwIDAwOjAwOjAwIC0wMDAwDQorKysgbmV0Zmls dGVyL3VzZXJzcGFjZS9leHRlbnNpb25zL2xpYmlwdF9DT05OTUFSSy5jCTEw IE1heSAyMDAxIDA2OjU1OjUwIC0wMDAwDQpAQCAtMCwwICsxLDE2NyBAQA0K Ky8qIFNoYXJlZCBsaWJyYXJ5IGFkZC1vbiB0byBpcHRhYmxlcyB0byBhZGQg Q09OTk1BUksgdGFyZ2V0IHN1cHBvcnQuICovDQorI2luY2x1ZGUgPHN0ZGlv Lmg+DQorI2luY2x1ZGUgPHN0cmluZy5oPg0KKyNpbmNsdWRlIDxzdGRsaWIu aD4NCisjaW5jbHVkZSA8Z2V0b3B0Lmg+DQorDQorI2luY2x1ZGUgPGlwdGFi bGVzLmg+DQorI2luY2x1ZGUgPGxpbnV4L25ldGZpbHRlcl9pcHY0L2lwX3Rh Ymxlcy5oPg0KKyNpbmNsdWRlIDxsaW51eC9uZXRmaWx0ZXJfaXB2NC9pcHRf Q09OTk1BUksuaD4NCisNCisjaWYgMA0KK3N0cnVjdCBtYXJraW5mbyB7DQor CXN0cnVjdCBpcHRfZW50cnlfdGFyZ2V0IHQ7DQorCXN0cnVjdCBpcHRfY29u bm1hcmtfdGFyZ2V0X2luZm8gbWFyazsNCit9Ow0KKyNlbmRpZg0KKw0KKy8q IEZ1bmN0aW9uIHdoaWNoIHByaW50cyBvdXQgdXNhZ2UgbWVzc2FnZS4gKi8N CitzdGF0aWMgdm9pZA0KK2hlbHAodm9pZCkNCit7DQorCXByaW50ZigNCisi Q09OTk1BUksgdGFyZ2V0IHYlcyBvcHRpb25zOlxuIg0KKyIgIC0tc2V0LW1h cmsgdmFsdWUgICAgICAgICAgICAgIFNldCBjb25udHJhY2sgbWFyayB2YWx1 ZVxuIg0KKyIgIC0tc2F2ZS1tYXJrICAgICAgICAgICAgICAgICAgIFNhdmUg dGhlIHBhY2tldCBuZm1hcmsgb24gdGhlIGNvbm5lY3Rpb25cbiINCisiICAt LXJlc3RvcmUtbWFyayAgICAgICAgICAgICAgICBSZXN0b3JlIHNhdmVkIG5m bWFyayB2YWx1ZVxuIg0KKyJcbiIsDQorTkVURklMVEVSX1ZFUlNJT04pOw0K K30NCisNCitzdGF0aWMgc3RydWN0IG9wdGlvbiBvcHRzW10gPSB7DQorCXsg InNldC1tYXJrIiwgMSwgMCwgJzEnIH0sDQorCXsgInNhdmUtbWFyayIsIDAs IDAsICcyJyB9LA0KKwl7ICJyZXN0b3JlLW1hcmsiLCAwLCAwLCAnMycgfSwN CisJeyAwIH0NCit9Ow0KKw0KKy8qIEluaXRpYWxpemUgdGhlIHRhcmdldC4g Ki8NCitzdGF0aWMgdm9pZA0KK2luaXQoc3RydWN0IGlwdF9lbnRyeV90YXJn ZXQgKnQsIHVuc2lnbmVkIGludCAqbmZjYWNoZSkNCit7DQorfQ0KKw0KKy8q IEZ1bmN0aW9uIHdoaWNoIHBhcnNlcyBjb21tYW5kIG9wdGlvbnM7IHJldHVy bnMgdHJ1ZSBpZiBpdA0KKyAgIGF0ZSBhbiBvcHRpb24gKi8NCitzdGF0aWMg aW50DQorcGFyc2UoaW50IGMsIGNoYXIgKiphcmd2LCBpbnQgaW52ZXJ0LCB1 bnNpZ25lZCBpbnQgKmZsYWdzLA0KKyAgICAgIGNvbnN0IHN0cnVjdCBpcHRf ZW50cnkgKmVudHJ5LA0KKyAgICAgIHN0cnVjdCBpcHRfZW50cnlfdGFyZ2V0 ICoqdGFyZ2V0KQ0KK3sNCisJc3RydWN0IGlwdF9jb25ubWFya190YXJnZXRf aW5mbyAqbWFya2luZm8NCisJCT0gKHN0cnVjdCBpcHRfY29ubm1hcmtfdGFy Z2V0X2luZm8gKikoKnRhcmdldCktPmRhdGE7DQorDQorCXN3aXRjaCAoYykg ew0KKwkJY2hhciAqZW5kOw0KKwljYXNlICcxJzoNCisJCW1hcmtpbmZvLT5t b2RlID0gSVBUX0NPTk5NQVJLX1NFVDsNCisJCW1hcmtpbmZvLT5tYXJrID0g c3RydG91bChvcHRhcmcsICZlbmQsIDApOw0KKwkJaWYgKCplbmQgIT0gJ1ww JyB8fCBlbmQgPT0gb3B0YXJnKQ0KKwkJCWV4aXRfZXJyb3IoUEFSQU1FVEVS X1BST0JMRU0sICJCYWQgTUFSSyB2YWx1ZSBgJXMnIiwgb3B0YXJnKTsNCisJ CWlmICgqZmxhZ3MpDQorCQkJZXhpdF9lcnJvcihQQVJBTUVURVJfUFJPQkxF TSwNCisJCQkgICAgICAgICAgICJDT05OTUFSSyB0YXJnZXQ6IENhbid0IHNw ZWNpZnkgLS1zZXQtbWFyayB0d2ljZSIpOw0KKwkJKmZsYWdzID0gMTsNCisJ CWJyZWFrOw0KKwljYXNlICcyJzoNCisJCW1hcmtpbmZvLT5tb2RlID0gSVBU X0NPTk5NQVJLX1NBVkU7DQorCQlpZiAoKmZsYWdzKQ0KKwkJCWV4aXRfZXJy b3IoUEFSQU1FVEVSX1BST0JMRU0sDQorCQkJICAgICAgICAgICAiQ09OTk1B UksgdGFyZ2V0OiBDYW4ndCBzcGVjaWZ5IC0tc2F2ZS1tYXJrIHR3aWNlIik7 DQorCQkqZmxhZ3MgPSAxOw0KKwkJYnJlYWs7DQorCWNhc2UgJzMnOg0KKwkJ bWFya2luZm8tPm1vZGUgPSBJUFRfQ09OTk1BUktfUkVTVE9SRTsNCisJCWlm ICgqZmxhZ3MpDQorCQkJZXhpdF9lcnJvcihQQVJBTUVURVJfUFJPQkxFTSwN CisJCQkgICAgICAgICAgICJDT05OTUFSSyB0YXJnZXQ6IENhbid0IHNwZWNp ZnkgLS1yZXN0b3JlLW1hcmsgdHdpY2UiKTsNCisJCSpmbGFncyA9IDE7DQor CQlicmVhazsNCisJZGVmYXVsdDoNCisJCXJldHVybiAwOw0KKwl9DQorDQor CXJldHVybiAxOw0KK30NCisNCitzdGF0aWMgdm9pZA0KK2ZpbmFsX2NoZWNr KHVuc2lnbmVkIGludCBmbGFncykNCit7DQorCWlmICghZmxhZ3MpDQorCQll eGl0X2Vycm9yKFBBUkFNRVRFUl9QUk9CTEVNLA0KKwkJICAgICAgICAgICAi Q09OTk1BUksgdGFyZ2V0OiBQYXJhbWV0ZXIgLS1zZXQtbWFyayBpcyByZXF1 aXJlZCIpOw0KK30NCisNCitzdGF0aWMgdm9pZA0KK3ByaW50X21hcmsodW5z aWduZWQgbG9uZyBtYXJrLCBpbnQgbnVtZXJpYykNCit7DQorCXByaW50Zigi MHglbHggIiwgbWFyayk7DQorfQ0KKw0KKy8qIFByaW50cyBvdXQgdGhlIHRh cmdpbmZvLiAqLw0KK3N0YXRpYyB2b2lkDQorcHJpbnQoY29uc3Qgc3RydWN0 IGlwdF9pcCAqaXAsDQorICAgICAgY29uc3Qgc3RydWN0IGlwdF9lbnRyeV90 YXJnZXQgKnRhcmdldCwNCisgICAgICBpbnQgbnVtZXJpYykNCit7DQorCWNv bnN0IHN0cnVjdCBpcHRfY29ubm1hcmtfdGFyZ2V0X2luZm8gKm1hcmtpbmZv ID0NCisJCShjb25zdCBzdHJ1Y3QgaXB0X2Nvbm5tYXJrX3RhcmdldF9pbmZv ICopdGFyZ2V0LT5kYXRhOw0KKwlzd2l0Y2ggKG1hcmtpbmZvLT5tb2RlKSB7 DQorCWNhc2UgSVBUX0NPTk5NQVJLX1NFVDoNCisJICAgIHByaW50ZigiQ09O Tk1BUksgc2V0ICIpOw0KKwkgICAgcHJpbnRfbWFyayhtYXJraW5mby0+bWFy aywgbnVtZXJpYyk7DQorCSAgICBicmVhazsNCisJY2FzZSBJUFRfQ09OTk1B UktfU0FWRToNCisJICAgIHByaW50ZigiQ09OTk1BUksgc2F2ZSAiKTsNCisJ ICAgIGJyZWFrOw0KKwljYXNlIElQVF9DT05OTUFSS19SRVNUT1JFOg0KKwkg ICAgcHJpbnRmKCJDT05OTUFSSyByZXN0b3JlICIpOw0KKwkgICAgYnJlYWs7 DQorCWRlZmF1bHQ6DQorCSAgICBwcmludGYoIkVSUk9SOiBVTktOT1dOIENP Tk5NQVJLIE1PREUgIik7DQorCSAgICBicmVhazsNCisJfQ0KK30NCisNCisv KiBTYXZlcyB0aGUgdW5pb24gaXB0X3RhcmdpbmZvIGluIHBhcnNhYmxlIGZv cm0gdG8gc3Rkb3V0LiAqLw0KK3N0YXRpYyB2b2lkDQorc2F2ZShjb25zdCBz dHJ1Y3QgaXB0X2lwICppcCwgY29uc3Qgc3RydWN0IGlwdF9lbnRyeV90YXJn ZXQgKnRhcmdldCkNCit7DQorCWNvbnN0IHN0cnVjdCBpcHRfY29ubm1hcmtf dGFyZ2V0X2luZm8gKm1hcmtpbmZvID0NCisJCShjb25zdCBzdHJ1Y3QgaXB0 X2Nvbm5tYXJrX3RhcmdldF9pbmZvICopdGFyZ2V0LT5kYXRhOw0KKw0KKwlz d2l0Y2ggKG1hcmtpbmZvLT5tb2RlKSB7DQorCWNhc2UgSVBUX0NPTk5NQVJL X1NFVDoNCisJICAgIHByaW50ZigiLS1zZXQtbWFyayAweCVseCAiLCBtYXJr aW5mby0+bWFyayk7DQorCSAgICBicmVhazsNCisJY2FzZSBJUFRfQ09OTk1B UktfU0FWRToNCisJICAgIHByaW50ZigiLS1zYXZlLW1hcmsgIik7DQorCSAg ICBicmVhazsNCisJY2FzZSBJUFRfQ09OTk1BUktfUkVTVE9SRToNCisJICAg IHByaW50ZigiLS1yZXN0b3JlLW1hcmsgIik7DQorCSAgICBicmVhazsNCisJ ZGVmYXVsdDoNCisJICAgIHByaW50ZigiRVJST1I6IFVOS05PV04gQ09OTk1B UksgTU9ERSAiKTsNCisJICAgIGJyZWFrOw0KKwl9DQorfQ0KKw0KK3N0cnVj dCBpcHRhYmxlc190YXJnZXQgbWFyaw0KKz0geyBOVUxMLA0KKyAgICAiQ09O Tk1BUksiLA0KKyAgICBORVRGSUxURVJfVkVSU0lPTiwNCisgICAgSVBUX0FM SUdOKHNpemVvZihzdHJ1Y3QgaXB0X2Nvbm5tYXJrX3RhcmdldF9pbmZvKSks DQorICAgIElQVF9BTElHTihzaXplb2Yoc3RydWN0IGlwdF9jb25ubWFya190 YXJnZXRfaW5mbykpLA0KKyAgICAmaGVscCwNCisgICAgJmluaXQsDQorICAg ICZwYXJzZSwNCisgICAgJmZpbmFsX2NoZWNrLA0KKyAgICAmcHJpbnQsDQor ICAgICZzYXZlLA0KKyAgICBvcHRzDQorfTsNCisNCit2b2lkIF9pbml0KHZv aWQpDQorew0KKwlyZWdpc3Rlcl90YXJnZXQoJm1hcmspOw0KK30NCkluZGV4 OiBuZXRmaWx0ZXIvdXNlcnNwYWNlL2V4dGVuc2lvbnMvbGliaXB0X2Nvbm5t YXJrLmMNCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0NClJDUyBmaWxlOiBuZXRm aWx0ZXIvdXNlcnNwYWNlL2V4dGVuc2lvbnMvbGliaXB0X2Nvbm5tYXJrLmMN CmRpZmYgLU4gbmV0ZmlsdGVyL3VzZXJzcGFjZS9leHRlbnNpb25zL2xpYmlw dF9jb25ubWFyay5jDQotLS0gL2Rldi9udWxsCTEgSmFuIDE5NzAgMDA6MDA6 MDAgLTAwMDANCisrKyBuZXRmaWx0ZXIvdXNlcnNwYWNlL2V4dGVuc2lvbnMv bGliaXB0X2Nvbm5tYXJrLmMJMTAgTWF5IDIwMDEgMDY6NTU6NTAgLTAwMDAN CkBAIC0wLDAgKzEsMTI5IEBADQorLyogU2hhcmVkIGxpYnJhcnkgYWRkLW9u IHRvIGlwdGFibGVzIHRvIGFkZCBDT05OTUFSSyBtYXRjaGluZyBzdXBwb3J0 LiAqLw0KKyNpbmNsdWRlIDxzdGRpby5oPg0KKyNpbmNsdWRlIDxuZXRkYi5o Pg0KKyNpbmNsdWRlIDxzdHJpbmcuaD4NCisjaW5jbHVkZSA8c3RkbGliLmg+ DQorI2luY2x1ZGUgPGdldG9wdC5oPg0KKw0KKyNpbmNsdWRlIDxpcHRhYmxl cy5oPg0KKyNpbmNsdWRlIDxsaW51eC9uZXRmaWx0ZXJfaXB2NC9pcHRfY29u bm1hcmsuaD4NCisNCisvKiBGdW5jdGlvbiB3aGljaCBwcmludHMgb3V0IHVz YWdlIG1lc3NhZ2UuICovDQorc3RhdGljIHZvaWQNCitoZWxwKHZvaWQpDQor ew0KKwlwcmludGYoDQorIkNPTk5NQVJLIG1hdGNoIHYlcyBvcHRpb25zOlxu Ig0KKyJbIV0gLS1tYXJrIHZhbHVlWy9tYXNrXSAgICAgICAgIE1hdGNoIG5m bWFyayB2YWx1ZSB3aXRoIG9wdGlvbmFsIG1hc2tcbiINCisiXG4iLA0KK05F VEZJTFRFUl9WRVJTSU9OKTsNCit9DQorDQorc3RhdGljIHN0cnVjdCBvcHRp b24gb3B0c1tdID0gew0KKwl7ICJtYXJrIiwgMSwgMCwgJzEnIH0sDQorCXsw fQ0KK307DQorDQorLyogSW5pdGlhbGl6ZSB0aGUgbWF0Y2guICovDQorc3Rh dGljIHZvaWQNCitpbml0KHN0cnVjdCBpcHRfZW50cnlfbWF0Y2ggKm0sIHVu c2lnbmVkIGludCAqbmZjYWNoZSkNCit7DQorCS8qIENhbid0IGNhY2hlIHRo aXMuICovDQorCSpuZmNhY2hlIHw9IE5GQ19VTktOT1dOOw0KK30NCisNCisv KiBGdW5jdGlvbiB3aGljaCBwYXJzZXMgY29tbWFuZCBvcHRpb25zOyByZXR1 cm5zIHRydWUgaWYgaXQNCisgICBhdGUgYW4gb3B0aW9uICovDQorc3RhdGlj IGludA0KK3BhcnNlKGludCBjLCBjaGFyICoqYXJndiwgaW50IGludmVydCwg dW5zaWduZWQgaW50ICpmbGFncywNCisgICAgICBjb25zdCBzdHJ1Y3QgaXB0 X2VudHJ5ICplbnRyeSwNCisgICAgICB1bnNpZ25lZCBpbnQgKm5mY2FjaGUs DQorICAgICAgc3RydWN0IGlwdF9lbnRyeV9tYXRjaCAqKm1hdGNoKQ0KK3sN CisJc3RydWN0IGlwdF9jb25ubWFya19pbmZvICptYXJraW5mbyA9IChzdHJ1 Y3QgaXB0X2Nvbm5tYXJrX2luZm8gKikoKm1hdGNoKS0+ZGF0YTsNCisNCisJ c3dpdGNoIChjKSB7DQorCQljaGFyICplbmQ7DQorCWNhc2UgJzEnOg0KKwkJ aWYgKGNoZWNrX2ludmVyc2Uob3B0YXJnLCAmaW52ZXJ0KSkNCisJCQlvcHRp bmQrKzsNCisJCW1hcmtpbmZvLT5tYXJrID0gc3RydG91bChvcHRhcmcsICZl bmQsIDApOw0KKwkJaWYgKCplbmQgPT0gJy8nKSB7DQorCQkJbWFya2luZm8t Pm1hc2sgPSBzdHJ0b3VsKGVuZCsxLCAmZW5kLCAwKTsNCisJCX0gZWxzZQ0K KwkJCW1hcmtpbmZvLT5tYXNrID0gMHhmZmZmZmZmZjsNCisJCWlmICgqZW5k ICE9ICdcMCcgfHwgZW5kID09IG9wdGFyZykNCisJCQlleGl0X2Vycm9yKFBB UkFNRVRFUl9QUk9CTEVNLCAiQmFkIE1BUksgdmFsdWUgYCVzJyIsIG9wdGFy Zyk7DQorCQlpZiAoaW52ZXJ0KQ0KKwkJCW1hcmtpbmZvLT5pbnZlcnQgPSAx Ow0KKwkJKmZsYWdzID0gMTsNCisJCWJyZWFrOw0KKw0KKwlkZWZhdWx0Og0K KwkJcmV0dXJuIDA7DQorCX0NCisJcmV0dXJuIDE7DQorfQ0KKw0KK3N0YXRp YyB2b2lkDQorcHJpbnRfbWFyayh1bnNpZ25lZCBsb25nIG1hcmssIHVuc2ln bmVkIGxvbmcgbWFzaywgaW50IGludmVydCwgaW50IG51bWVyaWMpDQorew0K KwlpZiAoaW52ZXJ0KQ0KKwkJZnB1dGMoJyEnLCBzdGRvdXQpOw0KKw0KKwlp ZihtYXNrICE9IDB4ZmZmZmZmZmYpDQorCQlwcmludGYoIjB4JWx4LzB4JWx4 ICIsIG1hcmssIG1hc2spOw0KKwllbHNlDQorCQlwcmludGYoIjB4JWx4ICIs IG1hcmspOw0KK30NCisNCisvKiBGaW5hbCBjaGVjazsgbXVzdCBoYXZlIHNw ZWNpZmllZCAtLW1hcmsuICovDQorc3RhdGljIHZvaWQNCitmaW5hbF9jaGVj ayh1bnNpZ25lZCBpbnQgZmxhZ3MpDQorew0KKwlpZiAoIWZsYWdzKQ0KKwkJ ZXhpdF9lcnJvcihQQVJBTUVURVJfUFJPQkxFTSwNCisJCQkgICAiTUFSSyBt YXRjaDogWW91IG11c3Qgc3BlY2lmeSBgLS1tYXJrJyIpOw0KK30NCisNCisv KiBQcmludHMgb3V0IHRoZSBtYXRjaGluZm8uICovDQorc3RhdGljIHZvaWQN CitwcmludChjb25zdCBzdHJ1Y3QgaXB0X2lwICppcCwNCisgICAgICBjb25z dCBzdHJ1Y3QgaXB0X2VudHJ5X21hdGNoICptYXRjaCwNCisgICAgICBpbnQg bnVtZXJpYykNCit7DQorCXByaW50ZigiQ09OTk1BUksgbWF0Y2ggIik7DQor CXByaW50X21hcmsoKChzdHJ1Y3QgaXB0X2Nvbm5tYXJrX2luZm8gKiltYXRj aC0+ZGF0YSktPm1hcmssDQorCQkgICgoc3RydWN0IGlwdF9jb25ubWFya19p bmZvICopbWF0Y2gtPmRhdGEpLT5tYXNrLA0KKwkJICAoKHN0cnVjdCBpcHRf Y29ubm1hcmtfaW5mbyAqKW1hdGNoLT5kYXRhKS0+aW52ZXJ0LCBudW1lcmlj KTsNCit9DQorDQorLyogU2F2ZXMgdGhlIHVuaW9uIGlwdF9tYXRjaGluZm8g aW4gcGFyc2FibGUgZm9ybSB0byBzdGRvdXQuICovDQorc3RhdGljIHZvaWQN CitzYXZlKGNvbnN0IHN0cnVjdCBpcHRfaXAgKmlwLCBjb25zdCBzdHJ1Y3Qg aXB0X2VudHJ5X21hdGNoICptYXRjaCkNCit7DQorCXByaW50ZigiLS1tYXJr ICIpOw0KKwlwcmludF9tYXJrKCgoc3RydWN0IGlwdF9jb25ubWFya19pbmZv ICopbWF0Y2gtPmRhdGEpLT5tYXJrLA0KKwkJICAoKHN0cnVjdCBpcHRfY29u bm1hcmtfaW5mbyAqKW1hdGNoLT5kYXRhKS0+bWFzaywNCisJCSAgKChzdHJ1 Y3QgaXB0X2Nvbm5tYXJrX2luZm8gKiltYXRjaC0+ZGF0YSktPmludmVydCwg MCk7DQorfQ0KKw0KK3N0cnVjdCBpcHRhYmxlc19tYXRjaCBtYXJrDQorPSB7 IE5VTEwsDQorICAgICJjb25ubWFyayIsDQorICAgIE5FVEZJTFRFUl9WRVJT SU9OLA0KKyAgICBJUFRfQUxJR04oc2l6ZW9mKHN0cnVjdCBpcHRfY29ubm1h cmtfaW5mbykpLA0KKyAgICBJUFRfQUxJR04oc2l6ZW9mKHN0cnVjdCBpcHRf Y29ubm1hcmtfaW5mbykpLA0KKyAgICAmaGVscCwNCisgICAgJmluaXQsDQor ICAgICZwYXJzZSwNCisgICAgJmZpbmFsX2NoZWNrLA0KKyAgICAmcHJpbnQs DQorICAgICZzYXZlLA0KKyAgICBvcHRzDQorfTsNCisNCit2b2lkIF9pbml0 KHZvaWQpDQorew0KKwlyZWdpc3Rlcl9tYXRjaCgmbWFyayk7DQorfQ0KSW5k ZXg6IG5ldGZpbHRlci91c2Vyc3BhY2UvcGF0Y2gtby1tYXRpYy9DT05OTUFS Sy5wYXRjaA0KPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQ0KUkNTIGZpbGU6IG5l dGZpbHRlci91c2Vyc3BhY2UvcGF0Y2gtby1tYXRpYy9DT05OTUFSSy5wYXRj aA0KZGlmZiAtTiBuZXRmaWx0ZXIvdXNlcnNwYWNlL3BhdGNoLW8tbWF0aWMv Q09OTk1BUksucGF0Y2gNCi0tLSAvZGV2L251bGwJMSBKYW4gMTk3MCAwMDow MDowMCAtMDAwMA0KKysrIG5ldGZpbHRlci91c2Vyc3BhY2UvcGF0Y2gtby1t YXRpYy9DT05OTUFSSy5wYXRjaAkxMCBNYXkgMjAwMSAwNjo1NTo1MCAtMDAw MA0KQEAgLTAsMCArMSwyMjAgQEANCitkaWZmIC11TiBsaW51eC0yLjQuMy1w cmUzL2luY2x1ZGUvbGludXgvbmV0ZmlsdGVyX2lwdjQvaXBfY29ubnRyYWNr LmggbGludXgtMi40LjMtcHJlMy11bWwvaW5jbHVkZS9saW51eC9uZXRmaWx0 ZXJfaXB2NC9pcF9jb25udHJhY2suaA0KKy0tLSBsaW51eC0yLjQuMy1wcmUz L2luY2x1ZGUvbGludXgvbmV0ZmlsdGVyX2lwdjQvaXBfY29ubnRyYWNrLmgJ RnJpIE1hciAgOSAyMTo0MzoyOCAyMDAxDQorKysrIGxpbnV4LTIuNC4zLXBy ZTMtdW1sL2luY2x1ZGUvbGludXgvbmV0ZmlsdGVyX2lwdjQvaXBfY29ubnRy YWNrLmgJV2VkIE1hciAyMSAxMzoyMDozNyAyMDAxDQorQEAgLTE0Nyw2ICsx NDcsOSBAQA0KKyAJfSBuYXQ7DQorICNlbmRpZiAvKiBDT05GSUdfSVBfTkZf TkFUX05FRURFRCAqLw0KKyANCisrI2lmIGRlZmluZWQoQ09ORklHX0lQX05G X0NPTk5UUkFDS19NQVJLKQ0KKysJdW5zaWduZWQgbG9uZyBtYXJrOw0KKysj ZW5kaWYNCisgfTsNCisgDQorIC8qIEFsdGVyIHJlcGx5IHR1cGxlIChtYXli ZSBhbHRlciBoZWxwZXIpLiAgSWYgaXQncyBhbHJlYWR5IHRha2VuLA0KK2Rp ZmYgLXVOIC0tZXhjbHVkZSAuKiAtLWV4Y2x1ZGUgKi5vIGxpbnV4LTIuNC4z LXByZTMvbmV0L2lwdjQvbmV0ZmlsdGVyL2lwX2Nvbm50cmFja19zdGFuZGFs b25lLmMgbGludXgtMi40LjMtcHJlMy11bWwvbmV0L2lwdjQvbmV0ZmlsdGVy L2lwX2Nvbm50cmFja19zdGFuZGFsb25lLmMNCistLS0gbGludXgtMi40LjMt cHJlMy9uZXQvaXB2NC9uZXRmaWx0ZXIvaXBfY29ubnRyYWNrX3N0YW5kYWxv bmUuYwlUaHUgQXVnIDEwIDIxOjM1OjE1IDIwMDANCisrKysgbGludXgtMi40 LjMtcHJlMy11bWwvbmV0L2lwdjQvbmV0ZmlsdGVyL2lwX2Nvbm50cmFja19z dGFuZGFsb25lLmMJV2VkIE1hciAyMSAxMzowNDoxOSAyMDAxDQorQEAgLTky LDYgKzkyLDkgQEANCisgCQlsZW4gKz0gc3ByaW50ZihidWZmZXIgKyBsZW4s ICJbVU5DT05GSVJNRURdICIpOw0KKyAJbGVuICs9IHNwcmludGYoYnVmZmVy ICsgbGVuLCAidXNlPSV1ICIsDQorIAkJICAgICAgIGF0b21pY19yZWFkKCZj b25udHJhY2stPmN0X2dlbmVyYWwudXNlKSk7DQorKyNpZiBkZWZpbmVkKENP TkZJR19JUF9ORl9DT05OVFJBQ0tfTUFSSykNCisrCWxlbiArPSBzcHJpbnRm KGJ1ZmZlciArIGxlbiwgIm1hcms9JWQgIiwgY29ubnRyYWNrLT5tYXJrKTsN CisrI2VuZGlmDQorIAlsZW4gKz0gc3ByaW50ZihidWZmZXIgKyBsZW4sICJc biIpOw0KKyANCisgCXJldHVybiBsZW47DQorLS0tIGxpbnV4LTIuNC40LXBy ZTEtaG5vL25ldC9pcHY0L25ldGZpbHRlci9pcF9jb25udHJhY2tfY29yZS5j CVR1ZSBBcHIgMTAgMjI6MzM6MjEgMjAwMQ0KKysrKyBsaW51eC0yLjQuNC1w cmUxLXVtbC9uZXQvaXB2NC9uZXRmaWx0ZXIvaXBfY29ubnRyYWNrX2NvcmUu YwlNb24gQXByIDE2IDAwOjIzOjAwIDIwMDENCitAQCAtNTIzLDYgKzUyMyw5 IEBADQorIAkJY29ubnRyYWNrLT5zdGF0dXMgPSBJUFNfRVhQRUNURUQ7DQor IAkJY29ubnRyYWNrLT5tYXN0ZXIubWFzdGVyID0gJmV4cGVjdGVkLT5leHBl Y3RhbnQtPmN0X2dlbmVyYWw7DQorIAkJSVBfTkZfQVNTRVJUKGNvbm50cmFj ay0+bWFzdGVyLm1hc3Rlcik7DQorKyNpZiBDT05GSUdfSVBfTkZfQ09OTlRS QUNLX01BUksNCisrCQljb25udHJhY2stPm1hcmsgPSBleHBlY3RlZC0+ZXhw ZWN0YW50LT5tYXJrOw0KKysjZW5kaWYNCisgCQlMSVNUX0RFTEVURSgmZXhw ZWN0X2xpc3QsIGV4cGVjdGVkKTsNCisgCQlleHBlY3RlZC0+ZXhwZWN0YW50 ID0gTlVMTDsNCisgCQluZl9jb25udHJhY2tfZ2V0KCZjb25udHJhY2stPm1h c3Rlcik7DQorZGlmZiAtdU4gbGludXgtMi40LjMtcHJlMy9pbmNsdWRlL2xp bnV4L25ldGZpbHRlcl9pcHY0L2lwdF9jb25ubWFyay5oIGxpbnV4LTIuNC4z LXByZTMtdW1sL2luY2x1ZGUvbGludXgvbmV0ZmlsdGVyX2lwdjQvaXB0X2Nv bm5tYXJrLmgNCistLS0gbGludXgtMi40LjMtcHJlMy9pbmNsdWRlL2xpbnV4 L25ldGZpbHRlcl9pcHY0L2lwdF9jb25ubWFyay5oCVRodSBKYW4gIDEgMDE6 MDA6MDAgMTk3MA0KKysrKyBsaW51eC0yLjQuMy1wcmUzLXVtbC9pbmNsdWRl L2xpbnV4L25ldGZpbHRlcl9pcHY0L2lwdF9jb25ubWFyay5oCVdlZCBNYXIg MjEgMTE6Mzg6NDYgMjAwMQ0KK0BAIC0wLDAgKzEsOSBAQA0KKysjaWZuZGVm IF9JUFRfQ09OTk1BUktfSA0KKysjZGVmaW5lIF9JUFRfQ09OTk1BUktfSA0K KysNCisrc3RydWN0IGlwdF9jb25ubWFya19pbmZvIHsNCisrICAgIHVuc2ln bmVkIGxvbmcgbWFyaywgbWFzazsNCisrICAgIHVfaW50OF90IGludmVydDsN CisrfTsNCisrDQorKyNlbmRpZiAvKl9JUFRfQ09OTk1BUktfSCovDQorZGlm ZiAtdU4gLS1leGNsdWRlIC4qIC0tZXhjbHVkZSAqLm8gbGludXgtMi40LjMt cHJlMy9uZXQvaXB2NC9uZXRmaWx0ZXIvaXB0X2Nvbm5tYXJrLmMgbGludXgt Mi40LjMtcHJlMy11bWwvbmV0L2lwdjQvbmV0ZmlsdGVyL2lwdF9jb25ubWFy ay5jDQorLS0tIGxpbnV4LTIuNC4zLXByZTMvbmV0L2lwdjQvbmV0ZmlsdGVy L2lwdF9jb25ubWFyay5jCVRodSBKYW4gIDEgMDE6MDA6MDAgMTk3MA0KKysr KyBsaW51eC0yLjQuMy1wcmUzLXVtbC9uZXQvaXB2NC9uZXRmaWx0ZXIvaXB0 X2Nvbm5tYXJrLmMJV2VkIE1hciAyMSAxMzoyMzozMyAyMDAxDQorQEAgLTAs MCArMSw1NSBAQA0KKysvKiBLZXJuZWwgbW9kdWxlIHRvIG1hdGNoIGNvbm5l Y3Rpb24gbWFyayB2YWx1ZXMuICovDQorKyNpbmNsdWRlIDxsaW51eC9tb2R1 bGUuaD4NCisrI2luY2x1ZGUgPGxpbnV4L3NrYnVmZi5oPg0KKysNCisrI2lu Y2x1ZGUgPGxpbnV4L25ldGZpbHRlcl9pcHY0L2lwX3RhYmxlcy5oPg0KKysj aW5jbHVkZSA8bGludXgvbmV0ZmlsdGVyX2lwdjQvaXB0X2Nvbm5tYXJrLmg+ DQorKyNpbmNsdWRlIDxsaW51eC9uZXRmaWx0ZXJfaXB2NC9pcF9jb25udHJh Y2suaD4NCisrDQorK3N0YXRpYyBpbnQNCisrbWF0Y2goY29uc3Qgc3RydWN0 IHNrX2J1ZmYgKnNrYiwNCisrICAgICAgY29uc3Qgc3RydWN0IG5ldF9kZXZp Y2UgKmluLA0KKysgICAgICBjb25zdCBzdHJ1Y3QgbmV0X2RldmljZSAqb3V0 LA0KKysgICAgICBjb25zdCB2b2lkICptYXRjaGluZm8sDQorKyAgICAgIGlu dCBvZmZzZXQsDQorKyAgICAgIGNvbnN0IHZvaWQgKmhkciwNCisrICAgICAg dV9pbnQxNl90IGRhdGFsZW4sDQorKyAgICAgIGludCAqaG90ZHJvcCkNCisr ew0KKysJY29uc3Qgc3RydWN0IGlwdF9jb25ubWFya19pbmZvICppbmZvID0g bWF0Y2hpbmZvOw0KKysJZW51bSBpcF9jb25udHJhY2tfaW5mbyBjdGluZm87 DQorKwlzdHJ1Y3QgaXBfY29ubnRyYWNrICpjdCA9IGlwX2Nvbm50cmFja19n ZXQoKHN0cnVjdCBza19idWZmICopc2tiLCAmY3RpbmZvKTsNCisrCWlmICgh Y3QpDQorKwkgICAgcmV0dXJuIDA7DQorKw0KKysJcmV0dXJuICgoY3QtPm1h cmsgJiBpbmZvLT5tYXNrKSA9PSBpbmZvLT5tYXJrKSBeIGluZm8tPmludmVy dDsNCisrfQ0KKysNCisrc3RhdGljIGludA0KKytjaGVja2VudHJ5KGNvbnN0 IGNoYXIgKnRhYmxlbmFtZSwNCisrICAgICAgICAgICBjb25zdCBzdHJ1Y3Qg aXB0X2lwICppcCwNCisrICAgICAgICAgICB2b2lkICptYXRjaGluZm8sDQor KyAgICAgICAgICAgdW5zaWduZWQgaW50IG1hdGNoc2l6ZSwNCisrICAgICAg ICAgICB1bnNpZ25lZCBpbnQgaG9va19tYXNrKQ0KKyt7DQorKwlpZiAobWF0 Y2hzaXplICE9IElQVF9BTElHTihzaXplb2Yoc3RydWN0IGlwdF9jb25ubWFy a19pbmZvKSkpDQorKwkJcmV0dXJuIDA7DQorKw0KKysJcmV0dXJuIDE7DQor K30NCisrDQorK3N0YXRpYyBzdHJ1Y3QgaXB0X21hdGNoIGNvbm5tYXJrX21h dGNoDQorKz0geyB7IE5VTEwsIE5VTEwgfSwgImNvbm5tYXJrIiwgJm1hdGNo LCAmY2hlY2tlbnRyeSwgTlVMTCwgVEhJU19NT0RVTEUgfTsNCisrDQorK3N0 YXRpYyBpbnQgX19pbml0IGluaXQodm9pZCkNCisrew0KKysJcmV0dXJuIGlw dF9yZWdpc3Rlcl9tYXRjaCgmY29ubm1hcmtfbWF0Y2gpOw0KKyt9DQorKw0K KytzdGF0aWMgdm9pZCBfX2V4aXQgZmluaSh2b2lkKQ0KKyt7DQorKwlpcHRf dW5yZWdpc3Rlcl9tYXRjaCgmY29ubm1hcmtfbWF0Y2gpOw0KKyt9DQorKw0K Kyttb2R1bGVfaW5pdChpbml0KTsNCisrbW9kdWxlX2V4aXQoZmluaSk7DQor ZGlmZiAtdU4gbGludXgtMi40LjMtcHJlMy9pbmNsdWRlL2xpbnV4L25ldGZp bHRlcl9pcHY0L2lwdF9DT05OTUFSSy5oIGxpbnV4LTIuNC4zLXByZTMtdW1s L2luY2x1ZGUvbGludXgvbmV0ZmlsdGVyX2lwdjQvaXB0X0NPTk5NQVJLLmgN CistLS0gbGludXgtMi40LjMtcHJlMy9pbmNsdWRlL2xpbnV4L25ldGZpbHRl cl9pcHY0L2lwdF9DT05OTUFSSy5oCVRodSBKYW4gIDEgMDE6MDA6MDAgMTk3 MA0KKysrKyBsaW51eC0yLjQuMy1wcmUzLXVtbC9pbmNsdWRlL2xpbnV4L25l dGZpbHRlcl9pcHY0L2lwdF9DT05OTUFSSy5oCVdlZCBNYXIgMjEgMTI6MjU6 MjAgMjAwMQ0KK0BAIC0wLDAgKzEsMTUgQEANCisrI2lmbmRlZiBfSVBUX0NP Tk5NQVJLX0hfdGFyZ2V0DQorKyNkZWZpbmUgX0lQVF9DT05OTUFSS19IX3Rh cmdldA0KKysNCisrZW51bSB7DQorKyAgICBJUFRfQ09OTk1BUktfU0VUID0g MCwNCisrICAgIElQVF9DT05OTUFSS19TQVZFLA0KKysgICAgSVBUX0NPTk5N QVJLX1JFU1RPUkUNCisrfTsNCisrDQorK3N0cnVjdCBpcHRfY29ubm1hcmtf dGFyZ2V0X2luZm8gew0KKysJdW5zaWduZWQgbG9uZyBtYXJrOw0KKysJdV9p bnQ4X3QgbW9kZTsNCisrfTsNCisrDQorKyNlbmRpZiAvKl9JUFRfQ09OTk1B UktfSF90YXJnZXQqLw0KK2RpZmYgLXVOIC0tZXhjbHVkZSAuKiAtLWV4Y2x1 ZGUgKi5vIGxpbnV4LTIuNC4zLXByZTMvbmV0L2lwdjQvbmV0ZmlsdGVyL2lw dF9DT05OTUFSSy5jIGxpbnV4LTIuNC4zLXByZTMtdW1sL25ldC9pcHY0L25l dGZpbHRlci9pcHRfQ09OTk1BUksuYw0KKy0tLSBsaW51eC0yLjQuMy1wcmUz L25ldC9pcHY0L25ldGZpbHRlci9pcHRfQ09OTk1BUksuYwlUaHUgSmFuICAx IDAxOjAwOjAwIDE5NzANCisrKysgbGludXgtMi40LjMtcHJlMy11bWwvbmV0 L2lwdjQvbmV0ZmlsdGVyL2lwdF9DT05OTUFSSy5jCVdlZCBNYXIgMjEgMTM6 MjM6MjIgMjAwMQ0KK0BAIC0wLDAgKzEsODcgQEANCisrLyogVGhpcyBpcyBh IG1vZHVsZSB3aGljaCBpcyB1c2VkIGZvciBzZXR0aW5nL3JlbWVtYmVyaW5n IHRoZSBtYXJrIGZpZWxkIG9mDQorKyAqIGFuIGNvbm5lY3Rpb24sIG9yIG9w dGlvbmFsbHkgcmVzdG9yZSBpdCB0byB0aGUgc2tiDQorKyAqLw0KKysjaW5j bHVkZSA8bGludXgvbW9kdWxlLmg+DQorKyNpbmNsdWRlIDxsaW51eC9za2J1 ZmYuaD4NCisrI2luY2x1ZGUgPGxpbnV4L2lwLmg+DQorKyNpbmNsdWRlIDxu ZXQvY2hlY2tzdW0uaD4NCisrDQorKyNpbmNsdWRlIDxsaW51eC9uZXRmaWx0 ZXJfaXB2NC9pcF90YWJsZXMuaD4NCisrI2luY2x1ZGUgPGxpbnV4L25ldGZp bHRlcl9pcHY0L2lwdF9DT05OTUFSSy5oPg0KKysjaW5jbHVkZSA8bGludXgv bmV0ZmlsdGVyX2lwdjQvaXBfY29ubnRyYWNrLmg+DQorKw0KKytzdGF0aWMg dW5zaWduZWQgaW50DQorK3RhcmdldChzdHJ1Y3Qgc2tfYnVmZiAqKnBza2Is DQorKyAgICAgICB1bnNpZ25lZCBpbnQgaG9va251bSwNCisrICAgICAgIGNv bnN0IHN0cnVjdCBuZXRfZGV2aWNlICppbiwNCisrICAgICAgIGNvbnN0IHN0 cnVjdCBuZXRfZGV2aWNlICpvdXQsDQorKyAgICAgICBjb25zdCB2b2lkICp0 YXJnaW5mbywNCisrICAgICAgIHZvaWQgKnVzZXJpbmZvKQ0KKyt7DQorKwlj b25zdCBzdHJ1Y3QgaXB0X2Nvbm5tYXJrX3RhcmdldF9pbmZvICptYXJraW5m byA9IHRhcmdpbmZvOw0KKysNCisrCWVudW0gaXBfY29ubnRyYWNrX2luZm8g Y3RpbmZvOw0KKysJc3RydWN0IGlwX2Nvbm50cmFjayAqY3QgPSBpcF9jb25u dHJhY2tfZ2V0KCgqcHNrYiksICZjdGluZm8pOw0KKysJaWYgKGN0KSB7DQor KwkgICAgc3dpdGNoKG1hcmtpbmZvLT5tb2RlKSB7DQorKwkgICAgY2FzZSBJ UFRfQ09OTk1BUktfU0VUOg0KKysJCWN0LT5tYXJrID0gbWFya2luZm8tPm1h cms7DQorKwkJYnJlYWs7DQorKwkgICAgY2FzZSBJUFRfQ09OTk1BUktfU0FW RToNCisrCQljdC0+bWFyayA9ICgqcHNrYiktPm5mbWFyazsNCisrCQlicmVh azsNCisrCSAgICBjYXNlIElQVF9DT05OTUFSS19SRVNUT1JFOg0KKysJCWlm IChjdC0+bWFyayAhPSAoKnBza2IpLT5uZm1hcmspIHsNCisrCQkgICAgY3Qt Pm1hcmsgPSAoKnBza2IpLT5uZm1hcms7DQorKwkJICAgICgqcHNrYiktPm5m Y2FjaGUgfD0gTkZDX0FMVEVSRUQ7DQorKwkJfQ0KKysJCWJyZWFrOw0KKysJ ICAgIH0NCisrCX0NCisrDQorKwlyZXR1cm4gSVBUX0NPTlRJTlVFOw0KKyt9 DQorKw0KKytzdGF0aWMgaW50DQorK2NoZWNrZW50cnkoY29uc3QgY2hhciAq dGFibGVuYW1lLA0KKysJICAgY29uc3Qgc3RydWN0IGlwdF9lbnRyeSAqZSwN CisrICAgICAgICAgICB2b2lkICp0YXJnaW5mbywNCisrICAgICAgICAgICB1 bnNpZ25lZCBpbnQgdGFyZ2luZm9zaXplLA0KKysgICAgICAgICAgIHVuc2ln bmVkIGludCBob29rX21hc2spDQorK3sNCisrCXN0cnVjdCBpcHRfY29ubm1h cmtfdGFyZ2V0X2luZm8gKm1hdGNoaW5mbyA9IHRhcmdpbmZvOw0KKysJaWYg KHRhcmdpbmZvc2l6ZSAhPSBJUFRfQUxJR04oc2l6ZW9mKHN0cnVjdCBpcHRf Y29ubm1hcmtfdGFyZ2V0X2luZm8pKSkgew0KKysJCXByaW50ayhLRVJOX1dB Uk5JTkcgIkNPTk5NQVJLOiB0YXJnaW5mb3NpemUgJXUgIT0gJVp1XG4iLA0K KysJCSAgICAgICB0YXJnaW5mb3NpemUsDQorKwkJICAgICAgIElQVF9BTElH TihzaXplb2Yoc3RydWN0IGlwdF9jb25ubWFya190YXJnZXRfaW5mbykpKTsN CisrCQlyZXR1cm4gMDsNCisrCX0NCisrDQorKwlpZiAobWF0Y2hpbmZvLT5t b2RlID09IElQVF9DT05OTUFSS19SRVNUT1JFKSB7DQorKwkgICAgaWYgKHN0 cmNtcCh0YWJsZW5hbWUsICJtYW5nbGUiKSAhPSAwKSB7DQorKwkJICAgIHBy aW50ayhLRVJOX1dBUk5JTkcgIkNPTk5NQVJLOiByZXN0b3JlIGNhbiBvbmx5 IGJlIGNhbGxlZCBmcm9tIFwibWFuZ2xlXCIgdGFibGUsIG5vdCBcIiVzXCJc biIsIHRhYmxlbmFtZSk7DQorKwkJICAgIHJldHVybiAwOw0KKysJICAgIH0N CisrCX0NCisrDQorKwlyZXR1cm4gMTsNCisrfQ0KKysNCisrc3RhdGljIHN0 cnVjdCBpcHRfdGFyZ2V0IGlwdF9jb25ubWFya19yZWcNCisrPSB7IHsgTlVM TCwgTlVMTCB9LCAiQ09OTk1BUksiLCB0YXJnZXQsIGNoZWNrZW50cnksIE5V TEwsIFRISVNfTU9EVUxFIH07DQorKw0KKytzdGF0aWMgaW50IF9faW5pdCBp bml0KHZvaWQpDQorK3sNCisrCWlmIChpcHRfcmVnaXN0ZXJfdGFyZ2V0KCZp cHRfY29ubm1hcmtfcmVnKSkNCisrCQlyZXR1cm4gLUVJTlZBTDsNCisrDQor KwlyZXR1cm4gMDsNCisrfQ0KKysNCisrc3RhdGljIHZvaWQgX19leGl0IGZp bmkodm9pZCkNCisrew0KKysJaXB0X3VucmVnaXN0ZXJfdGFyZ2V0KCZpcHRf Y29ubm1hcmtfcmVnKTsNCisrfQ0KKysNCisrbW9kdWxlX2luaXQoaW5pdCk7 DQorK21vZHVsZV9leGl0KGZpbmkpOw0KSW5kZXg6IG5ldGZpbHRlci91c2Vy c3BhY2UvcGF0Y2gtby1tYXRpYy9DT05OTUFSSy5wYXRjaC5jb25maWcuaW4N Cj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT0NClJDUyBmaWxlOiBuZXRmaWx0ZXIv dXNlcnNwYWNlL3BhdGNoLW8tbWF0aWMvQ09OTk1BUksucGF0Y2guY29uZmln LmluDQpkaWZmIC1OIG5ldGZpbHRlci91c2Vyc3BhY2UvcGF0Y2gtby1tYXRp Yy9DT05OTUFSSy5wYXRjaC5jb25maWcuaW4NCi0tLSAvZGV2L251bGwJMSBK YW4gMTk3MCAwMDowMDowMCAtMDAwMA0KKysrIG5ldGZpbHRlci91c2Vyc3Bh Y2UvcGF0Y2gtby1tYXRpYy9DT05OTUFSSy5wYXRjaC5jb25maWcuaW4JMTAg TWF5IDIwMDEgMDY6NTU6NTAgLTAwMDANCkBAIC0wLDAgKzEsMiBAQA0KKyAg ZGVwX3RyaXN0YXRlICcgIEZUUCBwcm90b2NvbCBzdXBwb3J0JyBDT05GSUdf SVBfTkZfRlRQICRDT05GSUdfSVBfTkZfQ09OTlRSQUNLDQorICBib29sICcg IENvbm5lY3Rpb24gbWFyayB0cmFja2luZyBzdXBwb3J0JyBDT05GSUdfSVBf TkZfQ09OTlRSQUNLX01BUksNCkluZGV4OiBuZXRmaWx0ZXIvdXNlcnNwYWNl L3BhdGNoLW8tbWF0aWMvQ09OTk1BUksucGF0Y2guY29uZmlnLmluLTINCj09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT0NClJDUyBmaWxlOiBuZXRmaWx0ZXIvdXNl cnNwYWNlL3BhdGNoLW8tbWF0aWMvQ09OTk1BUksucGF0Y2guY29uZmlnLmlu LTINCmRpZmYgLU4gbmV0ZmlsdGVyL3VzZXJzcGFjZS9wYXRjaC1vLW1hdGlj L0NPTk5NQVJLLnBhdGNoLmNvbmZpZy5pbi0yDQotLS0gL2Rldi9udWxsCTEg SmFuIDE5NzAgMDA6MDA6MDAgLTAwMDANCisrKyBuZXRmaWx0ZXIvdXNlcnNw YWNlL3BhdGNoLW8tbWF0aWMvQ09OTk1BUksucGF0Y2guY29uZmlnLmluLTIJ MTAgTWF5IDIwMDEgMDY6NTU6NTAgLTAwMDANCkBAIC0wLDAgKzEsNCBAQA0K KyAgZGVwX3RyaXN0YXRlICcgIExPRyB0YXJnZXQgc3VwcG9ydCcgQ09ORklH X0lQX05GX1RBUkdFVF9MT0cgJENPTkZJR19JUF9ORl9JUFRBQkxFUw0KKyAg aWYgWyAiJENPTkZJR19JUF9ORl9DT05OVFJBQ0tfTUFSSyIgIT0gIm4iIF07 IHRoZW4NCisgICAgZGVwX3RyaXN0YXRlICcgIENPTk5NQVJLIHRhcmdldCBz dXBwb3J0JyBDT05GSUdfSVBfTkZfVEFSR0VUX0NPTk5NQVJLICRDT05GSUdf SVBfTkZfSVBUQUJMRVMNCisgIGZpDQpJbmRleDogbmV0ZmlsdGVyL3VzZXJz cGFjZS9wYXRjaC1vLW1hdGljL0NPTk5NQVJLLnBhdGNoLmNvbmZpZy5pbi0z DQo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09DQpSQ1MgZmlsZTogbmV0ZmlsdGVy L3VzZXJzcGFjZS9wYXRjaC1vLW1hdGljL0NPTk5NQVJLLnBhdGNoLmNvbmZp Zy5pbi0zDQpkaWZmIC1OIG5ldGZpbHRlci91c2Vyc3BhY2UvcGF0Y2gtby1t YXRpYy9DT05OTUFSSy5wYXRjaC5jb25maWcuaW4tMw0KLS0tIC9kZXYvbnVs bAkxIEphbiAxOTcwIDAwOjAwOjAwIC0wMDAwDQorKysgbmV0ZmlsdGVyL3Vz ZXJzcGFjZS9wYXRjaC1vLW1hdGljL0NPTk5NQVJLLnBhdGNoLmNvbmZpZy5p bi0zCTEwIE1heSAyMDAxIDA2OjU1OjUwIC0wMDAwDQpAQCAtMCwwICsxLDQg QEANCisgICAgZGVwX3RyaXN0YXRlICcgIENvbm5lY3Rpb24gc3RhdGUgbWF0 Y2ggc3VwcG9ydCcgQ09ORklHX0lQX05GX01BVENIX1NUQVRFICRDT05GSUdf SVBfTkZfQ09OTlRSQUNLICRDT05GSUdfSVBfTkZfSVBUQUJMRVMgDQorICAg IGlmIFsgIiRDT05GSUdfSVBfTkZfQ09OTlRSQUNLX01BUksiICE9ICJuIiBd OyB0aGVuDQorICAgICAgZGVwX3RyaXN0YXRlICcgIENvbm5lY3Rpb24gbWFy ayBtYXRjaCBzdXBwb3J0JyBDT05GSUdfSVBfTkZfTUFUQ0hfQ09OTk1BUksg JENPTkZJR19JUF9ORl9JUFRBQkxFUw0KKyAgICBmaQ0KSW5kZXg6IG5ldGZp bHRlci91c2Vyc3BhY2UvcGF0Y2gtby1tYXRpYy9DT05OTUFSSy5wYXRjaC5j b25maWd1cmUuaGVscA0KPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQ0KUkNTIGZp bGU6IG5ldGZpbHRlci91c2Vyc3BhY2UvcGF0Y2gtby1tYXRpYy9DT05OTUFS Sy5wYXRjaC5jb25maWd1cmUuaGVscA0KZGlmZiAtTiBuZXRmaWx0ZXIvdXNl cnNwYWNlL3BhdGNoLW8tbWF0aWMvQ09OTk1BUksucGF0Y2guY29uZmlndXJl LmhlbHANCi0tLSAvZGV2L251bGwJMSBKYW4gMTk3MCAwMDowMDowMCAtMDAw MA0KKysrIG5ldGZpbHRlci91c2Vyc3BhY2UvcGF0Y2gtby1tYXRpYy9DT05O TUFSSy5wYXRjaC5jb25maWd1cmUuaGVscAkxMCBNYXkgMjAwMSAwNjo1NTo1 MCAtMDAwMA0KQEAgLTAsMCArMSwyMiBAQA0KK0NPTkZJR19JUF9ORl9GVFAN CitQZXIgY29ubmVjdGlvbiBtYXJrIHN1cHBvcnQNCitDT05GSUdfSVBfTkZf Q09OTlRSQUNLX01BUksNCisgIFRoaXMgb3B0aW9uIGVuYWJsZXMgc3VwcG9y dCBmb3IgY29ubmVjdGlvbiBtYXJrcywgdXNlZCBieSB0aGUNCisgIGBDT05O TUFSSycgdGFyZ2V0IGFuZCBgY29ubm1hcmsnIG1hdGNoLiBTaW1pbGFyIHRv IHRoZSBtYXJrIHZhbHVlDQorICBvZiBwYWNrZXRzLCBidXQgdGhpcyBtYXJr IHZhbHVlIGlzIGtlcHQgaW4gdGhlIGNvbm50cmFjayBzZXNzaW9uDQorICBp bnN0ZWFkIG9mIHRoZSBpbmRpdmlkdWFsIHBhY2tldHMuDQorDQorQ09OTk1B UksgdGFyZ2V0IHN1cHBvcnQNCitDT05GSUdfSVBfTkZfVEFSR0VUX0NPTk5N QVJLDQorICBUaGlzIG9wdGlvbiBhZGRzIGEgYENPTk5NQVJLJyB0YXJnZXQs IHdoaWNoIGFsbG93cyBvbmUgdG8gbWFuaXB1bGF0ZQ0KKyAgdGhlIGNvbm5l Y3Rpb24gbWFyayB2YWx1ZS4gIFNpbWlsYXIgdG8gdGhlIE1BUksgdGFyZ2V0 LCBidXQNCisgIGFmZmVjdHMgdGhlIGNvbm5lY3Rpb24gbWFyayB2YWx1ZSBy YXRoZXIgdGhhbiB0aGUgcGFja2V0IG1hcmsgdmFsdWUuDQorDQorICBJZiB5 b3Ugd2FudCB0byBjb21waWxlIGl0IGFzIGEgbW9kdWxlLCBzYXkgTSBoZXJl IGFuZCByZWFkDQorICBEb2N1bWVudGF0aW9uL21vZHVsZXMudHh0LiAgVGhl IG1vZHVsZSB3aWxsIGJlIGNhbGxlZA0KKyAgaXB0X0NPTk5NQVJLLm8uICBJ ZiB1bnN1cmUsIHNheSBgTicuDQorDQorY29ubm1hcmsgbWF0Y2ggc3VwcG9y dA0KK0NPTkZJUF9JUF9ORl9NQVRDSF9DT05OTUFSSw0KKyAgVGhpcyBvcHRp b24gYWRkcyBhIGBjb25ubWFyaycgbWF0Y2gsIHdoaWNoIGFsbG93cyB5b3Ug dG8gbWF0Y2ggdGhlDQorICBjb25uZWN0aW9uIG1hcmsgdmFsdWUgcHJldmlv dXNseSBzZXQgZm9yIHRoZSBzZXNzaW9uIGJ5IGBDT05OTUFSSycuIA0KSW5k ZXg6IG5ldGZpbHRlci91c2Vyc3BhY2UvcGF0Y2gtby1tYXRpYy9DT05OTUFS Sy5wYXRjaC5oZWxwDQo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09DQpSQ1MgZmls ZTogbmV0ZmlsdGVyL3VzZXJzcGFjZS9wYXRjaC1vLW1hdGljL0NPTk5NQVJL LnBhdGNoLmhlbHANCmRpZmYgLU4gbmV0ZmlsdGVyL3VzZXJzcGFjZS9wYXRj aC1vLW1hdGljL0NPTk5NQVJLLnBhdGNoLmhlbHANCi0tLSAvZGV2L251bGwJ MSBKYW4gMTk3MCAwMDowMDowMCAtMDAwMA0KKysrIG5ldGZpbHRlci91c2Vy c3BhY2UvcGF0Y2gtby1tYXRpYy9DT05OTUFSSy5wYXRjaC5oZWxwCTEwIE1h eSAyMDAxIDA2OjU1OjUwIC0wMDAwDQpAQCAtMCwwICsxLDM1IEBADQorQXV0 aG9yOiBIZW5yaWsgTm9yZHN0cm9tIDxobm9AbWFyYXN5c3RlbXMuY29tPg0K K1N0YXR1czogd29ya2luZw0KKw0KK1RoaXMgcGF0Y2ggYWRkcyBwZXIgY29u bmVjdGlvbiBtYXJrcywgYW5kIGEgdGFyZ2V0IChDT05OTUFSSykNCityZXNw ZWN0aXZlIGEgbWF0Y2ggKGNvbm5tYXJrKSBmb3IgdXNpbmcgdGhlc2UuDQor DQorVXNhZ2U6DQorDQorICAgY29ubm1hcmsNCisgICAgICAgVGhpcyAgbW9k dWxlICBtYXRjaGVzICB0aGUgbmV0ZmlsdGVyIG1hcmsgZmllbGQgYXNzb2Np YXRlZA0KKyAgICAgICB3aXRoIGEgY29ubmVjdGlvbiAod2hpY2ggY2FuIGJl ICBzZXQgIHVzaW5nICB0aGUgIENPTk5NQVJLDQorICAgICAgIHRhcmdldCBi ZWxvdykuDQorDQorICAgICAgIC0tbWFyayB2YWx1ZVsvbWFza10NCisgICAg ICAgICAgICAgIE1hdGNoZXMgIHBhY2tldHMgIGluICBjb25uZWN0aW9ucyAg d2l0aCAgdGhlICBnaXZlbg0KKyAgICAgICAgICAgICAgdW5zaWduZWQgbWFy ayB2YWx1ZSAoaWYgYSBtYXNrIGlzICBzcGVjaWZpZWQsICB0aGlzDQorICAg ICAgICAgICAgICBpcyBsb2dpY2FsbHkgQU5EZWQgd2l0aCB0aGUgbWFyayBi ZWZvcmUgdGhlIGNvbXBhcq0NCisgICAgICAgICAgICAgIGlzb24pLg0KKw0K Kw0KKyAgIENPTk5NQVJLDQorICAgICAgIFRoaXMgIGlzICB1c2VkICB0byBz ZXQgdGhlIG5ldGZpbHRlciBtYXJrIHZhbHVlIGFzc29jaWF0ZWQNCisgICAg ICAgd2l0aCB0aGUgY29ubmVjdGlvbg0KKw0KKyAgICAgICAtLXNldC1tYXJr IG1hcmsNCisgICAgICAgICAgICAgIFNldCBjb25uZWN0aW9uIG1hcmsNCisN CisgICAgICAgLS1zYXZlLW1hcmsNCisgICAgICAgICAgICAgIFNldCBjb25u ZWN0aW9uIG1hcmsgdG8gdGhlIHNhbWUgYXMgdGhlIG9uZSAgb24gIHRoZQ0K KyAgICAgICAgICAgICAgcGFja2V0DQorDQorICAgICAgIC0tcmVzdG9yZS1t YXJrDQorICAgICAgICAgICAgICBTZXQgIHRoZSAgbmV0ZmlsdGVyICBwYWNr ZXQgIG1hcmsgIHZhbHVlIHRvIHRoZSBvbmUNCisgICAgICAgICAgICAgIGFz c29jaWF0ZWQgd2l0aCB0aGUgY29ubmVjdGlvbi4gVGhpcyBpcyBvbmx5ICB2 YWxpZA0KKyAgICAgICAgICAgICAgaW4gdGhlIG1hbmdsZSB0YWJsZS4NCklu ZGV4OiBuZXRmaWx0ZXIvdXNlcnNwYWNlL3BhdGNoLW8tbWF0aWMvQ09OTk1B UksucGF0Y2gubWFrZWZpbGUNCj09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0NClJD UyBmaWxlOiBuZXRmaWx0ZXIvdXNlcnNwYWNlL3BhdGNoLW8tbWF0aWMvQ09O Tk1BUksucGF0Y2gubWFrZWZpbGUNCmRpZmYgLU4gbmV0ZmlsdGVyL3VzZXJz cGFjZS9wYXRjaC1vLW1hdGljL0NPTk5NQVJLLnBhdGNoLm1ha2VmaWxlDQot LS0gL2Rldi9udWxsCTEgSmFuIDE5NzAgMDA6MDA6MDAgLTAwMDANCisrKyBu ZXRmaWx0ZXIvdXNlcnNwYWNlL3BhdGNoLW8tbWF0aWMvQ09OTk1BUksucGF0 Y2gubWFrZWZpbGUJMTAgTWF5IDIwMDEgMDY6NTU6NTAgLTAwMDANCkBAIC0w LDAgKzEsMiBAQA0KK29iai0kKENPTkZJR19JUF9ORl9NQVRDSF9TVEFURSkg Kz0gaXB0X3N0YXRlLm8NCitvYmotJChDT05GSUdfSVBfTkZfTUFUQ0hfQ09O Tk1BUkspICs9IGlwdF9jb25ubWFyay5vDQpJbmRleDogbmV0ZmlsdGVyL3Vz ZXJzcGFjZS9wYXRjaC1vLW1hdGljL0NPTk5NQVJLLnBhdGNoLm1ha2VmaWxl LTINCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT0NClJDUyBmaWxlOiBuZXRmaWx0 ZXIvdXNlcnNwYWNlL3BhdGNoLW8tbWF0aWMvQ09OTk1BUksucGF0Y2gubWFr ZWZpbGUtMg0KZGlmZiAtTiBuZXRmaWx0ZXIvdXNlcnNwYWNlL3BhdGNoLW8t bWF0aWMvQ09OTk1BUksucGF0Y2gubWFrZWZpbGUtMg0KLS0tIC9kZXYvbnVs bAkxIEphbiAxOTcwIDAwOjAwOjAwIC0wMDAwDQorKysgbmV0ZmlsdGVyL3Vz ZXJzcGFjZS9wYXRjaC1vLW1hdGljL0NPTk5NQVJLLnBhdGNoLm1ha2VmaWxl LTIJMTAgTWF5IDIwMDEgMDY6NTU6NTAgLTAwMDANCkBAIC0wLDAgKzEsMiBA QA0KK29iai0kKENPTkZJR19JUF9ORl9UQVJHRVRfTE9HKSArPSBpcHRfTE9H Lm8NCitvYmotJChDT05GSUdfSVBfTkZfVEFSR0VUX0NPTk5NQVJLKSArPSBp cHRfQ09OTk1BUksubw0K ---1463811832-849060058-1004108540=:22037-- From owner-netdev@oss.sgi.com Fri Oct 26 08:20:09 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9QFK9h20307 for netdev-outgoing; Fri, 26 Oct 2001 08:20:09 -0700 Received: from mail.cgn.kopernikus.de (ns01.passionet.de [62.153.93.33]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9QFK3020303 for ; Fri, 26 Oct 2001 08:20:03 -0700 Received: by mail.cgn.kopernikus.de (Postfix, from userid 1) id D3AEA2B3B6; Fri, 26 Oct 2001 17:20:33 +0200 (CEST) Received: from f190 (dialin-212-144-147-079.arcor-ip.net [212.144.147.79]) by mail.cgn.kopernikus.de (Postfix.smtp) with ESMTP id D6F4D17BA2; Fri, 26 Oct 2001 17:20:26 +0200 (CEST) Date: Fri, 26 Oct 2001 17:19:46 +0200 From: Manon Goo To: Martin Josefsson Cc: Michael Richardson , design@lists.freeswan.org, netdev@oss.sgi.com Subject: Re: [Design] skb->security and friends Message-ID: <5452570.1004116785@f190> In-Reply-To: References: X-Mailer: Mulberry/2.1.1 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-AntiVirus: scanned for viruses by AMaViS 0.2.1 (http://amavis.org/) Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1542 Lines: 54 --On Freitag, 26. Oktober 2001 17:02 +0200 Martin Josefsson wrote: > On Fri, 26 Oct 2001, Manon F. Goo wrote: > >> >> > >> > Aha, RGB! a customer for the skb->{security,ipcb,fwmark} mechanism. >> > Well maybe. >> > skb->security (16-bit) >> > skb->nfmark (much contention for this field) >> >> is it planed to be able to set nfmark value per connecction for later >> processing with iptables ? would it not be more convinient to define the netfilter mark for the ipsec connection so when the connection is setup it is automaticly marked an can be procesed in the fw rule set. > > There is an iptablesmodule called CONNMARK for this purpose :) > you mark the connection with a mark and all packets in that connection > inherit that mark. But I don't think CONNMARK is part of the patch-o-matic > :( So you'll have to search the netfilter-devel archives I think. > > Ahh I actually had the patch here... it's a patch against the netfilter > CVS, it's probably not up to date so you might have to apply some hunks by > hand. And there's a bug in this patch... > > ++ case IPT_CONNMARK_SAVE: > ++ ct->mark = (*pskb)->nfmark; > ++ break; > > that should read > > ++ case IPT_CONNMARK_SAVE: > ++ (*pskb)->nfmark = ct->mark; > ++ break; > > I've never actually used it but people have said that it works :) > > Good luck! > > /Martin > > Never argue with an idiot. They drag you down to their level, then beat > you with experience. From owner-netdev@oss.sgi.com Fri Oct 26 08:52:42 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9QFqg621290 for netdev-outgoing; Fri, 26 Oct 2001 08:52:42 -0700 Received: from noxmail.sandelman.ottawa.on.ca (cyphermail.sandelman.ottawa.on.ca [192.139.46.78]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9QFqc021269 for ; Fri, 26 Oct 2001 08:52:38 -0700 Received: from marajade.sandelman.ottawa.on.ca ([2001:410:402:2:204:76ff:fe2d:8c]) by noxmail.sandelman.ottawa.on.ca (8.11.6/8.11.6) with ESMTP id f9QFqWn01997 (using TLSv1/SSLv3 with cipher EDH-RSA-DES-CBC3-SHA (168 bits) verified OK); Fri, 26 Oct 2001 11:52:34 -0400 (EDT) Received: from marajade.sandelman.ottawa.on.ca (localhost [[UNIX: localhost]]) by marajade.sandelman.ottawa.on.ca (8.11.6/8.11.0) with ESMTP id f9QFp5S06303; Fri, 26 Oct 2001 11:51:08 -0400 (EDT) Message-Id: <200110261551.f9QFp5S06303@marajade.sandelman.ottawa.on.ca> To: "Manon F. Goo" cc: design@lists.freeswan.org, netdev@oss.sgi.com Subject: Re: [Design] skb->security and friends In-reply-to: Your message of "Fri, 26 Oct 2001 11:20:31 +0200." <223034286.1004095231@f190> Mime-Version: 1.0 (generated by tm-edit 7.108) Content-Type: text/plain; charset=US-ASCII Date: Fri, 26 Oct 2001 11:51:04 -0400 From: Michael Richardson Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1430 Lines: 35 -----BEGIN PGP SIGNED MESSAGE----- >>>>> "Manon" == Manon F Goo writes: >> Aha, RGB! a customer for the skb->{security,ipcb,fwmark} mechanism. >> Well maybe. skb-> security (16-bit) skb-> nfmark (much contention for this field) Manon> is it planed to be able to set nfmark value per connecction for later Manon> processing with iptables ? No. The value that we would need to set is a 16-bit or more value. Setting a single bit is meaningless since different tunnels may have different policies. The intention is that you can use "security" (or whatever field is decided) to do filtering. (If pushed into a corner, we may resort to stomping on most of nfmark, which would be unfriendly, but nfmark has to be fixed...) ] ON HUMILITY: to err is human. To moo, bovine. | firewalls [ ] Michael Richardson, Sandelman Software Works, Ottawa, ON |net architect[ ] mcr@sandelman.ottawa.on.ca http://www.sandelman.ottawa.on.ca/ |device driver[ ] panic("Just another NetBSD/notebook using, kernel hacking, security guy"); [ -----BEGIN PGP SIGNATURE----- Version: 2.6.3ia Charset: latin1 Comment: Finger me for keys iQCVAwUBO9mGZ4qHRg3pndX9AQEzLgP/c5D5NW1OHMPXfACnd5fALj76De1W/T+d rEJCFA+dhMeAGPblcLdSED2HgJ+pzgLa6ZzxWpSPx5XHlxd5F8hiawpuYr3TQUKl vgU3UW78lrIHqLZNL0Nmmv5NU6ZRxjwqUr8XIgdZNHfbjVz6nrekNZGiA+8jxZUU 7w/NvypTjpc= =vRC9 -----END PGP SIGNATURE----- From owner-netdev@oss.sgi.com Fri Oct 26 09:03:45 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9QG3jb22873 for netdev-outgoing; Fri, 26 Oct 2001 09:03:45 -0700 Received: from noxmail.sandelman.ottawa.on.ca (cyphermail.sandelman.ottawa.on.ca [192.139.46.78]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9QG3e022868 for ; Fri, 26 Oct 2001 09:03:40 -0700 Received: from marajade.sandelman.ottawa.on.ca ([2001:410:402:2:204:76ff:fe2d:8c]) by noxmail.sandelman.ottawa.on.ca (8.11.6/8.11.6) with ESMTP id f9QG3en02032 (using TLSv1/SSLv3 with cipher EDH-RSA-DES-CBC3-SHA (168 bits) verified OK); Fri, 26 Oct 2001 12:03:42 -0400 (EDT) Received: from marajade.sandelman.ottawa.on.ca (localhost [[UNIX: localhost]]) by marajade.sandelman.ottawa.on.ca (8.11.6/8.11.0) with ESMTP id f9QG3Rn06463; Fri, 26 Oct 2001 12:03:31 -0400 (EDT) Message-Id: <200110261603.f9QG3Rn06463@marajade.sandelman.ottawa.on.ca> To: Martin Josefsson cc: "Manon F. Goo" , design@lists.freeswan.org, netdev@oss.sgi.com Subject: Re: [Design] skb->security and friends In-reply-to: Your message of "Fri, 26 Oct 2001 17:02:20 +0200." Mime-Version: 1.0 (generated by tm-edit 7.108) Content-Type: text/plain; charset=US-ASCII Date: Fri, 26 Oct 2001 12:03:27 -0400 From: Michael Richardson Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1956 Lines: 47 -----BEGIN PGP SIGNED MESSAGE----- >>>>> "Martin" == Martin Josefsson writes: Martin> [1 ] Martin> On Fri, 26 Oct 2001, Manon F. Goo wrote: >> >> > >> > Aha, RGB! a customer for the skb->{security,ipcb,fwmark} mechanism. >> > Well maybe. >> > skb->security (16-bit) >> > skb->nfmark (much contention for this field) >> >> is it planed to be able to set nfmark value per connecction for later >> processing with iptables ? Martin> There is an iptablesmodule called CONNMARK for this purpose :) Martin> you mark the connection with a mark and all packets in that connection Martin> inherit that mark. But I don't think CONNMARK is part of the patch-o-matic Martin> :( So you'll have to search the netfilter-devel archives I think. The term "connection" as used by Manon referes to an IPsec SA. The packets that emerge from the IPsec tunnel have never been seen by the system before (they were hidden by encryption) Conntrack will likely prove useful to short-circuit IPsec SPD (inbound) tunnel processing, particularly for Opportunistic Encryption (which is /32<->/32) uses, but we need to convince ourselves that there are no cache coherency problems with this. ] ON HUMILITY: to err is human. To moo, bovine. | firewalls [ ] Michael Richardson, Sandelman Software Works, Ottawa, ON |net architect[ ] mcr@sandelman.ottawa.on.ca http://www.sandelman.ottawa.on.ca/ |device driver[ ] panic("Just another NetBSD/notebook using, kernel hacking, security guy"); [ -----BEGIN PGP SIGNATURE----- Version: 2.6.3ia Charset: latin1 Comment: Finger me for keys iQCVAwUBO9mJTYqHRg3pndX9AQFykwQAvP2OE4UbgPB4cuIWGxGm+a9hLhznmQS9 GL0/FBBGLD+atE9By0x1qj5cd8sazRwMLuVLAY27xsyNL2x2MlGTr2Wkf6PKPmxH E9mNY3VRYayUn7A+JqVh8ti89Op8ljyzPsiX6D0UybmLhXYTLxq7uH2N6iUGAuRH 9Jv2QHhtxx0= =FDUE -----END PGP SIGNATURE----- From owner-netdev@oss.sgi.com Fri Oct 26 09:15:05 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9QGF5223392 for netdev-outgoing; Fri, 26 Oct 2001 09:15:05 -0700 Received: from noxmail.sandelman.ottawa.on.ca (cyphermail.sandelman.ottawa.on.ca [192.139.46.78]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9QGEv023375 for ; Fri, 26 Oct 2001 09:14:57 -0700 Received: from marajade.sandelman.ottawa.on.ca ([2001:410:402:2:204:76ff:fe2d:8c]) by noxmail.sandelman.ottawa.on.ca (8.11.6/8.11.6) with ESMTP id f9QGEvn02059 (using TLSv1/SSLv3 with cipher EDH-RSA-DES-CBC3-SHA (168 bits) verified OK); Fri, 26 Oct 2001 12:14:59 -0400 (EDT) Received: from marajade.sandelman.ottawa.on.ca (localhost [[UNIX: localhost]]) by marajade.sandelman.ottawa.on.ca (8.11.6/8.11.0) with ESMTP id f9QGEjs06538; Fri, 26 Oct 2001 12:14:47 -0400 (EDT) Message-Id: <200110261614.f9QGEjs06538@marajade.sandelman.ottawa.on.ca> To: Manon Goo cc: Martin Josefsson , design@lists.freeswan.org, netdev@oss.sgi.com Subject: Re: [Design] skb->security and friends In-reply-to: Your message of "Fri, 26 Oct 2001 17:19:46 +0200." <5452570.1004116785@f190> Mime-Version: 1.0 (generated by tm-edit 7.108) Content-Type: text/plain; charset=US-ASCII Date: Fri, 26 Oct 2001 12:14:45 -0400 From: Michael Richardson Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 2551 Lines: 65 -----BEGIN PGP SIGNED MESSAGE----- >>>>> "Manon" == Manon Goo writes: Manon> --On Freitag, 26. Oktober 2001 17:02 +0200 Martin Josefsson Manon> wrote: >> On Fri, 26 Oct 2001, Manon F. Goo wrote: >> >>> >>> > >>> > Aha, RGB! a customer for the skb->{security,ipcb,fwmark} mechanism. >>> > Well maybe. >>> > skb->security (16-bit) >>> > skb->nfmark (much contention for this field) >>> >>> is it planed to be able to set nfmark value per connecction for later >>> processing with iptables ? Manon> would it not be more convinient to define the netfilter mark for the ipsec Manon> connection Manon> so when the connection is setup it is automaticly marked an can be procesed Manon> in the Manon> fw rule set. There are three problems: 1) allocation of nfmark bits. Is this a solved problem yet? 2) IPsec needs more than 1 bit to indicate the tunnel number. 3) it doesn't solve the problem of marking skb's that need to go into a tunnel. Perhaps many people think that they can live with a bit field to say "encrypted" or "not", but we generally find that to be a very poor mechanism. We eventually need to indicate at least four different results of policy: 1) the packet arrived in the clear. 2) the packet arrived authenticated, but in the clear. 3) the packet arrived encrypted, via un/weakly authenticated opportunistic tunnel. 4) the packet arrived encrypted with a strongly trusted tunnel. We would prefer to indicate more than 4. Inability to distinguish between #3 and #4 will prevent people from deploying Opportunistic Encryption on the same machines which they currently do configured tunnels. (We may also provide other mechanism to distinguish between #3/#4, but they likely will have scaling challenges) ] ON HUMILITY: to err is human. To moo, bovine. | firewalls [ ] Michael Richardson, Sandelman Software Works, Ottawa, ON |net architect[ ] mcr@sandelman.ottawa.on.ca http://www.sandelman.ottawa.on.ca/ |device driver[ ] panic("Just another NetBSD/notebook using, kernel hacking, security guy"); [ -----BEGIN PGP SIGNATURE----- Version: 2.6.3ia Charset: latin1 Comment: Finger me for keys iQCVAwUBO9mL8IqHRg3pndX9AQHLPwQA25ljvkPr65Y1F6lcyaXMOkiMIhMNf1aK vJhxJoZ0CSQ4mtB0yzLUt+QAn4Nzn1eLINtGbeWEj5O5HAkEScztGJ2DQD+se84r 4DBttWVuv5xUtC0R3CckAYMgKs/M1Atw4sIAIgc5XBIMmhugJi/AQorisJp+RN0k YlBZzS13gGg= =I8uE -----END PGP SIGNATURE----- From owner-netdev@oss.sgi.com Fri Oct 26 12:25:46 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9QJPk005019 for netdev-outgoing; Fri, 26 Oct 2001 12:25:46 -0700 Received: from attila.bofh.it (postfix@attila.bofh.it [213.92.8.2]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9QJPh005016 for ; Fri, 26 Oct 2001 12:25:43 -0700 Received: by attila.bofh.it (Postfix, from userid 10) id 6F31E607EA; Fri, 26 Oct 2001 21:25:39 +0200 (CEST) Received: by wonderland.linux.it (Postfix/Md, from userid 1001) id DAF0017EAB; Fri, 26 Oct 2001 21:25:14 +0200 (CEST) Date: Fri, 7 Sep 2001 01:37:14 +0200 From: "Marco d'Itri" To: netdev@oss.sgi.com Subject: automatic default route Message-ID: <20010907013714.A8122@wonderland.linux.it> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.3.18i Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 948 Lines: 30 When I set up the ethernet card in my boot scripts: ip link set up dev eth0 ip addr add 10.0.0.1/24 brd + dev eth0 label eth0 ip addr add 10.0.0.9/24 brd + dev eth0 label eth0 I get this message: Sep 7 01:27:59 wonderland kernel: eth0: no IPv6 routers present and an extra default route for ipv6 is added by the kernel: root@wonderland:vc/2:~#ip -6 ro fe80::/10 dev eth0 proto kernel metric 256 mtu 1500 ff00::/8 dev eth0 proto kernel metric 256 mtu 1500 default dev eth0 proto kernel metric 256 mtu 1500 unreachable default dev lo metric -1 error -101 root@wonderland:vc/2:~# This is very annoying because IPv6 enabled programs then will try connectiong to IPv6 addresses and will have to wait the timeout to try with the IPv4 address, and this means I can't use a kernel with IPv6 support on my IPv4 only hosts (or on IPv6 hosts without a configured tunnel). How can I stop the kernel adding this route? -- ciao, Marco From owner-netdev@oss.sgi.com Fri Oct 26 12:42:44 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9QJgir05694 for netdev-outgoing; Fri, 26 Oct 2001 12:42:44 -0700 Received: from Cantor.suse.de (ns.suse.de [213.95.15.193]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9QJgf005691 for ; Fri, 26 Oct 2001 12:42:41 -0700 Received: from Hermes.suse.de (Hermes.suse.de [213.95.15.136]) by Cantor.suse.de (Postfix) with ESMTP id 1ECD41E613; Fri, 26 Oct 2001 21:42:36 +0200 (MEST) Date: Fri, 26 Oct 2001 21:42:35 +0200 From: Andi Kleen To: Michael Richardson Cc: design@lists.freeswan.org, netdev@oss.sgi.com Subject: Re: skb->security and friends Message-ID: <20011026214235.A5375@wotan.suse.de> References: <200110251841.f9PIfMM08793@marajade.sandelman.ottawa.on.ca> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.3.16i In-Reply-To: <200110251841.f9PIfMM08793@marajade.sandelman.ottawa.on.ca>; from mcr@sandelman.ottawa.on.ca on Thu, Oct 25, 2001 at 02:41:22PM -0400 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 788 Lines: 19 > We are seeking opinions. nfmark has the advantage that the routing code knows about it and can manage the destination cache based on it (very useful for pmtu management) security is basically on its way out; it was for a never completely merged ipsec implementation from the fi/sinus firewalls guys and is largely bitrotted now (e.g. a lot of stack modules won't maintain it correctly anymore and probably never have) If you wanted to use it you would need to fix it first. ->cb is free for your use as long as you have the skb queued privately, but it'll be destroyed as soon as you give it away. I don't understand your 64k comment. I would recommend to use nfmark. as far as I can see you'll need destination cache support anyways, and it gets you that for free. -Andi From owner-netdev@oss.sgi.com Fri Oct 26 13:31:20 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9QKVKx07773 for netdev-outgoing; Fri, 26 Oct 2001 13:31:20 -0700 Received: from netcore.fi (netcore.fi [193.94.160.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9QKVF007770 for ; Fri, 26 Oct 2001 13:31:15 -0700 Received: from localhost (pekkas@localhost) by netcore.fi (8.11.6/8.11.6) with ESMTP id f9QKUmM02910; Fri, 26 Oct 2001 23:30:48 +0300 Date: Fri, 26 Oct 2001 23:30:48 +0300 (EEST) From: Pekka Savola To: "Marco d'Itri" cc: Subject: Re: automatic default route In-Reply-To: <20010907013714.A8122@wonderland.linux.it> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 2031 Lines: 51 On Fri, 7 Sep 2001, Marco d'Itri wrote: > When I set up the ethernet card in my boot scripts: > > ip link set up dev eth0 > ip addr add 10.0.0.1/24 brd + dev eth0 label eth0 > ip addr add 10.0.0.9/24 brd + dev eth0 label eth0 > > I get this message: > > Sep 7 01:27:59 wonderland kernel: eth0: no IPv6 routers present The commands above do not trigger the message in itself. You've either compiled in IPv6 support, or it's loaded as the module. After the module is loaded and an interface is up, it takes about ~10-15 seconds to get this message (configurable) if there are no IPv6 routers present. > and an extra default route for ipv6 is added by the kernel: > > root@wonderland:vc/2:~#ip -6 ro > fe80::/10 dev eth0 proto kernel metric 256 mtu 1500 > ff00::/8 dev eth0 proto kernel metric 256 mtu 1500 > default dev eth0 proto kernel metric 256 mtu 1500 > unreachable default dev lo metric -1 error -101 > root@wonderland:vc/2:~# You mean the third one? It is required by the specification; if no better route is known, all destinations are considered to be on-link. > This is very annoying because IPv6 enabled programs then will try > connectiong to IPv6 addresses and will have to wait the timeout to try > with the IPv4 address, and this means I can't use a kernel with IPv6 > support on my IPv4 only hosts (or on IPv6 hosts without a configured > tunnel). My experience is that the delay is very small. E.g. one of my workstations has IPv6 enabled but not even global addresses. When SSH'ing into a system which has both IPv4 and IPv6 in DNS, it takes about a second at the maximum to realize the IPv6 will not be reachable. > How can I stop the kernel adding this route? Do not load IPv6 module (remove net-pf-10 definition from /etc/modules.conf to avoid it being autoloaded). -- Pekka Savola "Tell me of difficulties surmounted, Netcore Oy not those you stumble over and fall" Systems. Networks. Security. -- Robert Jordan: A Crown of Swords From owner-netdev@oss.sgi.com Fri Oct 26 21:24:47 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9R4OlB26586 for netdev-outgoing; Fri, 26 Oct 2001 21:24:47 -0700 Received: from noxmail.sandelman.ottawa.on.ca (cyphermail.sandelman.ottawa.on.ca [192.139.46.78]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9R4Oc026583 for ; Fri, 26 Oct 2001 21:24:40 -0700 Received: from marajade.sandelman.ottawa.on.ca ([2001:410:402:2:204:76ff:fe2d:8c]) by noxmail.sandelman.ottawa.on.ca (8.11.6/8.11.6) with ESMTP id f9R4OYn04070 (using TLSv1/SSLv3 with cipher EDH-RSA-DES-CBC3-SHA (168 bits) verified OK); Sat, 27 Oct 2001 00:24:37 -0400 (EDT) Received: from marajade.sandelman.ottawa.on.ca (localhost [[UNIX: localhost]]) by marajade.sandelman.ottawa.on.ca (8.11.6/8.11.0) with ESMTP id f9R4N3m09808; Sat, 27 Oct 2001 00:23:05 -0400 (EDT) Message-Id: <200110270423.f9R4N3m09808@marajade.sandelman.ottawa.on.ca> To: Andi Kleen cc: design@lists.freeswan.org, netdev@oss.sgi.com Subject: Re: skb->security and friends In-reply-to: Your message of "Fri, 26 Oct 2001 21:42:35 +0200." <20011026214235.A5375@wotan.suse.de> Mime-Version: 1.0 (generated by tm-edit 7.108) Content-Type: text/plain; charset=US-ASCII Date: Sat, 27 Oct 2001 00:23:02 -0400 From: Michael Richardson Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 2290 Lines: 62 -----BEGIN PGP SIGNED MESSAGE----- >>>>> "Andi" == Andi Kleen writes: >> We are seeking opinions. Andi> nfmark has the advantage that the routing code knows about it and Andi> can manage the destination cache based on it (very useful for pmtu Andi> management) Andi> security is basically on its way out; it was for a never completely Andi> merged ipsec implementation from the fi/sinus firewalls guys and is Andi> largely bitrotted now (e.g. a lot of stack modules won't maintain Andi> it correctly anymore and probably never have) If you wanted to use Andi> it you would need to fix it first. I see. What maintenance does it need? skb_copy() seems to copy it, and it seems to get initialized to zero. I think that is sufficient for our use. -> cb is free for your use as long as you have the skb queued privately, Andi> but it'll be destroyed as soon as you give it away. 1) We wish to set something in netfilter and/or advanced routing and examine it in dev xmit. (for entering the tunnel) 2) We wish to set something in dev recv, and examine it in netfilter. (for checking that the packet that exited the tunnel complied to policy) Andi> I don't Andi> understand your 64k comment. We need at least 16 bits of value to set/examine. This is the security policy identifier :-) Andi> I would recommend to use nfmark. as far as I can see you'll need Andi> destination cache support anyways, and it gets you that for free. Thanks. We'll use nfmark. What will you guys use? We'll need between 16 and 32 bits of nfmark :-) ] ON HUMILITY: to err is human. To moo, bovine. | firewalls [ ] Michael Richardson, Sandelman Software Works, Ottawa, ON |net architect[ ] mcr@sandelman.ottawa.on.ca http://www.sandelman.ottawa.on.ca/ |device driver[ ] panic("Just another NetBSD/notebook using, kernel hacking, security guy"); [ -----BEGIN PGP SIGNATURE----- Version: 2.6.3ia Charset: latin1 Comment: Finger me for keys iQCVAwUBO9o2o4qHRg3pndX9AQHLQwQAuUKMkFI7cPqH3M82xpT4SVvdSsi8cZJF 4qUIgIhXz0aKxtyYxTb1sL5ZUYm5gQEzsLDpzmKZ6wt0jGHBRiwqazjAGN+eGMSg 7Y3UKIWN1rKSF1K/gYOb+3D6uBqkuviULdtnhXRdyQ6pUTxZ1ThOxVvQC0WT9s6t DhRd1IoJQ3g= =+knd -----END PGP SIGNATURE----- From owner-netdev@oss.sgi.com Fri Oct 26 22:59:53 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9R5xr627930 for netdev-outgoing; Fri, 26 Oct 2001 22:59:53 -0700 Received: from blackbird.intercode.com.au (blackbird.intercode.com.au [203.32.101.10]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9R5xl027927 for ; Fri, 26 Oct 2001 22:59:48 -0700 Received: from localhost (jmorris@localhost) by blackbird.intercode.com.au (8.9.3/8.9.3) with ESMTP id PAA17507; Sat, 27 Oct 2001 15:58:22 +1000 Date: Sat, 27 Oct 2001 15:58:22 +1000 (EST) From: James Morris To: Andi Kleen cc: Michael Richardson , , , Subject: Re: skb->security and friends In-Reply-To: <20011026214235.A5375@wotan.suse.de> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 775 Lines: 23 On Fri, 26 Oct 2001, Andi Kleen wrote: > security is basically on its way out; it was for a never completely merged > ipsec implementation from the fi/sinus firewalls guys and is largely bitrotted > now (e.g. a lot of stack modules won't maintain it correctly anymore and > probably never have) > If you wanted to use it you would need to fix it first. [note: lsm added to the cc list] I was hoping that skb->security could be reassigned as a void pointer for use by LSM in 2.5, if LSM is accepted into the kernel. This would be used by LSM modules for maintaining security attributes between layers. Note that this may also be useful for Freeswan, as it should be possible now to implement ipsec as an LSM module. - James -- James Morris From owner-netdev@oss.sgi.com Sat Oct 27 04:30:18 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9RBUIf01003 for netdev-outgoing; Sat, 27 Oct 2001 04:30:18 -0700 Received: from Cantor.suse.de (ns.suse.de [213.95.15.193]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9RBUE000999 for ; Sat, 27 Oct 2001 04:30:15 -0700 Received: from Hermes.suse.de (Hermes.suse.de [213.95.15.136]) by Cantor.suse.de (Postfix) with ESMTP id 59BFD1E52D; Sat, 27 Oct 2001 13:30:09 +0200 (MEST) Date: Sat, 27 Oct 2001 13:30:00 +0200 From: Andi Kleen To: James Morris Cc: Andi Kleen , Michael Richardson , design@lists.freeswan.org, netdev@oss.sgi.com, linux-security-module@wirex.com Subject: Re: skb->security and friends Message-ID: <20011027133000.A20848@wotan.suse.de> References: <20011026214235.A5375@wotan.suse.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.3.16i In-Reply-To: ; from jmorris@intercode.com.au on Sat, Oct 27, 2001 at 03:58:22PM +1000 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1159 Lines: 28 On Sat, Oct 27, 2001 at 03:58:22PM +1000, James Morris wrote: > On Fri, 26 Oct 2001, Andi Kleen wrote: > > > security is basically on its way out; it was for a never completely merged > > ipsec implementation from the fi/sinus firewalls guys and is largely bitrotted > > now (e.g. a lot of stack modules won't maintain it correctly anymore and > > probably never have) > > If you wanted to use it you would need to fix it first. > > [note: lsm added to the cc list] > > I was hoping that skb->security could be reassigned as a void pointer > for use by LSM in 2.5, if LSM is accepted into the kernel. void pointer alone without any rules for freeing and reference counting (e.g. what to do with it on a skb_clone() or a skb_copy()) would not make too much sense. Getting that right would be probably ugly (similar to rusty's old abandoned ->cb attribute allocator) > > This would be used by LSM modules for maintaining security attributes > between layers. Note that this may also be useful for Freeswan, as it > should be possible now to implement ipsec as an LSM module. Could you give a more detailed scenario what it would be needed for? -Andi From owner-netdev@oss.sgi.com Sat Oct 27 04:34:46 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9RBYk201129 for netdev-outgoing; Sat, 27 Oct 2001 04:34:46 -0700 Received: from Cantor.suse.de (ns.suse.de [213.95.15.193]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9RBYg001126 for ; Sat, 27 Oct 2001 04:34:43 -0700 Received: from Hermes.suse.de (Hermes.suse.de [213.95.15.136]) by Cantor.suse.de (Postfix) with ESMTP id 52B481E52D; Sat, 27 Oct 2001 13:34:37 +0200 (MEST) Date: Sat, 27 Oct 2001 13:34:37 +0200 From: Andi Kleen To: Michael Richardson Cc: Andi Kleen , design@lists.freeswan.org, netdev@oss.sgi.com Subject: Re: skb->security and friends Message-ID: <20011027133437.B20848@wotan.suse.de> References: <20011026214235.A5375@wotan.suse.de> <200110270423.f9R4N3m09808@marajade.sandelman.ottawa.on.ca> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.3.16i In-Reply-To: <200110270423.f9R4N3m09808@marajade.sandelman.ottawa.on.ca>; from mcr@sandelman.ottawa.on.ca on Sat, Oct 27, 2001 at 12:23:02AM -0400 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1140 Lines: 27 On Sat, Oct 27, 2001 at 12:23:02AM -0400, Michael Richardson wrote: > 1) We wish to set something in netfilter and/or advanced routing and examine > it in dev xmit. (for entering the tunnel) > > 2) We wish to set something in dev recv, and examine it in netfilter. > (for checking that the packet that exited the tunnel complied to policy) netfilter is not a layer in this definition, so ->cb is not free for your use. It would be e.g. if you're a device driver and manage the skb in queue or if you're TCP/IP and also manage it in your queues. > Andi> I would recommend to use nfmark. as far as I can see you'll need > Andi> destination cache support anyways, and it gets you that for free. > > Thanks. We'll use nfmark. > > What will you guys use? We'll need between 16 and 32 bits of nfmark :-) For the current kernel nfmark is just an opaque value with no policy. ipchains/tables currently expose 32bit to the administrators for firewall purposes; if you don't want to wrestle with the admins about these bits it may be needed to expand it to 64bit and reserve the upper 32bits for kernel uses. -Andi From owner-netdev@oss.sgi.com Sun Oct 28 12:29:54 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9SKTsm05209 for netdev-outgoing; Sun, 28 Oct 2001 12:29:54 -0800 Received: from comunit.de (comunit.de [195.21.213.33]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9SKTk005205 for ; Sun, 28 Oct 2001 12:29:46 -0800 Received: (qmail 21308 invoked by uid 517); 28 Oct 2001 20:29:44 -0000 Date: Sun, 28 Oct 2001 21:29:44 +0100 (CET) From: Sven Koch X-X-Sender: haegar@space.comunit.de To: Pekka Savola cc: "Marco d'Itri" , Subject: Re: automatic default route In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1916 Lines: 60 On Fri, 26 Oct 2001, Pekka Savola wrote: > The commands above do not trigger the message in itself. You've either > compiled in IPv6 support, or it's loaded as the module. > > After the module is loaded and an interface is up, it takes about ~10-15 > seconds to get this message (configurable) if there are no IPv6 routers > present. How can I turn off IPv6 Router-Discovery for one pcmcia-scheme? I need to set my ipv6-addresses by hand (no router-advertising possible in my local lan, and won't be possible anytime soon), and add the ipv6-default-route by hand too. it's very bad to do from /etc/pcmcia/network.opts /sbin/ip addr add xxxx:yyy:zzz:1001::a/64 dev $DEVICE /sbin/ip -f inet6 route del default 2>/dev/null /sbin/ip -f inet6 route add ::/0 via xxxx:yyy:zzz:1001::1 { sleep 20 /sbin/ip -f inet6 route del default dev $DEVICE 2>/dev/null } & doing for d in all default $DEVICE ; do echo 0 >/proc/sys/net/ipv6/conf/$d/autoconf echo 0 >/proc/sys/net/ipv6/conf/$d/accept_ra done seems not to work, the router-discovery is done, and the broken default-route is set after 15 to 20 seconds :( > > and an extra default route for ipv6 is added by the kernel: > > > > root@wonderland:vc/2:~#ip -6 ro > > fe80::/10 dev eth0 proto kernel metric 256 mtu 1500 > > ff00::/8 dev eth0 proto kernel metric 256 mtu 1500 > > default dev eth0 proto kernel metric 256 mtu 1500 > > unreachable default dev lo metric -1 error -101 > > root@wonderland:vc/2:~# > > You mean the third one? It is required by the specification; if no better > route is known, all destinations are considered to be on-link. but when I set a valid ipv6-defaultroute before router-discovery finishes, a better route is known, and the broken defaultroute should imho not be set. c'ya sven -- The Internet treats censorship as a routing problem, and routes around it. (John Gilmore on http://www.cygnus.com/~gnu/) From owner-netdev@oss.sgi.com Sun Oct 28 12:37:08 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9SKb8905389 for netdev-outgoing; Sun, 28 Oct 2001 12:37:08 -0800 Received: from netcore.fi (netcore.fi [193.94.160.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9SKb1005383 for ; Sun, 28 Oct 2001 12:37:01 -0800 Received: from localhost (pekkas@localhost) by netcore.fi (8.11.6/8.11.6) with ESMTP id f9SKakD24184; Sun, 28 Oct 2001 22:36:47 +0200 Date: Sun, 28 Oct 2001 22:36:46 +0200 (EET) From: Pekka Savola To: Sven Koch cc: "Marco d'Itri" , Subject: Re: automatic default route In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 2607 Lines: 72 On Sun, 28 Oct 2001, Sven Koch wrote: > On Fri, 26 Oct 2001, Pekka Savola wrote: > > > The commands above do not trigger the message in itself. You've either > > compiled in IPv6 support, or it's loaded as the module. > > > > After the module is loaded and an interface is up, it takes about ~10-15 > > seconds to get this message (configurable) if there are no IPv6 routers > > present. > > How can I turn off IPv6 Router-Discovery for one pcmcia-scheme? > Not as you mean it (except by not enabling IPv6). However.. > I need to set my ipv6-addresses by hand (no router-advertising possible > in my local lan, and won't be possible anytime soon), and add the > ipv6-default-route by hand too. > > it's very bad to do from /etc/pcmcia/network.opts > > /sbin/ip addr add xxxx:yyy:zzz:1001::a/64 dev $DEVICE > /sbin/ip -f inet6 route del default 2>/dev/null > /sbin/ip -f inet6 route add ::/0 via xxxx:yyy:zzz:1001::1 > { > sleep 20 > /sbin/ip -f inet6 route del default dev $DEVICE 2>/dev/null > } & > > > doing > > for d in all default $DEVICE ; do > echo 0 >/proc/sys/net/ipv6/conf/$d/autoconf > echo 0 >/proc/sys/net/ipv6/conf/$d/accept_ra > done > > seems not to work, the router-discovery is done, and the broken > default-route is set after 15 to 20 seconds :( with this, you can toggle whether autoconfiguration happens or not; the "broken default" you speak of it irrespective of that. Please also note that... > > > and an extra default route for ipv6 is added by the kernel: > > > > > > root@wonderland:vc/2:~#ip -6 ro > > > fe80::/10 dev eth0 proto kernel metric 256 mtu 1500 > > > ff00::/8 dev eth0 proto kernel metric 256 mtu 1500 > > > default dev eth0 proto kernel metric 256 mtu 1500 > > > unreachable default dev lo metric -1 error -101 > > > root@wonderland:vc/2:~# > > > > You mean the third one? It is required by the specification; if no better > > route is known, all destinations are considered to be on-link. > > but when I set a valid ipv6-defaultroute before router-discovery finishes, > a better route is known, and the broken defaultroute should imho not be > set. ... the broken default route is set with metric 256; any routes you add will override it automatically. So it doesn't need to be removed. You _may_ be a seeing a known phenomenon that default route will be ignored for forwarded packets if forwarding is enabled. -- Pekka Savola "Tell me of difficulties surmounted, Netcore Oy not those you stumble over and fall" Systems. Networks. Security. -- Robert Jordan: A Crown of Swords From owner-netdev@oss.sgi.com Sun Oct 28 20:21:50 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9T4Lo419770 for netdev-outgoing; Sun, 28 Oct 2001 20:21:50 -0800 Received: from front.ozlabs.ibm.com (CPE-61-9-150-4.vic.bigpond.net.au [61.9.150.4]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9T4LQ019764 for ; Sun, 28 Oct 2001 20:21:26 -0800 Received: from wagner (thingo.ozlabs.ibm.com [10.161.2.77]) by front.ozlabs.ibm.com (Postfix) with ESMTP id 94E2C23F00; Mon, 29 Oct 2001 15:21:18 +1100 (EST) Received: from wagner ([127.0.0.1]) by wagner with esmtp (Exim 3.32 #1 (Debian)) id 15y2WI-0005jn-00; Mon, 29 Oct 2001 13:51:22 +1100 From: Rusty Russell To: Sridhar Samudrala Cc: netfilter-devel@lists.samba.org, netdev@oss.sgi.com Subject: Re: netfilter related issues with PAQ patch In-reply-to: Your message of "Sun, 14 Oct 2001 23:20:00 PDT." Date: Mon, 29 Oct 2001 13:51:22 +1100 Message-Id: Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 11317 Lines: 264 In message you write: > Hi Rusty, > > Sometime back i contacted you to inform about a performance issue that causes > zerocopy sends to be disabled when iptables is configured. You said that you > had a patch to fix this issue, but it needs more work and asked me to remind > you in a month. Hope you have time to look into this now. Alexey accused the netfilter team of being lazy: that hurt, because it's true (for me at least 8). Please find attached a patch v2.4.13 (should apply to any recent kernel) which pushes the responsibility for linearizing the skb to the individual hooks. This patch changes nothing BUT it allows for hooks which don't need to linearize not to (actually, the ingress code may now run a little faster). For iptables, it could be pushed down furthur to the individual modules, but that would break source compatibility for 2.4, so I'm not likely to do that. If this works fine for everyone, I'll send it to Harald Welte for entry into the official kernel. Cheers, Rusty. -- Premature optmztion is rt of all evl. --DK diff -urN -I \$.*\$ --exclude TAGS -X /home/rusty/devel/kernel/kernel-patches/current-dontdiff --minimal linux-2.4.13-official/net/core/netfilter.c working-2.4.13-nfnonlinear/net/core/netfilter.c --- linux-2.4.13-official/net/core/netfilter.c Sat Apr 28 07:15:01 2001 +++ working-2.4.13-nfnonlinear/net/core/netfilter.c Mon Oct 29 12:03:41 2001 @@ -451,11 +451,6 @@ unsigned int verdict; int ret = 0; - /* This stopgap cannot be removed until all the hooks are audited. */ - if (skb_is_nonlinear(skb) && skb_linearize(skb, GFP_ATOMIC) != 0) { - kfree_skb(skb); - return -ENOMEM; - } if (skb->ip_summed == CHECKSUM_HW) { if (outdev == NULL) { skb->ip_summed = CHECKSUM_NONE; diff -urN -I \$.*\$ --exclude TAGS -X /home/rusty/devel/kernel/kernel-patches/current-dontdiff --minimal linux-2.4.13-official/net/ipv4/netfilter/ip_conntrack_core.c working-2.4.13-nfnonlinear/net/ipv4/netfilter/ip_conntrack_core.c --- linux-2.4.13-official/net/ipv4/netfilter/ip_conntrack_core.c Wed Aug 8 01:30:50 2001 +++ working-2.4.13-nfnonlinear/net/ipv4/netfilter/ip_conntrack_core.c Mon Oct 29 13:14:28 2001 @@ -631,6 +631,9 @@ int set_reply; int ret; + if (skb_is_nonlinear(*pskb) && skb_linearize(*pskb, GFP_ATOMIC) != 0) + return NF_DROP; + /* FIXME: Do this right please. --RR */ (*pskb)->nfcache |= NFC_UNKNOWN; diff -urN -I \$.*\$ --exclude TAGS -X /home/rusty/devel/kernel/kernel-patches/current-dontdiff --minimal linux-2.4.13-official/net/ipv4/netfilter/ip_conntrack_standalone.c working-2.4.13-nfnonlinear/net/ipv4/netfilter/ip_conntrack_standalone.c --- linux-2.4.13-official/net/ipv4/netfilter/ip_conntrack_standalone.c Mon Oct 1 05:26:08 2001 +++ working-2.4.13-nfnonlinear/net/ipv4/netfilter/ip_conntrack_standalone.c Mon Oct 29 13:13:28 2001 @@ -169,6 +169,9 @@ const struct net_device *out, int (*okfn)(struct sk_buff *)) { + if (skb_is_nonlinear(*pskb) && skb_linearize(*pskb, GFP_ATOMIC) != 0) + return NF_DROP; + /* We've seen it coming out the other side: confirm it */ return ip_conntrack_confirm(*pskb); } @@ -181,6 +184,9 @@ { struct rtable *rt = (struct rtable *)(*pskb)->dst; + if (skb_is_nonlinear(*pskb) && skb_linearize(*pskb, GFP_ATOMIC) != 0) + return NF_DROP; + /* We've seen it coming out the other side: confirm */ if (ip_confirm(hooknum, pskb, in, out, okfn) != NF_ACCEPT) return NF_DROP; @@ -202,6 +208,8 @@ const struct net_device *out, int (*okfn)(struct sk_buff *)) { + if (skb_is_nonlinear(*pskb) && skb_linearize(*pskb, GFP_ATOMIC) != 0) + return NF_DROP; /* root is playing with raw sockets. */ if ((*pskb)->len < sizeof(struct iphdr) || (*pskb)->nh.iph->ihl * 4 < sizeof(struct iphdr)) { diff -urN -I \$.*\$ --exclude TAGS -X /home/rusty/devel/kernel/kernel-patches/current-dontdiff --minimal linux-2.4.13-official/net/ipv4/netfilter/ip_fw_compat.c working-2.4.13-nfnonlinear/net/ipv4/netfilter/ip_fw_compat.c --- linux-2.4.13-official/net/ipv4/netfilter/ip_fw_compat.c Sat Apr 28 07:15:01 2001 +++ working-2.4.13-nfnonlinear/net/ipv4/netfilter/ip_fw_compat.c Mon Oct 29 13:30:44 2001 @@ -79,6 +79,9 @@ int ret = FW_BLOCK; u_int16_t redirpt; + if (skb_is_nonlinear(*pskb) && skb_linearize(*pskb, GFP_ATOMIC) != 0) + return NF_DROP; + /* Assume worse case: any hook could change packet */ (*pskb)->nfcache |= NFC_UNKNOWN | NFC_ALTERED; if ((*pskb)->ip_summed == CHECKSUM_HW) @@ -183,6 +186,8 @@ const struct net_device *out, int (*okfn)(struct sk_buff *)) { + if (skb_is_nonlinear(*pskb) && skb_linearize(*pskb, GFP_ATOMIC) != 0) + return NF_DROP; return ip_conntrack_confirm(*pskb); } diff -urN -I \$.*\$ --exclude TAGS -X /home/rusty/devel/kernel/kernel-patches/current-dontdiff --minimal linux-2.4.13-official/net/ipv4/netfilter/ip_nat_standalone.c working-2.4.13-nfnonlinear/net/ipv4/netfilter/ip_nat_standalone.c --- linux-2.4.13-official/net/ipv4/netfilter/ip_nat_standalone.c Mon Oct 1 05:26:08 2001 +++ working-2.4.13-nfnonlinear/net/ipv4/netfilter/ip_nat_standalone.c Mon Oct 29 13:27:13 2001 @@ -56,6 +56,9 @@ /* maniptype == SRC for postrouting. */ enum ip_nat_manip_type maniptype = HOOK2MANIP(hooknum); + if (skb_is_nonlinear(*pskb) && skb_linearize(*pskb, GFP_ATOMIC) != 0) + return NF_DROP; + /* We never see fragments: conntrack defrags on pre-routing and local-out, and ip_nat_out protects post-routing. */ IP_NF_ASSERT(!((*pskb)->nh.iph->frag_off @@ -141,6 +144,9 @@ const struct net_device *out, int (*okfn)(struct sk_buff *)) { + if (skb_is_nonlinear(*pskb) && skb_linearize(*pskb, GFP_ATOMIC) != 0) + return NF_DROP; + /* root is playing with raw sockets. */ if ((*pskb)->len < sizeof(struct iphdr) || (*pskb)->nh.iph->ihl * 4 < sizeof(struct iphdr)) @@ -203,6 +209,9 @@ { u_int32_t saddr, daddr; unsigned int ret; + + if (skb_is_nonlinear(*pskb) && skb_linearize(*pskb, GFP_ATOMIC) != 0) + return NF_DROP; /* root is playing with raw sockets. */ if ((*pskb)->len < sizeof(struct iphdr) diff -urN -I \$.*\$ --exclude TAGS -X /home/rusty/devel/kernel/kernel-patches/current-dontdiff --minimal linux-2.4.13-official/net/ipv4/netfilter/ip_tables.c working-2.4.13-nfnonlinear/net/ipv4/netfilter/ip_tables.c --- linux-2.4.13-official/net/ipv4/netfilter/ip_tables.c Mon Oct 1 05:26:08 2001 +++ working-2.4.13-nfnonlinear/net/ipv4/netfilter/ip_tables.c Mon Oct 29 13:19:31 2001 @@ -264,6 +264,11 @@ void *table_base; struct ipt_entry *e, *back; + /* This check cannot be removed until all the extensions are + audited. */ + if (skb_is_nonlinear(*pskb)) + BUG(); + /* Initialization */ ip = (*pskb)->nh.iph; protohdr = (u_int32_t *)ip + ip->ihl; diff -urN -I \$.*\$ --exclude TAGS -X /home/rusty/devel/kernel/kernel-patches/current-dontdiff --minimal linux-2.4.13-official/net/ipv4/netfilter/iptable_filter.c working-2.4.13-nfnonlinear/net/ipv4/netfilter/iptable_filter.c --- linux-2.4.13-official/net/ipv4/netfilter/iptable_filter.c Mon Oct 1 05:26:08 2001 +++ working-2.4.13-nfnonlinear/net/ipv4/netfilter/iptable_filter.c Mon Oct 29 13:22:26 2001 @@ -93,6 +93,11 @@ const struct net_device *out, int (*okfn)(struct sk_buff *)) { + /* This stopgap cannot be removed until all the extensions are + audited. */ + if (skb_is_nonlinear(*pskb) && skb_linearize(*pskb, GFP_ATOMIC) != 0) + return NF_DROP; + return ipt_do_table(pskb, hook, in, out, &packet_filter, NULL); } @@ -103,6 +108,11 @@ const struct net_device *out, int (*okfn)(struct sk_buff *)) { + /* This stopgap cannot be removed until all the extensions are + audited. */ + if (skb_is_nonlinear(*pskb) && skb_linearize(*pskb, GFP_ATOMIC) != 0) + return NF_DROP; + /* root is playing with raw sockets. */ if ((*pskb)->len < sizeof(struct iphdr) || (*pskb)->nh.iph->ihl * 4 < sizeof(struct iphdr)) { diff -urN -I \$.*\$ --exclude TAGS -X /home/rusty/devel/kernel/kernel-patches/current-dontdiff --minimal linux-2.4.13-official/net/ipv4/netfilter/iptable_mangle.c working-2.4.13-nfnonlinear/net/ipv4/netfilter/iptable_mangle.c --- linux-2.4.13-official/net/ipv4/netfilter/iptable_mangle.c Mon Oct 1 05:26:08 2001 +++ working-2.4.13-nfnonlinear/net/ipv4/netfilter/iptable_mangle.c Mon Oct 29 13:23:57 2001 @@ -89,6 +89,10 @@ const struct net_device *out, int (*okfn)(struct sk_buff *)) { + /* This stopgap cannot be removed until all the extensions are + audited. */ + if (skb_is_nonlinear(*pskb) && skb_linearize(*pskb, GFP_ATOMIC) != 0) + return NF_DROP; return ipt_do_table(pskb, hook, in, out, &packet_mangler, NULL); } @@ -131,6 +135,11 @@ u_int8_t tos; u_int32_t saddr, daddr; unsigned long nfmark; + + /* This stopgap cannot be removed until all the extensions are + audited. */ + if (skb_is_nonlinear(*pskb) && skb_linearize(*pskb, GFP_ATOMIC) != 0) + return NF_DROP; /* root is playing with raw sockets. */ if ((*pskb)->len < sizeof(struct iphdr) diff -urN -I \$.*\$ --exclude TAGS -X /home/rusty/devel/kernel/kernel-patches/current-dontdiff --minimal linux-2.4.13-official/net/ipv6/netfilter/ip6table_filter.c working-2.4.13-nfnonlinear/net/ipv6/netfilter/ip6table_filter.c --- linux-2.4.13-official/net/ipv6/netfilter/ip6table_filter.c Mon Oct 1 05:26:08 2001 +++ working-2.4.13-nfnonlinear/net/ipv6/netfilter/ip6table_filter.c Mon Oct 29 13:32:01 2001 @@ -93,6 +93,10 @@ const struct net_device *out, int (*okfn)(struct sk_buff *)) { + /* This stopgap cannot be removed until all the extensions are + audited. */ + if (skb_is_nonlinear(*pskb) && skb_linearize(*pskb, GFP_ATOMIC) != 0) + return NF_DROP; return ip6t_do_table(pskb, hook, in, out, &packet_filter, NULL); } @@ -103,6 +107,10 @@ const struct net_device *out, int (*okfn)(struct sk_buff *)) { + /* This stopgap cannot be removed until all the extensions are + audited. */ + if (skb_is_nonlinear(*pskb) && skb_linearize(*pskb, GFP_ATOMIC) != 0) + return NF_DROP; #if 0 /* root is playing with raw sockets. */ if ((*pskb)->len < sizeof(struct iphdr) diff -urN -I \$.*\$ --exclude TAGS -X /home/rusty/devel/kernel/kernel-patches/current-dontdiff --minimal linux-2.4.13-official/net/ipv6/netfilter/ip6table_mangle.c working-2.4.13-nfnonlinear/net/ipv6/netfilter/ip6table_mangle.c --- linux-2.4.13-official/net/ipv6/netfilter/ip6table_mangle.c Mon Oct 1 05:26:09 2001 +++ working-2.4.13-nfnonlinear/net/ipv6/netfilter/ip6table_mangle.c Mon Oct 29 13:34:39 2001 @@ -89,6 +89,10 @@ const struct net_device *out, int (*okfn)(struct sk_buff *)) { + /* This stopgap cannot be removed until all the extensions are + audited. */ + if (skb_is_nonlinear(*pskb) && skb_linearize(*pskb, GFP_ATOMIC) != 0) + return NF_DROP; return ip6t_do_table(pskb, hook, in, out, &packet_mangler, NULL); } @@ -106,6 +110,10 @@ u_int8_t hop_limit; u_int32_t flowlabel; + /* This stopgap cannot be removed until all the extensions are + audited. */ + if (skb_is_nonlinear(*pskb) && skb_linearize(*pskb, GFP_ATOMIC) != 0) + return NF_DROP; #if 0 /* root is playing with raw sockets. */ if ((*pskb)->len < sizeof(struct iphdr) From owner-netdev@oss.sgi.com Mon Oct 29 01:23:33 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9T9NX127736 for netdev-outgoing; Mon, 29 Oct 2001 01:23:33 -0800 Received: from moutvdom01.kundenserver.de (moutvdom01.kundenserver.de [195.20.224.200]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9T9NP027726 for ; Mon, 29 Oct 2001 01:23:25 -0800 Received: from [195.20.224.204] (helo=mrvdom00.schlund.de) by moutvdom01.kundenserver.de with esmtp (Exim 2.12 #2) id 15y8df-0006OA-00; Mon, 29 Oct 2001 10:23:23 +0100 Received: from pd9506322.dip0.t-ipconnect.de ([217.80.99.34] helo=student.uni-ulm.de) by mrvdom00.schlund.de with esmtp (Exim 2.12 #2) id 15y8de-0007xI-00; Mon, 29 Oct 2001 10:23:23 +0100 Message-ID: <3BDD2001.CFA880EB@student.uni-ulm.de> Date: Mon, 29 Oct 2001 10:23:13 +0100 From: =?iso-8859-15?Q?J=FCrgen?= Nagler X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.4.7-4GB i586) X-Accept-Language: en MIME-Version: 1.0 To: netdev@oss.sgi.com Subject: IPv6 next hop routing by static host routes Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 2670 Lines: 71 This email refers to two topics: 1. /proc/sys/net/ipv6/conf/*/forwarding 2. ICMPv6 Redirects I'm experimenting in the MANET (Mobile Ad Hoc Networks) field and adopted some IPv4 routing protocols for IPv6. All of them used route add IP.of.the.destination gw IP.of.next.hop for creating an IPv4 route of Destination Gateway Genmask Flags Metric Ref Use Iface IP.of.the.dest IP.of.next.hop 255.255.255.255 UGH 0 0 0 eth1 on every node from source to destination. This works fine. Now doing the same for IPv6 with three nodes and a static configuration: node_left intermediate_node node_right MAC 00:02:2d:02:06:f3 00:02:2d:0d:b9:4a 00:02:2d:05:40:0c EUI 202:2dff:fe02:6f3 202:2dff:fe0d:b94a 202:2dff:fe05:400c I used site local adresses (fec0:: prefix) for MANET nodes: node_left: ifconfig eth1 add fec0::202:2dff:fe02:6f3 route -A inet6 add fec0::202:2dff:fe0d:b94a dev eth1 route -A inet6 add fec0::202:2dff:fe05:400c gw fec0::202:2dff:fe0d:b94a intermediate_node: ifconfig eth1 add fec0::202:2dff:fe0d:b94a route -A inet6 add fec0::202:2dff:fe02:6f3 dev eth1 route -A inet6 add fec0::202:2dff:fe05:400c dev eth1 node_right: ifconfig eth1 add fec0::202:2dff:fe05:400c route -A inet6 add fec0::202:2dff:fe0d:b94a dev eth1 route -A inet6 add fec0::202:2dff:fe02:6f3 gw fec0::202:2dff:fe0d:b94a This showed to things: 1. the intermediate host is only forwarding if /proc/sys/net/ipv6/conf/all/forwarding is set to 1. It isn't enough setting .../conf/eth1/forwarding to 1 (with eth1 being the used interface for forwarding). You must start setting .../all/forwarding to 1, after that it's safe to set 0 for all unused interfaces but vice versa setting .../all/forwarding to 0 and setting the only used interface to 1 doesn't work. Why not? 2. after the first ping6 Echo request/reply pair (node_left pinged node_right) the intermediate node sends a redirect for every request/reply back to the source saying it should send it the direct way. The source ignores the redirect (not sending a new direct packet or changing his behavior the next time) and the intermediate node is nevertheless forwarding the packet. Is this correct? I'm using SuSE Linux 7.2 with a 2.4.7 Kernel. Do you need more information about my installation? You can download an ethereal dump of this redirect behavior (missing the first Neighbor solicitation/advertisement pair from node_left to intermediate_node as the MAC of intermediate_node was already cached) at http://medien.informatik.uni-ulm.de/~nagler/projects/DA/redirect.libpcap.gz Thanks in advance, Juergen Nagler From owner-netdev@oss.sgi.com Mon Oct 29 04:18:24 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9TCIO531736 for netdev-outgoing; Mon, 29 Oct 2001 04:18:24 -0800 Received: from blackbird.intercode.com.au (blackbird.intercode.com.au [203.32.101.10]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9TCIG031733 for ; Mon, 29 Oct 2001 04:18:16 -0800 Received: from localhost (jmorris@localhost) by blackbird.intercode.com.au (8.9.3/8.9.3) with ESMTP id XAA24787; Mon, 29 Oct 2001 23:17:43 +1100 Date: Mon, 29 Oct 2001 23:17:43 +1100 (EST) From: James Morris To: Andi Kleen cc: , , Subject: Re: skb->security and friends In-Reply-To: <20011027133000.A20848@wotan.suse.de> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 2589 Lines: 70 On Sat, 27 Oct 2001, Andi Kleen wrote: > On Sat, Oct 27, 2001 at 03:58:22PM +1000, James Morris wrote: > > > > I was hoping that skb->security could be reassigned as a void pointer > > for use by LSM in 2.5, if LSM is accepted into the kernel. > > void pointer alone without any rules for freeing and reference counting > (e.g. what to do with it on a skb_clone() or a skb_copy()) would not > make too much sense. Getting that right would be probably ugly > (similar to rusty's old abandoned ->cb attribute allocator) > This is a little different to skb field reservation. LSM is a general kernel framework which allows different security models to be implemented via a hook-based API. It is up to specific implementations to implement a policy (if any) for the skb security pointer. The LSM API provides a mechanism for this via hooks which are called at important points in the lifecycle of an skb, such as clone and copy as you mention. The hook declarations and documentation may be viewed at: http://sejanus.wirex.com:5555/anno/include/linux/security.h@1.93?nav=index.html|src/.|src/include|src/include/linux > > > > This would be used by LSM modules for maintaining security attributes > > between layers. Note that this may also be useful for Freeswan, as it > > should be possible now to implement ipsec as an LSM module. > > Could you give a more detailed scenario what it would be needed for? > One scenario would be the use of packet labeling to implement mandatory access controls for IP networking. Labeling and access decisions may need to be made based on security conditions at various layers of the stack at specific points in time. For example, an outgoing packet might be labeled with the security attributes of the sending process and/or the object being transmitted. These attributes would need to be propagated between layers, and maintained throughout the life of the packet within the system. The latter is particularly critical in supporting security policy change, which may involve re/invalidating security attributes associated with packets. This scenario might also be extended to security processing performed in hardware, so there is a possibility of interelated labeling and access control decisions at every layer. (I'd prefer to show you some code, but I'm not aware of any projects of this nature which are ready for public consumption yet). Note that if someone was to come up with an acceptable skb field reservation scheme, then LSM would probably be able to make use of it. - James -- James Morris From owner-netdev@oss.sgi.com Mon Oct 29 09:05:58 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9TH5wY22516 for netdev-outgoing; Mon, 29 Oct 2001 09:05:58 -0800 Received: from netcore.fi (netcore.fi [193.94.160.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9TH5q022506 for ; Mon, 29 Oct 2001 09:05:52 -0800 Received: from localhost (pekkas@localhost) by netcore.fi (8.11.6/8.11.6) with ESMTP id f9TH5dL31837; Mon, 29 Oct 2001 19:05:39 +0200 Date: Mon, 29 Oct 2001 19:05:38 +0200 (EET) From: Pekka Savola To: =?iso-8859-15?Q?J=FCrgen?= Nagler cc: Subject: Re: IPv6 next hop routing by static host routes In-Reply-To: <3BDD2001.CFA880EB@student.uni-ulm.de> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=ISO-8859-1 X-MIME-Autoconverted: from 8bit to quoted-printable by netcore.fi id f9TH5dL31837 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by oss.sgi.com id f9TH5r022507 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 2141 Lines: 51 On Mon, 29 Oct 2001, [iso-8859-15] Jürgen Nagler wrote: > This showed to things: > > 1. the intermediate host is only forwarding if > /proc/sys/net/ipv6/conf/all/forwarding is set to 1. It isn't enough > setting .../conf/eth1/forwarding to 1 (with eth1 being the used > interface for forwarding). You must start setting .../all/forwarding to > 1, after that it's safe to set 0 for all unused interfaces but vice > versa setting .../all/forwarding to 0 and setting the only used > interface to 1 doesn't work. Why not? Current design is such that all/forwarding is a general setting for enabling the forwarding itself, eth0/forwarding etc. just enable certain IPv6 features of forwarding (like responding to router solicitations). Please see Documentation/networking/ip-sysctl.txt. > 2. after the first ping6 Echo request/reply pair (node_left pinged > node_right) the intermediate node sends a redirect for every > request/reply back to the source saying it should send it the direct > way. The source ignores the redirect (not sending a new direct packet or > changing his behavior the next time) and the intermediate node is > nevertheless forwarding the packet. Is this correct? When configuring routes: node_left: ifconfig eth1 add fec0::202:2dff:fe02:6f3 route -A inet6 add fec0::202:2dff:fe0d:b94a dev eth1 route -A inet6 add fec0::202:2dff:fe05:400c gw fec0::202:2dff:fe0d:b94a ^^^^ please try using fe80 instead; next-hops should be link-local addresses. The behaviour you're seeing (that is, not accepting the redirects), could be caused by the fact that redirects are sent from the link-local address but the next-hop is site-local; these are compared when receiving the redirect and they don't match. Actually one should not accept non - link-local nexthop's (there is a comment about that in route.c), and this could be one issue big issue caused by that. -- Pekka Savola "Tell me of difficulties surmounted, Netcore Oy not those you stumble over and fall" Systems. Networks. Security. -- Robert Jordan: A Crown of Swords From owner-netdev@oss.sgi.com Mon Oct 29 10:10:30 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9TIAUs31412 for netdev-outgoing; Mon, 29 Oct 2001 10:10:30 -0800 Received: from junk.nocrew.org (mail@junk.nocrew.org [212.73.17.42]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9TIAR031409 for ; Mon, 29 Oct 2001 10:10:27 -0800 Received: from lars by junk.nocrew.org with local (Exim 3.31 #1 (Debian)) for netdev@oss.sgi.com id 15yGrg-0008HR-00; Mon, 29 Oct 2001 19:10:24 +0100 To: netdev@oss.sgi.com Subject: af_packet bug? From: Lars Brinkhoff Organization: nocrew Date: 29 Oct 2001 19:10:24 +0100 Message-ID: <85bsiq2u7j.fsf@junk.nocrew.org> User-Agent: Gnus/5.0808 (Gnus v5.8.8) Emacs/20.7 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 462 Lines: 12 Hello, (Is this list alive?) I have an application that sends raw ethernet frames to a network interface using the af_packet module. This works well except when sending a packet intended for the host that the application is running on. Apparently the outbound frame is never considered for input to the TCP/IP stack. Is this considered a bug or a feature? Is there a way to alter this behaviour so that the host and the application can talk to each other? From owner-netdev@oss.sgi.com Mon Oct 29 10:44:13 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9TIiDE01120 for netdev-outgoing; Mon, 29 Oct 2001 10:44:13 -0800 Received: from ms2.inr.ac.ru (minus.inr.ac.ru [193.233.7.97]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9TIi9001117 for ; Mon, 29 Oct 2001 10:44:09 -0800 Received: (from kuznet@localhost) by ms2.inr.ac.ru (8.6.13/ANK) id VAA01641; Mon, 29 Oct 2001 21:43:46 +0300 From: kuznet@ms2.inr.ac.ru Message-Id: <200110291843.VAA01641@ms2.inr.ac.ru> Subject: Re: af_packet bug? To: lars@nocrew.ORG (Lars Brinkhoff) Date: Mon, 29 Oct 2001 21:43:46 +0300 (MSK) Cc: netdev@oss.sgi.com In-Reply-To: <85bsiq2u7j.fsf@junk.nocrew.org> from "Lars Brinkhoff" at Oct 29, 1 09:15:00 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 780 Lines: 20 Hello! > I have an application that sends raw ethernet frames to a network > interface using the af_packet module. This works well except when > sending a packet intended for the host that the application is running > on. Apparently the outbound frame is never considered for input to > the TCP/IP stack. > > Is this considered a bug or a feature? Neither or them. It is simply the only expected behaviour. Ethernet is simplex media, it does not hear itself and it is impossible to talk to self using it. Well, if you do not believe, try to use single phone to call to yourself. Apparently you want to send to loopback. Continuing analogy: to talk to self, you need not any phone at all and even better not to strain tongue frightening people with selftalking. :-) Alexey From owner-netdev@oss.sgi.com Mon Oct 29 10:58:36 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9TIwaR01989 for netdev-outgoing; Mon, 29 Oct 2001 10:58:36 -0800 Received: from mout03.kundenserver.de (mout03.kundenserver.de [195.20.224.218]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9TIwU001986 for ; Mon, 29 Oct 2001 10:58:30 -0800 Received: from [195.20.224.220] (helo=mrvdom04.kundenserver.de) by mout03.kundenserver.de with esmtp (Exim 2.12 #2) id 15yHc8-0005vn-00; Mon, 29 Oct 2001 19:58:24 +0100 Received: from pd9506310.dip0.t-ipconnect.de ([217.80.99.16] helo=student.uni-ulm.de) by mrvdom04.kundenserver.de with esmtp (Exim 2.12 #2) id 15yHc7-0001Dk-00; Mon, 29 Oct 2001 19:58:24 +0100 Message-ID: <3BDDA6C7.B71CBBD5@student.uni-ulm.de> Date: Mon, 29 Oct 2001 19:58:15 +0100 From: =?iso-8859-15?Q?J=FCrgen?= Nagler X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.4.7-4GB i586) X-Accept-Language: en MIME-Version: 1.0 To: Pekka Savola CC: netdev@oss.sgi.com Subject: Re: IPv6 next hop routing by static host routes References: Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1833 Lines: 53 > Current design is such that all/forwarding is a general setting for > enabling the forwarding itself, eth0/forwarding etc. just enable > certain IPv6 features of forwarding (like responding to router > solicitations). Please see Documentation/networking/ip-sysctl.txt. Thanks, this helped much :-) > When configuring routes: > > node_left: > ifconfig eth1 add fec0::202:2dff:fe02:6f3 > route -A inet6 add fec0::202:2dff:fe0d:b94a dev eth1 > route -A inet6 add fec0::202:2dff:fe05:400c gw fec0::202:2dff:fe0d:b94a > ^^^^ > > please try using fe80 instead; next-hops should be link-local addresses. I knew and tried before but always with the same result: # route -A inet6 add fec0::202:2dff:fe0 gw fe80::202:2dff:fe0d:b94a SIOCADDRT: Invalid argument > The behaviour you're seeing (that is, not accepting the redirects), could > be caused by the fact that redirects are sent from the link-local address > but the next-hop is site-local; these are compared when receiving the > redirect and they don't match. Ok, but if I get the link-local address configured as next hop and the nodes would follow the redirect they wouldn't reach their destination (they don't have a direct link) so why does the intermediate_node recommends the redirect? > Actually one should not accept non - link-local nexthop's (there is a > comment about that in route.c), and this could be one issue big issue > caused by that. Yes, I found this part before but not having the chance to use the fe80 address as next hop I stayed with the fec0 address. Would be this the appropriate ip call? # ip -6 route add fec0::202:2dff:fe05:400c via fe80::202:2dff:fe0d:b94a RTNETLINK answers: Invalid argument You see, it failed also. Can anybody help in this? Thanks in advance, Juergen Nagler From owner-netdev@oss.sgi.com Mon Oct 29 11:08:39 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9TJ8dF02328 for netdev-outgoing; Mon, 29 Oct 2001 11:08:39 -0800 Received: from netcore.fi (netcore.fi [193.94.160.1]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9TJ8Y002325 for ; Mon, 29 Oct 2001 11:08:34 -0800 Received: from localhost (pekkas@localhost) by netcore.fi (8.11.6/8.11.6) with ESMTP id f9TJ8IH32617; Mon, 29 Oct 2001 21:08:18 +0200 Date: Mon, 29 Oct 2001 21:08:18 +0200 (EET) From: Pekka Savola To: =?iso-8859-15?Q?J=FCrgen?= Nagler cc: Subject: Re: IPv6 next hop routing by static host routes In-Reply-To: <3BDDA6C7.B71CBBD5@student.uni-ulm.de> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=ISO-8859-1 X-MIME-Autoconverted: from 8bit to quoted-printable by netcore.fi id f9TJ8IH32617 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by oss.sgi.com id f9TJ8Z002326 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1704 Lines: 47 On Mon, 29 Oct 2001, [iso-8859-15] Jürgen Nagler wrote: > > please try using fe80 instead; next-hops should be link-local addresses. > > I knew and tried before but always with the same result: > > # route -A inet6 add fec0::202:2dff:fe0 gw > fe80::202:2dff:fe0d:b94a > SIOCADDRT: Invalid argument Don't forget 'dev' argument; should work then. > > The behaviour you're seeing (that is, not accepting the redirects), could > > be caused by the fact that redirects are sent from the link-local address > > but the next-hop is site-local; these are compared when receiving the > > redirect and they don't match. > > Ok, but if I get the link-local address configured as next hop and the > nodes would follow the redirect they wouldn't reach their destination > (they don't have a direct link) so why does the intermediate_node > recommends the redirect? Ah, the problem is different than I thought. What you're trying to accomplish, is so-called "multi-link subnet", that is, assigning a single prefix on many links. The router then acts as a proxy between them. For more info: http://www.ietf.org/internet-drafts/draft-thaler-ipngwg-multilink-subnets-01.txt I think the router, if it's set up properly (that is, have host routes to the addresses on the other interface), shouldn't be sending redirects. > Would be this the appropriate ip call? > > # ip -6 route add fec0::202:2dff:fe05:400c via fe80::202:2dff:fe0d:b94a > RTNETLINK answers: Invalid argument Remember 'dev'. -- Pekka Savola "Tell me of difficulties surmounted, Netcore Oy not those you stumble over and fall" Systems. Networks. Security. -- Robert Jordan: A Crown of Swords From owner-netdev@oss.sgi.com Mon Oct 29 12:20:15 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9TKKFX05215 for netdev-outgoing; Mon, 29 Oct 2001 12:20:15 -0800 Received: from e34.bld.us.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9TKKA005206 for ; Mon, 29 Oct 2001 12:20:11 -0800 Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e34.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id PAA50676; Mon, 29 Oct 2001 15:17:37 -0500 Received: from d03nm036.boulder.ibm.com (d03nm036.boulder.ibm.com [9.99.140.36]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id f9TKIrp194708; Mon, 29 Oct 2001 13:18:53 -0700 From: "Venkata Jagana" Importance: Normal Subject: Re: IPv6 next hop routing by static host routes To: =?iso-8859-1?Q?J=FCrgen_Nagler?= Cc: Pekka Savola , netdev@oss.sgi.com X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: Date: Mon, 29 Oct 2001 12:18:36 -0800 X-MIMETrack: Serialize by Router on D03NM036/03/M/IBM(Release 5.0.8 |June 18, 2001) at 10/29/2001 01:18:52 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1198 Lines: 42 Hello Juergen, >> When configuring routes: >> >> node_left: >> ifconfig eth1 add fec0::202:2dff:fe02:6f3 >> route -A inet6 add fec0::202:2dff:fe0d:b94a dev eth1 >> route -A inet6 add fec0::202:2dff:fe05:400c gw fec0::202:2dff:fe0d:b94a >> ^^^^ >> >> please try using fe80 instead; next-hops should be link-local addresses. > I knew and tried before but always with the same result: ># route -A inet6 add fec0::202:2dff:fe0 gw > fe80::202:2dff:fe0d:b94a > SIOCADDRT: Invalid argument I guess the problem is not the link-local address, you can specify site-local addresses too and I had created route entries statically earlier for site-local addresses without any problem. Have you ever tried providing the prefix for the destination address (or network) in your route command? Not sure what the length of your network prefix is but here is the example For eg. route -A inet6 add fec0::202:2dff:fe0/ gw fe80::202:2dff:fe0d:b94a Thanks, Venkat ---------------------------------------------------------------- Venkata R Jagana IBM Linux Technology Centre, Beaverton jagana@us.ibm.com Tel: (503) 578 3430 T/L 775-3430 From owner-netdev@oss.sgi.com Mon Oct 29 15:46:18 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9TNkIH27143 for netdev-outgoing; Mon, 29 Oct 2001 15:46:18 -0800 Received: from moutvdom01.kundenserver.de (moutvdom01.kundenserver.de [195.20.224.200]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9TNkC027134 for ; Mon, 29 Oct 2001 15:46:13 -0800 Received: from [195.20.224.209] (helo=mrvdom02.schlund.de) by moutvdom01.kundenserver.de with esmtp (Exim 2.12 #2) id 15yM6d-00049w-00; Tue, 30 Oct 2001 00:46:11 +0100 Received: from pd9506331.dip0.t-ipconnect.de ([217.80.99.49] helo=student.uni-ulm.de) by mrvdom02.schlund.de with esmtp (Exim 2.12 #2) id 15yM6c-0007Rp-00; Tue, 30 Oct 2001 00:46:10 +0100 Message-ID: <3BDDEA3B.7A0336A@student.uni-ulm.de> Date: Tue, 30 Oct 2001 00:46:03 +0100 From: =?iso-8859-15?Q?J=FCrgen?= Nagler X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.4.7-4GB i586) X-Accept-Language: en MIME-Version: 1.0 To: Pekka Savola CC: netdev@oss.sgi.com Subject: Re: IPv6 next hop routing by static host routes References: Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: 7bit Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1838 Lines: 56 Pekka Savola wrote: > > > I knew and tried before but always with the same result: > > > > # route -A inet6 add fec0::202:2dff:fe0 gw > > fe80::202:2dff:fe0d:b94a > > SIOCADDRT: Invalid argument > > Don't forget 'dev' argument; should work then. Yes, again you're right, this was the problem :-) Thanks! > > > The behaviour you're seeing (that is, not accepting the redirects), could > > > be caused by the fact that redirects are sent from the link-local address > > > but the next-hop is site-local; these are compared when receiving the > > > redirect and they don't match. > > > > Ok, but if I get the link-local address configured as next hop and the > > nodes would follow the redirect they wouldn't reach their destination > > (they don't have a direct link) so why does the intermediate_node > > recommends the redirect? > > Ah, the problem is different than I thought. > > What you're trying to accomplish, is so-called "multi-link subnet", that > is, assigning a single prefix on many links. The router then acts as a > proxy between them. I understand, the router sees the same prefix and that's the reason for the redirects. > For more info: > > http://www.ietf.org/internet-drafts/draft-thaler-ipngwg-multilink-subnets-01.txt Ok, I'll deepen into that. > > Would be this the appropriate ip call? > > > > # ip -6 route add fec0::202:2dff:fe05:400c via fe80::202:2dff:fe0d:b94a > > RTNETLINK answers: Invalid argument > > Remember 'dev'. Appending "dev eth1" to both "route -A inet6 add" and "ip -6 route add" created an appropriate route entry but a little bit different. The route command created Flags "UGH" and Metric 1, the ip command created Flags "UG" and Metric 1024. I'm just wondering but will use the route command whose result is ok for me. Thanks all for your great help! Regards, Juergen Nagler From owner-netdev@oss.sgi.com Mon Oct 29 23:39:22 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9U7dMT03953 for netdev-outgoing; Mon, 29 Oct 2001 23:39:22 -0800 Received: from junk.nocrew.org (mail@junk.nocrew.org [212.73.17.42]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9U7dI003949 for ; Mon, 29 Oct 2001 23:39:18 -0800 Received: from lars by junk.nocrew.org with local (Exim 3.31 #1 (Debian)) id 15yTUP-0005sL-00; Tue, 30 Oct 2001 08:39:13 +0100 To: kuznet@ms2.inr.ac.ru Cc: netdev@oss.sgi.com Subject: Re: af_packet bug? References: <200110291843.VAA01641@ms2.inr.ac.ru> From: Lars Brinkhoff Organization: nocrew Date: 30 Oct 2001 08:39:13 +0100 In-Reply-To: <200110291843.VAA01641@ms2.inr.ac.ru> Message-ID: <85lmht1sri.fsf@junk.nocrew.org> User-Agent: Gnus/5.0808 (Gnus v5.8.8) Emacs/20.7 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1369 Lines: 27 kuznet@ms2.inr.ac.ru writes: > Lars Brinkhoff writes: > > I have an application that sends raw ethernet frames to a network > > interface using the af_packet module. This works well except when > > sending a packet intended for the host that the application is > > running on. Apparently the outbound frame is never considered for > > input to the TCP/IP stack. Is this considered a bug or a feature? > > Neither or them. It is simply the only expected behaviour. Ethernet > is simplex media, it does not hear itself and it is impossible to > talk to self using it. Apparently you want to send to loopback. The application is in fact a computer emulator. Among other things, the emulator emulates a network interface using the af_packet facility. The emulator can run an operating system with a TCP/IP stack, and the emulated machine will have a different IP number than the Linux host. The emulated machine communicates with other network hosts by sending ethernet packets using af_packet. But since this doesn't work for talking to the host machine the emulator is running on, the emulator must strip off the ethernet header and send those packets to the loopback device. Does that sound like a plausible solution? -- Lars Brinkhoff http://lars.nocrew.org/ Linux, GCC, PDP-10 Brinkhoff Consulting http://www.brinkhoff.se/ programming From owner-netdev@oss.sgi.com Tue Oct 30 07:44:20 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9UFiKL28341 for netdev-outgoing; Tue, 30 Oct 2001 07:44:20 -0800 Received: from Cantor.suse.de (ns.suse.de [213.95.15.193]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9UFiI028337 for ; Tue, 30 Oct 2001 07:44:18 -0800 Received: from Hermes.suse.de (Hermes.suse.de [213.95.15.136]) by Cantor.suse.de (Postfix) with ESMTP id 523081E144; Tue, 30 Oct 2001 16:44:12 +0100 (MET) Date: Tue, 30 Oct 2001 16:44:11 +0100 From: Andi Kleen To: Lars Brinkhoff Cc: kuznet@ms2.inr.ac.ru, netdev@oss.sgi.com Subject: Re: af_packet bug? Message-ID: <20011030164411.A10318@wotan.suse.de> References: <200110291843.VAA01641@ms2.inr.ac.ru> <85lmht1sri.fsf@junk.nocrew.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.3.16i In-Reply-To: <85lmht1sri.fsf@junk.nocrew.org>; from lars.spam@nocrew.org on Tue, Oct 30, 2001 at 08:39:13AM +0100 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 440 Lines: 11 > The emulated machine communicates with other network hosts by sending > ethernet packets using af_packet. But since this doesn't work for > talking to the host machine the emulator is running on, the emulator > must strip off the ethernet header and send those packets to the > loopback device. Does that sound like a plausible solution? The tun and/or ethertap devices have been exactly designed to make such things possible. -Andi From owner-netdev@oss.sgi.com Tue Oct 30 09:52:24 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9UHqOx32204 for netdev-outgoing; Tue, 30 Oct 2001 09:52:24 -0800 Received: from ms2.inr.ac.ru (minus.inr.ac.ru [193.233.7.97]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9UHqM032198 for ; Tue, 30 Oct 2001 09:52:22 -0800 Received: (from kuznet@localhost) by ms2.inr.ac.ru (8.6.13/ANK) id UAA19512; Tue, 30 Oct 2001 20:52:08 +0300 From: kuznet@ms2.inr.ac.ru Message-Id: <200110301752.UAA19512@ms2.inr.ac.ru> Subject: Re: af_packet bug? To: lars.spam@nocrew.org (Lars Brinkhoff) Date: Tue, 30 Oct 2001 20:52:08 +0300 (MSK) Cc: netdev@oss.sgi.com In-Reply-To: <85lmht1sri.fsf@junk.nocrew.org> from "Lars Brinkhoff" at Oct 30, 1 08:39:13 am X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 199 Lines: 10 Hello! > Does that sound like a plausible solution? No. It looks like you use packet socket for a purpose where it cannot be useful at all. This cannot work. I guess you want ethertap. Alexey From owner-netdev@oss.sgi.com Tue Oct 30 17:37:41 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.2/8.11.3) id f9V1bf917103 for netdev-outgoing; Tue, 30 Oct 2001 17:37:41 -0800 Received: from noxmail.sandelman.ottawa.on.ca (cyphermail.sandelman.ottawa.on.ca [192.139.46.78]) by oss.sgi.com (8.11.2/8.11.3) with SMTP id f9V1bb017100 for ; Tue, 30 Oct 2001 17:37:38 -0800 Received: from marajade.sandelman.ottawa.on.ca ([2001:410:402:2:204:76ff:fe2d:8c]) by noxmail.sandelman.ottawa.on.ca (8.11.6/8.11.6) with ESMTP id f9V1bYR04113 (using TLSv1/SSLv3 with cipher EDH-RSA-DES-CBC3-SHA (168 bits) verified OK); Tue, 30 Oct 2001 20:37:36 -0500 (EST) Received: from marajade.sandelman.ottawa.on.ca (localhost [[UNIX: localhost]]) by marajade.sandelman.ottawa.on.ca (8.11.6/8.11.0) with ESMTP id f9V1Zrx24494; Tue, 30 Oct 2001 20:35:56 -0500 (EST) Message-Id: <200110310135.f9V1Zrx24494@marajade.sandelman.ottawa.on.ca> To: Lars Brinkhoff cc: netdev@oss.sgi.com Subject: Re: af_packet bug? In-reply-to: Your message of "30 Oct 2001 08:39:13 +0100." <85lmht1sri.fsf@junk.nocrew.org> Mime-Version: 1.0 (generated by tm-edit 7.108) Content-Type: text/plain; charset=US-ASCII Date: Tue, 30 Oct 2001 20:35:53 -0500 From: Michael Richardson Sender: owner-netdev@oss.sgi.com Precedence: bulk Content-Length: 1257 Lines: 24 >>>>> "Lars" == Lars Brinkhoff writes: Lars> The application is in fact a computer emulator. Among other Lars> things, the emulator emulates a network interface using the Lars> af_packet facility. The emulator can run an operating system with Lars> a TCP/IP stack, and the emulated machine will have a different IP Lars> number than the Linux host. Lars> The emulated machine communicates with other network hosts by Lars> sending ethernet packets using af_packet. But since this doesn't Lars> work for talking to the host machine the emulator is running on, Lars> the emulator must strip off the ethernet header and send those Lars> packets to the loopback device. Does that sound like a plausible Lars> solution? Sure... Or, use ethertap0, and use bridging on the host to get onto the real wire. That will solve all the problems with host communications. ] ON HUMILITY: to err is human. To moo, bovine. | firewalls [ ] Michael Richardson, Sandelman Software Works, Ottawa, ON |net architect[ ] mcr@sandelman.ottawa.on.ca http://www.sandelman.ottawa.on.ca/ |device driver[ ] panic("Just another NetBSD/notebook using, kernel hacking, security guy"); [