From owner-stp@oss.sgi.com Thu Feb 10 16:11:37 2000 Received: by oss.sgi.com id ; Thu, 10 Feb 2000 16:11:27 -0800 Received: from pneumatic-tube.sgi.com ([204.94.214.22]:2422 "EHLO pneumatic-tube.sgi.com") by oss.sgi.com with ESMTP id ; Thu, 10 Feb 2000 16:11:12 -0800 Received: from lhotse.engr.sgi.com (lhotse.engr.sgi.com [150.166.40.83]) by pneumatic-tube.sgi.com (980327.SGI.8.8.8-aspam/980310.SGI-aspam) via ESMTP id QAA03727 for ; Thu, 10 Feb 2000 16:13:59 -0800 (PST) mail_from (aman@cthulhu.engr.sgi.com) Received: from engr.sgi.com (localhost [127.0.0.1]) by lhotse.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id QAA17214; Thu, 10 Feb 2000 16:09:36 -0800 (PST) Message-ID: <38A35340.C5886C5D@engr.sgi.com> Date: Thu, 10 Feb 2000 16:09:36 -0800 From: Aman Singla Organization: SGI X-Mailer: Mozilla 4.7C-SGI [en] (X11; I; IRIX 6.5 IP32) X-Accept-Language: en MIME-Version: 1.0 To: "Mailing List, Linux Kernel" CC: "Mailing List, Acenic" , stp@oss.sgi.com Subject: Scheduled Transfer Protocol on Linux Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-stp@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;stp-outgoing Hi, I'd like to announce the source code release for Scheduled Transfer Protocol (STP) support in Linux. SGI played a major role in the development of STP to address 2 sides of the performance coin at the same time: high bandwidth and low latency. Current GSN hardware with STP achieves 720MB/sec on a single stream in bandwidth and 6 usec in latency validating these combined design goals. SGI is interested in bringing the STP technology to Linux, and to lower cost interfaces like Gigabit Ethernet. This is an experimental 0.1 alpha release providing some pieces towards this goal. This release is not intended to be integrated into the kernel in its current state of development. We'd like this release to serve as the framework to define our goals for the project and the issues we're attempting to address, get feedback regarding our approach for its continued development, explore avenues to leverage off and provide leverage to any related work, and to invite contributions from everyone interested. The code constitutes a minimal patch to the kernel to enable support for STP. The core of the STP functionality can be loaded/unloaded as modules (or linked into the kernel). Please refer to http://oss.sgi.com/projects/stp to download the code, to get more information about STP and the project, or to join the STP development mailing list. thanks, Aman Singla (aman@sgi.com) SGI From owner-stp@oss.sgi.com Thu Feb 17 09:54:18 2000 Received: by oss.sgi.com id ; Thu, 17 Feb 2000 09:54:09 -0800 Received: from pneumatic-tube.sgi.com ([204.94.214.22]:2589 "EHLO pneumatic-tube.sgi.com") by oss.sgi.com with ESMTP id ; Thu, 17 Feb 2000 09:53:49 -0800 Received: from lhotse.engr.sgi.com (lhotse.engr.sgi.com [163.154.35.41]) by pneumatic-tube.sgi.com (980327.SGI.8.8.8-aspam/980310.SGI-aspam) via ESMTP id JAA04124; Thu, 17 Feb 2000 09:56:43 -0800 (PST) mail_from (aman@cthulhu.engr.sgi.com) Received: from engr.sgi.com (localhost [127.0.0.1]) by lhotse.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id JAA03501; Thu, 17 Feb 2000 09:52:32 -0800 (PST) Message-ID: <38AC355F.2363E9A8@engr.sgi.com> Date: Thu, 17 Feb 2000 09:52:31 -0800 From: Aman Singla Organization: SGI X-Mailer: Mozilla 4.7C-SGI [en] (X11; I; IRIX 6.5 IP32) X-Accept-Language: en MIME-Version: 1.0 To: Ralf Baechle CC: stp@oss.sgi.com Subject: Re: Scheduled Transfer Protocol on Linux References: <200002132058.MAA22259@work.bitmover.com> <38A85787.FB76CFCC@engr.sgi.com> <20000216161038.F3472@uni-koblenz.de> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-stp@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;stp-outgoing Ralf, STP does require putting some of the protocol state in the NIC- be it in h/w (GSN) or in firmware (like we're attempting to do with the Alteon GbE cards). But in order to support STP over any unmodified NIC and to use as a development tool, I've implemented a virtual device driver (stvd), which simulates pieces of STP functionality required in the NIC in s/w. This can be used to run STP over most NICs without modifying their driver/firmware. But the performance in this case would be at best comparable to TCP/IP. I'll try to find out about the GENROCO HIPPI NIC - in terms of any STP support it might offer, and get back to you. :a > Quick question - is STP intended to be implemented in software or in > hardware? I do have Genroco HFP832 HIPPI NICs in my pair of Origins > which I'm using and I'm somewhat interested in playing with STP on > that platform base as soon as my Linux port to the O200 stabilizes. From owner-stp@oss.sgi.com Thu Feb 17 10:09:18 2000 Received: by oss.sgi.com id ; Thu, 17 Feb 2000 10:08:59 -0800 Received: from deliverator.sgi.com ([204.94.214.10]:39536 "EHLO deliverator.sgi.com") by oss.sgi.com with ESMTP id ; Thu, 17 Feb 2000 10:08:41 -0800 Received: from nodin.corp.sgi.com (nodin.corp.sgi.com [192.26.51.193]) by deliverator.sgi.com (980309.SGI.8.8.8-aspam-6.2/980310.SGI-aspam) via ESMTP id KAA15793 for ; Thu, 17 Feb 2000 10:04:10 -0800 (PST) mail_from (aman@cthulhu.engr.sgi.com) Received: from lhotse.engr.sgi.com (lhotse.engr.sgi.com [163.154.35.41]) by nodin.corp.sgi.com (980427.SGI.8.8.8/980728.SGI.AUTOCF) via ESMTP id KAA82023 for ; Thu, 17 Feb 2000 10:08:40 -0800 (PST) Received: from engr.sgi.com (localhost [127.0.0.1]) by lhotse.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id KAA03532; Thu, 17 Feb 2000 10:06:06 -0800 (PST) Message-ID: <38AC388E.B96032E2@engr.sgi.com> Date: Thu, 17 Feb 2000 10:06:06 -0800 From: Aman Singla Organization: SGI X-Mailer: Mozilla 4.7C-SGI [en] (X11; I; IRIX 6.5 IP32) X-Accept-Language: en MIME-Version: 1.0 To: Michael Werner CC: stp@oss.sgi.com Subject: Re: stp for linux References: <38A8955A.F11DD254@quantum.ece.ucsb.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-stp@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;stp-outgoing Hi Michael- Thanks for a brief description of your project. Regarding routing, we do support IP encapsulation of STP - when the upper layer protocol using STP uses INET addresses. The various sequences defined for STP make it useful for both high b/w and low latency scenarios - whether through the kernel or from user space. Of course, I'll need more details about your project to say something more sensible, but this is what I can come up with at the top of my head: STP establishes connections from a NIC to NIC; and the address space is the ports (in h/w or s/w) on the NIC. The higher layer addressing (like INET) is dealt with by the upper layer protocol e.g. we have an implementation that uses INET addressing on STP; SCSI might want to use different addressing (don't know). For your purpose - I could see your protocol being an upper layer protocol on STP; you want to use something other than INET addresses, and that's possible. This will let you leverage off the low latency, high b/w transport mechanism that STP provides (on a variety of interfaces, hopefully :-) Let me know if this makes sense. I'd be very interested in discussing this in more detail with you. thanks, :a > > Hi Aman, > in your original post there was some comment above collaborating with > other projects which I like to see too. > Recently, I have started writing a network protocol for linux but it is > in its early stages ;-) > I am interested in distributed computing, clusters, distributed > operating systems, group communication, etc. > Given that many users will want to put together linux clusters of some > sort it is appropriate > to provide the necessary mechanisms in the kernel to do this in a nice > way. One way that is interesting > to pursue is to add some mecahanisms for location transparency which are > lacking in Unix. I have > started to do this for Linux. The basic idea is simple and comes from > the FLIP protocol of Amoeba. > Replace IP network layer with another protocol which works without the > upper layers knowing > which machine a process is on. Of course, one wants to have the nice > features which you have seen (?) for > STP as well -- large bandwidth and low latency. I have added code to > Linux 2.3.44 which adds > a random address to every process created called the flip address (64 > bits). I have departed from the > original protocol when it comes to routing though. My guess is that > every Linux machine in a cluster > will have an IP stack anyway and so why not use it for routing. Also, I > propose that the routing table > for flip addresses on a machine only contains nonlocal addresses. The > process table itself with suitable > hashing should be enough for flip to work on a single machine. This > again departs from the original. > Upper protocol layers needn't care about location. For a tightly coupled > cluster this should be efficient too > but the basic abstraction makes building distributed services simpler I > am guessing though since I haven't > actually done it. Many of the features I wanted for this FLIP/Linux > protocol apart from addressing appear > in STP. I wonder whether the too can be blended to give a rather nice > hybrid for clusters. > > Yours, > Mike From owner-stp@oss.sgi.com Thu Feb 17 15:52:54 2000 Received: by oss.sgi.com id ; Thu, 17 Feb 2000 15:52:45 -0800 Received: from pneumatic-tube.sgi.com ([204.94.214.22]:37726 "EHLO pneumatic-tube.sgi.com") by oss.sgi.com with ESMTP id ; Thu, 17 Feb 2000 15:52:32 -0800 Received: from lhotse.engr.sgi.com (lhotse.engr.sgi.com [163.154.35.41]) by pneumatic-tube.sgi.com (980327.SGI.8.8.8-aspam/980310.SGI-aspam) via ESMTP id PAA00200 for ; Thu, 17 Feb 2000 15:55:27 -0800 (PST) mail_from (aman@cthulhu.engr.sgi.com) Received: from engr.sgi.com (localhost [127.0.0.1]) by lhotse.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id PAA04541; Thu, 17 Feb 2000 15:50:54 -0800 (PST) Message-ID: <38AC895E.840B8227@engr.sgi.com> Date: Thu, 17 Feb 2000 15:50:54 -0800 From: Aman Singla Organization: SGI X-Mailer: Mozilla 4.7C-SGI [en] (X11; I; IRIX 6.5 IP32) X-Accept-Language: en MIME-Version: 1.0 To: michael@quantum.ece.ucsb.edu CC: stp@oss.sgi.com Subject: Re: stp for linux References: <38A8955A.F11DD254@quantum.ece.ucsb.edu> <38AC388E.B96032E2@engr.sgi.com> <38AC84C5.FD7A412F@quantum.ece.ucsb.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-stp@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;stp-outgoing > Thanks for your comments. It would be wonderful to use STP > as the underlying protocol with a FLIP addressing scheme rather > than INET. FLIP was originally designed for Amoeba which uses > cryptographically secure ports for naming and communication end-points > by the RPC layer above FLIP. This is described briefly in Tanenbaum's > book on "Distributed Operating Systems". Since STP uses ports on the NIC's > for communication one needs a mapping between MAC addresses and STP ports. Note that those are STP ports on a particular NIC. So the address space from a host point of view is mac_addr+stp_port. Of course the mac_addr is determined by the NIC that is used to make the connection. > FLIP would then need to map its addresses to STP ports instead of INET > addresses > which isn't a problem. This mapping would need to be dynamic to provide the > necessary location transparency for distributed systems. This sounds really > cool to me. > Unfortunately, a quick glance over your code was tough going because it is > interwined > with regular INET4 linux code. I started from scratch for FLIP and don't call > any INET > routines currently. But I haven't got that far either :-) Sorry about that; infact the intent IS to make the stp code totally upper layer protocol clean! but since INET is the only upper layer protocol in this release its isn't very so; will start happening as we start supporting scsi over STP (or your work could be the reason :-) In terms of reading the code - in the core directory, any filename which is stp*inet* named, consider it to be part of the INET upper layer; rest of the code is mostly stp clean - except for back calls to the inet upper layer which will have to be abstracted out into a data structure some time. > Keep in touch. > By the way I develop kernel code on a 21264 alpha system so it is 64 bit > clean hopefully. Have been playing with this only on IA32 so far.. but would be very interested an responsive to 64 bit bugs (and fixes :-) :a From owner-stp@oss.sgi.com Sun Feb 20 21:38:43 2000 Received: by oss.sgi.com id ; Sun, 20 Feb 2000 21:38:34 -0800 Received: from quantum.ece.ucsb.edu ([128.111.185.95]:48907 "EHLO quantum.ece.ucsb.edu") by oss.sgi.com with ESMTP id ; Sun, 20 Feb 2000 21:38:16 -0800 Received: (from michael@localhost) by quantum.ece.ucsb.edu (8.8.6 (PHNE_17135)/8.8.6) id VAA10007 for stp@oss.sgi.com; Sun, 20 Feb 2000 21:38:15 -0800 (PST) Date: Sun, 20 Feb 2000 21:38:15 -0800 (PST) From: Michael Werner Message-Id: <200002210538.VAA10007@quantum.ece.ucsb.edu> To: stp@oss.sgi.com Subject: 64 bit cleanup Mime-Version: 1.0 Content-Type: text/plain; charset=X-roman8 Content-Transfer-Encoding: 7bit Sender: owner-stp@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;stp-outgoing filename: stp_inet.c linenumber: 290 original: if ((err = (int)sk->prot->accept(sk, flags, &err))) release_sock(sk); return err; new: if ((newsk = sk->prot->accept(sk, flags, &err))==NULL) { release_sock(sk); return -EINVAL;  eror value probably wrong -- need to check it From owner-stp@oss.sgi.com Mon Feb 21 17:32:11 2000 Received: by oss.sgi.com id ; Mon, 21 Feb 2000 17:32:01 -0800 Received: from pneumatic-tube.sgi.com ([204.94.214.22]:25441 "EHLO pneumatic-tube.sgi.com") by oss.sgi.com with ESMTP id ; Mon, 21 Feb 2000 17:31:46 -0800 Received: from dsl-lhotse.corp.sgi.com ([192.132.126.170]) by pneumatic-tube.sgi.com (980327.SGI.8.8.8-aspam/980310.SGI-aspam) via SMTP id RAA03096 for ; Mon, 21 Feb 2000 17:34:44 -0800 (PST) mail_from (aman@engr.sgi.com) Received: from engr.sgi.com (localhost [127.0.0.1]) by dsl-lhotse.corp.sgi.com (950413.SGI.8.6.12/970903.SGI.AUTOCF) via ESMTP id BAA04301; Tue, 22 Feb 2000 01:27:37 GMT Message-ID: <38B1E608.CF4554AC@engr.sgi.com> Date: Mon, 21 Feb 2000 17:27:36 -0800 From: Aman Singla Organization: Silicon Graphics, Inc. X-Mailer: Mozilla 4.51C-SGI [en] (X11; I; IRIX 6.2 IP22) X-Accept-Language: en MIME-Version: 1.0 To: Michael Werner CC: stp@oss.sgi.com Subject: Re: 64 bit cleanup References: <200002210538.VAA10007@quantum.ece.ucsb.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-stp@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;stp-outgoing Michael- My original intent in writing stp/core/stp_inet.c instead of patching net/ipv4/af_inet.c was to develop more generic code which both TCP/STP could leverage off without requiring duplication. The current af_inet is very TCP specific and thus required me to splice in the stp protocol through the proto_ops vector instead of the proto vector. Thus in stp_inet_accept I've tried to pull in functionality common to most INET protocols - tcp and stp in this case. Accordingly, as the comment at the beginning of stp_inet_accept says, I'd prefer the sk->prot->accept to just return the error code instead of the newsk (the newsk will be picked off a list in (stp)_inet_accept itself; but I didn't want to define my own proto vector type (because the current declaration of prot->accept is to return a sock*). Since, currently stp is the only protocol to use stp_inet.c and its prot->accept is NULL, it shouldn't matter; but for another protocol to leverage off stp_inet.c it should define its prot->accept to return an errorcode (or 0) instead of a newsk. Hope this explanation makes it clearer. Please let me know if you have any issues with it. thanks, :a Michael Werner wrote: > > filename: stp_inet.c > linenumber: 290 > > original: if ((err = (int)sk->prot->accept(sk, flags, &err))) > release_sock(sk); > return err; > > new: if ((newsk = sk->prot->accept(sk, flags, &err))==NULL) { > release_sock(sk); > return -EINVAL; >  > eror value probably wrong -- need to check it From owner-stp@oss.sgi.com Tue Feb 29 13:18:49 2000 Received: by oss.sgi.com id ; Tue, 29 Feb 2000 13:18:40 -0800 Received: from mars.iol.unh.edu ([132.177.121.222]:41243 "EHLO iol.unh.edu") by oss.sgi.com with ESMTP id ; Tue, 29 Feb 2000 13:18:35 -0800 Received: from localhost (cwl@localhost) by iol.unh.edu (8.9.3/8.9.3) with ESMTP id QAA16862 for ; Tue, 29 Feb 2000 16:18:06 -0500 (EST) Date: Tue, 29 Feb 2000 16:18:06 -0500 From: Chris Loveland To: stp@oss.sgi.com Subject: scsi on stp Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-stp@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;stp-outgoing i have a couple questions regarding the currently defined mapping of scsi on stp. im primarily thinking of stp running on gigabit ethernet. the SST standard defines the error recovery mechanism to be based on the transaction. a scsi transaction running on ethernet is likely to consist of a decent number of data frames, maybe on the order of 50 or so. my understanding of things is that the loss of a single data frame will require the entire transaction to be retried. am i correct in all this? one scenario which i would think would be fairly common is for an initiator to make data requests of a number of targets at the same time. if several targets respond at the same time you have a situation where multiple flows of data are arriving at the switch and are destined for the same port, resulting in packet loss if the switch's buffering is not sufficient. are there reasons why this sort of thing is not likely to occur? i am basically wondering what the reasons are for feeling that recovery at the level of a transaction is sufficient when running on a low level protocol such as ethernet. if it appears that i am misunderstanding something about STP or STT, which is quite possible, please correct me. chris From owner-stp@oss.sgi.com Tue Feb 29 17:54:52 2000 Received: by oss.sgi.com id ; Tue, 29 Feb 2000 17:54:42 -0800 Received: from pneumatic-tube.sgi.com ([204.94.214.22]:48719 "EHLO pneumatic-tube.sgi.com") by oss.sgi.com with ESMTP id ; Tue, 29 Feb 2000 17:54:28 -0800 Received: from nodin.corp.sgi.com (nodin.corp.sgi.com [192.26.51.193]) by pneumatic-tube.sgi.com (980327.SGI.8.8.8-aspam/980310.SGI-aspam) via ESMTP id RAA02078 for ; Tue, 29 Feb 2000 17:57:35 -0800 (PST) mail_from (aman@cthulhu.engr.sgi.com) Received: from lhotse.engr.sgi.com (lhotse.engr.sgi.com [163.154.35.41]) by nodin.corp.sgi.com (980427.SGI.8.8.8/980728.SGI.AUTOCF) via ESMTP id RAA14560 for ; Tue, 29 Feb 2000 17:54:27 -0800 (PST) Received: from engr.sgi.com (localhost [127.0.0.1]) by lhotse.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id RAA04435; Tue, 29 Feb 2000 17:52:04 -0800 (PST) Message-ID: <38BC77C4.EF66A13@engr.sgi.com> Date: Tue, 29 Feb 2000 17:52:04 -0800 From: Aman Singla Organization: SGI X-Mailer: Mozilla 4.7C-SGI [en] (X11; I; IRIX 6.5 IP32) X-Accept-Language: en MIME-Version: 1.0 To: Chris Loveland CC: stp@oss.sgi.com Subject: Re: scsi on stp References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-stp@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;stp-outgoing Chris, The data hierarchy associated with Long Message transfers in STP is transfer->block->STU; a transfer consists of one or many blocks and a block consists of one or many STUs. A SCSI transaction maps to a STP transfer. The STP stack takes care of retransmissions for missed/dropped blocks based on timeouts (or any other mechanism like missed ordering of blocks etc.). Further the NIC h/w or firmware, if capable, may take care of STU retransmission for dropped STUs. For example on GbE a frame would correspond to a STU, and lets say a block corresponds to 64 STUs; now, if a frame is dropped/lost, the media/physical layer/NIC, if capable, could have the remote NIC resend the STU - generally resulting in the protocol stack on the host always getting all the blocks; but if the NIC can't support STU retransmission, the protocol stack will observe a dropped block (a block isn't deemed recd. until all STUs are recd) and would request retransmission of the block by resending a CTS. The entire transfer will (generally speaking) never have to be redone. Hope this helps, :a > > i have a couple questions regarding the currently defined mapping of scsi > on stp. im primarily thinking of stp running on gigabit ethernet. the > SST standard defines the error recovery mechanism to be based on the > transaction. a scsi transaction running on ethernet is likely to consist > of a decent number of data frames, maybe on the order of 50 or so. my > understanding of things is that the loss of a single data frame will > require the entire transaction to be retried. am i correct in all > this? one scenario which i would think would be fairly common is for an > initiator to make data requests of a number of targets at the same > time. if several targets respond at the same time you have a situation > where multiple flows of data are arriving at the switch and are destined > for the same port, resulting in packet loss if the switch's buffering is > not sufficient. are there reasons why this sort of thing is not likely to > occur? i am basically wondering what the reasons are for feeling that > recovery at the level of a transaction is sufficient when running on a low > level protocol such as ethernet. if it appears that i am misunderstanding > something about STP or STT, which is quite possible, please correct me. > > chris