From owner-kaio@oss.sgi.com Tue Apr 3 13:05:36 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.3/8.11.3) id f33K5aB16721 for kaio-outgoing; Tue, 3 Apr 2001 13:05:36 -0700 Received: from firewall.philstone.com (cy57850-a.rdondo1.ca.home.com [24.5.132.106]) by oss.sgi.com (8.11.3/8.11.3) with ESMTP id f33K5ZM16718 for ; Tue, 3 Apr 2001 13:05:35 -0700 Received: from hellman (unknown [192.168.1.2]) by firewall.philstone.com (Postfix) with ESMTP id 3334886AC for ; Tue, 3 Apr 2001 13:05:30 -0700 (PDT) Date: Mon, 02 Apr 2001 18:53:22 -0700 From: Christopher Smith To: kaio@oss.sgi.com Subject: Finally got things right... ;-) (signals & threads in LInux) Message-ID: <7310000.986262802@hellman> X-Mailer: Mulberry/2.0.6 (Linux/x86 Demo) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="==========1811599384==========" Sender: owner-kaio@oss.sgi.com Precedence: bulk --==========1811599384========== Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Hey there! I've been playing around with KAIO for a while now, and I've been somewhat frustrated with it. Not so much because of shortcomings in KAIO itself, but with the Linux kernel. One of the annoying things about using KAIO (or any other poll/signal approach to IO) is that you'd really like it if a thread B could receive the signal indicating completion of an I/O request posted by thread A. Realtime POSIX extensions and POSIX threads make this possible with AIO, but unfortunately Linux's implementation of threading causes a signal to always be returned to the invoking thread. To make it clear what I'm trying to accomplish I've included a really basic test program which demonstrates how things are supposed to work (it should work on a POSIX compliant OS, but it won't work on standard Linux with the KAIO patch). Please excuse all the commented code as I've been experimenting with Linux's broken thread implementation. Right now I've got this working by doing a small hack to the 2.4.3 kernel and the KAIO code. I'm going to tidy this up and maybe move part of the fixing into LinuxThreads (right now LinuxThreads does not use the new CLONE_THREAD flag and it should). Either way the patch is a complete hack, but it does serve as a demonstration of how efficient this code could be. --Chris --==========1811599384========== Content-Type: text/x-csrc; charset=us-ascii; name="test.c" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="test.c"; size=3128 #include #include #include #include #include #include #include #include #include #include #define USE_AIO int TEST_SIGNAL; char *buffer; int fd; pthread_t thread; #ifndef USE_AIO int readFile(int fd, char* buffer, int size) { return read(fd, buffer, size); } #else int readFile(int fd, char* buffer, int size) { struct aiocb cb; struct sigevent sigevt; sigset_t set, oset; siginfo_t info; int error; cb.aio_fildes = fd; cb.aio_offset = 0; cb.aio_buf = buffer; cb.aio_nbytes = size; sigevt.sigev_notify = SIGEV_SIGNAL; sigevt.sigev_signo = TEST_SIGNAL; sigevt.sigev_value = (sigval_t) 42; cb.aio_sigevent = sigevt; sigemptyset(&set); sigaddset(&set, TEST_SIGNAL); // error = sigprocmask(SIG_BLOCK, &set, &oset); error = pthread_sigmask(SIG_BLOCK, &set, &oset); if (error != 0) { perror("Error setting mainthread signal mask"); } printf("Reader pid: %d\n", getpid()); pthread_sigmask(SIG_BLOCK, NULL, &oset); // sigprocmask(SIG_BLOCK, NULL, &oset); if (sigismember(&oset, TEST_SIGNAL)) { printf("We are blocking TEST_SIGNAL\n"); } aio_read(&cb); pthread_sigmask(SIG_BLOCK, NULL, &oset); // sigprocmask(SIG_BLOCK, NULL, &oset); if (sigismember(&oset, TEST_SIGNAL)) { printf("We are still blocking TEST_SIGNAL in process %d\n", getpid()); } // sigwaitinfo(&set, &info); // printf("The magic number is: %d\n", info.si_value.sival_int); aio_return(&cb); // printf("Signalling thread\n"); // pthread_kill(thread, TEST_SIGNAL); return 1; } void* setupWaiter(void *args) { sigset_t set, oset; siginfo_t info; int error; printf("Waiter pid: %d\n", getpid()); sigemptyset(&set); sigaddset(&set, TEST_SIGNAL); error = pthread_sigmask(SIG_BLOCK, &set, &oset); if (error != 0) { perror("Error setting thread signal mask"); } pthread_sigmask(SIG_BLOCK, NULL, &oset); if (sigismember(&oset, TEST_SIGNAL)) { printf("Waiter is also bocking TEST_SIGNAL\n"); } sigwaitinfo(&set, &info); printf("The magic number is: %d\n", info.si_value.sival_int); printf("Contents: %s\n", buffer); close(fd); return 0; } #endif int main(int argc, char** argv) { void *an_answer; int size; int answer; struct stat fd_stat; TEST_SIGNAL = SIGRTMIN; printf("Argc: %d\n", argc); printf("PID: %d\n", getpid()); fd = open("test.c", O_RDONLY); answer = fstat(fd, &fd_stat); if (answer != 0) { perror("Error stating file"); } else { size = (int) fd_stat.st_size; printf("size: %d\n", size); } buffer = (char*) malloc((size+1)*sizeof(char)); #ifdef USE_AIO { pthread_attr_t attrib; pthread_attr_init(&attrib); pthread_create(&thread, &attrib, &setupWaiter, NULL); } #endif answer = readFile(fd, buffer, size); #ifndef USE_AIO printf("Contents: %s\n", buffer); close(fd); #else pthread_join(thread, &an_answer); #endif return answer; } --==========1811599384==========-- From owner-kaio@oss.sgi.com Wed Apr 4 03:35:50 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.3/8.11.3) id f34AZoS09896 for kaio-outgoing; Wed, 4 Apr 2001 03:35:50 -0700 Received: from turing.xman.org (adsl-63-198-73-118.dsl.lsan03.pacbell.net [63.198.73.118]) by oss.sgi.com (8.11.3/8.11.3) with ESMTP id f34AZmM09893 for ; Wed, 4 Apr 2001 03:35:48 -0700 Received: from hellman (unknown [192.168.1.29]) by turing.xman.org (Postfix) with ESMTP id 7204B56A5 for ; Wed, 4 Apr 2001 03:35:48 -0700 (PDT) Date: Wed, 04 Apr 2001 03:33:24 -0700 From: Christopher Smith To: kaio@oss.sgi.com Subject: Re: Finally got things right... ;-) (signals & threads in LInux) Message-ID: <48140000.986380404@hellman> In-Reply-To: <7310000.986262802@hellman> X-Mailer: Mulberry/2.0.8 (Linux/x86 Demo) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="==========1852429384==========" Sender: owner-kaio@oss.sgi.com Precedence: bulk --==========1852429384========== Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline --On Monday, April 02, 2001 18:53:22 -0700 Christopher Smith wrote: > Right now I've got this working by doing a small hack to the 2.4.3 kernel > and the KAIO code. I'm going to tidy this up and maybe move part of the > fixing into LinuxThreads (right now LinuxThreads does not use the new > CLONE_THREAD flag and it should). Either way the patch is a complete > hack, but it does serve as a demonstration of how efficient this code > could be. As promised, I'm including the patch. Run this patch against 2.4.3 after applying the KAIO patch. Note that it also fixes a small problem compiling the patch against 2.4.3 (the last hunk of the patch covers it). I managed to make the patch work without using tgid, instead relying on the "mm" pointer in task_struct. It's too bad VM's don't have pointers to all the tasks which are using them (instead of that silly counter). Anyway, with this patch the code block I sent out will work, as will any application which wants to use real-time signals to queue up AIO completion notifications for another thread. Comments welcome. --Chris --==========1852429384========== Content-Type: text/plain; charset=iso-8859-1; name="patch-posix-sig.txt" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="patch-posix-sig.txt"; size=1756 diff -urb linux/kernel/signal.c linux-sig/kernel/signal.c --- linux/kernel/signal.c Wed Jan 3 20:45:26 2001 +++ linux-sig/kernel/signal.c Wed Apr 4 03:10:44 2001 @@ -606,6 +606,35 @@ } = /* + * kill_posix_vm_info() send a signal to a thread which isn't blocking the = signal + * and which is executing in the current process. This is what is used + * to get POSIX signal semantics. + */ + +int kill_posix_vm_info(int sig, struct siginfo *info, struct mm_struct* = vm_ptr) { + int retval =3D -EINVAL; + if (vm_ptr !=3D NULL) { + struct task_struct *p; + + retval =3D -ESRCH; + read_lock(&tasklist_lock); + for_each_task(p) { + if (p->mm =3D=3D vm_ptr) { + if (!sigismember(&(p->blocked), sig)) { + int err =3D send_sig_info(sig, info, p); + + if (retval) + retval =3D err; + break; + } + } + } + read_unlock(&tasklist_lock); + } + return retval; +} + +/* * kill_sl_info() sends a signal to the session leader: this is used * to send SIGHUP to the controlling process of a terminal when * the connection is lost. diff -urb linux/mm/filemap.c linux-sig/mm/filemap.c --- linux/mm/filemap.c Wed Apr 4 02:59:10 2001 +++ linux-sig/mm/filemap.c Wed Apr 4 02:49:47 2001 @@ -1544,7 +1544,10 @@ sinfo.si_code =3D SI_ASYNCIO; sinfo.si_value =3D sigev->sigev_value; = - send_sig_info(sigev->sigev_signo, &sinfo, task); +#ifdef AIO_DEBUG + printk("Sending signal to mm %x of process %d\n", task->mm, task->pid); +#endif + kill_posix_vm_info(sigev->sigev_signo, &sinfo, task->mm); } } = @@ -2178,7 +2181,7 @@ */ if (!page_cache) { spin_unlock(&pagecache_lock); - page_cache =3D page_cache_alloc(); + page_cache =3D page_cache_alloc(NULL); /* * That could have slept, so go around to the * very beginning.. --==========1852429384==========-- From owner-kaio@oss.sgi.com Fri Apr 6 15:08:35 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.3/8.11.3) id f36M8Zd08786 for kaio-outgoing; Fri, 6 Apr 2001 15:08:35 -0700 Received: from turing.xman.org (adsl-63-198-73-118.dsl.lsan03.pacbell.net [63.198.73.118]) by oss.sgi.com (8.11.3/8.11.3) with ESMTP id f36M8YM08783 for ; Fri, 6 Apr 2001 15:08:34 -0700 Received: from hellman (unknown [192.168.1.41]) by turing.xman.org (Postfix) with ESMTP id E09AB56A5 for ; Fri, 6 Apr 2001 15:08:33 -0700 (PDT) Date: Fri, 06 Apr 2001 15:06:09 -0700 From: Christopher Smith To: kaio@oss.sgi.com Subject: aio_error() returning positive values??? Message-ID: <57100000.986594769@hellman> X-Mailer: Mulberry/2.0.8 (Linux/x86 Demo) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Sender: owner-kaio@oss.sgi.com Precedence: bulk If I understand how aio_error() is supposed to work, it should return 0 in the event of a successful AIO operation. Unfortunately, this is what's happening to me: sigtimedwait(...) //waiting for completion signal int err = aio_error(&aiocb); err is turing out to be "29". I thought the value would either be 0 or negative. What does positive mean? --Chris From owner-kaio@oss.sgi.com Fri Apr 6 17:54:36 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.3/8.11.3) id f370sa611472 for kaio-outgoing; Fri, 6 Apr 2001 17:54:36 -0700 Received: from turing.xman.org (adsl-63-198-73-118.dsl.lsan03.pacbell.net [63.198.73.118]) by oss.sgi.com (8.11.3/8.11.3) with ESMTP id f370sZM11469 for ; Fri, 6 Apr 2001 17:54:35 -0700 Received: from hellman (unknown [192.168.1.41]) by turing.xman.org (Postfix) with ESMTP id 309BB56A5 for ; Fri, 6 Apr 2001 17:54:33 -0700 (PDT) Date: Fri, 06 Apr 2001 17:52:08 -0700 From: Christopher Smith To: kaio@oss.sgi.com Subject: Question about AIO behavior.... Message-ID: <13200000.986604728@hellman> X-Mailer: Mulberry/2.0.8 (Linux/x86 Demo) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Sender: owner-kaio@oss.sgi.com Precedence: bulk Okay, I am seeing some behavior from KAIO which doesn't seem entirely right to me. If I do a read() on a socket which has no data, it blocks until there is some data available. However, if I do an aio_read() on the socket, it responds immediately, reading zero data, giving me that error code "29" I mentioned previously (aio_return() gives me 0, so it really thinks it didn't get any data). Is this actually correct AIO behavior? I thought AIO was supposed to be more of an equivlanet to read(). --Chris From owner-kaio@oss.sgi.com Fri Apr 6 18:32:17 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.3/8.11.3) id f371WHh11950 for kaio-outgoing; Fri, 6 Apr 2001 18:32:17 -0700 Received: from horus.its.uow.edu.au (horus.its.uow.edu.au [130.130.68.25]) by oss.sgi.com (8.11.3/8.11.3) with ESMTP id f371WFM11947 for ; Fri, 6 Apr 2001 18:32:15 -0700 Received: from uow.edu.au (wumpus.its.uow.edu.au [130.130.68.12]) by horus.its.uow.edu.au (8.9.3/8.9.3) with ESMTP id LAA22641; Sat, 7 Apr 2001 11:32:06 +1000 (EST) Message-ID: <3ACE0B48.382BE62B@uow.edu.au> Date: Fri, 06 Apr 2001 11:30:32 -0700 From: Andrew Morton X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.2.18-0.22 i686) X-Accept-Language: en MIME-Version: 1.0 To: Christopher Smith CC: kaio@oss.sgi.com Subject: Re: Finally got things right... ;-) (signals & threads in LInux) References: <7310000.986262802@hellman> <48140000.986380404@hellman> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-kaio@oss.sgi.com Precedence: bulk Very sane patch,IMO. Small bug: +int kill_posix_vm_info(int sig, struct siginfo *info, struct mm_struct* vm_ptr) { + int retval = -EINVAL; + if (vm_ptr != NULL) { + struct task_struct *p; + + retval = -ESRCH; + read_lock(&tasklist_lock); + for_each_task(p) { + if (p->mm == vm_ptr) { + if (!sigismember(&(p->blocked), sig)) { + int err = send_sig_info(sig, info, p); + + if (retval) -->> if (err) + retval = err; + break; + } + } + } + read_unlock(&tasklist_lock); From owner-kaio@oss.sgi.com Fri Apr 6 18:52:56 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.3/8.11.3) id f371quO12213 for kaio-outgoing; Fri, 6 Apr 2001 18:52:56 -0700 Received: from turing.xman.org (adsl-63-198-73-118.dsl.lsan03.pacbell.net [63.198.73.118]) by oss.sgi.com (8.11.3/8.11.3) with ESMTP id f371qtM12210 for ; Fri, 6 Apr 2001 18:52:55 -0700 Received: from hellman (unknown [192.168.1.45]) by turing.xman.org (Postfix) with ESMTP id 0E21D56A5; Fri, 6 Apr 2001 18:52:55 -0700 (PDT) Date: Fri, 06 Apr 2001 18:50:30 -0700 From: Christopher Smith To: Andrew Morton Cc: kaio@oss.sgi.com Subject: Re: Finally got things right... ;-) (signals & threads in LInux) Message-ID: <2350000.986608230@hellman> In-Reply-To: <3ACE0B48.382BE62B@uow.edu.au> X-Mailer: Mulberry/2.0.8 (Linux/x86 Demo) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Sender: owner-kaio@oss.sgi.com Precedence: bulk --On Friday, April 06, 2001 11:30:32 -0700 Andrew Morton wrote: > + if (retval) > > -->> if (err) Ouch, how embarassing. ;-) Thanks for catching that. --Chris From owner-kaio@oss.sgi.com Sat Apr 7 22:15:00 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.3/8.11.3) id f385F0c04758 for kaio-outgoing; Sat, 7 Apr 2001 22:15:00 -0700 Received: from turing.xman.org (adsl-63-198-73-118.dsl.lsan03.pacbell.net [63.198.73.118]) by oss.sgi.com (8.11.3/8.11.3) with ESMTP id f385ExM04754 for ; Sat, 7 Apr 2001 22:14:59 -0700 Received: from hellman (unknown [192.168.1.9]) by turing.xman.org (Postfix) with ESMTP id BEEB356A5 for ; Sat, 7 Apr 2001 22:14:55 -0700 (PDT) Date: Sat, 07 Apr 2001 22:12:28 -0700 From: Christopher Smith To: kaio@oss.sgi.com Subject: Re: Question about AIO behavior.... Message-ID: <39120000.986706748@hellman> In-Reply-To: <13200000.986604728@hellman> X-Mailer: Mulberry/2.0.8 (Linux/x86 Demo) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Sender: owner-kaio@oss.sgi.com Precedence: bulk --On Friday, April 06, 2001 17:52:08 -0700 Christopher Smith wrote: > Okay, I am seeing some behavior from KAIO which doesn't seem entirely > right to me. If I do a read() on a socket which has no data, it blocks > until there is some data available. However, if I do an aio_read() on the > socket, it responds immediately, reading zero data, giving me that error > code "29" I mentioned previously (aio_return() gives me 0, so it really > thinks it didn't get any data). > > Is this actually correct AIO behavior? I thought AIO was supposed to be > more of an equivlanet to read(). Okay, I've learned a bit more about what's going on now. It appears that the latest patch tries to do a seek if you do a read() on a socket (and presumably on a pipe as well). This results in a seek error (error 29). I'm going to look into patching this myself, but I thought I'd notify the group in the event that I was either wrong in my analysis or if someone had a fix (or got one done before I did). --Chris From owner-kaio@oss.sgi.com Sun Apr 8 06:34:50 2001 Received: (from majordomo@localhost) by oss.sgi.com (8.11.3/8.11.3) id f38DYoA12967 for kaio-outgoing; Sun, 8 Apr 2001 06:34:50 -0700 Received: from turing.xman.org (adsl-63-198-73-118.dsl.lsan03.pacbell.net [63.198.73.118]) by oss.sgi.com (8.11.3/8.11.3) with ESMTP id f38DYnM12963 for ; Sun, 8 Apr 2001 06:34:49 -0700 Received: from hellman (unknown [192.168.1.19]) by turing.xman.org (Postfix) with ESMTP id 43E7156A5 for ; Sun, 8 Apr 2001 06:34:49 -0700 (PDT) Date: Sun, 08 Apr 2001 06:32:25 -0700 From: Christopher Smith To: kaio@oss.sgi.com Subject: Fix to make KAIO work with sockets (and likely pipes) again [was Re: Question about AIO behavior....] Message-ID: <3080000.986736745@hellman> In-Reply-To: <39120000.986706748@hellman> X-Mailer: Mulberry/2.0.8 (Linux/x86 Demo) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="==========1807369384==========" Sender: owner-kaio@oss.sgi.com Precedence: bulk --==========1807369384========== Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline --On Saturday, April 07, 2001 22:12:28 -0700 Christopher Smith wrote: > Okay, I've learned a bit more about what's going on now. It appears that > the latest patch tries to do a seek if you do a read() on a socket (and > presumably on a pipe as well). This results in a seek error (error 29). > > I'm going to look into patching this myself, but I thought I'd notify the > group in the event that I was either wrong in my analysis or if someone > had a fix (or got one done before I did). Please find attached a one-line patch which, if applied to the kaio-kern-1.3.1-2.4.2 patch file will correct the bug in the patch. This bug appears to also exist in the 1.3 patch as well. Perhaps a sockets test should be added to the regression tests? ;-) --Chris --==========1807369384========== Content-Type: application/octet-stream; name=fixer Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename=fixer; size=513 LS0tIGthaW8ta2Vybi0xLjMuMS0yLjQuMglNb24gTWFyIDE5IDE0OjUzOjIzIDIwMDEKKysrIGth aW8ta2Vybi0xLjMuMi0yLjQuMglTdW4gQXByICA4IDA2OjI4OjA4IDIwMDEKQEAgLTE0NTYsNyAr MTQ1Niw3IEBACiArCWludCByZXQ7CQkJCQkJCSAgICAgIFwKICsJKGspLT5rYWlvX2ZpbHAtPmZf cG9zID0gKGspLT5rYWlvX29mZnNldDsgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFwKICsJ cmV0ID0gKGspLT5rYWlvX2ZpbHAtPmZfb3AtPiAjIyBvcCAoKGspLT5rYWlvX2ZpbHAsICAgICAg ICAgICAgICAgICAgIFwKLSsJCShrKS0+a2Fpb19idWYsIChrKS0+a2Fpb19uYnl0ZXMsICYoaykt PmthaW9fb2Zmc2V0KTsgICAgIFwKKysJCShrKS0+a2Fpb19idWYsIChrKS0+a2Fpb19uYnl0ZXMs ICYoayktPmthaW9fZmlscC0+Zl9wb3MpOyAgICAgXAogKwlrYWlvX2lvX2NvbXBsZXRlKChrKSwg cmV0IDwgMCA/IDAgOiByZXQsIHJldCA+IDAgPyAwIDogcmV0LCAwKTsJICAgICAgXAogK30KICsK --==========1807369384==========--