From mtobias@alfonso.wustl.edu Mon May 3 08:18:19 2004 Received: with ECARTIS (v1.0.0; list pagg); Mon, 03 May 2004 08:18:24 -0700 (PDT) Received: from alfonso.wustl.edu (alfonso.wustl.edu [128.252.35.168]) by oss.sgi.com (8.12.10/8.12.9) with SMTP id i43FIIKO027485 for ; Mon, 3 May 2004 08:18:18 -0700 Received: from alfonso.wustl.edu (IDENT:1110@localhost [127.0.0.1]) by alfonso.wustl.edu (8.12.10/8.12.8) with ESMTP id i43FICG2002195 for ; Mon, 3 May 2004 10:18:12 -0500 Received: from localhost (localhost [[UNIX: localhost]]) by alfonso.wustl.edu (8.12.10/8.12.8/Submit) id i43FICOB002194 for pagg@oss.sgi.com; Mon, 3 May 2004 10:18:12 -0500 From: Malcolm Tobias To: pagg@oss.sgi.com Subject: troubles building job-1.4.0-1 Date: Mon, 3 May 2004 10:18:12 -0500 User-Agent: KMail/1.5.3 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200405031018.12521.mtobias@wustl.edu> X-archive-position: 8 X-ecartis-version: Ecartis v1.0.0 Sender: pagg-bounce@oss.sgi.com Errors-to: pagg-bounce@oss.sgi.com X-original-sender: mtobias@wustl.edu Precedence: bulk X-list: pagg I've succeeded in using patches linux-2.4.19-job/pagg.patch to build a new kernel, but I'm now running into troubles trying to build job-1.4.0-1 from source. -In the Makefile, KINCLUDES attempts to match anything under /usr/src/ that begins with linux-*. This causes problems if there are any tar-balls in that directory as it tries to include the tar-ball (e.g. -I/usr/src/linux-xfs-1.3.1.patch.gz). Maybe you should just try including /usr/src/linux assuming a soft-link has been made? -job/include/jobctl.h attempts to include linux/paggctl.h which doesn't appear in either of the patches. I guess this version of job is only compatible with patches 2.4.24 and greater? This would be good to put in the README. Malcolm -- Malcolm Tobias 314.935.5289 From erikj@efs.americas.sgi.com Tue May 4 09:37:14 2004 Received: with ECARTIS (v1.0.0; list pagg); Tue, 04 May 2004 09:37:18 -0700 (PDT) Received: from omx1.americas.sgi.com (cfcafw.sgi.com [198.149.23.1]) by oss.sgi.com (8.12.10/8.12.9) with SMTP id i44GbDKO010909 for ; Tue, 4 May 2004 09:37:13 -0700 Received: from flecktone.americas.sgi.com (flecktone.americas.sgi.com [192.48.203.135]) by omx1.americas.sgi.com (8.12.10/8.12.9/linux-outbound_gateway-1.1) with ESMTP id i44GYpT9019525 for ; Tue, 4 May 2004 11:34:51 -0500 Received: from thistle-e236.americas.sgi.com (thistle-e236.americas.sgi.com [128.162.236.204]) by flecktone.americas.sgi.com (8.12.9/8.12.10/SGI_generic_relay-1.2) with ESMTP id i44GYoKe37953026; Tue, 4 May 2004 11:34:50 -0500 (CDT) Received: from efs.americas.sgi.com (efs.americas.sgi.com [128.162.236.150]) by thistle-e236.americas.sgi.com (8.12.9/SGI-server-1.8) with ESMTP id i44GYoGa3936032; Tue, 4 May 2004 11:34:50 -0500 (CDT) Received: from efs.americas.sgi.com (localhost [127.0.0.1]) by efs.americas.sgi.com (SGI-8.12.5/8.12.5/erikj-IRIX6519-news) with ESMTP id i44GYoLN005018; Tue, 4 May 2004 11:34:50 -0500 (CDT) Received: from localhost (erikj@localhost) by efs.americas.sgi.com (SGI-8.12.5/8.12.5/Submit) with ESMTP id i44GYnUw005014; Tue, 4 May 2004 11:34:49 -0500 (CDT) Date: Tue, 4 May 2004 11:34:49 -0500 From: Erik Jacobson To: Malcolm Tobias cc: pagg@oss.sgi.com Subject: Re: troubles building job-1.4.0-1 In-Reply-To: <200405031018.12521.mtobias@wustl.edu> Message-ID: References: <200405031018.12521.mtobias@wustl.edu> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 9 X-ecartis-version: Ecartis v1.0.0 Sender: pagg-bounce@oss.sgi.com Errors-to: pagg-bounce@oss.sgi.com X-original-sender: erikj@efs.americas.sgi.com Precedence: bulk X-list: pagg > I've succeeded in using patches linux-2.4.19-job/pagg.patch to build a new > kernel, but I'm now running into troubles trying to build job-1.4.0-1 from > source. You were correct in that only the newer patches work with job 1.4. I have added a README to the download directory. I've also added an "Install Notes" entry to the left hand nav bar on the pagg web site. It just includes the README from the download directory. The README lays out: - Which patches are needed to work with job 1.4 - Order of patch application - Some notes on which patches began to include some recent community feedback that affects PAGG macros and usage. It also appears the 'news' section isn't very new :) I'll try to tidy that up now. Personally, I'd prefer not to have a 'news' and just have people look at the mailing list archives. Let me know if this doesn't sound acceptable. If there is strong demand, I can try to make some pagg/job patches for older kernels that have the latest updates. Let me know. > -In the Makefile, KINCLUDES attempts to match anything under /usr/src/ > that begins with linux-*. This causes problems if there are any tar-balls > in that directory as it tries to include the tar-ball (e.g. > -I/usr/src/linux-xfs-1.3.1.patch.gz). > Maybe you should just try including /usr/src/linux assuming a soft-link has > been made? I'll file a bug on this internally so we can find a better way to handle this. -- Erik Jacobson - Linux System Software - Silicon Graphics - Eagan, Minnesota From mtobias@alfonso.wustl.edu Tue May 4 09:42:13 2004 Received: with ECARTIS (v1.0.0; list pagg); Tue, 04 May 2004 09:42:16 -0700 (PDT) Received: from alfonso.wustl.edu (alfonso.wustl.edu [128.252.35.168]) by oss.sgi.com (8.12.10/8.12.9) with SMTP id i44GgCKO012242 for ; Tue, 4 May 2004 09:42:13 -0700 Received: from alfonso.wustl.edu (IDENT:1110@localhost [127.0.0.1]) by alfonso.wustl.edu (8.12.10/8.12.8) with ESMTP id i44Gg45J002814; Tue, 4 May 2004 11:42:04 -0500 Received: from localhost (localhost [[UNIX: localhost]]) by alfonso.wustl.edu (8.12.10/8.12.8/Submit) id i44Gg49G002813; Tue, 4 May 2004 11:42:04 -0500 From: Malcolm Tobias To: Erik Jacobson Subject: Re: troubles building job-1.4.0-1 Date: Tue, 4 May 2004 11:42:04 -0500 User-Agent: KMail/1.5.3 Cc: pagg@oss.sgi.com References: <200405031018.12521.mtobias@wustl.edu> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200405041142.04299.mtobias@wustl.edu> X-archive-position: 10 X-ecartis-version: Ecartis v1.0.0 Sender: pagg-bounce@oss.sgi.com Errors-to: pagg-bounce@oss.sgi.com X-original-sender: mtobias@wustl.edu Precedence: bulk X-list: pagg On Tuesday 04 May 2004 11:34 am, Erik Jacobson wrote: > If there is strong demand, I can try to make some pagg/job patches for > older kernels that have the latest updates. Let me know. No need to patch the old kernels on my account. I was trying to get PAGG to work on an Itanium box, so I needed to have the IA64 kernel patch, as well as the XFS kernel patches. Trying to find a complete set of patches for the same kernel version is asking a bit much ;-) The install notes look great. Hopefully they'll help the next person. Cheers, Malcolm -- Malcolm Tobias 314.935.5289 From limin@dbear.engr.sgi.com Tue May 4 10:17:30 2004 Received: with ECARTIS (v1.0.0; list pagg); Tue, 04 May 2004 10:17:34 -0700 (PDT) Received: from omx2.sgi.com (mtvcafw.sgi.com [192.48.171.6]) by oss.sgi.com (8.12.10/8.12.9) with SMTP id i44HHUKO013888 for ; Tue, 4 May 2004 10:17:30 -0700 Received: from cthulhu.engr.sgi.com (cthulhu.engr.sgi.com [192.26.80.2]) by omx2.sgi.com (8.12.11/8.12.9/linux-outbound_gateway-1.1) with ESMTP id i44HJM5F027132 for ; Tue, 4 May 2004 10:19:22 -0700 Received: from dbear.engr.sgi.com (dbear.engr.sgi.com [163.154.18.85]) by cthulhu.engr.sgi.com (SGI-8.12.5/8.12.5) with ESMTP id i44HHO078101999 for ; Tue, 4 May 2004 10:17:25 -0700 (PDT) Received: (from limin@localhost) by dbear.engr.sgi.com (8.11.0/8.11.0) id i44HHOe15469; Tue, 4 May 2004 10:17:24 -0700 From: Limin Gu Message-Id: <200405041717.i44HHOe15469@dbear.engr.sgi.com> Subject: test, please ignore. To: pagg@oss.sgi.com Date: Tue, 4 May 2004 10:17:24 -0700 (PDT) Cc: limin@dbear.engr.sgi.com (Limin Gu) X-Mailer: ELM [version 2.5 PL3] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-archive-position: 11 X-ecartis-version: Ecartis v1.0.0 Sender: pagg-bounce@oss.sgi.com Errors-to: pagg-bounce@oss.sgi.com X-original-sender: limin@dbear.engr.sgi.com Precedence: bulk X-list: pagg test only --Limin From erikj@efs.americas.sgi.com Tue May 4 12:55:46 2004 Received: with ECARTIS (v1.0.0; list pagg); Tue, 04 May 2004 12:55:51 -0700 (PDT) Received: from zok.sgi.com (mtvcafw.sgi.com [192.48.171.6]) by oss.sgi.com (8.12.10/8.12.9) with SMTP id i44JtkKO021916 for ; Tue, 4 May 2004 12:55:46 -0700 Received: from nodin.corp.sgi.com (fddi-nodin.corp.sgi.com [198.29.75.193]) by zok.sgi.com (8.12.9/8.12.9/linux-outbound_gateway-1.1) with ESMTP id i44Jlshv027933 for ; Tue, 4 May 2004 12:47:54 -0700 Received: from omx2.sgi.com (omx2.sgi.com [198.149.32.25]) by nodin.corp.sgi.com (8.12.9/8.12.10/SGI_generic_relay-1.2) with ESMTP id i44Jlq9f4732895 for ; Tue, 4 May 2004 12:47:52 -0700 (PDT) Received: from flecktone.americas.sgi.com (flecktone.americas.sgi.com [192.48.203.135]) by omx2.sgi.com (8.12.11/8.12.9/linux-outbound_gateway-1.1) with ESMTP id i44JnorD013928 for ; Tue, 4 May 2004 12:49:50 -0700 Received: from thistle-e236.americas.sgi.com (thistle-e236.americas.sgi.com [128.162.236.204]) by flecktone.americas.sgi.com (8.12.9/8.12.10/SGI_generic_relay-1.2) with ESMTP id i44JlpKe37947584 for ; Tue, 4 May 2004 14:47:51 -0500 (CDT) Received: from efs.americas.sgi.com (efs.americas.sgi.com [128.162.236.150]) by thistle-e236.americas.sgi.com (8.12.9/SGI-server-1.8) with ESMTP id i44JlpGa4046682 for ; Tue, 4 May 2004 14:47:51 -0500 (CDT) Received: from efs.americas.sgi.com (localhost [127.0.0.1]) by efs.americas.sgi.com (SGI-8.12.5/8.12.5/erikj-IRIX6519-news) with ESMTP id i44JlpLN008204 for ; Tue, 4 May 2004 14:47:51 -0500 (CDT) Received: from localhost (erikj@localhost) by efs.americas.sgi.com (SGI-8.12.5/8.12.5/Submit) with ESMTP id i44JloG1008200 for ; Tue, 4 May 2004 14:47:50 -0500 (CDT) Date: Tue, 4 May 2004 14:47:50 -0500 From: Erik Jacobson To: pagg@oss.sgi.com Subject: New pagg/job patches available Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 12 X-ecartis-version: Ecartis v1.0.0 Sender: pagg-bounce@oss.sgi.com Errors-to: pagg-bounce@oss.sgi.com X-original-sender: erikj@efs.americas.sgi.com Precedence: bulk X-list: pagg There are new pagg and job patches available for 2.4.26 and 2.6.5 kernels. I went through a couple sets of changes based on suggestions from LMKL with most of the suggestions coming from Christoph Hellwig. Because changes include removing and renaming macros and functions, the new job patch requires the new pagg patch. In other words, if you wish to use the job patch, be sure you match it with the proper version of the PAGG. I updated the README to show the pairs. I have one more batch of changes to implement yet. Thanks! You can find the new patches on oss.sgi.com/projects/pagg Click on 'download' on the left. -- Erik Jacobson - Linux System Software - Silicon Graphics - Eagan, Minnesota From pwil3058@bigpond.net.au Tue May 4 23:57:49 2004 Received: with ECARTIS (v1.0.0; list pagg); Tue, 04 May 2004 23:57:54 -0700 (PDT) Received: from gizmo02bw.bigpond.com (gizmo02bw.bigpond.com [144.140.70.12]) by oss.sgi.com (8.12.10/8.12.9) with SMTP id i456vmKO012686 for ; Tue, 4 May 2004 23:57:48 -0700 Received: (qmail 12015 invoked from network); 5 May 2004 06:54:57 -0000 Received: from unknown (HELO bwmam01.bigpond.com) (144.135.24.69) by gizmo02bw.bigpond.com with SMTP; 5 May 2004 06:54:57 -0000 Received: from cpe-203-45-91-55.nsw.bigpond.net.au ([203.45.91.55]) by bwmam01.bigpond.com(MAM REL_3_4_2 8/907985) with SMTP id 907985; Wed, 05 May 2004 16:57:41 +1000 Message-ID: <40989065.4080905@bigpond.net.au> Date: Wed, 05 May 2004 16:57:41 +1000 From: Peter Williams User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Erik Jacobson CC: pagg@oss.sgi.com Subject: Re: New pagg/job patches available References: In-Reply-To: Content-Type: multipart/mixed; boundary="------------020606080500000802090905" X-archive-position: 13 X-ecartis-version: Ecartis v1.0.0 Sender: pagg-bounce@oss.sgi.com Errors-to: pagg-bounce@oss.sgi.com X-original-sender: pwil3058@bigpond.net.au Precedence: bulk X-list: pagg This is a multi-part message in MIME format. --------------020606080500000802090905 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Erik Jacobson wrote: > There are new pagg and job patches available for 2.4.26 and 2.6.5 > kernels. > > I went through a couple sets of changes based on suggestions from LMKL > with most of the suggestions coming from Christoph Hellwig. > > Because changes include removing and renaming macros and functions, the > new job patch requires the new pagg patch. In other words, if you wish > to use the job patch, be sure you match it with the proper version of > the PAGG. > > I updated the README to show the pairs. > > I have one more batch of changes to implement yet. > > Thanks! > > You can find the new patches on oss.sgi.com/projects/pagg > Click on 'download' on the left. > > -- > Erik Jacobson - Linux System Software - Silicon Graphics - Eagan, Minnesota > > Please find attached a patch which is a first pass at a "safe to block" method for calling the init() function during client registration. The way in which this patch handles kmalloc() failures needs to be improved but I think this could benefit from input from someone more familiar with PAGG than I am. Peter --------------020606080500000802090905 Content-Type: text/plain; name="pagg_init.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="pagg_init.patch" Index: kernel/pagg.c =================================================================== RCS file: /export/cvsroot/Linux-2.6.5/kernel/Attic/pagg.c,v retrieving revision 1.1.4.1 diff -c -r1.1.4.1 pagg.c *** kernel/pagg.c 5 May 2004 05:36:45 -0000 1.1.4.1 --- kernel/pagg.c 5 May 2004 06:48:03 -0000 *************** *** 202,207 **** --- 202,305 ---- /* Okay, we can insert into the pagg hook list */ list_add_tail(&pagg_hook_new->entry, &pagg_hook_list); + /* Now we can call the initialiser function (if present) for each task */ + if (pagg_hook_new->init != NULL) { + int num_inited = 0; + + /* Because of internal race conditions we can't gaurantee + * getting every task in just one pass so we just keep going + * until we don't find any unitialised tasks. The inefficiency + * of this should be tempered by the fact that this happens + * at most once for each registered client. + */ + do { + struct task_struct *p = NULL; + int *live_pids; + int live_pids_sz; + int i; + int failed_pid_mallocs = 0; + int failed_pagg_mallocs = 0; + + retry_malloc: + live_pids_sz = nr_threads + 16; + live_pids = kmalloc(sizeof(int) * live_pids_sz, GFP_KERNEL); + if (live_pids == NULL) { + /* This should be changed to abort the registration + * and undo anything that's been done. Undoing the + * mess may be difficult so we'll just retry for the + * time being. + */ + if (failed_pid_mallocs < 10) { + failed_pid_mallocs++; + yield(); + goto retry_malloc; + } else { + /* we can't return an error value here + * as it would cause the module load to + * fail while we (possibly) still hold + * malloced memory. So just warn that + * initialisation has failed. This is + * no worse than completely ignoring + * the initialisation function. + */ + printk(KERN_WARNING "Insufficient memory" + " to initialise" + " PAGG support (name=%s)\n", + pagg_hook_new->name); + break; + } + } + read_lock(&tasklist_lock); + if (nr_threads > live_pids_sz) { + read_unlock(&tasklist_lock); + kfree(live_pids); + goto retry_malloc; + } + num_inited = 0; + for_each_process(p) { + struct pagg *paggp; + + down_read(&p->pagg_list.sem); + paggp = pagg_get(p, pagg_hook_new->name); + up_read(&p->pagg_list.sem); + + if (paggp == NULL) { + /* this one hasn't been initialised yet */ + live_pids[num_inited] = p->pid; + num_inited++; + } + } + read_unlock(&tasklist_lock); + for (i = 0; i < num_inited; i++) { + read_lock(&tasklist_lock); + if (likely((p = find_task_by_pid(i)) != NULL)) + get_task_struct(p); + read_unlock(&tasklist_lock); + if (likely(p != NULL)) { + struct pagg *paggp; + + down_write(&p->pagg_list.sem); + paggp = pagg_alloc(p, pagg_hook_new); + if (paggp != NULL) + pagg_hook_new->init(p, paggp); + else + failed_pagg_mallocs++; + up_write(&p->pagg_list.sem); + put_task_struct(p); + } + } + kfree(live_pids); + if (failed_pagg_mallocs > 10) { + /* we can't return an error value here + * for the same reason as above. + */ + printk(KERN_WARNING "Insufficient memory" + " to initialise PAGG support (name=%s)\n", + pagg_hook_new->name); + break; + } + } while (num_inited > 0); + } up_write(&pagg_hook_list_sem); printk(KERN_INFO "Registering PAGG support for (name=%s)\n", --------------020606080500000802090905-- From erikj@subway.americas.sgi.com Wed May 5 05:28:12 2004 Received: with ECARTIS (v1.0.0; list pagg); Wed, 05 May 2004 05:28:17 -0700 (PDT) Received: from zok.sgi.com (mtvcafw.sgi.com [192.48.171.6]) by oss.sgi.com (8.12.10/8.12.9) with SMTP id i45CSBKO030698 for ; Wed, 5 May 2004 05:28:11 -0700 Received: from flecktone.americas.sgi.com (flecktone.americas.sgi.com [192.48.203.135]) by zok.sgi.com (8.12.9/8.12.9/linux-outbound_gateway-1.1) with ESMTP id i45CDWhv011941 for ; Wed, 5 May 2004 05:13:32 -0700 Received: from thistle-e236.americas.sgi.com (thistle-e236.americas.sgi.com [128.162.236.204]) by flecktone.americas.sgi.com (8.12.9/8.12.10/SGI_generic_relay-1.2) with ESMTP id i45CDVKe37906544; Wed, 5 May 2004 07:13:31 -0500 (CDT) Received: from subway.americas.sgi.com (subway.americas.sgi.com [128.162.236.152]) by thistle-e236.americas.sgi.com (8.12.9/SGI-server-1.8) with ESMTP id i45CDVGa4069379; Wed, 5 May 2004 07:13:31 -0500 (CDT) Received: from subway.americas.sgi.com (localhost [127.0.0.1]) by subway.americas.sgi.com (SGI-8.12.5/8.12.5/erikj-IRIX6519-news) with ESMTP id i45CDVS2050409; Wed, 5 May 2004 07:13:31 -0500 (CDT) Received: from localhost (erikj@localhost) by subway.americas.sgi.com (SGI-8.12.5/8.12.5/Submit) with ESMTP id i45CDTVd050393; Wed, 5 May 2004 07:13:30 -0500 (CDT) Date: Wed, 5 May 2004 07:13:29 -0500 From: Erik Jacobson To: Peter Williams cc: pagg@oss.sgi.com Subject: Re: New pagg/job patches available In-Reply-To: <40989065.4080905@bigpond.net.au> Message-ID: References: <40989065.4080905@bigpond.net.au> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 14 X-ecartis-version: Ecartis v1.0.0 Sender: pagg-bounce@oss.sgi.com Errors-to: pagg-bounce@oss.sgi.com X-original-sender: erikj@subway.americas.sgi.com Precedence: bulk X-list: pagg Thanks. Let me digest this and perhaps have some time to discuss it with a couple of my co-workers. I did get this and will get back to you. Erik On Wed, 5 May 2004, Peter Williams wrote: > Erik Jacobson wrote: > > There are new pagg and job patches available for 2.4.26 and 2.6.5 > > kernels. > > > > I went through a couple sets of changes based on suggestions from LMKL > > with most of the suggestions coming from Christoph Hellwig. > > > > Because changes include removing and renaming macros and functions, the > > new job patch requires the new pagg patch. In other words, if you wish > > to use the job patch, be sure you match it with the proper version of > > the PAGG. > > > > I updated the README to show the pairs. > > > > I have one more batch of changes to implement yet. > > > > Thanks! > > > > You can find the new patches on oss.sgi.com/projects/pagg > > Click on 'download' on the left. > > > > -- > > Erik Jacobson - Linux System Software - Silicon Graphics - Eagan, Minnesota > > > > > > Please find attached a patch which is a first pass at a "safe to block" > method for calling the init() function during client registration. The > way in which this patch handles kmalloc() failures needs to be improved > but I think this could benefit from input from someone more familiar > with PAGG than I am. > > Peter > -- Erik Jacobson - Linux System Software - Silicon Graphics - Eagan, Minnesota From pwil3058@bigpond.net.au Wed May 5 19:48:14 2004 Received: with ECARTIS (v1.0.0; list pagg); Wed, 05 May 2004 19:48:17 -0700 (PDT) Received: from gizmo08bw.bigpond.com (gizmo08bw.bigpond.com [144.140.70.18]) by oss.sgi.com (8.12.10/8.12.9) with SMTP id i462mCKO001673 for ; Wed, 5 May 2004 19:48:12 -0700 Received: (qmail 31955 invoked from network); 6 May 2004 02:37:22 -0000 Received: from unknown (HELO bwmam01.bigpond.com) (144.135.24.69) by gizmo08bw.bigpond.com with SMTP; 6 May 2004 02:37:22 -0000 Received: from cpe-203-45-91-55.nsw.bigpond.net.au ([203.45.91.55]) by bwmam01.bigpond.com(MAM REL_3_4_2 8/1512673) with SMTP id 1512673; Thu, 06 May 2004 12:48:04 +1000 Message-ID: <4099A764.3000201@bigpond.net.au> Date: Thu, 06 May 2004 12:48:04 +1000 From: Peter Williams User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Erik Jacobson CC: pagg@oss.sgi.com Subject: Re: New pagg/job patches available References: <40989065.4080905@bigpond.net.au> In-Reply-To: Content-Type: multipart/mixed; boundary="------------030605080503040001000808" X-archive-position: 15 X-ecartis-version: Ecartis v1.0.0 Sender: pagg-bounce@oss.sgi.com Errors-to: pagg-bounce@oss.sgi.com X-original-sender: pwil3058@bigpond.net.au Precedence: bulk X-list: pagg This is a multi-part message in MIME format. --------------030605080503040001000808 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Erik Jacobson wrote: > Thanks. > > Let me digest this and perhaps have some time to discuss it with a couple > of my co-workers. I did get this and will get back to you. > > Erik > > On Wed, 5 May 2004, Peter Williams wrote: > > >>Erik Jacobson wrote: >> >>>There are new pagg and job patches available for 2.4.26 and 2.6.5 >>>kernels. >>> >>>I went through a couple sets of changes based on suggestions from LMKL >>>with most of the suggestions coming from Christoph Hellwig. >>> >>>Because changes include removing and renaming macros and functions, the >>>new job patch requires the new pagg patch. In other words, if you wish >>>to use the job patch, be sure you match it with the proper version of >>>the PAGG. >>> >>>I updated the README to show the pairs. >>> >>>I have one more batch of changes to implement yet. >>> >>>Thanks! >>> >>>You can find the new patches on oss.sgi.com/projects/pagg >>>Click on 'download' on the left. >>> >>>-- >>>Erik Jacobson - Linux System Software - Silicon Graphics - Eagan, Minnesota >>> >>> >> >>Please find attached a patch which is a first pass at a "safe to block" >>method for calling the init() function during client registration. The >>way in which this patch handles kmalloc() failures needs to be improved >>but I think this could benefit from input from someone more familiar >>with PAGG than I am. >> OK. In addition to execve, I think that events such as changes to real uid, real gid and cpu affinity would be of (potential) interest to PAGG clients. To that end I enclose a set of patches that adds that functionality. This patch assumes that my init patch has already been applied but should patch successfully (with offsets) if it has not. Enjoy Peter --------------030605080503040001000808 Content-Type: text/plain; name="pagg_extras.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="pagg_extras.patch" Index: Linux-2.6.5/Documentation/pagg.txt diff -c Linux-2.6.5/Documentation/pagg.txt:1.1.4.1 Linux-2.6.5/Documentation/pagg.txt:1.1.4.1.4.1 *** Linux-2.6.5/Documentation/pagg.txt:1.1.4.1 Wed May 5 15:36:45 2004 --- Linux-2.6.5/Documentation/pagg.txt Thu May 6 12:33:18 2004 *************** *** 32,37 **** --- 32,48 ---- used, for example, by other kernel modules that wish to do advanced CPU placement on multi-processor systems (just one example). + The set_user function has been modified to support an optional callout + that can be run when a process in a pagg list changes its real uid. + + The sys_setresgid, sy_setregid and sys_setgid functions have been modified + to support optional callouts that can be run when a process in a pagg list changes + its real gid. + + The set_cpus_allowed function has been modified to support an optional callout + that can be run when a process in a pagg list changes its cpu affinity. It could be + used, for example, to implement CPU sets. + Additional details concerning this implementation of the process aggregates infrastructure are described in the sections that follow. *************** *** 51,56 **** --- 62,69 ---- - kernel/Makefile - kernel/exit.c - kernel/fork.c + - kernel/sched.c + - kernel/sys.c - fs/exec.c - init/Kconfig *************** *** 96,101 **** --- 109,117 ---- void *data; /* Module specific data */ struct list_head entry; /* List connection */ void (*exec)(struct task_struct *, struct pagg *); /* exec func ptr */ + void (*setruid)(struct task_struct *, struct pagg *); /* setruid func ptr */ + void (*setrgid)(struct task_struct *, struct pagg *); /* setrgid func ptr */ + void (*setcpuaffinity)(struct task_struct *, struct pagg *); /* setcpuaffinity func ptr */ The pagg structure provides the process' reference to the PAGG containers provided by the PAGG modules. The attach function pointer *************** *** 104,110 **** the referenced PAGG container that the process is exiting or otherwise detaching from the container. The exec function pointer is used when a process in the pagg container exec's a new process. This is optional and ! may be set to NULL if it is not needed by the pagg module. The pagg_hook structure provides the reference to the module that implements a type of PAGG container. In addition to the function pointers --- 120,134 ---- the referenced PAGG container that the process is exiting or otherwise detaching from the container. The exec function pointer is used when a process in the pagg container exec's a new process. This is optional and ! may be set to NULL if it is not needed by the pagg module. The setruid ! function pointer is used when a process in the pagg container changes its ! real uid. This is optional and may be set to NULL if it is not needed by the ! pagg module. The setrgid function pointer is used when a process in the ! pagg container changes its real gid. This is optional and may be set to ! NULL if it is not needed by the pagg module. The setcpuaffinity function ! pointer is used when a process in the pagg container changes its cpu ! affinity. This is optional and may be set to NULL if it is not needed by the ! pagg module. The pagg_hook structure provides the reference to the module that implements a type of PAGG container. In addition to the function pointers *************** *** 129,134 **** --- 153,167 ---- Call detach_pagg_list function with current task_struct - sys_execve: (fs/exec.c) /* When a process in a pagg exec's, an optional callout can be run. This + is implemented with an optional function pointer in the pagg_hook. */ + - set_user: (kernel/sys.c) + /* When a process in a pagg sets its real uid, an optional callout can be run. This + is implemented with an optional function pointer in the pagg_hook. */ + - sys_setresgid, sy_setregid and sys_setgid: (kernel/sys.c) + /* When a process in a pagg sets its real gid, an optional callout can be run. This + is implemented with an optional function pointer in the pagg_hook. */ + - set_cpus_allowed: (kernel/sched.c) + /* When a process in a pagg changes cpu affinity, an optional callout can be run. This is implemented with an optional function pointer in the pagg_hook. */ 2.6 New Functions Index: Linux-2.6.5/include/linux/pagg.h diff -c Linux-2.6.5/include/linux/pagg.h:1.1.4.1 Linux-2.6.5/include/linux/pagg.h:1.1.4.1.4.1 *** Linux-2.6.5/include/linux/pagg.h:1.1.4.1 Wed May 5 15:36:45 2004 --- Linux-2.6.5/include/linux/pagg.h Thu May 6 12:33:18 2004 *************** *** 121,126 **** --- 121,138 ---- * in the pagg container exec's a new process. This * is optional and may be set to NULL if it is not * needed by the pagg module. + * setruid: Function pointer to function used when a process + * in the pagg container changes its real uid. This + * is optional and may be set to NULL if it is not + * needed by the pagg module. + * setrgid: Function pointer to function used when a process + * in the pagg container changes its real gid. This + * is optional and may be set to NULL if it is not + * needed by the pagg module. + * setcpuaffinity: Function pointer to function used when a process + * in the pagg container changes its cpu affinity. This + * is optional and may be set to NULL if it is not + * needed by the pagg module. */ struct pagg_hook { struct module *module; *************** *** 131,136 **** --- 143,151 ---- int (*attach)(struct task_struct *, struct pagg *, void*); int (*detach)(struct task_struct *, struct pagg *); void (*exec)(struct task_struct *, struct pagg *); + void (*setruid)(struct task_struct *, struct pagg *); + void (*setrgid)(struct task_struct *, struct pagg *); + void (*setcpuaffinity)(struct task_struct *, struct pagg *); }; *************** *** 145,150 **** --- 160,168 ---- struct task_struct *from_task); extern int __pagg_detach(struct task_struct *task); extern int __pagg_exec(struct task_struct *task); + extern int __pagg_setruid(struct task_struct *task); + extern int __pagg_setrgid(struct task_struct *task); + extern int __pagg_setcpuaffinity(struct task_struct *task); /* macros used when a child process must inherit attachment to pagg * containers from the parent. *************** *** 182,187 **** --- 200,235 ---- __pagg_exec(task); \ } while(0) + /* + * macro used when a process changes real uid. + * + */ + #define pagg_setruid(task) \ + do { \ + if (!list_empty(&task->pagg_list.head)) \ + __pagg_setruid(task); \ + } while(0) + + /* + * macro used when a process changes real gid. + * + */ + #define pagg_setrgid(task) \ + do { \ + if (!list_empty(&task->pagg_list.head)) \ + __pagg_setrgid(task); \ + } while(0) + + /* + * macro used when a process changes cpu affinity. + * + */ + #define pagg_setrcpuaffinity(task) \ + do { \ + if (!list_empty(&task->pagg_list.head)) \ + __pagg_setrcpuaffinity(task); \ + } while(0) + /* The static inlines commented out for now with the ifdef below */ #ifdef NOTDEFINED *************** *** 217,222 **** --- 265,300 ---- if (!list_empty(&task->pagg_list.head)) __pagg_exec(task); } + + /* + * function used when a process changes real uid. + * + */ + static inline void pagg_setruid(struct task_struct *task) + { + if (!list_empty(&task->pagg_list.head)) + __pagg_setruid(task); + } + + /* + * function used when a process changes real gid. + * + */ + static inline void pagg_setrgid(struct task_struct *task) + { + if (!list_empty(&task->pagg_list.head)) + __pagg_setrgid(task); + } + + /* + * function used when a process changes cpu affinity. + * + */ + static inline void pagg_setcpuaffinity(struct task_struct *task) + { + if (!list_empty(&task->pagg_list.head)) + __pagg_setcpuaffinity(task); + } #endif /* NOT-DEFINED just comment out the block above for now */ /* *************** *** 240,245 **** --- 318,326 ---- #define pagg_attach(ct, pt) do { } while(0) #define pagg_detach(t) do { } while(0) #define pagg_exec(t) do { } while(0) + #define pagg_setruid(t) do { } while(0) + #define pagg_setrgid(t) do { } while(0) + #define pagg_setcpuaffinity(t) do { } while(0) #endif /* CONFIG_PAGG */ Index: Linux-2.6.5/kernel/pagg.c diff -c Linux-2.6.5/kernel/pagg.c:1.1.4.1.2.1 Linux-2.6.5/kernel/pagg.c:1.1.4.1.2.1.2.1 *** Linux-2.6.5/kernel/pagg.c:1.1.4.1.2.1 Thu May 6 10:41:14 2004 --- Linux-2.6.5/kernel/pagg.c Thu May 6 12:33:18 2004 *************** *** 497,502 **** --- 497,568 ---- } + /* + * __pagg_setruid + * + * Used to process a task's pagg list when changes real user id. + * + */ + int __pagg_setruid(struct task_struct *task) + { + struct pagg *pagg; + + down_read(&task->pagg_list.sem); /* lock the pagg list */ + + list_for_each_entry(pagg, &task->pagg_list.head, entry) { + if (pagg->hook->setruid) /* conditional because it's optional */ + pagg->hook->setruid(task, pagg); + } + + up_read(&task->pagg_list.sem); /* unlock the pagg list */ + return 0; + } + + + /* + * __pagg_setrgid + * + * Used to process a task's pagg list when it changes real group id. + * + */ + int __pagg_setrgid(struct task_struct *task) + { + struct pagg *pagg; + + down_read(&task->pagg_list.sem); /* lock the pagg list */ + + list_for_each_entry(pagg, &task->pagg_list.head, entry) { + if (pagg->hook->setrgid) /* conditional because it's optional */ + pagg->hook->setrgid(task, pagg); + } + + up_read(&task->pagg_list.sem); /* unlock the pagg list */ + return 0; + } + + + /* + * __pagg_setcpuaffinity + * + * Used to process a task's pagg list when it changes its cpu affinity. + * + */ + int __pagg_setcpuaffinity(struct task_struct *task) + { + struct pagg *pagg; + + down_read(&task->pagg_list.sem); /* lock the pagg list */ + + list_for_each_entry(pagg, &task->pagg_list.head, entry) { + if (pagg->hook->setcpuaffinity) /* conditional because it's optional */ + pagg->hook->setcpuaffinity(task, pagg); + } + + up_read(&task->pagg_list.sem); /* unlock the pagg list */ + return 0; + } + + EXPORT_SYMBOL(pagg_get); EXPORT_SYMBOL(pagg_alloc); EXPORT_SYMBOL(pagg_free); Index: Linux-2.6.5/kernel/sched.c diff -c Linux-2.6.5/kernel/sched.c:1.1.1.2 Linux-2.6.5/kernel/sched.c:1.1.1.2.16.1 *** Linux-2.6.5/kernel/sched.c:1.1.1.2 Wed Mar 31 13:31:56 2004 --- Linux-2.6.5/kernel/sched.c Thu May 6 12:33:18 2004 *************** *** 2712,2725 **** int set_cpus_allowed(task_t *p, cpumask_t new_mask) { unsigned long flags; - int ret = 0; migration_req_t req; runqueue_t *rq; rq = task_rq_lock(p, &flags); if (any_online_cpu(new_mask) == NR_CPUS) { ! ret = -EINVAL; ! goto out; } if (__set_cpus_allowed(p, new_mask, &req)) { --- 2712,2724 ---- int set_cpus_allowed(task_t *p, cpumask_t new_mask) { unsigned long flags; migration_req_t req; runqueue_t *rq; rq = task_rq_lock(p, &flags); if (any_online_cpu(new_mask) == NR_CPUS) { ! task_rq_unlock(rq, &flags); ! return -EINVAL; } if (__set_cpus_allowed(p, new_mask, &req)) { *************** *** 2727,2737 **** task_rq_unlock(rq, &flags); wake_up_process(rq->migration_thread); wait_for_completion(&req.done); return 0; } - out: task_rq_unlock(rq, &flags); ! return ret; } EXPORT_SYMBOL_GPL(set_cpus_allowed); --- 2726,2737 ---- task_rq_unlock(rq, &flags); wake_up_process(rq->migration_thread); wait_for_completion(&req.done); + pagg_setcpuaffinity(p); return 0; } task_rq_unlock(rq, &flags); ! pagg_setcpuaffinity(p); ! return 0; } EXPORT_SYMBOL_GPL(set_cpus_allowed); Index: Linux-2.6.5/kernel/sys.c diff -c Linux-2.6.5/kernel/sys.c:1.1.1.1 Linux-2.6.5/kernel/sys.c:1.1.1.1.38.1 *** Linux-2.6.5/kernel/sys.c:1.1.1.1 Sun Mar 21 11:09:00 2004 --- Linux-2.6.5/kernel/sys.c Thu May 6 12:33:18 2004 *************** *** 582,587 **** --- 582,588 ---- current->fsgid = new_egid; current->egid = new_egid; current->gid = new_rgid; + pagg_setrgid(current); return 0; } *************** *** 607,612 **** --- 608,614 ---- wmb(); } current->gid = current->egid = current->sgid = current->fsgid = gid; + pagg_setrgid(current); } else if ((gid == current->gid) || (gid == current->sgid)) { *************** *** 645,650 **** --- 647,653 ---- wmb(); } current->uid = new_ruid; + pagg_setruid(current); return 0; } *************** *** 843,850 **** current->egid = egid; } current->fsgid = current->egid; ! if (rgid != (gid_t) -1) current->gid = rgid; if (sgid != (gid_t) -1) current->sgid = sgid; return 0; --- 846,855 ---- current->egid = egid; } current->fsgid = current->egid; ! if (rgid != (gid_t) -1) { current->gid = rgid; + pagg_setrgid(current); + } if (sgid != (gid_t) -1) current->sgid = sgid; return 0; --------------030605080503040001000808-- From pwil3058@bigpond.net.au Fri May 7 04:22:04 2004 Received: with ECARTIS (v1.0.0; list pagg); Fri, 07 May 2004 04:22:31 -0700 (PDT) Received: from gizmo03bw.bigpond.com (gizmo03bw.bigpond.com [144.140.70.13]) by oss.sgi.com (8.12.10/8.12.9) with SMTP id i47BM1KO005242 for ; Fri, 7 May 2004 04:22:04 -0700 Received: (qmail 22950 invoked from network); 7 May 2004 11:13:04 -0000 Received: from unknown (HELO bwmam01.bigpond.com) (144.135.24.69) by gizmo03bw.bigpond.com with SMTP; 7 May 2004 11:13:04 -0000 Received: from cpe-203-45-91-55.nsw.bigpond.net.au ([203.45.91.55]) by bwmam01.bigpond.com(MAM REL_3_4_2 8/2686276) with SMTP id 2686276; Fri, 07 May 2004 21:21:54 +1000 Message-ID: <409B7152.2010808@bigpond.net.au> Date: Fri, 07 May 2004 21:21:54 +1000 From: Peter Williams User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 X-Accept-Language: en-us, en, en-au, fr-fr, es-ar MIME-Version: 1.0 To: Erik Jacobson CC: pagg@oss.sgi.com Subject: Re: New pagg/job patches available References: In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 16 X-ecartis-version: Ecartis v1.0.0 Sender: pagg-bounce@oss.sgi.com Errors-to: pagg-bounce@oss.sgi.com X-original-sender: pwil3058@bigpond.net.au Precedence: bulk X-list: pagg Erik Jacobson wrote: > There are new pagg and job patches available for 2.4.26 and 2.6.5 > kernels. > > I went through a couple sets of changes based on suggestions from LMKL > with most of the suggestions coming from Christoph Hellwig. > > Because changes include removing and renaming macros and functions, the > new job patch requires the new pagg patch. In other words, if you wish > to use the job patch, be sure you match it with the proper version of > the PAGG. > > I updated the README to show the pairs. > > I have one more batch of changes to implement yet. > > Thanks! > > You can find the new patches on oss.sgi.com/projects/pagg > Click on 'download' on the left. > > -- > Erik Jacobson - Linux System Software - Silicon Graphics - Eagan, Minnesota > > I've just been reading the "job" patch and would like to suggest that the file include/linux/paggctl.h should be renamed include/linux/jobctl.h as it is about control of the "job" PAGG client rather than PAGG itself which is what its current name suggests. Peter From erikj@subway.americas.sgi.com Fri May 7 05:30:41 2004 Received: with ECARTIS (v1.0.0; list pagg); Fri, 07 May 2004 05:30:43 -0700 (PDT) Received: from omx2.sgi.com (mtvcafw.sgi.com [192.48.171.6]) by oss.sgi.com (8.12.10/8.12.9) with SMTP id i47CUeKO006306 for ; Fri, 7 May 2004 05:30:40 -0700 Received: from flecktone.americas.sgi.com (flecktone.americas.sgi.com [192.48.203.135]) by omx2.sgi.com (8.12.11/8.12.9/linux-outbound_gateway-1.1) with ESMTP id i47CWv1k000633 for ; Fri, 7 May 2004 05:32:57 -0700 Received: from thistle-e236.americas.sgi.com (thistle-e236.americas.sgi.com [128.162.236.204]) by flecktone.americas.sgi.com (8.12.9/8.12.10/SGI_generic_relay-1.2) with ESMTP id i47CUXKe38038461; Fri, 7 May 2004 07:30:33 -0500 (CDT) Received: from subway.americas.sgi.com (subway.americas.sgi.com [128.162.236.152]) by thistle-e236.americas.sgi.com (8.12.9/SGI-server-1.8) with ESMTP id i47CUXGa4085361; Fri, 7 May 2004 07:30:33 -0500 (CDT) Received: from subway.americas.sgi.com (localhost [127.0.0.1]) by subway.americas.sgi.com (SGI-8.12.5/8.12.5/erikj-IRIX6519-news) with ESMTP id i47CUWS2177292; Fri, 7 May 2004 07:30:32 -0500 (CDT) Received: from localhost (erikj@localhost) by subway.americas.sgi.com (SGI-8.12.5/8.12.5/Submit) with ESMTP id i47CUSFv177250; Fri, 7 May 2004 07:30:32 -0500 (CDT) Date: Fri, 7 May 2004 07:30:28 -0500 From: Erik Jacobson To: Peter Williams cc: pagg@oss.sgi.com, Limin Gu Subject: Re: New pagg/job patches available In-Reply-To: <409B7152.2010808@bigpond.net.au> Message-ID: References: <409B7152.2010808@bigpond.net.au> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 17 X-ecartis-version: Ecartis v1.0.0 Sender: pagg-bounce@oss.sgi.com Errors-to: pagg-bounce@oss.sgi.com X-original-sender: erikj@subway.americas.sgi.com Precedence: bulk X-list: pagg > I've just been reading the "job" patch and would like to suggest that > the file include/linux/paggctl.h should be renamed > include/linux/jobctl.h as it is about control of the "job" PAGG client > rather than PAGG itself which is what its current name suggests. Sounds good to me personally. I think I came up with the bad name. It really is job talking to pagg but I agree the name is confusing. One issue is that it would require us to re-spin the job package as, at this moment, it includes paggctl.h. If you look at the job package, it itself has a header called 'jobctl.h' that defines how the ioctl is done. So it might be confusing to have two jobctl.h headers (one in the kernel, one in userland) but maybe we can come up with something else. Limin is starting to take over the job side of things so I'll let her comment and look in to the change. -Erik -- Erik Jacobson - Linux System Software - Silicon Graphics - Eagan, Minnesota From erikj@subway.americas.sgi.com Fri May 7 06:22:10 2004 Received: with ECARTIS (v1.0.0; list pagg); Fri, 07 May 2004 06:22:20 -0700 (PDT) Received: from omx1.americas.sgi.com (cfcafw.sgi.com [198.149.23.1]) by oss.sgi.com (8.12.10/8.12.9) with SMTP id i47DM9KO007174 for ; Fri, 7 May 2004 06:22:10 -0700 Received: from flecktone.americas.sgi.com (flecktone.americas.sgi.com [192.48.203.135]) by omx1.americas.sgi.com (8.12.10/8.12.9/linux-outbound_gateway-1.1) with ESMTP id i47D9dT9010175 for ; Fri, 7 May 2004 08:09:39 -0500 Received: from thistle-e236.americas.sgi.com (thistle-e236.americas.sgi.com [128.162.236.204]) by flecktone.americas.sgi.com (8.12.9/8.12.10/SGI_generic_relay-1.2) with ESMTP id i47D9cKe38004173 for ; Fri, 7 May 2004 08:09:38 -0500 (CDT) Received: from subway.americas.sgi.com (subway.americas.sgi.com [128.162.236.152]) by thistle-e236.americas.sgi.com (8.12.9/SGI-server-1.8) with ESMTP id i47D9cGa4196382 for ; Fri, 7 May 2004 08:09:38 -0500 (CDT) Received: from subway.americas.sgi.com (localhost [127.0.0.1]) by subway.americas.sgi.com (SGI-8.12.5/8.12.5/erikj-IRIX6519-news) with ESMTP id i47D9cS2177549 for ; Fri, 7 May 2004 08:09:38 -0500 (CDT) Received: from localhost (erikj@localhost) by subway.americas.sgi.com (SGI-8.12.5/8.12.5/Submit) with ESMTP id i47D9R9X177483 for ; Fri, 7 May 2004 08:09:37 -0500 (CDT) Date: Fri, 7 May 2004 08:09:27 -0500 From: Erik Jacobson To: pagg@oss.sgi.com Subject: BUG 914213 - PAGG feedback III - final round from the recent LKML discussion (fwd) Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 18 X-ecartis-version: Ecartis v1.0.0 Sender: pagg-bounce@oss.sgi.com Errors-to: pagg-bounce@oss.sgi.com X-original-sender: erikj@subway.americas.sgi.com Precedence: bulk X-list: pagg Hi there. We don't have a public bug tracking system (at least not yet). So I'm using our internal system. I filed this bug. If you're interested in it, you'd need to use the LKML discussion as a guide to what I"m talking about. I plan on implementing these issues over the next couple days. ---------- Forwarded message ---------- Date: Fri, 7 May 2004 06:04:02 -0700 (PDT) From: "erikj@sgi.com via BugWorks" Reply-To: sgi.bugs.snlinux@fido.engr.sgi.com Subject: BUG 914213 - PAGG feedback III - final round from the recent LKML discussion View Incident: http://co-op.engr.sgi.com/BugWorks/query.cgi/914213 Status: open Priority: 2 Assigned Group: linux-acct Project: snlinux Assigned Engineer: erikj Submitter: erikj CC List: jlan limin Opened Date: 05/07/04 This is the hopefully final (and hardest) feedback round. Some of the things to look at here include things I had trouble implementing before and will probably need help with. - There are two list_for_each's left in pagg.c. One I need to leave as-is, one can be changed to list_for_each_entry - I just missed it last round and didn't want to sneak it in after people had done reviews. - braces around '_l' in INIT_PAGG_LIST - I just couldn't get that to work and will need help. - I did change the macro names as suggested, but it was also suggested to change the macros to static inlines. I had trouble implementing this. I got some suggestions but still had some trouble. I need to look further. - It was commenting that the "Registering PAGG support for X" is too verbose but I couldn't come up with a less verbose message that still made sense to me. Suggestions welcome. - Biggest Issue: LOCKING LOCKING LOCKING.... I fixed up locking some, but Chris Wright and Christoph Hellwig have had suggestions. They wonder if I can switch from semaphores to spinlocks. Robin Holt wasn't convinced this was a good idea. However, having semaphores makes the code more complicated. In any case, we need to look at Chris Wright's feedback on locking and figure out what to do. We should look in to the same adjustments for JOB where necessary. From limin@engr.sgi.com Fri May 7 12:59:53 2004 Received: with ECARTIS (v1.0.0; list pagg); Fri, 07 May 2004 13:00:08 -0700 (PDT) Received: from zok.sgi.com (mtvcafw.sgi.com [192.48.171.6]) by oss.sgi.com (8.12.10/8.12.9) with SMTP id i47JxrKO001102 for ; Fri, 7 May 2004 12:59:53 -0700 Received: from cthulhu.engr.sgi.com (cthulhu.engr.sgi.com [192.26.80.2]) by zok.sgi.com (8.12.9/8.12.9/linux-outbound_gateway-1.1) with ESMTP id i47Jxmhv028120 for ; Fri, 7 May 2004 12:59:48 -0700 Received: from engr.sgi.com (redhat-linux.engr.sgi.com [163.154.6.101]) by cthulhu.engr.sgi.com (SGI-8.12.5/8.12.5) with ESMTP id i47JxP078784621; Fri, 7 May 2004 12:59:30 -0700 (PDT) Message-ID: <409BDE4D.2010500@engr.sgi.com> Date: Fri, 07 May 2004 12:06:53 -0700 From: Limin Gu User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/20030819 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Erik Jacobson CC: Peter Williams , pagg@oss.sgi.com, Limin Gu Subject: Re: New pagg/job patches available References: <409B7152.2010808@bigpond.net.au> In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 19 X-ecartis-version: Ecartis v1.0.0 Sender: pagg-bounce@oss.sgi.com Errors-to: pagg-bounce@oss.sgi.com X-original-sender: limin@engr.sgi.com Precedence: bulk X-list: pagg Erik Jacobson wrote: >>I've just been reading the "job" patch and would like to suggest that >>the file include/linux/paggctl.h should be renamed >>include/linux/jobctl.h as it is about control of the "job" PAGG client >>rather than PAGG itself which is what its current name suggests. >> >> > >Sounds good to me personally. I think I came up with the bad name. >It really is job talking to pagg but I agree the name is confusing. > > I agree, jobctl.h is a less confusing name for that header file. >One issue is that it would require us to re-spin the job package as, at this >moment, it includes paggctl.h. If you look at the job package, it itself has >a header called 'jobctl.h' that defines how the ioctl is done. So it might >be confusing to have two jobctl.h headers (one in the kernel, one in >userland) but maybe we can come up with something else. > > We may change the name of user land jobctl.h to something like jobctl_internal.h. I'll file a bug in SGI to have those file names changed. Thanks, --Limin >Limin is starting to take over the job side of things so I'll let her >comment and look in to the change. -Erik > >-- >Erik Jacobson - Linux System Software - Silicon Graphics - Eagan, Minnesota > > From peterw@mudlark.pw.nest.sgi.com Wed May 19 03:34:51 2004 Received: with ECARTIS (v1.0.0; list pagg); Wed, 19 May 2004 03:35:01 -0700 (PDT) Received: from smtp.sw.oz.au (IDENT:FWUSER@alt.aurema.com [203.217.18.57]) by oss.sgi.com (8.12.10/8.12.9) with SMTP id i4JAYiKO016287 for ; Wed, 19 May 2004 03:34:48 -0700 Received: from mudlark.pw.nest (swag.sw.oz.au [192.41.203.35]) by smtp.sw.oz.au with ESMTP id i4JAXA4X018623; Wed, 19 May 2004 20:33:12 +1000 (EST) Message-ID: <40AB37E6.6050407@mudlark.pw.nest> Date: Wed, 19 May 2004 20:33:10 +1000 From: Peter Williams User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Erik Jacobson CC: Peter Williams , pagg@oss.sgi.com Subject: Re: New pagg/job patches available References: <40989065.4080905@bigpond.net.au> In-Reply-To: Content-Type: multipart/mixed; boundary="------------000507090300050300090005" X-Scanned-By: MIMEDefang 2.42 X-archive-position: 20 X-ecartis-version: Ecartis v1.0.0 Sender: pagg-bounce@oss.sgi.com Errors-to: pagg-bounce@oss.sgi.com X-original-sender: peterw@mudlark.pw.nest.sgi.com Precedence: bulk X-list: pagg This is a multi-part message in MIME format. --------------000507090300050300090005 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Erik Jacobson wrote: > Thanks. > > Let me digest this and perhaps have some time to discuss it with a couple > of my co-workers. I did get this and will get back to you. > > Erik > > On Wed, 5 May 2004, Peter Williams wrote: > > >>Erik Jacobson wrote: >> >>>There are new pagg and job patches available for 2.4.26 and 2.6.5 >>>kernels. >>> >>>I went through a couple sets of changes based on suggestions from LMKL >>>with most of the suggestions coming from Christoph Hellwig. >>> >>>Because changes include removing and renaming macros and functions, the >>>new job patch requires the new pagg patch. In other words, if you wish >>>to use the job patch, be sure you match it with the proper version of >>>the PAGG. >>> >>>I updated the README to show the pairs. >>> >>>I have one more batch of changes to implement yet. >>> >>>Thanks! >>> >>>You can find the new patches on oss.sgi.com/projects/pagg >>>Click on 'download' on the left. >>> >>>-- >>>Erik Jacobson - Linux System Software - Silicon Graphics - Eagan, Minnesota >>> >>> >> >>Please find attached a patch which is a first pass at a "safe to block" >>method for calling the init() function during client registration. The >>way in which this patch handles kmalloc() failures needs to be improved >>but I think this could benefit from input from someone more familiar >>with PAGG than I am. >> I finally got around to testing this patch and found a few errors (not the least of which was calling semaphores inside a spinlock :-(). The enclosed patch fixes those problems and has passed some gentle testing. --------------000507090300050300090005 Content-Type: text/plain; name="pagg_init.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="pagg_init.patch" Index: Linux-2.6.5/kernel/pagg.c diff -c Linux-2.6.5/kernel/pagg.c:1.1.4.1 Linux-2.6.5/kernel/pagg.c:1.1.4.1.2.2 *** Linux-2.6.5/kernel/pagg.c:1.1.4.1 Wed May 5 15:36:45 2004 --- Linux-2.6.5/kernel/pagg.c Wed May 19 20:24:46 2004 *************** *** 202,207 **** --- 202,304 ---- /* Okay, we can insert into the pagg hook list */ list_add_tail(&pagg_hook_new->entry, &pagg_hook_list); + /* Now we can call the initialiser function (if present) for each task */ + if (pagg_hook_new->init != NULL) { + int num_inited = 0; + + /* Because of internal race conditions we can't gaurantee + * getting every task in just one pass so we just keep going + * until we don't find any unitialised tasks. The inefficiency + * of this should be tempered by the fact that this happens + * at most once for each registered client. + */ + do { + struct task_struct *p = NULL; + int *live_pids; + int live_pids_sz; + int i, nump; + int failed_pid_mallocs = 0; + int failed_pagg_mallocs = 0; + + retry_malloc: + live_pids_sz = nr_threads + 16; + live_pids = kmalloc(sizeof(int) * live_pids_sz, GFP_KERNEL); + if (live_pids == NULL) { + /* This should be changed to abort the registration + * and undo anything that's been done. Undoing the + * mess may be difficult so we'll just retry for the + * time being. + */ + if (failed_pid_mallocs < 10) { + failed_pid_mallocs++; + yield(); + goto retry_malloc; + } else { + /* we can't return an error value here + * as it would cause the module load to + * fail while we (possibly) still hold + * malloced memory. So just warn that + * initialisation has failed. This is + * no worse than completely ignoring + * the initialisation function. + */ + printk(KERN_WARNING "Insufficient memory" + " to initialise" + " PAGG support (name=%s)\n", + pagg_hook_new->name); + break; + } + } + read_lock(&tasklist_lock); + if (nr_threads > live_pids_sz) { + read_unlock(&tasklist_lock); + kfree(live_pids); + goto retry_malloc; + } + nump = 0; + for_each_process(p) { + live_pids[nump] = p->pid; + nump++; + } + read_unlock(&tasklist_lock); + num_inited = 0; + for (i = 0; i < nump; i++) { + read_lock(&tasklist_lock); + if (likely((p = find_task_by_pid(live_pids[i])) != NULL)) + get_task_struct(p); + read_unlock(&tasklist_lock); + if (likely(p != NULL)) { + struct pagg *paggp; + + down_read(&p->pagg_list.sem); + paggp = pagg_get(p, pagg_hook_new->name); + up_read(&p->pagg_list.sem); + + if (paggp == NULL) { + down_write(&p->pagg_list.sem); + paggp = pagg_alloc(p, pagg_hook_new); + if (paggp != NULL) + pagg_hook_new->init(p, paggp); + else + failed_pagg_mallocs++; + up_write(&p->pagg_list.sem); + num_inited++; + } + put_task_struct(p); + } + } + kfree(live_pids); + if (failed_pagg_mallocs > 10) { + /* we can't return an error value here + * for the same reason as above. + */ + printk(KERN_WARNING "Insufficient memory" + " to initialise PAGG support (name=%s)\n", + pagg_hook_new->name); + break; + } + } while (num_inited > 0); + } up_write(&pagg_hook_list_sem); printk(KERN_INFO "Registering PAGG support for (name=%s)\n", --------------000507090300050300090005-- From erikj@subway.americas.sgi.com Thu May 20 14:21:25 2004 Received: with ECARTIS (v1.0.0; list pagg); Thu, 20 May 2004 14:21:29 -0700 (PDT) Received: from omx1.americas.sgi.com (cfcafw.sgi.com [198.149.23.1]) by oss.sgi.com (8.12.10/8.12.9) with SMTP id i4KLLPKO024542 for ; Thu, 20 May 2004 14:21:25 -0700 Received: from flecktone.americas.sgi.com (flecktone.americas.sgi.com [192.48.203.135]) by omx1.americas.sgi.com (8.12.10/8.12.9/linux-outbound_gateway-1.1) with ESMTP id i4KLDPiv023285 for ; Thu, 20 May 2004 16:13:25 -0500 Received: from thistle-e236.americas.sgi.com (thistle-e236.americas.sgi.com [128.162.236.204]) by flecktone.americas.sgi.com (8.12.9/8.12.10/SGI_generic_relay-1.2) with ESMTP id i4KLDPKe38823997 for ; Thu, 20 May 2004 16:13:25 -0500 (CDT) Received: from subway.americas.sgi.com (subway.americas.sgi.com [128.162.236.152]) by thistle-e236.americas.sgi.com (8.12.9/SGI-server-1.8) with ESMTP id i4KLDOoo225110 for ; Thu, 20 May 2004 16:13:25 -0500 (CDT) Received: from subway.americas.sgi.com (localhost [127.0.0.1]) by subway.americas.sgi.com (SGI-8.12.5/8.12.5/erikj-IRIX6519-news) with ESMTP id i4KLDOq0213325 for ; Thu, 20 May 2004 16:13:24 -0500 (CDT) Received: from localhost (erikj@localhost) by subway.americas.sgi.com (SGI-8.12.5/8.12.5/Submit) with ESMTP id i4KLDOFN213159 for ; Thu, 20 May 2004 16:13:24 -0500 (CDT) Date: Thu, 20 May 2004 16:13:24 -0500 From: Erik Jacobson To: pagg@oss.sgi.com Subject: New 2.6.6 pagg and job patches available Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 21 X-ecartis-version: Ecartis v1.0.0 Sender: pagg-bounce@oss.sgi.com Errors-to: pagg-bounce@oss.sgi.com X-original-sender: erikj@subway.americas.sgi.com Precedence: bulk X-list: pagg New pagg and job patches are available from the pagg web site for the 2.6.6 kernel. oss.sgi.com/projects/pagg Click 'download' on the left. The patches I made available are: linux-2.6.6-pagg.patch and linux-2.6.6-job.patch Check out the README for some installation tips. Note: we haven't finished processing all the community feedback yet. So there will likely be at least one more 2.6.6 pagg patch to come. I also haven't implemented the patches from Peter Williams yet but hope to soon. Let me know if you have any questions. -- Erik Jacobson - Linux System Software - Silicon Graphics - Eagan, Minnesota From pwil3058@bigpond.net.au Thu May 20 19:25:46 2004 Received: with ECARTIS (v1.0.0; list pagg); Thu, 20 May 2004 19:26:01 -0700 (PDT) Received: from gizmo04ps.bigpond.com (gizmo04ps.bigpond.com [144.140.71.14]) by oss.sgi.com (8.12.10/8.12.9) with SMTP id i4L2PgKO001673 for ; Thu, 20 May 2004 19:25:44 -0700 Received: (qmail 22144 invoked from network); 21 May 2004 02:15:26 -0000 Received: from unknown (HELO psmam02.bigpond.com) (144.135.25.72) by gizmo04ps.bigpond.com with SMTP; 21 May 2004 02:15:26 -0000 Received: from cpe-203-45-91-55.nsw.bigpond.net.au ([203.45.91.55]) by psmam02.bigpond.com(MAM REL_3_4_2a 80/1132185) with SMTP id 1132185; Fri, 21 May 2004 12:25:32 +1000 Message-ID: <40AD689C.7030106@bigpond.net.au> Date: Fri, 21 May 2004 12:25:32 +1000 From: Peter Williams User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Erik Jacobson CC: pagg@oss.sgi.com Subject: Re: New 2.6.6 pagg and job patches available References: In-Reply-To: Content-Type: multipart/mixed; boundary="------------040808010600030601050201" X-archive-position: 22 X-ecartis-version: Ecartis v1.0.0 Sender: pagg-bounce@oss.sgi.com Errors-to: pagg-bounce@oss.sgi.com X-original-sender: pwil3058@bigpond.net.au Precedence: bulk X-list: pagg This is a multi-part message in MIME format. --------------040808010600030601050201 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Erik Jacobson wrote: > New pagg and job patches are available from the pagg web site for the > 2.6.6 kernel. > > oss.sgi.com/projects/pagg > > Click 'download' on the left. > > The patches I made available are: > linux-2.6.6-pagg.patch > and > linux-2.6.6-job.patch > > Check out the README for some installation tips. > > Note: we haven't finished processing all the community feedback yet. So > there will likely be at least one more 2.6.6 pagg patch to come. > I also haven't implemented the patches from Peter Williams yet but hope to > soon. Attached is a patch (against your latest 2.6.6 patch) to do the initialisation of tasks at registration and add real uid/gid and CPU affinity hooks. The "initialisation of tasks" code still needs some work so that it cleans up properly when faced with memory allocation failures. Probably best left to someone more familiar with PAGG than me. On a related note, shouldn't the deregistration code do something similar? i.e. call detach() on all of the deregistering clients paggs and remove form any task lists that they are on rather than just refusing to unload if there are any tasks with paggs belonging to the deregistering client. Otherwise every client will have to reinvent the wheel in order to do this when it is unloaded from the kernel. Peter -- Dr Peter Williams pwil3058@bigpond.net.au "Learning, n. The kind of ignorance distinguishing the studious." -- Ambrose Bierce --------------040808010600030601050201 Content-Type: text/plain; name="pagg_extras-2.6.6.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="pagg_extras-2.6.6.patch" Index: Linux-2.6.X/Documentation/pagg.txt diff -c Linux-2.6.X/Documentation/pagg.txt:1.1.4.1 Linux-2.6.X/Documentation/pagg.txt:1.1.4.1.4.1 *** Linux-2.6.X/Documentation/pagg.txt:1.1.4.1 Fri May 21 11:33:14 2004 --- Linux-2.6.X/Documentation/pagg.txt Fri May 21 12:03:40 2004 *************** *** 32,37 **** --- 32,48 ---- used, for example, by other kernel modules that wish to do advanced CPU placement on multi-processor systems (just one example). + The set_user function has been modified to support an optional callout + that can be run when a process in a pagg list changes its real uid. + + The sys_setresgid, sy_setregid and sys_setgid functions have been modified + to support optional callouts that can be run when a process in a pagg list changes + its real gid. + + The set_cpus_allowed function has been modified to support an optional callout + that can be run when a process in a pagg list changes its cpu affinity. It could be + used, for example, to implement CPU sets. + Additional details concerning this implementation of the process aggregates infrastructure are described in the sections that follow. *************** *** 51,56 **** --- 62,69 ---- - kernel/Makefile - kernel/exit.c - kernel/fork.c + - kernel/sched.c + - kernel/sys.c - fs/exec.c - init/Kconfig *************** *** 96,101 **** --- 109,117 ---- void *data; /* Module specific data */ struct list_head entry; /* List connection */ void (*exec)(struct task_struct *, struct pagg *); /* exec func ptr */ + void (*setruid)(struct task_struct *, struct pagg *); /* setruid func ptr */ + void (*setrgid)(struct task_struct *, struct pagg *); /* setrgid func ptr */ + void (*setcpuaffinity)(struct task_struct *, struct pagg *); /* setcpuaffinity func ptr */ The pagg structure provides the process' reference to the PAGG containers provided by the PAGG modules. The attach function pointer *************** *** 104,110 **** the referenced PAGG container that the process is exiting or otherwise detaching from the container. The exec function pointer is used when a process in the pagg container exec's a new process. This is optional and ! may be set to NULL if it is not needed by the pagg module. The pagg_hook structure provides the reference to the module that implements a type of PAGG container. In addition to the function pointers --- 120,134 ---- the referenced PAGG container that the process is exiting or otherwise detaching from the container. The exec function pointer is used when a process in the pagg container exec's a new process. This is optional and ! may be set to NULL if it is not needed by the pagg module. The setruid ! function pointer is used when a process in the pagg container changes its ! real uid. This is optional and may be set to NULL if it is not needed by the ! pagg module. The setrgid function pointer is used when a process in the ! pagg container changes its real gid. This is optional and may be set to ! NULL if it is not needed by the pagg module. The setcpuaffinity function ! pointer is used when a process in the pagg container changes its cpu ! affinity. This is optional and may be set to NULL if it is not needed by the ! pagg module. The pagg_hook structure provides the reference to the module that implements a type of PAGG container. In addition to the function pointers *************** *** 129,134 **** --- 153,167 ---- Call detach_pagg_list function with current task_struct - sys_execve: (fs/exec.c) /* When a process in a pagg exec's, an optional callout can be run. This + is implemented with an optional function pointer in the pagg_hook. */ + - set_user: (kernel/sys.c) + /* When a process in a pagg sets its real uid, an optional callout can be run. This + is implemented with an optional function pointer in the pagg_hook. */ + - sys_setresgid, sy_setregid and sys_setgid: (kernel/sys.c) + /* When a process in a pagg sets its real gid, an optional callout can be run. This + is implemented with an optional function pointer in the pagg_hook. */ + - set_cpus_allowed: (kernel/sched.c) + /* When a process in a pagg changes cpu affinity, an optional callout can be run. This is implemented with an optional function pointer in the pagg_hook. */ 2.6 New Functions Index: Linux-2.6.X/include/linux/pagg.h diff -c Linux-2.6.X/include/linux/pagg.h:1.1.4.1 Linux-2.6.X/include/linux/pagg.h:1.1.4.1.4.1 *** Linux-2.6.X/include/linux/pagg.h:1.1.4.1 Fri May 21 11:33:14 2004 --- Linux-2.6.X/include/linux/pagg.h Fri May 21 12:03:40 2004 *************** *** 121,126 **** --- 121,138 ---- * in the pagg container exec's a new process. This * is optional and may be set to NULL if it is not * needed by the pagg module. + * setruid: Function pointer to function used when a process + * in the pagg container changes its real uid. This + * is optional and may be set to NULL if it is not + * needed by the pagg module. + * setrgid: Function pointer to function used when a process + * in the pagg container changes its real gid. This + * is optional and may be set to NULL if it is not + * needed by the pagg module. + * setcpuaffinity: Function pointer to function used when a process + * in the pagg container changes its cpu affinity. This + * is optional and may be set to NULL if it is not + * needed by the pagg module. */ struct pagg_hook { struct module *module; *************** *** 131,136 **** --- 143,151 ---- int (*attach)(struct task_struct *, struct pagg *, void*); int (*detach)(struct task_struct *, struct pagg *); void (*exec)(struct task_struct *, struct pagg *); + void (*setruid)(struct task_struct *, struct pagg *); + void (*setrgid)(struct task_struct *, struct pagg *); + void (*setcpuaffinity)(struct task_struct *, struct pagg *); }; *************** *** 145,150 **** --- 160,168 ---- struct task_struct *from_task); extern int __pagg_detach(struct task_struct *task); extern int __pagg_exec(struct task_struct *task); + extern int __pagg_setruid(struct task_struct *task); + extern int __pagg_setrgid(struct task_struct *task); + extern int __pagg_setcpuaffinity(struct task_struct *task); /* macros used when a child process must inherit attachment to pagg * containers from the parent. *************** *** 182,187 **** --- 200,235 ---- __pagg_exec(task); \ } while(0) + /* + * macro used when a process changes real uid. + * + */ + #define pagg_setruid(task) \ + do { \ + if (!list_empty(&task->pagg_list.head)) \ + __pagg_setruid(task); \ + } while(0) + + /* + * macro used when a process changes real gid. + * + */ + #define pagg_setrgid(task) \ + do { \ + if (!list_empty(&task->pagg_list.head)) \ + __pagg_setrgid(task); \ + } while(0) + + /* + * macro used when a process changes cpu affinity. + * + */ + #define pagg_setrcpuaffinity(task) \ + do { \ + if (!list_empty(&task->pagg_list.head)) \ + __pagg_setrcpuaffinity(task); \ + } while(0) + /* The static inlines commented out for now with the ifdef below */ #ifdef NOTDEFINED *************** *** 217,222 **** --- 265,300 ---- if (!list_empty(&task->pagg_list.head)) __pagg_exec(task); } + + /* + * function used when a process changes real uid. + * + */ + static inline void pagg_setruid(struct task_struct *task) + { + if (!list_empty(&task->pagg_list.head)) + __pagg_setruid(task); + } + + /* + * function used when a process changes real gid. + * + */ + static inline void pagg_setrgid(struct task_struct *task) + { + if (!list_empty(&task->pagg_list.head)) + __pagg_setrgid(task); + } + + /* + * function used when a process changes cpu affinity. + * + */ + static inline void pagg_setcpuaffinity(struct task_struct *task) + { + if (!list_empty(&task->pagg_list.head)) + __pagg_setcpuaffinity(task); + } #endif /* NOT-DEFINED just comment out the block above for now */ /* *************** *** 240,245 **** --- 318,326 ---- #define pagg_attach(ct, pt) do { } while(0) #define pagg_detach(t) do { } while(0) #define pagg_exec(t) do { } while(0) + #define pagg_setruid(t) do { } while(0) + #define pagg_setrgid(t) do { } while(0) + #define pagg_setcpuaffinity(t) do { } while(0) #endif /* CONFIG_PAGG */ Index: Linux-2.6.X/kernel/pagg.c diff -c Linux-2.6.X/kernel/pagg.c:1.1.4.1 Linux-2.6.X/kernel/pagg.c:1.1.4.1.2.1.2.1 *** Linux-2.6.X/kernel/pagg.c:1.1.4.1 Fri May 21 11:33:14 2004 --- Linux-2.6.X/kernel/pagg.c Fri May 21 12:03:40 2004 *************** *** 202,207 **** --- 202,304 ---- /* Okay, we can insert into the pagg hook list */ list_add_tail(&pagg_hook_new->entry, &pagg_hook_list); + /* Now we can call the initialiser function (if present) for each task */ + if (pagg_hook_new->init != NULL) { + int num_inited = 0; + + /* Because of internal race conditions we can't gaurantee + * getting every task in just one pass so we just keep going + * until we don't find any unitialised tasks. The inefficiency + * of this should be tempered by the fact that this happens + * at most once for each registered client. + */ + do { + struct task_struct *p = NULL; + int *live_pids; + int live_pids_sz; + int i, nump; + int failed_pid_mallocs = 0; + int failed_pagg_mallocs = 0; + + retry_malloc: + live_pids_sz = nr_threads + 16; + live_pids = kmalloc(sizeof(int) * live_pids_sz, GFP_KERNEL); + if (live_pids == NULL) { + /* This should be changed to abort the registration + * and undo anything that's been done. Undoing the + * mess may be difficult so we'll just retry for the + * time being. + */ + if (failed_pid_mallocs < 10) { + failed_pid_mallocs++; + yield(); + goto retry_malloc; + } else { + /* we can't return an error value here + * as it would cause the module load to + * fail while we (possibly) still hold + * malloced memory. So just warn that + * initialisation has failed. This is + * no worse than completely ignoring + * the initialisation function. + */ + printk(KERN_WARNING "Insufficient memory" + " to initialise" + " PAGG support (name=%s)\n", + pagg_hook_new->name); + break; + } + } + read_lock(&tasklist_lock); + if (nr_threads > live_pids_sz) { + read_unlock(&tasklist_lock); + kfree(live_pids); + goto retry_malloc; + } + nump = 0; + for_each_process(p) { + live_pids[nump] = p->pid; + nump++; + } + read_unlock(&tasklist_lock); + num_inited = 0; + for (i = 0; i < nump; i++) { + read_lock(&tasklist_lock); + if (likely((p = find_task_by_pid(live_pids[i])) != NULL)) + get_task_struct(p); + read_unlock(&tasklist_lock); + if (likely(p != NULL)) { + struct pagg *paggp; + + down_read(&p->pagg_list.sem); + paggp = pagg_get(p, pagg_hook_new->name); + up_read(&p->pagg_list.sem); + + if (paggp == NULL) { + down_write(&p->pagg_list.sem); + paggp = pagg_alloc(p, pagg_hook_new); + if (paggp != NULL) + pagg_hook_new->init(p, paggp); + else + failed_pagg_mallocs++; + up_write(&p->pagg_list.sem); + num_inited++; + } + put_task_struct(p); + } + } + kfree(live_pids); + if (failed_pagg_mallocs > 10) { + /* we can't return an error value here + * for the same reason as above. + */ + printk(KERN_WARNING "Insufficient memory" + " to initialise PAGG support (name=%s)\n", + pagg_hook_new->name); + break; + } + } while (num_inited > 0); + } up_write(&pagg_hook_list_sem); printk(KERN_INFO "Registering PAGG support for (name=%s)\n", *************** *** 392,397 **** --- 489,560 ---- list_for_each_entry(pagg, &task->pagg_list.head, entry) { if (pagg->hook->exec) /* conditional because it's optional */ pagg->hook->exec(task, pagg); + } + + up_read(&task->pagg_list.sem); /* unlock the pagg list */ + return 0; + } + + + /* + * __pagg_setruid + * + * Used to process a task's pagg list when changes real user id. + * + */ + int __pagg_setruid(struct task_struct *task) + { + struct pagg *pagg; + + down_read(&task->pagg_list.sem); /* lock the pagg list */ + + list_for_each_entry(pagg, &task->pagg_list.head, entry) { + if (pagg->hook->setruid) /* conditional because it's optional */ + pagg->hook->setruid(task, pagg); + } + + up_read(&task->pagg_list.sem); /* unlock the pagg list */ + return 0; + } + + + /* + * __pagg_setrgid + * + * Used to process a task's pagg list when it changes real group id. + * + */ + int __pagg_setrgid(struct task_struct *task) + { + struct pagg *pagg; + + down_read(&task->pagg_list.sem); /* lock the pagg list */ + + list_for_each_entry(pagg, &task->pagg_list.head, entry) { + if (pagg->hook->setrgid) /* conditional because it's optional */ + pagg->hook->setrgid(task, pagg); + } + + up_read(&task->pagg_list.sem); /* unlock the pagg list */ + return 0; + } + + + /* + * __pagg_setcpuaffinity + * + * Used to process a task's pagg list when it changes its cpu affinity. + * + */ + int __pagg_setcpuaffinity(struct task_struct *task) + { + struct pagg *pagg; + + down_read(&task->pagg_list.sem); /* lock the pagg list */ + + list_for_each_entry(pagg, &task->pagg_list.head, entry) { + if (pagg->hook->setcpuaffinity) /* conditional because it's optional */ + pagg->hook->setcpuaffinity(task, pagg); } up_read(&task->pagg_list.sem); /* unlock the pagg list */ Index: Linux-2.6.X/kernel/sched.c diff -c Linux-2.6.X/kernel/sched.c:1.1.1.7 Linux-2.6.X/kernel/sched.c:1.1.1.7.12.1 *** Linux-2.6.X/kernel/sched.c:1.1.1.7 Thu May 6 18:43:49 2004 --- Linux-2.6.X/kernel/sched.c Fri May 21 12:03:40 2004 *************** *** 2722,2735 **** int set_cpus_allowed(task_t *p, cpumask_t new_mask) { unsigned long flags; - int ret = 0; migration_req_t req; runqueue_t *rq; rq = task_rq_lock(p, &flags); if (any_online_cpu(new_mask) == NR_CPUS) { ! ret = -EINVAL; ! goto out; } if (__set_cpus_allowed(p, new_mask, &req)) { --- 2722,2734 ---- int set_cpus_allowed(task_t *p, cpumask_t new_mask) { unsigned long flags; migration_req_t req; runqueue_t *rq; rq = task_rq_lock(p, &flags); if (any_online_cpu(new_mask) == NR_CPUS) { ! task_rq_unlock(rq, &flags); ! return -EINVAL; } if (__set_cpus_allowed(p, new_mask, &req)) { *************** *** 2737,2747 **** task_rq_unlock(rq, &flags); wake_up_process(rq->migration_thread); wait_for_completion(&req.done); return 0; } - out: task_rq_unlock(rq, &flags); ! return ret; } EXPORT_SYMBOL_GPL(set_cpus_allowed); --- 2736,2747 ---- task_rq_unlock(rq, &flags); wake_up_process(rq->migration_thread); wait_for_completion(&req.done); + pagg_setcpuaffinity(p); return 0; } task_rq_unlock(rq, &flags); ! pagg_setcpuaffinity(p); ! return 0; } EXPORT_SYMBOL_GPL(set_cpus_allowed); Index: Linux-2.6.X/kernel/sys.c diff -c Linux-2.6.X/kernel/sys.c:1.1.1.6 Linux-2.6.X/kernel/sys.c:1.1.1.6.16.1 *** Linux-2.6.X/kernel/sys.c:1.1.1.6 Thu May 6 18:43:49 2004 --- Linux-2.6.X/kernel/sys.c Fri May 21 12:03:40 2004 *************** *** 593,598 **** --- 593,599 ---- current->fsgid = new_egid; current->egid = new_egid; current->gid = new_rgid; + pagg_setrgid(current); return 0; } *************** *** 618,623 **** --- 619,625 ---- wmb(); } current->gid = current->egid = current->sgid = current->fsgid = gid; + pagg_setrgid(current); } else if ((gid == current->gid) || (gid == current->sgid)) { *************** *** 656,661 **** --- 658,664 ---- wmb(); } current->uid = new_ruid; + pagg_setruid(current); return 0; } *************** *** 854,861 **** current->egid = egid; } current->fsgid = current->egid; ! if (rgid != (gid_t) -1) current->gid = rgid; if (sgid != (gid_t) -1) current->sgid = sgid; return 0; --- 857,866 ---- current->egid = egid; } current->fsgid = current->egid; ! if (rgid != (gid_t) -1) { current->gid = rgid; + pagg_setrgid(current); + } if (sgid != (gid_t) -1) current->sgid = sgid; return 0; --------------040808010600030601050201-- From pwil3058@bigpond.net.au Wed May 26 20:17:57 2004 Received: with ECARTIS (v1.0.0; list pagg); Wed, 26 May 2004 20:18:04 -0700 (PDT) Received: from gizmo01bw.bigpond.com (gizmo01bw.bigpond.com [144.140.70.11]) by oss.sgi.com (8.12.10/8.12.9) with SMTP id i4R3Htgi001940 for ; Wed, 26 May 2004 20:17:56 -0700 Received: (qmail 32713 invoked from network); 27 May 2004 03:10:07 -0000 Received: from unknown (HELO bwmam01.bigpond.com) (144.135.24.69) by gizmo01bw.bigpond.com with SMTP; 27 May 2004 03:10:07 -0000 Received: from cpe-203-45-91-93.nsw.bigpond.net.au ([203.45.91.93]) by bwmam01.bigpond.com(MAM REL_3_4_2a 8/493621) with SMTP id 493621; Thu, 27 May 2004 13:17:48 +1000 Message-ID: <40B55DDB.5020108@bigpond.net.au> Date: Thu, 27 May 2004 13:17:47 +1000 From: Peter Williams User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Erik Jacobson CC: pagg@oss.sgi.com Subject: Re: New 2.6.6 pagg and job patches available References: <40AD689C.7030106@bigpond.net.au> In-Reply-To: <40AD689C.7030106@bigpond.net.au> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 23 X-ecartis-version: Ecartis v1.0.0 Sender: pagg-bounce@oss.sgi.com Errors-to: pagg-bounce@oss.sgi.com X-original-sender: pwil3058@bigpond.net.au Precedence: bulk X-list: pagg Peter Williams wrote: > Erik Jacobson wrote: > >> New pagg and job patches are available from the pagg web site for the >> 2.6.6 kernel. >> >> oss.sgi.com/projects/pagg >> >> Click 'download' on the left. >> >> The patches I made available are: >> linux-2.6.6-pagg.patch >> and >> linux-2.6.6-job.patch >> >> Check out the README for some installation tips. >> >> Note: we haven't finished processing all the community feedback yet. So >> there will likely be at least one more 2.6.6 pagg patch to come. >> I also haven't implemented the patches from Peter Williams yet but >> hope to >> soon. > > > Attached is a patch (against your latest 2.6.6 patch) to do the > initialisation of tasks at registration and add real uid/gid and CPU > affinity hooks. > > The "initialisation of tasks" code still needs some work so that it > cleans up properly when faced with memory allocation failures. Probably > best left to someone more familiar with PAGG than me. > > On a related note, shouldn't the deregistration code do something > similar? i.e. call detach() on all of the deregistering clients paggs > and remove form any task lists that they are on rather than just > refusing to unload if there are any tasks with paggs belonging to the > deregistering client. Otherwise every client will have to reinvent the > wheel in order to do this when it is unloaded from the kernel. I've just been perusing pagg_hook_unregister() with a view to implementing the above idea and noticed that there is a possible race condition in the code that checks for the presence of paggs from the client being deregistered in tasks. Because the tasklist_lock is released within the for loop it is possible for new tasks to be created and added in such a way that they miss being processed by this for loop. -- Dr Peter Williams pwil3058@bigpond.net.au "Learning, n. The kind of ignorance distinguishing the studious." -- Ambrose Bierce From erikj@subway.americas.sgi.com Thu May 27 06:28:29 2004 Received: with ECARTIS (v1.0.0; list pagg); Thu, 27 May 2004 06:28:41 -0700 (PDT) Received: from zok.sgi.com (mtvcafw.sgi.com [192.48.171.6]) by oss.sgi.com (8.12.10/8.12.9) with SMTP id i4RDSSgi013630 for ; Thu, 27 May 2004 06:28:28 -0700 Received: from flecktone.americas.sgi.com (flecktone.americas.sgi.com [192.48.203.135]) by zok.sgi.com (8.12.9/8.12.9/linux-outbound_gateway-1.1) with ESMTP id i4RDMahv001525 for ; Thu, 27 May 2004 06:22:36 -0700 Received: from thistle-e236.americas.sgi.com (thistle-e236.americas.sgi.com [128.162.236.204]) by flecktone.americas.sgi.com (8.12.9/8.12.10/SGI_generic_relay-1.2) with ESMTP id i4RDMaKe39288602 for ; Thu, 27 May 2004 08:22:36 -0500 (CDT) Received: from subway.americas.sgi.com (subway.americas.sgi.com [128.162.236.152]) by thistle-e236.americas.sgi.com (8.12.9/SGI-server-1.8) with ESMTP id i4RDMaoo631015 for ; Thu, 27 May 2004 08:22:36 -0500 (CDT) Received: from subway.americas.sgi.com (localhost [127.0.0.1]) by subway.americas.sgi.com (SGI-8.12.5/8.12.5/erikj-IRIX6519-news) with ESMTP id i4RDMZq0635577 for ; Thu, 27 May 2004 08:22:35 -0500 (CDT) Received: from localhost (erikj@localhost) by subway.americas.sgi.com (SGI-8.12.5/8.12.5/Submit) with ESMTP id i4RDMZvC635559 for ; Thu, 27 May 2004 08:22:35 -0500 (CDT) Date: Thu, 27 May 2004 08:22:35 -0500 From: Erik Jacobson To: pagg@oss.sgi.com Subject: New PAGG and Job patches available Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 24 X-ecartis-version: Ecartis v1.0.0 Sender: pagg-bounce@oss.sgi.com Errors-to: pagg-bounce@oss.sgi.com X-original-sender: erikj@subway.americas.sgi.com Precedence: bulk X-list: pagg Hi there. I have just made available new PAGG and job patches 2.4.26 and 2.6.6 kernels. These new patches are a result of community feedback on the Linux Kernel Mailling list. I believe we're mostly caught up with the mailing list feedback now. We're not caught up with accepting in new features yet and I hope to investigate that soon with some of my coworkers. Because the feedback changed a structure used by kernel modules making use of PAGG, it was necessary to also re-spin the job patch to match. These are the new pairs: 2.4.26: linux-2.4.26-pagg.patch-4 linux-2.4.26-job.patch-4 2.6.6: linux-2.6.6-pagg.patch-2 linux-2.6.6-job.patch-2 Please send a note to the list if you have any questions or problems. Thanks! -- Erik Jacobson - Linux System Software - Silicon Graphics - Eagan, Minnesota From peterw@aurema.com Fri May 28 02:29:20 2004 Received: with ECARTIS (v1.0.0; list pagg); Fri, 28 May 2004 02:29:23 -0700 (PDT) Received: from mudlark.pw.nest (CPE-203-45-91-93.nsw.bigpond.net.au [203.45.91.93]) by oss.sgi.com (8.12.10/8.12.9) with SMTP id i4S9THgi028998 for ; Fri, 28 May 2004 02:29:18 -0700 Received: from aurema.com (localhost.localdomain [127.0.0.1]) by mudlark.pw.nest (8.12.11/8.12.11) with ESMTP id i4S9TDXN003084; Fri, 28 May 2004 19:29:14 +1000 Message-ID: <40B70669.5000909@aurema.com> Date: Fri, 28 May 2004 19:29:13 +1000 From: Peter Williams Organization: Aurema Pty Ltd User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Erik Jacobson CC: pagg@oss.sgi.com Subject: Re: [PATCH] Process Aggregates (PAGG) update for 2.6.6 References: In-Reply-To: Content-Type: multipart/mixed; boundary="------------030808050206030304070703" X-archive-position: 25 X-ecartis-version: Ecartis v1.0.0 Sender: pagg-bounce@oss.sgi.com Errors-to: pagg-bounce@oss.sgi.com X-original-sender: peterw@aurema.com Precedence: bulk X-list: pagg This is a multi-part message in MIME format. --------------030808050206030304070703 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Erik Jacobson wrote: > Hi there. > > Attached is a fresh PAGG patch for the 2.6.6 kernel. > > This patch implements the LKML feedback received so far (with very few > exceptions). > > I would be happy to post the inescapable jobs patch (a user of PAGG) as well. > It can be found here (along with pagg): > > http://oss.sgi.com/projects/pagg/ Attached is a patch the implements per task initialisation during registration, detaches the relevant paggs from all tasks during unregistration of a client, and adds hooks for set real uid/gid and setting of CPU affinity. This patch is based on the one you've just posted and takes into account the various changes in technique that it incorporates. Enjoy Peter -- Dr Peter Williams, Chief Scientist peterw@aurema.com Aurema Pty Limited Tel:+61 2 9698 2322 PO Box 305, Strawberry Hills NSW 2012, Australia Fax:+61 2 9699 9174 79 Myrtle Street, Chippendale NSW 2008, Australia http://www.aurema.com --------------030808050206030304070703 Content-Type: text/plain; name="pagg-extras-2.6.6.patch" Content-Disposition: inline; filename="pagg-extras-2.6.6.patch" Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by mudlark.pw.nest id i4S9TDXN003084 Index: Linux-2.6.X/Documentation/pagg.txt diff -c Linux-2.6.X/Documentation/pagg.txt:1.1.6.1 Linux-2.6.X/Documentat= ion/pagg.txt:1.1.6.1.2.1 *** Linux-2.6.X/Documentation/pagg.txt:1.1.6.1 Fri May 28 11:26:10 2004 --- Linux-2.6.X/Documentation/pagg.txt Fri May 28 17:36:14 2004 *************** *** 30,32 **** --- 30,188 ---- used, for example, by other kernel modules that wish to do advanced CPU placement on multi-processor systems (just one example). =20 + The set_user function has been modified to support an optional callout + that can be run when a process in a pagg list changes its real uid. +=20 + The sys_setresgid, sy_setregid and sys_setgid functions have been modif= ied + to support optional callouts that can be run when a process in a pagg l= ist changes + its real gid. +=20 + The set_cpus_allowed function has been modified to support an optional = callout + that can be run when a process in a pagg list changes its cpu affinity.= It could be + used, for example, to implement CPU sets. +=20 + Additional details concerning this implementation of the process aggreg= ates + infrastructure are described in the sections that follow. +=20 +=20 + 2. Kernel Changes +=20 + This section describe the files and data strcutrues that are involved i= n this + implementation of PAGG. Both modified as well as new files and data + structures are discussed. +=20 + 2.1. Modified Files +=20 + The following files were modified to implement PAGG: +=20 + - include/linux/init_task.h + - include/linux/sched.h + - kernel/Makefile + - kernel/exit.c + - kernel/fork.c + - kernel/sched.c + - kernel/sys.c + - fs/exec.c + - init/Kconfig +=20 + 2.2. New Files +=20 + The following files were added to implement PAGG: +=20 + - Documentation/pagg.txt + - include/linux/pagg.h + - kernel/pagg.c +=20 +=20 + 2.3. Modified Data Structures +=20 + The following existing data structures were altered to implement PAGG. +=20 + - struct task_struct: (include/linux/sched.h) + struct pagg_list pagg_list; /* List of pagg containers */ +=20 + This new member in task_struct, pagg_list, points to the list of pagg + containers to which the process is currently attached. +=20 + 2.4. New Data Structures +=20 + The following new data structures were introduced to implement PAGG. +=20 + - struct pagg: (include/linux/pagg.h) + struct pagg_hook *hook /* Ptr to pagg module entry */ + void *data; /* Task specific data */ + struct list_head entry; /* List connection */=09 + =20 + - struct pagg_hook: (include/linux/pagg.h) + struct module *module; /* Ptr to PAGG module */ + char *name; /* PAGG hook name - restri= cted + * to 32 characters. */ + int (*attach)(struct task_struct *, /* Function to attach */ + struct pagg *, + void *); + int (*detach)(struct task_struct *, /* Function to detach */ + struct pagg *); + int (*init)(struct task_struct *, /* Load task init func. */ + struct pagg *); + void *data; /* Module specific data */ + struct list_head entry; /* List connection */ + void (*exec)(struct task_struct *, struct pagg *); /* exec func= ptr */ + void (*setruid)(struct task_struct *, struct pagg *); /* setruid= func ptr */ + void (*setrgid)(struct task_struct *, struct pagg *); /* setrgid= func ptr */ + void (*setcpuaffinity)(struct task_struct *, struct pagg *); /* = setcpuaffinity func ptr */ +=20 + The pagg structure provides the process' reference to the PAGG + containers provided by the PAGG modules. The attach function pointer + is the function used to notify the referenced PAGG container that the + process is being attached. The detach function pointer is used to noti= fy + the referenced PAGG container that the process is exiting or otherwise + detaching from the container. The exec function pointer is used when a + process in the pagg container exec's a new process. This is optional a= nd + may be set to NULL if it is not needed by the pagg module. The setruid + function pointer is used when a process in the pagg container changes i= ts + real uid. This is optional and may be set to NULL if it is not needed = by the + pagg module. The setrgid function pointer is used when a process in the + pagg container changes its real gid. This is optional and may be set t= o + NULL if it is not needed by the pagg module. The setcpuaffinity functio= n + pointer is used when a process in the pagg container changes its cpu + affinity. This is optional and may be set to NULL if it is not needed = by the + pagg module. +=20 + The pagg_hook structure provides the reference to the module that + implements a type of PAGG container. In addition to the function point= ers + described concerning pagg, this structure provides an addition + function pointer. The init function pointer is currently not used + but will be available in the future. Future use of the init function + will be optional and will used to attach currently running processes to + a default PAGG container when a PAGG module is loaded on a running syst= em. +=20 +=20 + 2.5. Modified Functions +=20 + The following functions were changed to implement PAGG: +=20 + - do_fork: (kernel/fork.c) + /* execute the following pseudocode before add to run-queue */ =20 + If parent process pagg list is not empty + Call attach_pagg_list function with child task_struct as argu= ment + - do_exit: (kernel/exit.c) + /* execute the following pseudocode prior to schedule call */ + If current process pagg list is not empty + Call detach_pagg_list function with current task_struct=20 + - sys_execve: (fs/exec.c) + /* When a process in a pagg exec's, an optional callout can be run= . This + is implemented with an optional function pointer in the pagg_ho= ok. */ + - set_user: (kernel/sys.c) + /* When a process in a pagg sets its real uid, an optional callout= can be run. This + is implemented with an optional function pointer in the pagg_ho= ok. */ + - sys_setresgid, sy_setregid and sys_setgid: (kernel/sys.c) + /* When a process in a pagg sets its real gid, an optional callout= can be run. This + is implemented with an optional function pointer in the pagg_ho= ok. */ + - set_cpus_allowed: (kernel/sched.c) + /* When a process in a pagg changes cpu affinity, an optional call= out can be run. This + is implemented with an optional function pointer in the pagg_ho= ok. */ +=20 + 2.6 New Functions +=20 + The following new functions were added to implement PAGG: +=20 + - int register_pagg_hook(struct pagg_hook *); (kernel/pagg.c) + Add module entry into table of pagg modules + - int unregister_pagg_hook(struct pagg_hook *); (kernel/pagg.c) + Find module entry in list of pagg modules + Foreach task + If task is attached to this pagg module + return error + If no tasks are referencing this module + remove module entry from list of pagg modules + - int attach_pagg_list(struct task_struct *); (kernel/pagg.c) + /* Assumed task pagg list pts to paggs that it attaches to */ + While another pagg container reference + Make copy of pagg container reference & insert into new list + Attach task to pagg container using new container reference + Get next pagg container reference + Make task pagg list use the new pagg list + - int detach_pagg_list(struct task_struct *); (kernel/pagg.c) + While another pagg container reference + Detach task from pagg container using reference +=20 Index: Linux-2.6.X/include/linux/pagg.h diff -c Linux-2.6.X/include/linux/pagg.h:1.1.6.1 Linux-2.6.X/include/linu= x/pagg.h:1.1.6.1.2.1 *** Linux-2.6.X/include/linux/pagg.h:1.1.6.1 Fri May 28 11:26:10 2004 --- Linux-2.6.X/include/linux/pagg.h Fri May 28 17:36:14 2004 *************** *** 105,122 **** * in the pagg container exec's a new process. This * is optional and may be set to NULL if it is not=20 * needed by the pagg module. * refcnt: Keep track of user count of the pagg hook */ struct pagg_hook { ! struct module *module; ! char *name; /* Name Key - restricted to 32 characters */ ! void *data; /* Opaque module specific data */ ! struct list_head entry; /* List pointers */ ! atomic_t refcnt; /* usage counter */ ! int (*init)(struct task_struct *, struct pagg *); ! int (*attach)(struct task_struct *, struct pagg *, void*); ! int (*detach)(struct task_struct *, struct pagg *); ! void (*exec)(struct task_struct *, struct pagg *); }; =20 =20 --- 105,137 ---- * in the pagg container exec's a new process. This * is optional and may be set to NULL if it is not=20 * needed by the pagg module. + * setruid: Function pointer to function used when a process + * in the pagg container changes its real uid. This + * is optional and may be set to NULL if it is not=20 + * needed by the pagg module. + * setrgid: Function pointer to function used when a process + * in the pagg container changes its real gid. This + * is optional and may be set to NULL if it is not=20 + * needed by the pagg module. + * setcpuaffinity: Function pointer to function used when a process + * in the pagg container changes its cpu affinity. = This + * is optional and may be set to NULL if it is not=20 + * needed by the pagg module. * refcnt: Keep track of user count of the pagg hook */ struct pagg_hook { ! struct module *module; ! char *name; /* Name Key - restricted to 32 characters */ ! void *data; /* Opaque module specific data */ ! struct list_head entry; /* List pointers */ ! atomic_t refcnt; /* usage counter */ ! int (*init)(struct task_struct *, struct pagg *); ! int (*attach)(struct task_struct *, struct pagg *, void*); ! int (*detach)(struct task_struct *, struct pagg *); ! void (*exec)(struct task_struct *, struct pagg *); ! void (*setruid)(struct task_struct *, struct pagg *); ! void (*setrgid)(struct task_struct *, struct pagg *); ! void (*setcpuaffinity)(struct task_struct *, struct pagg *); }; =20 =20 *************** *** 131,136 **** --- 146,154 ---- struct task_struct *from_task); extern int __pagg_detach(struct task_struct *task); extern int __pagg_exec(struct task_struct *task); + extern int __pagg_setruid(struct task_struct *task); + extern int __pagg_setrgid(struct task_struct *task); + extern int __pagg_setcpuaffinity(struct task_struct *task); =20 /* function used when a child process must inherit attachment to pagg * containers from the parent. *************** *** 166,171 **** --- 184,219 ---- __pagg_exec(task); } =20 + /*=20 + * function used when a process setruid's. + * + */ + static inline void pagg_setruid(struct task_struct *task) + { + if (!list_empty(&task->pagg_list)) + __pagg_setruid(task); + } +=20 + /*=20 + * function used when a process setrgid's. + * + */ + static inline void pagg_setrgid(struct task_struct *task) + { + if (!list_empty(&task->pagg_list)) + __pagg_setrgid(task); + } +=20 + /*=20 + * function used when a process's cpu affinity changes. + * + */ + static inline void pagg_setcpuaffinity(struct task_struct *task) + { + if (!list_empty(&task->pagg_list)) + __pagg_setcpuaffinity(task); + } +=20 /* * Marco Used in INIT_TASK to set the head and sem of pagg_list. * If CONFIG_PAGG is off, it is defined as an empty macro below. *************** *** 185,190 **** --- 233,241 ---- #define pagg_attach(ct, pt) do { } while(0) #define pagg_detach(t) do { } while(0) =20 #define pagg_exec(t) do { } while(0) =20 + #define pagg_setruid(t) do { } while(0) =20 + #define pagg_setrgid(t) do { } while(0) =20 + #define pagg_setcpuaffinity(t) do { } while(0) =20 =20 #endif /* CONFIG_PAGG */ =20 Index: Linux-2.6.X/kernel/pagg.c diff -c Linux-2.6.X/kernel/pagg.c:1.1.6.1 Linux-2.6.X/kernel/pagg.c:1.1.6= .1.2.2 *** Linux-2.6.X/kernel/pagg.c:1.1.6.1 Fri May 28 11:26:10 2004 --- Linux-2.6.X/kernel/pagg.c Fri May 28 19:15:01 2004 *************** *** 157,162 **** --- 157,203 ---- return NULL; } =20 + /* + * remove_named_pagg_from_all_tasks + *=20 + * Given a pagg hook name key, this function will remove all paggs + * associated with that pagg hook from all tasks calling the provided + * function on each pagg + */ + static void + remove_named_pagg_from_all_tasks(struct pagg_hook *php) + { + if (php =3D=3D NULL) + return; +=20 + /* Because of internal race conditions we can't gaurantee + * getting every task in just one pass so we just keep going=20 + * until there are no tasks with paggs from this hook attached. + * The inefficiency of this should be tempered by the fact that this h= appens + * at most once for each registered client. + */ + while (atomic_read(&php->refcnt) !=3D 0) { + struct task_struct *p =3D NULL; +=20 + read_lock(&tasklist_lock); + for_each_process(p) { + struct pagg *paggp; +=20 + get_task_struct(p); + read_unlock(&tasklist_lock); + down_write(&p->pagg_sem); + paggp =3D pagg_get(p, php->name); + if (paggp !=3D NULL) { + (void)php->detach(p, paggp); + pagg_free(paggp); + } + up_write(&p->pagg_sem); + read_lock(&tasklist_lock); + put_task_struct(p); + } + read_unlock(&tasklist_lock); + } + } =20 /* * pagg_hook_register *************** *** 208,213 **** --- 249,310 ---- atomic_set(&pagg_hook_new->refcnt, 0); /* printk("DEBUG - pagg hook register - refcnt now: %d\n",=20 atomic_read(&pagg_hook_new->refcnt)); */ +=20 + /* Now we can call the initialiser function (if present) for each task= */ + if (pagg_hook_new->init !=3D NULL) { + int num_inited =3D 0; + int malloc_failures =3D 0; + int init_result =3D 0; +=20 + /* Because of internal race conditions we can't gaurantee + * getting every task in just one pass so we just keep going=20 + * until we don't find any unitialised tasks. The inefficiency + * of this should be tempered by the fact that this happens + * at most once for each registered client. + */ + do { + struct task_struct *p =3D NULL; +=20 + num_inited =3D 0; + read_lock(&tasklist_lock); + for_each_process(p) { + struct pagg *paggp; +=20 + get_task_struct(p); + read_unlock(&tasklist_lock); + down_write(&p->pagg_sem); + paggp =3D pagg_get(p, pagg_hook_new->name); + if (paggp =3D=3D NULL) { + paggp =3D pagg_alloc(p, pagg_hook_new); + if (paggp !=3D NULL) + init_result =3D pagg_hook_new->init(p, paggp); + else + malloc_failures++; + num_inited++; + } + up_write(&p->pagg_sem); + read_lock(&tasklist_lock); + put_task_struct(p); + } + read_unlock(&tasklist_lock); + } while ((init_result =3D=3D 0) && (malloc_failures =3D=3D 0) && (num= _inited > 0)); +=20 + /* + * if anything went wrong during initialisation abandon the + * registration process + */ + if ((init_result !=3D 0) || (malloc_failures !=3D 0)) { + remove_named_pagg_from_all_tasks(pagg_hook_new); + list_del_init(&pagg_hook_new->entry); + up_write(&pagg_hook_list_sem); +=20 + printk(KERN_WARNING "Registering PAGG support for" + " (name=3D%s) failed\n", pagg_hook_new->name); +=20 + return malloc_failures ? -ENOMEM : init_result; /* success */ + } + } +=20 up_write(&pagg_hook_list_sem); =20 printk(KERN_INFO "Registering PAGG support for (name=3D%s)\n", *************** *** 247,258 **** */ =20 if (pagg_hook && pagg_hook =3D=3D pagg_hook_old) { ! /* Is the pagg hook busy? Check if the refcnt is zero */ ! if (atomic_read(&pagg_hook->refcnt) !=3D 0) { ! up_write(&pagg_hook_list_sem); ! printk(KERN_INFO "Failed attempt to unregister a PAGG hook from: %s\= n", pagg_hook_old->name); ! return -EBUSY; ! } list_del_init(&pagg_hook->entry); up_write(&pagg_hook_list_sem); =20 --- 344,350 ---- */ =20 if (pagg_hook && pagg_hook =3D=3D pagg_hook_old) { ! remove_named_pagg_from_all_tasks(pagg_hook); list_del_init(&pagg_hook->entry); up_write(&pagg_hook_list_sem); =20 *************** *** 378,383 **** --- 470,541 ---- } =20 =20 + /* + * __pagg_setruid + * + * Used to process a task's pagg list when changes real user id. + * + */ + int __pagg_setruid(struct task_struct *task)=20 + { + struct pagg *pagg; +=20 + down_read(&task->pagg_sem); /* lock the pagg list */ +=20 + list_for_each_entry(pagg, &task->pagg_list, entry) { + if (pagg->hook->setruid) /* conditional because it's optional */ + pagg->hook->setruid(task, pagg); + } +=20 + up_read(&task->pagg_sem); /* unlock the pagg list */ + return 0; + } +=20 +=20 + /* + * __pagg_setrgid + * + * Used to process a task's pagg list when it changes real group id. + * + */ + int __pagg_setrgid(struct task_struct *task)=20 + { + struct pagg *pagg; +=20 + down_read(&task->pagg_sem); /* lock the pagg list */ +=20 + list_for_each_entry(pagg, &task->pagg_list, entry) { + if (pagg->hook->setrgid) /* conditional because it's optional */ + pagg->hook->setrgid(task, pagg); + } +=20 + up_read(&task->pagg_sem); /* unlock the pagg list */ + return 0; + } +=20 +=20 + /* + * __pagg_setcpuaffinity + * + * Used to process a task's pagg list when it changes its cpu affinity. + * + */ + int __pagg_setcpuaffinity(struct task_struct *task)=20 + { + struct pagg *pagg; +=20 + down_read(&task->pagg_sem); /* lock the pagg list */ +=20 + list_for_each_entry(pagg, &task->pagg_list, entry) { + if (pagg->hook->setcpuaffinity) /* conditional because it's optional = */ + pagg->hook->setcpuaffinity(task, pagg); + } +=20 + up_read(&task->pagg_sem); /* unlock the pagg list */ + return 0; + } +=20 +=20 EXPORT_SYMBOL(pagg_get); EXPORT_SYMBOL(pagg_alloc); EXPORT_SYMBOL(pagg_free); Index: Linux-2.6.X/kernel/sched.c diff -c Linux-2.6.X/kernel/sched.c:1.1.1.7 Linux-2.6.X/kernel/sched.c:1.1= .1.7.16.1 *** Linux-2.6.X/kernel/sched.c:1.1.1.7 Thu May 6 18:43:49 2004 --- Linux-2.6.X/kernel/sched.c Fri May 28 17:36:14 2004 *************** *** 2722,2735 **** int set_cpus_allowed(task_t *p, cpumask_t new_mask) { unsigned long flags; - int ret =3D 0; migration_req_t req; runqueue_t *rq; =20 rq =3D task_rq_lock(p, &flags); if (any_online_cpu(new_mask) =3D=3D NR_CPUS) { ! ret =3D -EINVAL; ! goto out; } =20 if (__set_cpus_allowed(p, new_mask, &req)) { --- 2722,2734 ---- int set_cpus_allowed(task_t *p, cpumask_t new_mask) { unsigned long flags; migration_req_t req; runqueue_t *rq; =20 rq =3D task_rq_lock(p, &flags); if (any_online_cpu(new_mask) =3D=3D NR_CPUS) { ! task_rq_unlock(rq, &flags); ! return -EINVAL; } =20 if (__set_cpus_allowed(p, new_mask, &req)) { *************** *** 2737,2747 **** task_rq_unlock(rq, &flags); wake_up_process(rq->migration_thread); wait_for_completion(&req.done); return 0; } - out: task_rq_unlock(rq, &flags); ! return ret; } =20 EXPORT_SYMBOL_GPL(set_cpus_allowed); --- 2736,2747 ---- task_rq_unlock(rq, &flags); wake_up_process(rq->migration_thread); wait_for_completion(&req.done); + pagg_setcpuaffinity(p); return 0; } task_rq_unlock(rq, &flags); ! pagg_setcpuaffinity(p); ! return 0; } =20 EXPORT_SYMBOL_GPL(set_cpus_allowed); Index: Linux-2.6.X/kernel/sys.c diff -c Linux-2.6.X/kernel/sys.c:1.1.1.6 Linux-2.6.X/kernel/sys.c:1.1.1.6= .22.1 *** Linux-2.6.X/kernel/sys.c:1.1.1.6 Thu May 6 18:43:49 2004 --- Linux-2.6.X/kernel/sys.c Fri May 28 17:36:14 2004 *************** *** 23,28 **** --- 23,29 ---- #include #include #include + #include =20 #include #include *************** *** 593,598 **** --- 594,600 ---- current->fsgid =3D new_egid; current->egid =3D new_egid; current->gid =3D new_rgid; + pagg_setrgid(current); return 0; } =20 *************** *** 618,623 **** --- 620,626 ---- wmb(); } current->gid =3D current->egid =3D current->sgid =3D current->fsgid =3D= gid; + pagg_setrgid(current); } else if ((gid =3D=3D current->gid) || (gid =3D=3D current->sgid)) { *************** *** 656,661 **** --- 659,665 ---- wmb(); } current->uid =3D new_ruid; + pagg_setruid(current); return 0; } =20 *************** *** 854,861 **** current->egid =3D egid; } current->fsgid =3D current->egid; ! if (rgid !=3D (gid_t) -1) current->gid =3D rgid; if (sgid !=3D (gid_t) -1) current->sgid =3D sgid; return 0; --- 858,867 ---- current->egid =3D egid; } current->fsgid =3D current->egid; ! if (rgid !=3D (gid_t) -1) { current->gid =3D rgid; + pagg_setrgid(current); + } if (sgid !=3D (gid_t) -1) current->sgid =3D sgid; return 0; *************** *** 905,911 **** } =20 /* ! * Samma p=E5 svenska.. */ asmlinkage long sys_setfsgid(gid_t gid) { --- 911,917 ---- } =20 /* ! * Samma p=EF=BF=BD svenska.. */ asmlinkage long sys_setfsgid(gid_t gid) { --------------030808050206030304070703-- From erikj@subway.americas.sgi.com Fri May 28 12:36:29 2004 Received: with ECARTIS (v1.0.0; list pagg); Fri, 28 May 2004 12:36:31 -0700 (PDT) Received: from omx1.americas.sgi.com (cfcafw.sgi.com [198.149.23.1]) by oss.sgi.com (8.12.10/8.12.9) with SMTP id i4SJaTgi032731 for ; Fri, 28 May 2004 12:36:29 -0700 Received: from flecktone.americas.sgi.com (flecktone.americas.sgi.com [192.48.203.135]) by omx1.americas.sgi.com (8.12.10/8.12.9/linux-outbound_gateway-1.1) with ESMTP id i4SJP7iv021931 for ; Fri, 28 May 2004 14:25:07 -0500 Received: from thistle-e236.americas.sgi.com (thistle-e236.americas.sgi.com [128.162.236.204]) by flecktone.americas.sgi.com (8.12.9/8.12.10/SGI_generic_relay-1.2) with ESMTP id i4SJP6Ke39416905 for ; Fri, 28 May 2004 14:25:06 -0500 (CDT) Received: from subway.americas.sgi.com (subway.americas.sgi.com [128.162.236.152]) by thistle-e236.americas.sgi.com (8.12.9/SGI-server-1.8) with ESMTP id i4SJP6oo747109 for ; Fri, 28 May 2004 14:25:06 -0500 (CDT) Received: from subway.americas.sgi.com (localhost [127.0.0.1]) by subway.americas.sgi.com (SGI-8.12.5/8.12.5/erikj-IRIX6519-news) with ESMTP id i4SJP6q0725837 for ; Fri, 28 May 2004 14:25:06 -0500 (CDT) Received: from localhost (erikj@localhost) by subway.americas.sgi.com (SGI-8.12.5/8.12.5/Submit) with ESMTP id i4SJP5C8724756 for ; Fri, 28 May 2004 14:25:06 -0500 (CDT) Date: Fri, 28 May 2004 14:25:05 -0500 From: Erik Jacobson To: pagg@oss.sgi.com Subject: Re: New PAGG and Job patches available In-Reply-To: Message-ID: References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 26 X-ecartis-version: Ecartis v1.0.0 Sender: pagg-bounce@oss.sgi.com Errors-to: pagg-bounce@oss.sgi.com X-original-sender: erikj@subway.americas.sgi.com Precedence: bulk X-list: pagg I just re-spun the 2.6.6 patch. Patch linux-2.6.6-pagg.patch-3 is now available. Only comments were adjusted per a suggestion that we should be using kerneldoc style comments. No actual code was injured in the making of the new comments. On Thu, 27 May 2004, Erik Jacobson wrote: > Hi there. > > I have just made available new PAGG and job patches 2.4.26 and 2.6.6 kernels. > > These new patches are a result of community feedback on the Linux Kernel > Mailling list. > > I believe we're mostly caught up with the mailing list feedback now. We're > not caught up with accepting in new features yet and I hope to investigate > that soon with some of my coworkers. > > Because the feedback changed a structure used by kernel modules making > use of PAGG, it was necessary to also re-spin the job patch to match. > > These are the new pairs: > 2.4.26: linux-2.4.26-pagg.patch-4 linux-2.4.26-job.patch-4 > 2.6.6: linux-2.6.6-pagg.patch-2 linux-2.6.6-job.patch-2 > > Please send a note to the list if you have any questions or problems. > Thanks! > > -- > Erik Jacobson - Linux System Software - Silicon Graphics - Eagan, Minnesota > -- Erik Jacobson - Linux System Software - Silicon Graphics - Eagan, Minnesota