From mgoodwin@redhat.com Thu Jan 1 00:51:19 2015 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id E29887F3F for ; Thu, 1 Jan 2015 00:51:19 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id D14FD8F80AD for ; Wed, 31 Dec 2014 22:51:16 -0800 (PST) X-ASG-Debug-ID: 1420095075-04cbb0106916eb1d0001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 6cC7pogqmmiUtxxh (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 31 Dec 2014 22:51:15 -0800 (PST) X-Barracuda-Envelope-From: mgoodwin@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id t016pEnn010629 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Thu, 1 Jan 2015 01:51:14 -0500 Received: from [10.64.51.141] (vpn1-51-141.bne.redhat.com [10.64.51.141]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id t016pC7J009978; Thu, 1 Jan 2015 01:51:13 -0500 Message-ID: <54A4EE5F.2000108@redhat.com> Date: Thu, 01 Jan 2015 17:51:11 +1100 From: Mark Goodwin User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: pcp@oss.sgi.com CC: "shirshendu@riva.co >> Shirshendu Chakrabarti" Subject: pcp updates: fix kernel.pernode.cpu metrics on single CPU systems Content-Type: text/plain; charset=windows-1252; format=flowed X-ASG-Orig-Subj: pcp updates: fix kernel.pernode.cpu metrics on single CPU systems Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1420095075 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Fix for the kernel.pernode.cpu problem reported by Shirshendu Chakrabarti where kernel.pernode.cpu.user was incorrectly zero on single CPU systems. Plus a new QA test to verify the fix. Changes committed to git://git.performancecopilot.org/markgw/pcp/pcp.git dev commit 4ea1e0b7758660514fcae3117112bf1db4818ba0 Author: Mark Goodwin Date: Thu Jan 1 16:47:39 2015 +1100 Fix kernel.pernode.cpu.* for systems with only one CPU. On systems with only one CPU, the Linux PMDA re-uses kernel.all.cpu values for the kernel.percpu.cpu values (since there is only one CPU the values are the same), but neglected to also do this for the kernel.pernode.cpu values. Hence the per-node metrics were always zero on single CPU systems. Also, don't use /sys/devices/system/node/node[0-9]*/cpumap to determine the cpu:node mapping. The cpumap bitmap has different syntax depending on the kernel config options, e.g. an AWS kernel exports a different cpumap syntax to a Fedora or RHEL kernel. Instead, just use the cpu symlinks for each node (/sys/devices/system/node/node[0-9]*/cpu[0-9]*) to determine the mapping. modified: src/pmdas/linux/pmda.c modified: src/pmdas/linux/proc_cpuinfo.c modified: src/pmdas/linux/proc_stat.c commit 76dabf032b90c794fed3323e09ca8598571b6a03 Author: Mark Goodwin Date: Thu Jan 1 17:42:00 2015 +1100 QA to check kernel.pernode.cpu metrics on single CPU systems. new file: qa/873 new file: qa/873.out modified: qa/group From kenj@internode.on.net Thu Jan 1 14:21:53 2015 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 678597F55 for ; Thu, 1 Jan 2015 14:21:53 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 56493304039 for ; Thu, 1 Jan 2015 12:21:50 -0800 (PST) X-ASG-Debug-ID: 1420143704-04cbb0106b170b460001-S8gJnT Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by cuda.sgi.com with ESMTP id rURYtBI4tyFVgzUu for ; Thu, 01 Jan 2015 12:21:45 -0800 (PST) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.145 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AuQBAEerpVR20ZX6PGdsb2JhbAANT4pfwyWCTwKBGwEBAQEBBgEBAQE4hEgBAQQnEUARCxgJFg8JAwIBAgExFBMIAQG2RpR+AQEIAgEfj34WhBMBBJgVjQqDOYQkgx4BAQE Received: from ppp118-209-149-250.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.149.250]) by ipmail06.adl6.internode.on.net with ESMTP; 02 Jan 2015 06:51:43 +1030 Message-ID: <54A5AD0C.9010105@internode.on.net> Date: Fri, 02 Jan 2015 07:24:44 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: Re: [pcp] pcp updates: fix kernel.pernode.cpu metrics on single CPU systems References: <54A4EE5F.2000108@redhat.com> X-ASG-Orig-Subj: Re: [pcp] pcp updates: fix kernel.pernode.cpu metrics on single CPU systems In-Reply-To: <54A4EE5F.2000108@redhat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl6.internode.on.net[150.101.137.145] X-Barracuda-Start-Time: 1420143704 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.13778 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- On 01/01/15 17:51, Mark Goodwin wrote: > Fix for the kernel.pernode.cpu problem reported by Shirshendu Chakrabarti > where kernel.pernode.cpu.user was incorrectly zero on single CPU systems. > Plus a new QA test to verify the fix. > ... Mark, Thanks for this. I've cherry picked these commits into my tree (and will push them upstream after review). On a single CPU machine the QA tests fails before and pass after the pmda change, so that's good. For the code change, I have a couple of observations. For the most part these are not in your changes but in the context around them ... 1. The last sprintf() in map_cpu_nodes() should be a snprintf() ... I think this is part of our de facto virtual style guide, so I've made that change. 2. This part at the head of map_cpu_nodes() confused me ... pmdaIndom *idp = PMDAINDOM(NODE_INDOM); for (i = 0; i < proc_cpuinfo->cpuindom->it_numinst; i++) because proc_cpuinfo->cpuindom must be == idp for this to all make sense, so I'd expect the references to the indom control struct in this routine to all be consistent. I think using idp->it_numinst in the loop control will work, and removes the inconsistency. 3. But my real question is about the sizing of the two arrays (which need to end up the same), proc_cpuinfo->cpuinfo[] and idp->it_set[]. The latter is resized and initialized at the end of map_cpu_nodes() but on entry to map_cpu_nodes() there is an assumption that proc_cpuinfo->cpuinfo[] is already the big enough to accommodate all the cpus found in the directory traversal. I think there needs to be a comment describing why this assumption is expected to be true and/or a check in the loop before proc_cpuinfo->cpuinfo[cpu].node = node; to ensure cpu is not too large Cheers and Happy New Year, Ken. From mgoodwin@redhat.com Thu Jan 1 18:26:08 2015 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id E45EE7F55 for ; Thu, 1 Jan 2015 18:26:08 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id C4355304039 for ; Thu, 1 Jan 2015 16:26:05 -0800 (PST) X-ASG-Debug-ID: 1420158360-04cb6c0572279be10001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id FACdbSWC4KD4FOYf (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 01 Jan 2015 16:26:01 -0800 (PST) X-Barracuda-Envelope-From: mgoodwin@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id t020PuNo018125 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Thu, 1 Jan 2015 19:25:56 -0500 Received: from [10.64.51.142] (vpn1-51-142.bne.redhat.com [10.64.51.142]) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id t020PsNk004667; Thu, 1 Jan 2015 19:25:55 -0500 Message-ID: <54A5E591.3010309@redhat.com> Date: Fri, 02 Jan 2015 11:25:53 +1100 From: Mark Goodwin User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: Ken McDonell , pcp@oss.sgi.com Subject: Re: [pcp] pcp updates: fix kernel.pernode.cpu metrics on single CPU systems References: <54A4EE5F.2000108@redhat.com> <54A5AD0C.9010105@internode.on.net> X-ASG-Orig-Subj: Re: [pcp] pcp updates: fix kernel.pernode.cpu metrics on single CPU systems In-Reply-To: <54A5AD0C.9010105@internode.on.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1420158361 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On 01/02/2015 07:24 AM, Ken McDonell wrote: > On 01/01/15 17:51, Mark Goodwin wrote: >> Fix for the kernel.pernode.cpu problem reported by Shirshendu Chakrabarti >> where kernel.pernode.cpu.user was incorrectly zero on single CPU systems. >> Plus a new QA test to verify the fix. [...] > 1. The last sprintf() in map_cpu_nodes() should be a snprintf() ... I think this > is part of our de facto virtual style guide, so I've made that change. yep ok, thanks. (I just pulled from your tree, but you haven't committed yet?) > > 2. This part at the head of map_cpu_nodes() confused me ... > pmdaIndom *idp = PMDAINDOM(NODE_INDOM); > > for (i = 0; i < proc_cpuinfo->cpuindom->it_numinst; i++) > because proc_cpuinfo->cpuindom must be == idp for this to all make sense, so I'd > expect the references to the indom control struct in this routine to all be > consistent. I think using idp->it_numinst in the loop control will work, and > removes the inconsistency. yep, that's outside this bug fix, but agree and ok I'll make that change and a few others to make the code more robust in general. > > 3. But my real question is about the sizing of the two arrays (which need to end > up the same), proc_cpuinfo->cpuinfo[] and idp->it_set[]. The latter is resized > and initialized at the end of map_cpu_nodes() but on entry to map_cpu_nodes() > there is an assumption that proc_cpuinfo->cpuinfo[] is already the big enough to > accommodate all the cpus found in the directory traversal. I think there needs > to be a comment describing why this assumption is expected to be true and/or a > check in the loop before > proc_cpuinfo->cpuinfo[cpu].node = node; > to ensure cpu is not too large yes, this could be more robust and defensive too. If it's busted then the kernel sysfs code is busted (someone else's problem!). On the other hand, I'm not sure how this code would tolerate cpu-hotplug - the bug has been there since the numa epoch (circa 1995 or so, when cpu hotplug wasn't feasible) but I guess nobody noticed the per-node cpu metrics were broken on single CPU systems. Any thoughts on how we might test cpu hotplug/unplug? Do we need to even support it? I suspect that if the per-cpu and per-node indoms need to be dynamic then this isn't the only codxe that will need some attention ... > > Cheers and Happy New Year, Ken. and to you too :) Cheers -- Mark From wwwrun@oss.sgi.com Fri Jan 2 10:56:41 2015 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE,NO_RELAYS autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: by oss.sgi.com (Postfix, from userid 30) id ACB2A7F58; Fri, 2 Jan 2015 10:56:41 -0600 (CST) From: bugzilla-daemon@oss.sgi.com To: pcp@oss.sgi.com Subject: [Bug 1100] New: libpcp interp.c crash on archive scan Date: Fri, 02 Jan 2015 16:56:40 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Classification: Unclassified X-Bugzilla-Product: pcp X-Bugzilla-Component: pcp X-Bugzilla-Keywords: X-Bugzilla-Severity: major X-Bugzilla-Who: fche@redhat.com X-Bugzilla-Status: NEW X-Bugzilla-Priority: P5 X-Bugzilla-Assigned-To: pcp@kenj.com.au X-Bugzilla-Target-Milestone: --- X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter cc classification Message-ID: Content-Type: multipart/alternative; boundary="1420217801.f6d481.24382"; charset="us-ascii" X-Bugzilla-URL: http://oss.sgi.com/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 --1420217801.f6d481.24382 Date: Fri, 2 Jan 2015 10:56:41 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" http://oss.sgi.com/bugzilla/show_bug.cgi?id=1100 Bug ID: 1100 Summary: libpcp interp.c crash on archive scan Product: pcp Version: unspecified Hardware: All OS: Linux Status: NEW Severity: major Priority: P5 Component: pcp Assignee: pcp@kenj.com.au Reporter: fche@redhat.com CC: pcp@oss.sgi.com Classification: Unclassified Created attachment 326 --> http://oss.sgi.com/bugzilla/attachment.cgi?id=326&action=edit bad (?) archive % pmval -a *20141125*00009*meta network.interface.in.bytes crashes thusly: 21:01:43.810 No values available 21:01:44.810 No values available 21:01:45.810 No values available [Fri Jan 2 11:54:41] pmval(7977) Warning: __pmPinPDUBuf: 0x80000004e9b9d20 not in pool! 21:01:46.810 No values available [Fri Jan 2 11:54:41] pmval(7977) Warning: __pmPinPDUBuf: 0x80000004e9b9d20 not in pool! 21:01:47.810 No values available [Fri Jan 2 11:54:41] pmval(7977) Warning: __pmPinPDUBuf: 0x80000004e9b9d20 not in pool! 21:01:48.810 No values available [...] [Fri Jan 2 11:54:41] pmval(7977) Warning: __pmPinPDUBuf: 0x210000004e9d83d0 not in pool! 21:03:45.810 No values available [Fri Jan 2 11:54:41] pmval(7977) Warning: __pmPinPDUBuf: 0x210000004e9d83d0 not in pool! [Fri Jan 2 11:54:41] pmval(7977) Warning: __pmPinPDUBuf: 0x210000004e9d83d0 not in pool! pmval: pdubuf.c:123: __pmPinPDUBuf: Assertion `((__psint_t)handle % sizeof(int)) == 0' failed. [1] 7977 abort (core dumped) % pmval -Dall ... produces much text, then: [Fri Jan 2 11:36:24] pmval(3247) Warning: __pmPinPDUBuf: 0x80000004e9b9d20 not in pool! free pdubuf[size]: 0x13e2000[15360] 0x13de000[15360] 0x13d8000[15360] 0x13d5000[9216] 0x13dc000[5120] pinned pdubuf[size](pincnt): 0x13eb000...0x13eebff[15360](2) 0x13e7000...0x13eabff[15360](1) update+search pmid 60.3.0 inst 2 prior: t=4798.394739 next: t=4858.393740 [1] 3247 segmentation fault (core dumped) -- You are receiving this mail because: You are on the CC list for the bug. --1420217801.f6d481.24382 Date: Fri, 2 Jan 2015 10:56:41 -0600 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8"
Bug ID 1100
Summary libpcp interp.c crash on archive scan
Product pcp
Version unspecified
Hardware All
OS Linux
Status NEW
Severity major
Priority P5
Component pcp
Assignee pcp@kenj.com.au
Reporter fche@redhat.com
CC pcp@oss.sgi.com
Classification Unclassified 

Created attachment 326 [details]
bad (?) archive

% pmval -a *20141125*00009*meta network.interface.in.bytes

crashes thusly:

21:01:43.810  No values available
21:01:44.810  No values available
21:01:45.810  No values available
[Fri Jan  2 11:54:41] pmval(7977) Warning: __pmPinPDUBuf: 0x80000004e9b9d20 not
in pool!
21:01:46.810  No values available
[Fri Jan  2 11:54:41] pmval(7977) Warning: __pmPinPDUBuf: 0x80000004e9b9d20 not
in pool!
21:01:47.810  No values available
[Fri Jan  2 11:54:41] pmval(7977) Warning: __pmPinPDUBuf: 0x80000004e9b9d20 not
in pool!
21:01:48.810  No values available
[...]
[Fri Jan  2 11:54:41] pmval(7977) Warning: __pmPinPDUBuf: 0x210000004e9d83d0
not in pool!
21:03:45.810  No values available
[Fri Jan  2 11:54:41] pmval(7977) Warning: __pmPinPDUBuf: 0x210000004e9d83d0
not in pool!
[Fri Jan  2 11:54:41] pmval(7977) Warning: __pmPinPDUBuf: 0x210000004e9d83d0
not in pool!
pmval: pdubuf.c:123: __pmPinPDUBuf: Assertion `((__psint_t)handle %
sizeof(int)) == 0' failed.
[1]    7977 abort (core dumped)

% pmval -Dall ... 

produces much text, then:

[Fri Jan  2 11:36:24] pmval(3247) Warning: __pmPinPDUBuf: 0x80000004e9b9d20 not
in pool!
   free pdubuf[size]: 0x13e2000[15360] 0x13de000[15360] 0x13d8000[15360]
0x13d5000[9216] 0x13dc000[5120]
   pinned pdubuf[size](pincnt): 0x13eb000...0x13eebff[15360](2)
0x13e7000...0x13eabff[15360](1)
update+search pmid 60.3.0 inst 2 prior: t=4798.394739 <mark> next:
t=4858.393740
[1]    3247 segmentation fault (core dumped)

You are receiving this mail because:
  • You are on the CC list for the bug.
--1420217801.f6d481.24382-- From wwwrun@oss.sgi.com Fri Jan 2 12:28:40 2015 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE,NO_RELAYS autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: by oss.sgi.com (Postfix, from userid 30) id 78EBB7F56; Fri, 2 Jan 2015 12:28:40 -0600 (CST) From: bugzilla-daemon@oss.sgi.com To: pcp@oss.sgi.com Subject: [Bug 1101] New: security bug (information disclosure) in linux-proc pmda - case 1: new kernel Date: Fri, 02 Jan 2015 18:28:40 +0000 X-Bugzilla-Reason: CC AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Classification: Unclassified X-Bugzilla-Product: pcp X-Bugzilla-Component: pcp X-Bugzilla-Keywords: X-Bugzilla-Severity: major X-Bugzilla-Who: fche@redhat.com X-Bugzilla-Status: NEW X-Bugzilla-Priority: P5 X-Bugzilla-Assigned-To: pcp@oss.sgi.com X-Bugzilla-Target-Milestone: --- X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter cc classification Message-ID: Content-Type: multipart/alternative; boundary="1420223320.372DeAC00.30742"; charset="us-ascii" X-Bugzilla-URL: http://oss.sgi.com/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 --1420223320.372DeAC00.30742 Date: Fri, 2 Jan 2015 12:28:40 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" http://oss.sgi.com/bugzilla/show_bug.cgi?id=1101 Bug ID: 1101 Summary: security bug (information disclosure) in linux-proc pmda - case 1: new kernel Product: pcp Version: unspecified Hardware: All OS: Linux Status: NEW Severity: major Priority: P5 Component: pcp Assignee: pcp@oss.sgi.com Reporter: fche@redhat.com CC: pcp@oss.sgi.com Classification: Unclassified A relative to the finding at the tail end of http://oss.sgi.com/pipermail/pcp/2014-November/006062.html , the following variant happens with 3.10.1 relesed code on kernel 3.17.4-301.fc21.x86_64: % sudo service pmcd restart % cat /proc/1/maps cat: /proc/1/maps: Permission denied % pmval -s 1 -i 1 proc.memory.maps [...] "7f714c000000-7f714c029000 rw-p 00000000 00:00 0 7f714c029000-7f7150000000 ---p 00000000 00:00 0 7f7154000000-7f7154029000 rw-p 00000000 00:00 0 7f7154029000-7f7158000000 ---p 00000000 00:00 0 7f715b49c000-7f715b49d000 ---p 00000000 00:00 0 [...] 7f715ea33000-7f715ea34000 rw-p 00000000 00:00 0 7f715ea34000-7f715eb6b000 r-xp 00000000 fd:01 196126 /usr/lib/systemd/systemd 7f715eb6b000-7f715eb88000 r--p 00136000 fd:01 196126 /usr/lib/systemd/systemd 7f715eb88000-7f715eb89000 rw-p 00153000 fd:01 196126 /usr/lib/systemd/systemd [...] An strace of the pmdaproc binary indicates setresgid(-1, 100, -1) = 0 setresuid(-1, 500, -1) = 0 openat(AT_FDCWD, "/proc", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 5 getdents(5, /* 326 entries */, 32768) = 8880 getdents(5, /* 0 entries */, 32768) = 0 close(5) = 0 open("/proc/1/maps", O_RDONLY) = 5 read(5, "7f714c000000-7f714c029000 rw-p 0"..., 1024) = 1024 [...] setresuid(-1, 0, -1) = 0 setresgid(-1, 0, -1) = 0 So in this specific case, it appears to be a kernel check that permits /proc/1/maps to be opened, even with a procpmda effective-[ug]id set. the pmda's temporary-setuid machinery may need to set real*, not just effective*[ug]ids to be portable to this generation of kernels. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. --1420223320.372DeAC00.30742 Date: Fri, 2 Jan 2015 12:28:40 -0600 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8"
Bug ID 1101
Summary security bug (information disclosure) in linux-proc pmda - case 1: new kernel
Product pcp
Version unspecified
Hardware All
OS Linux
Status NEW
Severity major
Priority P5
Component pcp
Assignee pcp@oss.sgi.com
Reporter fche@redhat.com
CC pcp@oss.sgi.com
Classification Unclassified 

A relative to the finding at the tail end of 
http://oss.sgi.com/pipermail/pcp/2014-November/006062.html ,
the following variant happens with 3.10.1 relesed code on
kernel 3.17.4-301.fc21.x86_64:

% sudo service pmcd restart
% cat /proc/1/maps
cat: /proc/1/maps: Permission denied
% pmval -s 1 -i 1 proc.memory.maps
[...]
"7f714c000000-7f714c029000 rw-p 00000000 00:00 0 
7f714c029000-7f7150000000 ---p 00000000 00:00 0 
7f7154000000-7f7154029000 rw-p 00000000 00:00 0 
7f7154029000-7f7158000000 ---p 00000000 00:00 0 
7f715b49c000-7f715b49d000 ---p 00000000 00:00 0 
[...]
7f715ea33000-7f715ea34000 rw-p 00000000 00:00 0 
7f715ea34000-7f715eb6b000 r-xp 00000000 fd:01 196126                    
/usr/lib/systemd/systemd
7f715eb6b000-7f715eb88000 r--p 00136000 fd:01 196126                    
/usr/lib/systemd/systemd
7f715eb88000-7f715eb89000 rw-p 00153000 fd:01 196126                    
/usr/lib/systemd/systemd
[...]


An strace of the pmdaproc binary indicates 

setresgid(-1, 100, -1)                  = 0
setresuid(-1, 500, -1)                  = 0
openat(AT_FDCWD, "/proc", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 5
getdents(5, /* 326 entries */, 32768)   = 8880
getdents(5, /* 0 entries */, 32768)     = 0
close(5)                                = 0
open("/proc/1/maps", O_RDONLY)          = 5
read(5, "7f714c000000-7f714c029000 rw-p 0"..., 1024) = 1024
[...]
setresuid(-1, 0, -1)                    = 0
setresgid(-1, 0, -1)                    = 0

So in this specific case, it appears to be a kernel check that permits
/proc/1/maps to be opened, even with a procpmda effective-[ug]id set.
the pmda's temporary-setuid machinery may need to set real*, not just
effective*[ug]ids to be portable to this generation of kernels.

You are receiving this mail because:
  • You are on the CC list for the bug.
  • You are the assignee for the bug.
--1420223320.372DeAC00.30742-- From wwwrun@oss.sgi.com Fri Jan 2 12:35:03 2015 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE,NO_RELAYS autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: by oss.sgi.com (Postfix, from userid 30) id C8FF77F56; Fri, 2 Jan 2015 12:35:03 -0600 (CST) From: bugzilla-daemon@oss.sgi.com To: pcp@oss.sgi.com Subject: [Bug 1102] New: security bug (information disclosure) in linux-proc pmda - case 2: old kernel Date: Fri, 02 Jan 2015 18:35:03 +0000 X-Bugzilla-Reason: CC AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Classification: Unclassified X-Bugzilla-Product: pcp X-Bugzilla-Component: pcp X-Bugzilla-Keywords: X-Bugzilla-Severity: major X-Bugzilla-Who: fche@redhat.com X-Bugzilla-Status: NEW X-Bugzilla-Priority: P5 X-Bugzilla-Assigned-To: pcp@oss.sgi.com X-Bugzilla-Target-Milestone: --- X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter cc classification Message-ID: Content-Type: multipart/alternative; boundary="1420223703.a21C0.31279"; charset="us-ascii" X-Bugzilla-URL: http://oss.sgi.com/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 --1420223703.a21C0.31279 Date: Fri, 2 Jan 2015 12:35:03 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" http://oss.sgi.com/bugzilla/show_bug.cgi?id=1102 Bug ID: 1102 Summary: security bug (information disclosure) in linux-proc pmda - case 2: old kernel Product: pcp Version: unspecified Hardware: All OS: Linux Status: NEW Severity: major Priority: P5 Component: pcp Assignee: pcp@oss.sgi.com Reporter: fche@redhat.com CC: pcp@oss.sgi.com Classification: Unclassified The finding at the tail end of http://oss.sgi.com/pipermail/pcp/2014-November/006062.html is reproduced thusly on 3.9.10, on a 2.6.18-398.el5 kernel. (3.9.10 happens to be freshest on epel-testing.) (# - run as root; % - run as unprivileged user) # sudo service pmcd restart % cat /proc/1/maps cat: /proc/1/maps: Permission denied % pmval -s 1 -i 1 proc.memory.maps [...] 1 "" [i.e., value not available ... though no error, hm] # service pmcd restart # pmval -s 1 -i 1 proc.memory.maps [...shows results, as expected...] % pmval -s 1 -i 1 proc.memory.maps [...shows results, as unexpected!...] In this case, unlike for bug #1101, the kernel does report a read error (EACCES) for the proc/1/maps file descriptor, so the new client is not permitted to get new data. But... The pmda (due to its "refresh" caches?) goes on to supply stale and confidential data to the unprivileged pcp client. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. --1420223703.a21C0.31279 Date: Fri, 2 Jan 2015 12:35:03 -0600 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8"
Bug ID 1102
Summary security bug (information disclosure) in linux-proc pmda - case 2: old kernel
Product pcp
Version unspecified
Hardware All
OS Linux
Status NEW
Severity major
Priority P5
Component pcp
Assignee pcp@oss.sgi.com
Reporter fche@redhat.com
CC pcp@oss.sgi.com
Classification Unclassified 

The finding at the tail end of 
http://oss.sgi.com/pipermail/pcp/2014-November/006062.html
is reproduced thusly on 3.9.10, on a 2.6.18-398.el5 kernel.
(3.9.10 happens to be freshest on epel-testing.)
(# - run as root;  % - run as unprivileged user)


# sudo service pmcd restart
% cat /proc/1/maps
cat: /proc/1/maps: Permission denied
% pmval -s 1 -i 1 proc.memory.maps
[...]
          1
         ""
[i.e., value not available ... though no error, hm]
# service pmcd restart
# pmval -s 1 -i 1 proc.memory.maps
[...shows results, as expected...]
% pmval -s 1 -i 1 proc.memory.maps
[...shows results, as unexpected!...]


In this case, unlike for bug #1101, the kernel does report
a read error (EACCES) for the proc/1/maps file descriptor,
so the new client is not permitted to get new data.  But...
The pmda (due to its "refresh" caches?) goes on to supply
stale and confidential data to the unprivileged pcp client.

You are receiving this mail because:
  • You are on the CC list for the bug.
  • You are the assignee for the bug.
--1420223703.a21C0.31279-- From wwwrun@oss.sgi.com Fri Jan 2 13:25:50 2015 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE,NO_RELAYS autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: by oss.sgi.com (Postfix, from userid 30) id 2D9DA7F56; Fri, 2 Jan 2015 13:25:50 -0600 (CST) From: bugzilla-daemon@oss.sgi.com To: pcp@oss.sgi.com Subject: [Bug 1101] security bug (information disclosure) in linux-proc pmda - case 1: new kernel Date: Fri, 02 Jan 2015 19:25:50 +0000 X-Bugzilla-Reason: CC AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Classification: Unclassified X-Bugzilla-Product: pcp X-Bugzilla-Component: pcp X-Bugzilla-Keywords: X-Bugzilla-Severity: major X-Bugzilla-Who: fche@redhat.com X-Bugzilla-Status: NEW X-Bugzilla-Priority: P5 X-Bugzilla-Assigned-To: pcp@oss.sgi.com X-Bugzilla-Target-Milestone: --- X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: multipart/alternative; boundary="1420226750.e68F0.1965"; charset="us-ascii" X-Bugzilla-URL: http://oss.sgi.com/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 --1420226750.e68F0.1965 Date: Fri, 2 Jan 2015 13:25:50 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" http://oss.sgi.com/bugzilla/show_bug.cgi?id=1101 --- Comment #1 from Frank Ch. Eigler --- pcpfans.git commit 4cb333ed6a appears to correct this -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. --1420226750.e68F0.1965 Date: Fri, 2 Jan 2015 13:25:50 -0600 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8"

Comment # 1 on bug 1101 from 
pcpfans.git commit 4cb333ed6a appears to correct this

You are receiving this mail because:
  • You are on the CC list for the bug.
  • You are the assignee for the bug.
--1420226750.e68F0.1965-- From minnus@buffalo.edu Fri Jan 2 14:21:29 2015 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 4FCD37F55 for ; Fri, 2 Jan 2015 14:21:29 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id C421FAC001 for ; Fri, 2 Jan 2015 12:21:25 -0800 (PST) X-ASG-Debug-ID: 1420230079-04cb6c057127dcfc0001-S8gJnT Received: from mtareserve1.acsu.buffalo.edu (mtareserve8.acsu.buffalo.edu [128.205.6.19]) by cuda.sgi.com with ESMTP id tk6tli3fuTqqRN5i for ; Fri, 02 Jan 2015 12:21:19 -0800 (PST) X-Barracuda-Envelope-From: minnus@buffalo.edu X-Barracuda-Apparent-Source-IP: 128.205.6.19 Received: from localmailD.acsu.buffalo.edu (localmaild.acsu.buffalo.edu [128.205.5.208]) by mtareserve1.acsu.buffalo.edu (Postfix) with ESMTP id F0CE7156; Fri, 2 Jan 2015 15:21:18 -0500 (EST) Received: from localmailD.acsu.buffalo.edu (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id E65651F591; Fri, 2 Jan 2015 15:21:18 -0500 (EST) Received: from localmailD.acsu.buffalo.edu (localhost [127.0.0.1]) by localmailD.acsu.buffalo.edu (Postfix) with ESMTP id BE4CB1F587; Fri, 2 Jan 2015 15:21:16 -0500 (EST) Received: from smtp.buffalo.edu (smtp3.acsu.buffalo.edu [128.205.5.226]) by localmailD.acsu.buffalo.edu (Prefixe) with ESMTP id AF66D1F586; Fri, 2 Jan 2015 15:21:16 -0500 (EST) Received: from prince.ccr.buffalo.edu (prince.ccr.buffalo.edu [128.205.40.45]) (Authenticated sender: minnus@buffalo.edu) by smtp.buffalo.edu (Postfix) with ESMTPSA id 9EA82490C; Fri, 2 Jan 2015 15:21:16 -0500 (EST) Message-ID: <54A6FDBC.1040001@buffalo.edu> Date: Fri, 02 Jan 2015 15:21:16 -0500 From: Martins Innus User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: Ryan Doyle , "Frank Ch. Eigler" CC: pcp@oss.sgi.com Subject: Re: [pcp] [RFC] Using Vagrant for QA tests References: <1929973019.5471071.1418248733440.JavaMail.zimbra@aconex.com> <2122993129.5476735.1418259486798.JavaMail.zimbra@aconex.com> <1477093403.5527751.1418338412186.JavaMail.zimbra@aconex.com> X-ASG-Orig-Subj: Re: [pcp] [RFC] Using Vagrant for QA tests In-Reply-To: <1477093403.5527751.1418338412186.JavaMail.zimbra@aconex.com> Content-Type: multipart/mixed; boundary="------------080903040603050509040408" X-PM-EL-Spam-Prob: : 8% X-Barracuda-Connect: mtareserve8.acsu.buffalo.edu[128.205.6.19] X-Barracuda-Start-Time: 1420230079 X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.75 X-Barracuda-Spam-Status: No, SCORE=0.75 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_RULE_7580D, HTML_MESSAGE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.13829 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 HTML_MESSAGE BODY: HTML included in message 0.75 BSF_RULE_7580D Custom Rule 7580D This is a multi-part message in MIME format. --------------080903040603050509040408 Content-Type: multipart/alternative; boundary="------------070402090405040204030201" --------------070402090405040204030201 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Ryan, This is great! I had not thought of using vagrant in this way. In order to be able to test my own changes in as many environments as possible, I have started working with your vagrant file and added a bunch of VMs as you mentioned. Vagrantfile attached and work ongoing here: https://github.com/ubccr/pcp/tree/vagrant_test A "vagrant up" will start , provision and run qa on all the hosts. The provisioning happens in serial and the QA runs happen in parallel. I have a script that does the provisioning in parallel, but my host machine is not powerful enough to support that, kept getting disk errors. After the qa is done, any .bad files as well as the QA output itself are copied to a new "qaresults" hierarchy in the current directory. Probably could scripts something to do a simple analysis on the results. If you had a good enough host machine, you could probably run all the VMs at once, but vagrant up will also take a regex, so I tend to do something like: vagrant up /centos.*/ or depending on your shell vagrant up \/centos.\*\/ The boxes are all standard vagrant cloud images except opensuse. I couldn't get that to boot, so had to build my own from the bento project, so those are commented out in the vagrantfile. Some of the QA tests trigger the OOM killer with 512 MB VMs. It's likely that at least 1024 is needed. I haven't looked too much into that yet. You can configure the VM characteristics as well as the QA tests to run with a few variables at the top. Thanks Martins On 12/11/14 5:53 PM, Ryan Doyle wrote: > Hi Frank > > > Neat. One problem with an overly hardcoded approach like > > is that it > > can encourage a monoculture of pcp test environments (a particular > > ubuntu version etc. in this case). > > > > Luckily Vagrant can support multiple VMs. For example we could support > Ubuntu, FreeBSD, OmniOS etc... > > ------ > > config.vm.define "precise64" do |b| > b.vm.box = "precise64" > b.vm.box_url = > "https://cloud-images.ubuntu.com/vagrant/precise/current/precise-server-cloudimg-amd64-vagrant-disk1.box" > end > > config.vm.define "freebsd9" do |b| > b.vm.box = "freebsd9" > b.vm.box_url = > "http://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_freebsd-9.2_chef-provisionerless.box" > end > > config.vm.define "omnios" do |b| > b.vm.box = "omnios" > b.vm.box_url = "http://omnios.omniti.com/media/omnios-latest.box" > end > ----- > > --------------070402090405040204030201 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 8bit
Ryan,
    This is great!  I had not thought of using vagrant in this way. In order to be able to test my own changes in as many environments as possible, I have started working with your vagrant file and added a bunch of VMs as you mentioned.  Vagrantfile attached and work ongoing here:

https://github.com/ubccr/pcp/tree/vagrant_test

A "vagrant up" will start , provision and run qa on all the hosts.  The provisioning happens in serial and the QA runs happen in parallel.  I have a script that does the provisioning in parallel, but my host machine is not powerful enough to support that, kept getting disk errors.

After the qa is done, any .bad files as well as the QA output itself are copied to a new "qaresults" hierarchy in the current directory.  Probably could scripts something to do a simple analysis on the results.

If you had a good enough host machine, you could probably run all the VMs at once, but vagrant up will also take a regex, so I tend to do something like:

vagrant up /centos.*/
or depending on your shell
vagrant up \/centos.\*\/

The boxes are all standard vagrant cloud images except opensuse.  I couldn't get that to boot, so had to build my own from the bento project, so those are commented out in the vagrantfile.

Some of the QA tests trigger the OOM killer with 512 MB VMs.  It's likely that at least 1024 is needed.  I haven't looked too much into that yet.
You can configure the VM characteristics as well as the QA tests to run with a few variables at the top.

Thanks

Martins

On 12/11/14 5:53 PM, Ryan Doyle wrote:
Hi Frank

> Neat.  One problem with an overly hardcoded approach like
> <https://github.com/Aconex/pcp/blob/vagrant-qa/Vagrantfile> is that it
> can encourage a monoculture of pcp test environments (a particular
> ubuntu version etc. in this case). 



Luckily Vagrant can support multiple VMs. For example we could support Ubuntu, FreeBSD, OmniOS etc...

------

  config.vm.define "precise64" do |b| 
    b.vm.box = "precise64"
    b.vm.box_url = "https://cloud-images.ubuntu.com/vagrant/precise/current/precise-server-cloudimg-amd64-vagrant-disk1.box"
  end

  config.vm.define "freebsd9" do |b|
    b.vm.box = "freebsd9"
  end

  config.vm.define "omnios" do |b|
    b.vm.box = "omnios"
  end
-----



--------------070402090405040204030201-- --------------080903040603050509040408 Content-Type: text/plain; charset=UTF-8; x-mac-type="0"; x-mac-creator="0"; name="Vagrantfile" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="Vagrantfile" # -*- mode: ruby -*- # vi: set ft=ruby : # Leave this alone VAGRANTFILE_API_VERSION = "2" # VM Configs VM_MEM = "512" VM_CPU = "1" # QA Flags QA_FLAGS = "" #QA_FLAGS = "022" #QA_FLAGS = "-g pmda.linux" # Arch Specific Config Files ############################################################ # CentOS ############################################################ $script_centos = <