From kenj@internode.on.net Wed Oct 1 01:28:38 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 80D677F76 for ; Wed, 1 Oct 2014 01:28:38 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id F36ECAC00A for ; Tue, 30 Sep 2014 23:28:34 -0700 (PDT) X-ASG-Debug-ID: 1412144908-04cbb073016ab570001-S8gJnT Received: from ipmail07.adl2.internode.on.net (ipmail07.adl2.internode.on.net [150.101.137.131]) by cuda.sgi.com with ESMTP id IwHn3iohbbVMzZfN for ; Tue, 30 Sep 2014 23:28:28 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.131 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApQBAESeK1R20Xt7/2dsb2JhbAANU4Nhg1nGbYh3AYUoVTAGAgUWCwILAwIBAgE/GQYCAQGIR6dseJYVgSyPD4JigVMFliWgb1qCSgEBAQ Received: from ppp118-209-123-123.lns20.mel4.internode.on.net (HELO [192.168.1.100]) ([118.209.123.123]) by ipmail07.adl2.internode.on.net with ESMTP; 01 Oct 2014 15:58:27 +0930 Message-ID: <542B9F09.8000804@internode.on.net> Date: Wed, 01 Oct 2014 16:28:25 +1000 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: pcp updates Content-Type: text/plain; charset=utf-8 X-ASG-Orig-Subj: pcp updates Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail07.adl2.internode.on.net[150.101.137.131] X-Barracuda-Start-Time: 1412144908 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10086 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Nathan, I think it is safe to pull my changes up to and including this batch now. Changes committed to git://git.performancecopilot.org/kenj/pcp.git dev build/tar/postinstall.tail | 41 ++++++++++++++++++++++++++++++++++------- qa/022 | 1 + qa/102 | 3 +++ qa/232 | 4 ++-- qa/994 | 30 +++++++++++++++++++++++++----- qa/admin/myconfigure | 12 ++++++++++-- qa/admin/pcp-daily | 2 +- qa/check | 2 +- qa/common.check | 21 ++++++++++++++------- src/pmdas/freebsd/freebsd.c | 21 +++++++++++++++++---- 10 files changed, 108 insertions(+), 29 deletions(-) commit 1d2ed754679dfcc58048fad1c0e7225118601088 Author: Ken McDonell Date: Wed Oct 1 16:26:35 2014 +1000 QA - FreeBSD changes No changes for non-freebsd platforms, stuff to make -g sanity pass again on FreeBSD. commit 535f84adcced6c2928946a275f95d8ca7ede3ed4 Author: Ken McDonell Date: Wed Oct 1 16:26:05 2014 +1000 qa/102 - more verbose diags to 102.full commit 13ba953e33153e67af438e9f5d17e77e04113190 Author: Ken McDonell Date: Wed Oct 1 16:23:18 2014 +1000 tar packaging - postinstall changes to track ownership/mode changes Try and mimic the debian packaging rules for ownership and permissions and split pmie/pmlogger files into control files below $PCP_SYSCONF_DIR and config.default files below $PCP_VAR_DIR/config. commit 38604dc4c4d96dcc10e249b3c1621f9e4e1dccd1 Author: Ken McDonell Date: Wed Oct 1 16:19:59 2014 +1000 FreeBSD pmda - changes for 32-bit platforms Some kernel variables were incorrectly assumed to be 64-bit long, rather than "long", which broke on 32-bit platforms. commit debd76d63f3f948ce0d8b55bafb5e69e1276cf5e Author: Ken McDonell Date: Tue Sep 30 09:17:59 2014 +1000 qa/check & qa/common.check - be more careful with the config.default for pmlogger Additional ownership and permissions problems to be avoided. commit 70497fe320efe60eb8fe7a85ae68d70381655541 Author: Ken McDonell Date: Tue Sep 30 09:16:59 2014 +1000 qa/022 - duck and weave around more cgroups warnings commit c59c41418c6e42a78dcba3790313c6fa9f231b61 Author: Ken McDonell Date: Tue Sep 30 06:50:14 2014 +1000 qa/admin/pcp-daily - fix initial git checkout Needs --, not -f. From nscott@redhat.com Wed Oct 1 02:19:53 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 51D567F81 for ; Wed, 1 Oct 2014 02:19:53 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 3FA0B8F8035 for ; Wed, 1 Oct 2014 00:19:50 -0700 (PDT) X-ASG-Debug-ID: 1412147986-04cbb073026ad440001-S8gJnT Received: from mx6-phx2.redhat.com (mx6-phx2.redhat.com [209.132.183.39]) by cuda.sgi.com with ESMTP id ApDCgLSduV4snB0C (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 01 Oct 2014 00:19:46 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.39 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx6-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s917Jj8N009774 for ; Wed, 1 Oct 2014 03:19:45 -0400 Date: Wed, 1 Oct 2014 03:19:45 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: pcp@oss.sgi.com Message-ID: <1937764162.59495580.1412147985706.JavaMail.zimbra@redhat.com> In-Reply-To: <625932365.59245206.1412129611488.JavaMail.zimbra@redhat.com> Subject: pcp updates: docs, build (esp. python), qa MIME-Version: 1.0 X-ASG-Orig-Subj: pcp updates: docs, build (esp. python), qa Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: pcp updates: docs, build (esp. python), qa Thread-Index: 2GLHd+9YxqrYfvzxWUkdoIZElE4XZA== X-Barracuda-Connect: mx6-phx2.redhat.com[209.132.183.39] X-Barracuda-Start-Time: 1412147986 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10087 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Changes committed to git://git.pcp.io/pcp.git dev Nathan Scott (6): HTML 4.0 page validation for the tutorial content (5/4) man pages: migrate perl database PMDA man pages to new scheme Revert "Older python support" Revert "Older Python support ... some cases I missed earlier" build: bring consistency to the various configure-enabled PMDAs build: disable builds of python components for pre-2.6 python Marko Myllynen (1): guide.html: add few links to online {iostat,sar}2pcp(1) pages build/rpm/GNUmakefile | 6 build/rpm/pcp.spec.in | 36 - configure | 319 ++++++----- configure.ac | 160 +++-- man/html/guide.html | 20 man/html/howto.cpuperf.html | 4 man/html/howto.diskmodel.html | 4 man/html/howto.diskperf.html | 2 man/html/howto.enterprise.html | 4 man/html/index.html | 4 man/html/installation.html | 6 man/html/lab.auth.html | 4 man/html/lab.pmchart.html | 4 man/html/lab.pmdas.html | 2 man/html/lab.pmie.html | 6 man/html/lab.pmieconf.html | 2 man/html/lab.pmlogger.html | 3 man/html/lab.pmview.html | 2 man/html/lab.secure.html | 20 man/html/pcpintro.html | 8 man/html/pmchart.html | 2 man/html/timecontrol.html | 8 man/man1/pmdadmcache.1 | 64 -- man/man1/pmdagluster.1 | 89 --- man/man1/pmdazswap.1 | 66 -- qa/702 | 3 qa/707 | 3 qa/708 | 3 qa/709 | 9 qa/722 | 4 qa/737 | 4 qa/739 | 3 qa/741 | 3 qa/979 | 3 qa/980 | 3 src/include/builddefs.in | 22 src/include/buildrules | 14 src/pcp/dmcache/.gitignore | 1 src/pcp/dmcache/GNUmakefile | 6 src/pcp/dmcache/pcp-dmcache.py | 154 +++++ src/pcp/dmcache/pcp-dmcache.pyin | 154 ----- src/pcp/free/.gitignore | 1 src/pcp/free/GNUmakefile | 6 src/pcp/free/pcp-free.py | 216 +++++++ src/pcp/free/pcp-free.pyin | 216 ------- src/pcp/numastat/.gitignore | 1 src/pcp/numastat/GNUmakefile | 6 src/pcp/numastat/pcp-numastat.py | 157 +++++ src/pcp/numastat/pcp-numastat.pyin | 157 ----- src/pcp/uptime/.gitignore | 1 src/pcp/uptime/GNUmakefile | 6 src/pcp/uptime/pcp-uptime.py | 128 ++++ src/pcp/uptime/pcp-uptime.pyin | 128 ---- src/pmatop/.gitignore | 1 src/pmatop/GNUmakefile | 13 src/pmatop/pmatop.py | 917 +++++++++++++++++++++++++++++++++ src/pmatop/pmatop.pyin | 917 --------------------------------- src/pmcollectl/GNUmakefile | 2 src/pmdas/dbping/.gitignore | 2 src/pmdas/dbping/GNUmakefile | 18 src/pmdas/dbping/dbprobe.1 | 34 + src/pmdas/dbping/dbprobe.pl | 4 src/pmdas/dbping/pmdadbping.1 | 90 +++ src/pmdas/dbping/pmdadbping.pl | 90 --- src/pmdas/dmcache/.gitignore | 4 src/pmdas/dmcache/GNUmakefile | 9 src/pmdas/dmcache/pmdadmcache.1 | 64 ++ src/pmdas/gluster/.gitignore | 4 src/pmdas/gluster/GNUmakefile | 9 src/pmdas/gluster/pmdagluster.1 | 89 +++ src/pmdas/infiniband/GNUmakefile | 2 src/pmdas/mysql/.gitignore | 1 src/pmdas/mysql/GNUmakefile | 10 src/pmdas/mysql/pmdamysql.1 | 91 +++ src/pmdas/mysql/pmdamysql.pl | 100 --- src/pmdas/papi/GNUmakefile | 5 src/pmdas/postgresql/.gitignore | 1 src/pmdas/postgresql/GNUmakefile | 12 src/pmdas/postgresql/pmdapostgresql.1 | 74 ++ src/pmdas/postgresql/pmdapostgresql.pl | 79 -- src/pmdas/rpm/GNUmakefile | 4 src/pmdas/systemd/GNUmakefile | 2 src/pmdas/zimbra/.gitignore | 1 src/pmdas/zimbra/GNUmakefile | 11 src/pmdas/zimbra/pmdazimbra.1 | 58 ++ src/pmdas/zimbra/pmdazimbra.pl | 55 - src/pmdas/zswap/.gitignore | 4 src/pmdas/zswap/GNUmakefile | 9 src/pmdas/zswap/pmdazswap.1 | 66 ++ src/python/GNUmakefile | 10 src/python/pcp/.gitignore | 2 src/python/pcp/GNUmakefile | 10 src/python/pcp/pmcc.pyin | 620 ---------------------- src/python/pcp/pmsubsys.pyin | 355 ------------ 94 files changed, 2587 insertions(+), 3519 deletions(-) From mgoodwin@redhat.com Wed Oct 1 03:16:52 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: ** X-Spam-Status: No, score=3.0 required=5.0 tests=TVD_SUBJ_NUM_OBFU_MINFP autolearn=no version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 5E0D17F6D for ; Wed, 1 Oct 2014 03:16:52 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id ECD07AC009 for ; Wed, 1 Oct 2014 01:16:48 -0700 (PDT) X-ASG-Debug-ID: 1412151406-04cbb073016ae550001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 31zal6JvhfqyRzVA (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 01 Oct 2014 01:16:47 -0700 (PDT) X-Barracuda-Envelope-From: mgoodwin@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s918Gj8u031662 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Wed, 1 Oct 2014 04:16:46 -0400 Received: from [10.64.49.170] (vpn1-49-170.bne.redhat.com [10.64.49.170]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s918Gieb006296 for ; Wed, 1 Oct 2014 04:16:45 -0400 Message-ID: <542BB86B.1030902@redhat.com> Date: Wed, 01 Oct 2014 18:16:43 +1000 From: Mark Goodwin User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: pcp updates: support for pmiostat on archives converted with collectl2pcp Content-Type: text/plain; charset=ISO-8859-1; format=flowed X-ASG-Orig-Subj: pcp updates: support for pmiostat on archives converted with collectl2pcp Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412151407 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Changes committed to git://git.performancecopilot.org/markgw/pcp/pcp.git dev commit ee0d58dcbd5787ae9b3b9882e57acae114ad0f44 Author: Mark Goodwin Date: Wed Oct 1 17:49:54 2014 +1000 Add support for pmiostat on PCP archives converted from collectl. Add support for disk.dev.{read,write}_rawactive in converted archives, which were missing in the initial implementation. Note that collectl archives do not support any notion of dm persistent name mappings, hence there is no support for disk.dm.* metrics and thus "pmiostat -x dm" still fails (gracefully). commit c0c34756fdff615709cfed05629a368eb1ffba57 Author: Mark Goodwin Date: Wed Oct 1 18:14:09 2014 +1000 Check pmiostat works on archives converted with collectl2pcp. modified: qa/536 modified: qa/536.out From brolley@redhat.com Wed Oct 1 10:44:29 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 42A917F83 for ; Wed, 1 Oct 2014 10:44:29 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 0574F304070 for ; Wed, 1 Oct 2014 08:44:28 -0700 (PDT) X-ASG-Debug-ID: 1412178264-04bdf003a26b50a0001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id C7jsPAm4Nq4dBz7U (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 01 Oct 2014 08:44:24 -0700 (PDT) X-Barracuda-Envelope-From: brolley@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s91FiN28032549 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Wed, 1 Oct 2014 11:44:23 -0400 Received: from [10.15.16.139] (dhcp-10-15-16-139.yyz.redhat.com [10.15.16.139]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s91FiNor003455 for ; Wed, 1 Oct 2014 11:44:23 -0400 Message-ID: <542C21AE.1010504@redhat.com> Date: Wed, 01 Oct 2014 11:45:50 -0400 From: Dave Brolley User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: PCP Mailing List Subject: Multi-Volume Archive + Live Data Playback for PCP Client Tools Content-Type: multipart/alternative; boundary="------------080301000209050605030702" X-ASG-Orig-Subj: Multi-Volume Archive + Live Data Playback for PCP Client Tools X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412178264 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Barracuda-BRTS-Status: 1 X-Virus-Scanned: by bsmtpd at sgi.com This is a multi-part message in MIME format. --------------080301000209050605030702 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi, I've been learning about pmlogger archives, their file formats, the related archive management tools and the pcp clients which support the 'archive' context with the goal of coming up with a design for allowing the extraction of PCP metrics across archive boundaries. I first want to write down what I think I've learned followed by a couple of ideas for how this could be done from the user's point of view as well as from a technical point of view. Of course, I know that others among you have been thinking about this and have much more expertise, especially Ken, so please correct me where I have it wrong and add your own thoughts, ideas and comments! The current situation as I understand it: * PCP archives are created by pmlogger in distinct volumes due to various constraints, such as a maximum file size of 2GB, the desire to allow organization of the collected data, the desire to be able to manage data retention (i.e. log rotation) and, undoubtedly, for other reasons as well. * Some multi-volume support exists in the form of archive folios. These can be created by mkaf(1) but are also created by some other tools, such as pmchart(1) and pmview(1). Archive volumes in a folio may be examined using pmafm(1) using its 'replay', 'repeat' or 'run' commands. The latter two commands allow for repeated application of PCP client tools against one or more archives in the folio. * The archive management tool, pmlogextract and indirectly, pmlogger_daily and pmlogger_merge, provide the ability to extract data from multiple archives and combine that data into a single archive volume. * Otherwise, PCP client tools are currently restricted to extracting metrics from a single archive volume via PM_CONTEXT_ARCHIVE (the -a option). A single archive volume and an option time window is specified, which is applied against that single archive volume. What we would like have is for PCP client tools to have the ability to easily extract metrics from multiple archive volumes. Ultimately, we would also like tail-like following of an active archive volume with seamless transition from archived data to live data. Here are a few ideas for realizing these goals: *Client/tool interface:* Currently only a single archive volume may be specified by its base name (via PM_CONTEXT_ARCHIVE or -a). We could allow the specification of multiple archive specs, each of which could be: * an archive volume file base name -- same as now * the name of a directory containing a collection of PCP archive volumes * wildcarded names which resolve to a list of the above items For example, pminfo -a 20140930.0 -a 201408*.* -a /some/path/archives -a /another/path/archive* PM_CONTEXT_ARCHIVE could be extended to support more than one archive volume. *Extracting multi-volume data:* For PCP tools, one very simple idea for extracting data from multiple existing archive volumes would be to use pmlogextract(1) to consolidate the specified volumes into a single temporary one and then to operate against the temporary archive. Because pmlogextract(1) supports the -S and -T options, a time window which spans archive volumes would automatically be supported. I imagine that this is already done manually in order to consolidate metrics from multiple sources. This would just be a way to automate this process. If we were to implement this within libpcp, then no changes to the client tools would be necessary. PM_CONTEXT_ARCHIVE could do it under the covers, or could use internal logic similar to that used by pmlogextract(1) in order to consolidate the specified archive volumes. *Streaming live data:* While the above could get us very quickly to multi-volume support against existing archive volumes, it may not be helpful in reaching the subsequent goals of live-archive tailing and transitioning from archived to live data. For these, we need some way of streaming new data as it is generated. In order to make the transition from archived to live data, we must be able to identify the following: * When the archive volume we're reading from is live * When it becomes no longer live * What the next live archive volume is (if any), otherwise, which pmcd is the source of the live data. We could try to implement conventions within the archive file system for providing this information via new metadata or some similar mechanism and then write client-side code to handle polling for new data or transitioning to a pmcd once the end of a live archive has been reached. However, there is already a PCP component which knows about and manages all of this information. It is pmlogger(1). pmlogger(1) already knows the location of the archives it is creating, which one is the live one, whether there will be a new live archive once the current one is ended and which pmcd is the source of live data. One way to make all of this available to a client tool in a seamless way would be to allow pmlogger(1) to be a source of metrics in the same way that pmcd is, in addition to its logging function. That is, given a pmlogger(1) instance, a client/tool could connect in the same way that it would to a pmcd instance (call it PM_CONTEXT_LOGGER?). The client/tool could then specify a time window. If the time window reaches into the past then pmlogger(1) would then access the appropriate archive volume(s), as needed to extract the requested metrics up until the specified end time. If no end time is given by the client, or if the end time is in the future, then pmlogger(1) would automatically transition to relaying live data to the client/tool, in addition to logging it, once the end of the current live archive has been reached. If logging were to be terminated, then pmlogger(1) could continue to relay metrics to the client/tool without logging them. PM_CONTEXT_ARCHIVE would not be obsolete, since there may not be a pmlogger(1) instance running and the client may not want archive tailing or live data. For those cases an active pmlogger(1) instance would be necessary in any case. *Implementation:* The retrieval of archived metrics could be done on separate threads within pmlogger(1), one for each connecting client. Relaying of tailed or live data could be done on the main thread. There would be a list of fd's to write the data to, one of which could be the one for the currently active archive log file (if any). One choice to be made would be how to handle the case of a client connecting with no start time. This could either mean "extract metrics from the beginning of the known logs" or it could mean "live data only". I propose having it mean the former and having some special time value which means "now" (perhaps there is already one) which could be used as a start time to indicate "live data only". Similarly there could be a special end time value which means "forever" which could be explicitly used instead of omitting an end time. The above is only a rough sketch of how we could implement multi-volume+live metric playback with little impact on the existing tools. Thoughts, comments, ideas please! Dave --------------080301000209050605030702 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hi,

I've been learning about pmlogger archives, their file formats, the related archive management tools and the pcp clients which support the 'archive' context with the goal of coming up with a design for allowing the extraction of PCP metrics across archive boundaries. I first want to write down what I think I've learned followed by a couple of ideas for how this could be done from the user's point of view as well as from a technical point of view.

Of course, I know that others among you have been thinking about this and have much more expertise, especially Ken, so please correct me where I have it wrong and add your own thoughts, ideas and comments!

The current situation as I understand it:
  • PCP archives are created by pmlogger in distinct volumes due to various constraints, such as a maximum file size of 2GB, the desire to allow organization of the collected data, the desire to be able to manage data retention (i.e. log rotation) and, undoubtedly, for other reasons as well.

  • Some multi-volume support exists in the form of archive folios. These can be created by mkaf(1) but are also created by some other tools, such as pmchart(1) and pmview(1). Archive volumes in a folio may be examined using pmafm(1) using its 'replay', 'repeat' or 'run' commands. The latter two commands allow for repeated application of PCP client tools against one or more archives in the folio.

  • The archive management tool, pmlogextract and indirectly, pmlogger_daily and pmlogger_merge, provide the ability to extract data from multiple archives and combine that data into a single archive volume.

  • Otherwise, PCP client tools are currently restricted to extracting metrics from a single archive volume via PM_CONTEXT_ARCHIVE (the -a option). A single archive volume and an option time window is specified, which is applied against that single archive volume.
What we would like have is for PCP client tools to have the ability to easily extract metrics from multiple archive volumes. Ultimately, we would also like tail-like following of an active archive volume with seamless transition from archived data to live data.

Here are a few ideas for realizing these goals:

Client/tool interface:
Currently only a single archive volume may be specified by its base name (via PM_CONTEXT_ARCHIVE or -a). We could allow the specification of multiple archive specs, each of which could be:
  • an archive volume file base name -- same as now
  • the name of a directory containing a collection of PCP archive volumes
  • wildcarded names which resolve to a list of the above items
For example,

   pminfo -a 20140930.0 -a 201408*.* -a /some/path/archives -a /another/path/archive*

PM_CONTEXT_ARCHIVE could be extended to support more than one archive volume.

Extracting multi-volume data:
For PCP tools, one very simple idea for extracting data from multiple existing archive volumes would be to use pmlogextract(1) to consolidate the specified volumes into a single temporary one and then to operate against the temporary archive. Because pmlogextract(1) supports the -S and -T options, a time window which spans archive volumes would automatically be supported. I imagine that this is already done manually in order to consolidate metrics from multiple sources. This would just be a way to automate this process.

If we were to implement this within libpcp, then no changes to the client tools would be necessary.  PM_CONTEXT_ARCHIVE could do it under the covers, or could use internal logic similar to that used by pmlogextract(1) in order to consolidate the specified archive volumes.

Streaming live data:
While the above could get us very quickly to multi-volume support against existing archive volumes, it may not be helpful in reaching the subsequent goals of live-archive tailing and transitioning from archived to live data. For these, we need some way of streaming new data as it is generated. In order to make the transition from archived to live data, we must be able to identify the following:
  • When the archive volume we're reading from is live
  • When it becomes no longer live
  • What the next live archive volume is (if any), otherwise, which pmcd is the source of the live data.
We could try to implement conventions within the archive file system for providing this information via new metadata or some similar mechanism and then write client-side code to handle polling for new data or transitioning to a pmcd once the end of a live archive has been reached. However, there is already a PCP component which knows about and manages all of this information. It is pmlogger(1). pmlogger(1) already knows the location of the archives it is creating, which one is the live one, whether there will be a new live archive once the current one is ended and which pmcd is the source of live data.

One way to make all of this available to a client tool in a seamless way would be to allow pmlogger(1) to be a source of metrics in the same way that pmcd is, in addition to its logging function. That is, given a pmlogger(1) instance, a client/tool could connect in the same way that it would to a pmcd instance (call it PM_CONTEXT_LOGGER?). The client/tool could then specify a time window. If the time window reaches into the past then pmlogger(1) would then access the appropriate archive volume(s), as needed to extract the requested metrics up until the specified end time. If no end time is given by the client, or if the end time is in the future, then pmlogger(1) would automatically transition to relaying live data to the client/tool, in addition to logging it, once the end of the current live archive has been reached. If logging were to be terminated, then pmlogger(1) could continue to relay metrics to the client/tool without logging them.

PM_CONTEXT_ARCHIVE would not be obsolete, since there may not be a pmlogger(1) instance running and the client may not want archive tailing or live data. For those cases an active pmlogger(1) instance would be necessary in any case.

Implementation:
The retrieval of archived metrics could be done on separate threads within pmlogger(1), one for each connecting client. Relaying of tailed or live data could be done on the main thread. There would be a list of fd's to write the data to, one of which could be the one for the currently active archive log file (if any).

One choice to be made would be how to handle the case of a client connecting with no start time. This could either mean "extract metrics from the beginning of the known logs" or it could mean "live data only". I propose having it mean the former and having some special time value which means "now" (perhaps there is already one) which could be used as a start time to indicate "live data only". Similarly there could be a special end time value which means "forever" which could be explicitly used instead of omitting an end time.

The above is only a rough sketch of how we could implement multi-volume+live metric playback with little impact on the existing tools.

Thoughts, comments, ideas please!
Dave
--------------080301000209050605030702-- From fche@redhat.com Wed Oct 1 13:32:10 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id CB0707F76 for ; Wed, 1 Oct 2014 13:32:10 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id B992F8F8037 for ; Wed, 1 Oct 2014 11:32:10 -0700 (PDT) X-ASG-Debug-ID: 1412188326-04cb6c50e463b1c0001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id l90bqqXcUzuUcQYO (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 01 Oct 2014 11:32:06 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s91IW5WV018348 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Wed, 1 Oct 2014 14:32:06 -0400 Received: from fche.csb (vpn-230-163.phx2.redhat.com [10.3.230.163]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s91IW51W006183; Wed, 1 Oct 2014 14:32:05 -0400 Received: by fche.csb (Postfix, from userid 2569) id 94EAB58548; Wed, 1 Oct 2014 14:32:04 -0400 (EDT) To: Dave Brolley Cc: PCP Mailing List Subject: Re: Multi-Volume Archive + Live Data Playback for PCP Client Tools References: <542C21AE.1010504@redhat.com> X-ASG-Orig-Subj: Re: Multi-Volume Archive + Live Data Playback for PCP Client Tools From: fche@redhat.com (Frank Ch. Eigler) Date: Wed, 01 Oct 2014 14:32:04 -0400 In-Reply-To: <542C21AE.1010504@redhat.com> (Dave Brolley's message of "Wed, 01 Oct 2014 11:45:50 -0400") Message-ID: User-Agent: Gnus/5.1008 (Gnus v5.10.8) Emacs/21.4 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412188326 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Hi, Dave - > I've been learning about pmlogger archives, their file formats, the > related archive management tools and the pcp clients [...] Thanks! > * PCP archives are created by pmlogger in distinct volumes due to various > constraints, such as a maximum file size of 2GB, the desire to allow > organization of the collected data, the desire to be able to manage data > retention (i.e. log rotation) and, undoubtedly, for other reasons as well. Correction: archive *volumes* are just the .0 / .1 / .2 / .3 ... files that logically constitute a single *archive*. These are split only for the 2GB file limit reason (due to the 32-bit size of offsets in the meta/index files). Archives consisting of multiple volumes (.0-.N files) are already handled transparently in libpcp, for both reading and writing purposes. So where you use "multi-volume", you probably (should) mean "multi-archive" in the following. > * Some multi-volume support exists in the form of archive folios. [...] > [...] Folios are for grouping multiple archives together (each of which may have one or more .0-.N volumes). > What we would like have is for PCP client tools to have the ability > to easily extract metrics from multiple archive volumes. Ultimately, > we would also like tail-like following of an active archive volume > with seamless transition from archived data to live data. These are distinct steps in the 'grand unification' process; we can do one at a time. > Here are a few ideas for realizing these goals: > > Client/tool interface: > Currently only a single archive volume may be specified by its base name (via > PM_CONTEXT_ARCHIVE or -a). We could allow the specification of multiple archive > specs, each of which could be: > > * an archive volume file base name -- same as now > * the name of a directory containing a collection of PCP archive volumes > * wildcarded names which resolve to a list of the above items > > For example, > > pminfo -a 20140930.0 -a 201408*.* -a /some/path/archives -a /another/path/ > archive* This is possible, but we may get a long way without requiring extension of the user interface of the pmapi tools (namely, it's undesirable to have to use multiple -a flags, ie. multiple explicit contexts within the PMAPI client code.) Even just pminfo -a 'GLOB*' # note quoting pminfo -a /path/to/directory would be a big step forward, and can be done entirely within libpcp (no modification to pminfo etc.). > PM_CONTEXT_ARCHIVE could be extended to support more than one archive volume. > [...] > For PCP tools, one very simple idea for extracting data from multiple existing > archive volumes would be to use pmlogextract(1) to consolidate the specified > volumes into a single temporary one and then to operate against the temporary > archive. This is possible, but causes a potentially tragic amount I/O. What would be desirable is to extend the libpcp code for PM_CONTEXT_ARCHIVE handling to transparently jump from archive to archive, in much the same way it already jumps from volume to volume, as the current "time" changes. > Streaming live data: > [...] Lots of good ideas in there, but how about we leave this part until later? - FChE From brolley@redhat.com Wed Oct 1 13:56:50 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 6084A7F50 for ; Wed, 1 Oct 2014 13:56:50 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 4EC678F8035 for ; Wed, 1 Oct 2014 11:56:49 -0700 (PDT) X-ASG-Debug-ID: 1412189805-04cbb073046c9670001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id Tqv4Fa1EsgwnhKog (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 01 Oct 2014 11:56:46 -0700 (PDT) X-Barracuda-Envelope-From: brolley@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s91IujSK014847 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Wed, 1 Oct 2014 14:56:45 -0400 Received: from [10.15.16.139] (dhcp-10-15-16-139.yyz.redhat.com [10.15.16.139]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s91IugQG008508; Wed, 1 Oct 2014 14:56:44 -0400 Message-ID: <542C4EC1.4040407@redhat.com> Date: Wed, 01 Oct 2014 14:58:09 -0400 From: Dave Brolley User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: "Frank Ch. Eigler" CC: PCP Mailing List Subject: Re: Multi-Volume Archive + Live Data Playback for PCP Client Tools References: <542C21AE.1010504@redhat.com> X-ASG-Orig-Subj: Re: Multi-Volume Archive + Live Data Playback for PCP Client Tools In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412189805 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On 10/01/2014 02:32 PM, Frank Ch. Eigler wrote: > Thanks! Thanks for the quick feedback! >> * PCP archives are created by pmlogger in distinct volumes due to various >> constraints, such as a maximum file size of 2GB, the desire to allow >> organization of the collected data, the desire to be able to manage data >> retention (i.e. log rotation) and, undoubtedly, for other reasons as well. > Correction: archive *volumes* are just the .0 / .1 / .2 / .3 ... files > that logically constitute a single *archive*. These are split only > for the 2GB file limit reason (due to the 32-bit size of offsets in > the meta/index files). > > Archives consisting of multiple volumes (.0-.N files) are already > handled transparently in libpcp, for both reading and writing > purposes. So where you use "multi-volume", you probably (should) mean > "multi-archive" in the following. Oh, ok. I didn't realize that there was a distinction and I didn't know that libpcp already supported treating a collection of .0 - .N volumes as a single archive (didn't get that deep into the code). So part of what I thought we didn't support is already supported. Great! >> * Some multi-volume support exists in the form of archive folios. [...] >> [...] > Folios are for grouping multiple archives together (each of which may have > one or more .0-.N volumes). Got it. >> What we would like have is for PCP client tools to have the ability >> to easily extract metrics from multiple archive volumes. Ultimately, >> we would also like tail-like following of an active archive volume >> with seamless transition from archived data to live data. > These are distinct steps in the 'grand unification' process; we can do > one at a time. Yes, I not only see them as separate steps, but necessarily separate functionality all together, as can be seen by the way I proposed separate schemes for each. >> Here are a few ideas for realizing these goals: >> >> Client/tool interface: >> Currently only a single archive volume may be specified by its base name (via >> PM_CONTEXT_ARCHIVE or -a). We could allow the specification of multiple archive >> specs, each of which could be: >> >> * an archive volume file base name -- same as now >> * the name of a directory containing a collection of PCP archive volumes >> * wildcarded names which resolve to a list of the above items >> >> For example, >> >> pminfo -a 20140930.0 -a 201408*.* -a /some/path/archives -a /another/path/ >> archive* > This is possible, but we may get a long way without requiring > extension of the user interface of the pmapi tools (namely, it's > undesirable to have to use multiple -a flags, ie. multiple explicit > contexts within the PMAPI client code.) Even just > > pminfo -a 'GLOB*' # note quoting > pminfo -a /path/to/directory > > would be a big step forward, and can be done entirely within libpcp > (no modification to pminfo etc.). > >> PM_CONTEXT_ARCHIVE could be extended to support more than one archive volume. I wasn't actually proposing multiple explicit contexts within pmapi code, but rather extending a single PM_CONTEXT_ARCHIVE to be able to handle more than one archive. It would be done in a way that would allow existing clients to continue to work without changes. Given that clarification, do you still see multiple -a flags as undesirable? >> [...] >> For PCP tools, one very simple idea for extracting data from multiple existing >> archive volumes would be to use pmlogextract(1) to consolidate the specified >> volumes into a single temporary one and then to operate against the temporary >> archive. > This is possible, but causes a potentially tragic amount I/O. What > would be desirable is to extend the libpcp code for PM_CONTEXT_ARCHIVE > handling to transparently jump from archive to archive, in much the > same way it already jumps from volume to volume, as the current "time" > changes. Seems doable. >> Streaming live data: >> [...] > Lots of good ideas in there, but how about we leave this part until > later? Why wait to start developing ideas? Thanks, Dave From fche@redhat.com Wed Oct 1 14:54:34 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 59B987F5A for ; Wed, 1 Oct 2014 14:54:34 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 398FD8F8037 for ; Wed, 1 Oct 2014 12:54:31 -0700 (PDT) X-ASG-Debug-ID: 1412193269-04cb6c50e56401f0001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id DFBaSbEba6SpKNiy (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 01 Oct 2014 12:54:30 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s91JsTuF020484 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Wed, 1 Oct 2014 15:54:29 -0400 Received: from fche.csb (vpn-230-163.phx2.redhat.com [10.3.230.163]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s91JsTTl025320; Wed, 1 Oct 2014 15:54:29 -0400 Received: by fche.csb (Postfix, from userid 2569) id E089B58548; Wed, 1 Oct 2014 15:54:28 -0400 (EDT) To: Dave Brolley Cc: pcp@oss.sgi.com Subject: Re: Multi-Volume Archive + Live Data Playback for PCP Client Tools References: <542C21AE.1010504@redhat.com> <542C4EC1.4040407@redhat.com> X-ASG-Orig-Subj: Re: Multi-Volume Archive + Live Data Playback for PCP Client Tools From: fche@redhat.com (Frank Ch. Eigler) Date: Wed, 01 Oct 2014 15:54:28 -0400 In-Reply-To: <542C4EC1.4040407@redhat.com> (Dave Brolley's message of "Wed, 01 Oct 2014 14:58:09 -0400") Message-ID: User-Agent: Gnus/5.1008 (Gnus v5.10.8) Emacs/21.4 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412193270 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Dave Brolley writes: > [...] >>> For example, >>> pminfo -a 20140930.0 -a 201408*.* -a /some/path/archives -a /another/path/ >>> archive* >> This is possible, but we may get a long way without requiring >> extension of the user interface of the pmapi tools (namely, it's >> undesirable to have to use multiple -a flags [...] > I wasn't actually proposing multiple explicit contexts within pmapi > code, but rather extending a single PM_CONTEXT_ARCHIVE to be able to > handle more than one archive. It would be done in a way that would > allow existing clients to continue to work without changes. We're both on the same page here, but: > Given that clarification, do you still see multiple -a flags as > undesirable? The trick is that each -a/-h type flag normally maps to new individual pcp contexts! pminfo doesn't even accept multiple -a/-h's now. Consider what the pminfo.c code would look like. Would all the -a OPTION strings be concatenated somehow and given to a single rc = pmNewContext(PM_CONTEXT_ARCHIVE,bigString); PMAPI call? >>> Streaming live data: >>> [...] >> Lots of good ideas in there, but how about we leave this part until >> later? > Why wait to start developing ideas? Nothing wrong with developing ideas, even if premature with respect to timing of implementation. I just wouldn't want to get bogged down with detailed discussions, when there's already a specific precursor piece of work ready to bite down on. - FChE From nscott@redhat.com Wed Oct 1 19:01:45 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id A747C7F6B for ; Wed, 1 Oct 2014 19:01:45 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 865BB8F8035 for ; Wed, 1 Oct 2014 17:01:42 -0700 (PDT) X-ASG-Debug-ID: 1412208097-04bdf003a26cf610001-S8gJnT Received: from mx3-phx2.redhat.com (mx3-phx2.redhat.com [209.132.183.24]) by cuda.sgi.com with ESMTP id jVXVGJhpaX6fk4IO (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 01 Oct 2014 17:01:37 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.24 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx3-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s9201YMN029242; Wed, 1 Oct 2014 20:01:34 -0400 Date: Wed, 1 Oct 2014 20:01:34 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Ken McDonell Cc: PCP Message-ID: <675135471.60035316.1412208094589.JavaMail.zimbra@redhat.com> In-Reply-To: <900014462.60034609.1412207921331.JavaMail.zimbra@redhat.com> Subject: QA leaving dbus-daemon processes running MIME-Version: 1.0 X-ASG-Orig-Subj: QA leaving dbus-daemon processes running Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: QA leaving dbus-daemon processes running Thread-Index: yEq9UvQvGOJh8nvSxK9OB2HCbFPu1w== X-Barracuda-Connect: mx3-phx2.redhat.com[209.132.183.24] X-Barracuda-Start-Time: 1412208097 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10109 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Hi Ken, Noticing here that QA has started leaving quite a few dbus-daemon processes running at the end of each run. I guess this is from the common.qt changes that trigger dbus-launch if its found? I can't seem to find a way to make those processes not persist beyond the end of each qt test (other than a big killall kind of hammer, but that'd be dangerous for any invocations of pcp/qa as the desktop user I guess?). cheers. -- Nathan From pcp-announce-bounces@oss.sgi.com Wed Oct 1 21:31:02 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=RP_MATCHES_RCVD autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from oss.sgi.com (localhost [IPv6:::1]) by oss.sgi.com (Postfix) with ESMTP id 677FD7F6D; Wed, 1 Oct 2014 21:31:02 -0500 (CDT) X-Original-To: pcp-announce@oss.sgi.com Delivered-To: pcp-announce@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id B9B767F6B for ; Wed, 1 Oct 2014 21:31:00 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id A7EE08F8040 for ; Wed, 1 Oct 2014 19:31:00 -0700 (PDT) X-ASG-Debug-ID: 1412217054-04cbb073046dbd60001-87ZIJf Received: from mx6-phx2.redhat.com (mx6-phx2.redhat.com [209.132.183.39]) by cuda.sgi.com with ESMTP id AQps9wKF7umAgibg (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 01 Oct 2014 19:30:55 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.39 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx6-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s922UsQW015699 for ; Wed, 1 Oct 2014 22:30:54 -0400 Date: Wed, 1 Oct 2014 22:30:54 -0400 (EDT) From: Nathan Scott To: pcp-announce Message-ID: <1968683202.60097085.1412217054636.JavaMail.zimbra@redhat.com> In-Reply-To: <901130688.60042744.1412210950889.JavaMail.zimbra@redhat.com> MIME-Version: 1.0 X-ASG-Orig-Subj: PCP developers conference call X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: PCP developers conference call Thread-Index: is+tBAM7Gw61JXdO4KtrNRZTS9utJQ== X-Barracuda-Connect: mx6-phx2.redhat.com[209.132.183.39] X-Barracuda-Start-Time: 1412217055 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10114 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Subject: [pcp-announce] PCP developers conference call X-BeenThere: pcp-announce@oss.sgi.com X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Nathan Scott List-Id: Performance Co-Pilot announcements List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pcp-announce-bounces@oss.sgi.com Sender: pcp-announce-bounces@oss.sgi.com Hi all, It's time for our quarterly developers call - please join us! For those new to the game, we have these calls approximately once every 3 months - they are open to anyone who has even a passing interest in PCP development. Please let me know if you'll be on the call (we will wait a bit if we are expecting more people) and if there are any specific topics you would like to discuss. An initial draft agenda follows, below. The usual aftermath is a collation of any note-keeping effort anyone has made, and summaries emailed to the pcp list. I'll attempt to record the call once more and forward instructions for replaying it for those who can't make it in person (these recordings last a week or two, then are deleted, AIUI). Details for the call: - Starting at 7am on Thursday 9th October in Melbourne, track your local start date/time here: http://www.timeanddate.com/worldclock/fixedtime.html?msg=PCP+Developers+Meeting&iso=20141009T07&p1=152&ah=2 - The phone number to call from your location is here: https://www.intercallonline.com/listNumbersByCode.action?confCode=6409801839 - When prompted, enter the conference code: 6409801839 Initial agenda: - welcome - any general news - quality assurance - status [kenj, nathans] - pcp culture and zen of QA [nathans] - non-intrusive QA testing for pcp [nathans, fche, kenj] - source trees - code encumbrance concerns, a history [nathans] - issues with separate trees, pcp web status [fche, nathans] - WIP / future work - local-context pmlogger [mgoodwin] - multi-archive libpcp support [brolley] - generic json pmda, python pmda module work [dsmith?, fche] - smaller packages for containers [nathans] - simplifying PMAPI access (new APIs, caching) [fche, kenj?] - jvm instrumentation planning [nathans] - ... any other hacking in progress? - anyone need help with anything? - any other topics? cheers. -- Nathan _______________________________________________ pcp-announce mailing list pcp-announce@oss.sgi.com http://oss.sgi.com/mailman/listinfo/pcp-announce From nscott@redhat.com Wed Oct 1 21:37:13 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 44B767F55 for ; Wed, 1 Oct 2014 21:37:13 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 2294830406A for ; Wed, 1 Oct 2014 19:37:10 -0700 (PDT) X-ASG-Debug-ID: 1412217422-04cb6c50e564ff90001-S8gJnT Received: from mx5-phx2.redhat.com (mx5-phx2.redhat.com [209.132.183.37]) by cuda.sgi.com with ESMTP id i8FlTjDwZCgEOQRX (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 01 Oct 2014 19:37:03 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.37 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx5-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s922b2Ss005499 for ; Wed, 1 Oct 2014 22:37:02 -0400 Date: Wed, 1 Oct 2014 22:37:02 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: PCP Message-ID: <1688183028.60098659.1412217422719.JavaMail.zimbra@redhat.com> In-Reply-To: <595718694.60098550.1412217346036.JavaMail.zimbra@redhat.com> Subject: pcp updates: merges, qa MIME-Version: 1.0 X-ASG-Orig-Subj: pcp updates: merges, qa Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: pcp updates: merges, qa Thread-Index: boUq0xsVPhcLecqJWzXbT3ZDJKp2qA== X-Barracuda-Connect: mx5-phx2.redhat.com[209.132.183.37] X-Barracuda-Start-Time: 1412217423 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10114 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Changes committed to git://git.pcp.io/pcp.git dev Ken McDonell (26): qa/365 - minor fiddle Linux proc pmda - pmda.c - smack more PM_ERR_INST uses Linux proc pmda - proc_pid.c - parser rework qa/744 - a little assistance for sudo and newhelp pmstat - add pmlogger config as per man page qa - run 994 to check-n-repair permissions and ownership after each test man pages - rework pmie and pmlogger control and config files pmie - rework control and config files pmlogger - rework control and config files pmchart, pmview, hotproc pmda - cosmetic makefile changes qa/src/permslist - rework pmie and pmlogger control and config files debian packaging - rework pmie and pmlogger control and config files pmstat - add pmlogger config file qa/999 - add new expected error for proc.psinfo.cgroups qa/admin/allow-pmlc-access & qa/admin/check-vm - pmlogger config.default moved rpm/pcp.spec.in - tweak the save config logic qa - heaps - config file changes qa/1009 - dodge pmconfig -L curve ball qa/1009 and qa/1023 - dodge spaces in pmconfig -L output qa/admin/pcp-daily - fix initial git checkout qa/022 - duck and weave around more cgroups warnings qa/check & qa/common.check - be more careful with the config.default for pmlogger FreeBSD pmda - changes for 32-bit platforms tar packaging - postinstall changes to track ownership/mode changes qa/102 - more verbose diags to 102.full QA - FreeBSD changes Nathan Scott (4): build: disable remaining python components for pre-2.6 python qa: extend the notrun check for test qa/536 qa: fix some typos in qa/admin docs/scripts build: reinstate python code missing from git-revert action Mark Goodwin (2): Add support for pmiostat on PCP archives converted from collectl. Check pmiostat works on archives converted with collectl2pcp. build/rpm/pcp.spec.in | 2 build/tar/postinstall.tail | 41 ++ debian/pcp.postinst.tail | 40 +- man/html/pcpintro.html | 2 man/man1/pmafm.1 | 6 man/man1/pmie.1 | 12 man/man1/pmie_check.1 | 2 man/man1/pmlogger.1 | 16 man/man1/pmlogger_check.1 | 2 man/man1/pmnewlog.1 | 6 man/man1/pmstat.1 | 2 qa/022 | 2 qa/023 | 16 qa/066 | 17 - qa/067 | 11 qa/1009 | 7 qa/102 | 3 qa/1023 | 2 qa/110 | 16 qa/115 | 10 qa/159 | 8 qa/169 | 6 qa/170 | 4 qa/199 | 21 - qa/232 | 4 qa/241 | 15 qa/279 | 5 qa/296 | 5 qa/324 | 6 qa/326 | 6 qa/340 | 18 - qa/346 | 6 qa/347 | 6 qa/348 | 6 qa/349 | 6 qa/365 | 10 qa/365.out.ipv6 | 8 qa/365.out.nonipv6 | 8 qa/398 | 4 qa/427 | 12 qa/445 | 13 qa/455 | 6 qa/510 | 10 qa/536 | 4 qa/536.out | 8 qa/565 | 6 qa/578 | 17 - qa/587 | 5 qa/642 | 4 qa/643 | 4 qa/647 | 6 qa/648 | 16 qa/649 | 7 qa/660 | 7 qa/704 | 13 qa/710 | 3 qa/717 | 4 qa/725 | 18 - qa/729 | 3 qa/742 | 2 qa/743 | 2 qa/744 | 2 qa/823 | 11 qa/829 | 2 qa/832 | 16 qa/841 | 3 qa/842 | 2 qa/919 | 37 ++ qa/919.out | 8 qa/973 | 4 qa/986 | 14 qa/991 | 2 qa/994 | 51 ++- qa/999 | 3 qa/admin/README | 4 qa/admin/allow-pmlc-access | 14 qa/admin/check-vm | 10 qa/admin/myconfigure | 12 qa/admin/pcp-daily | 2 qa/check | 30 + qa/common | 1 qa/common.check | 86 +++-- qa/common.pcpweb | 7 qa/common.secure | 12 qa/group | 1 qa/src/permslist | 7 src/collectl2pcp/disk.c | 4 src/pmchart/views/GNUmakefile | 5 src/pmdas/freebsd/freebsd.c | 21 + src/pmdas/hotproc/GNUakefile | 14 src/pmdas/linux_proc/pmda.c | 42 +- src/pmdas/linux_proc/proc_pid.c | 226 ++++++++++--- src/pmie/GNUmakefile | 7 src/pmie/control | 2 src/pmie/pmie_check.sh | 2 src/pmie/src/pmie.c | 4 src/pmiostat/GNUmakefile | 3 src/pmlogger/GNUmakefile | 7 src/pmlogger/pmlogger_check.sh | 2 src/pmlogger/pmnewlog.sh | 4 src/pmlogger/src/pmlogger.c | 8 src/pmstat/GNUmakefile | 2 src/pmstat/pmstat.pmlogger | 24 + src/pmview/front-ends/GNUmakefile | 5 src/python/pcp/pmcc.py | 620 ++++++++++++++++++++++++++++++++++++++ src/python/pcp/pmsubsys.py | 355 +++++++++++++++++++++ 106 files changed, 1796 insertions(+), 439 deletions(-) From nscott@redhat.com Wed Oct 1 23:22:15 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id F3B0B7F50 for ; Wed, 1 Oct 2014 23:22:14 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id E1C3E8F8035 for ; Wed, 1 Oct 2014 21:22:11 -0700 (PDT) X-ASG-Debug-ID: 1412223729-04cb6c50e6653620001-S8gJnT Received: from mx4-phx2.redhat.com (mx4-phx2.redhat.com [209.132.183.25]) by cuda.sgi.com with ESMTP id 1TDrkrf1QmQ9ktYc (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 01 Oct 2014 21:22:10 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.25 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx4-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s924M9eT002984; Thu, 2 Oct 2014 00:22:09 -0400 Date: Thu, 2 Oct 2014 00:22:08 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Dave Brolley Cc: "Frank Ch. Eigler" , PCP Mailing List Message-ID: <408521091.60123629.1412223728964.JavaMail.zimbra@redhat.com> In-Reply-To: <542C4EC1.4040407@redhat.com> References: <542C21AE.1010504@redhat.com> <542C4EC1.4040407@redhat.com> Subject: Re: [pcp] Multi-Volume Archive + Live Data Playback for PCP Client Tools MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] Multi-Volume Archive + Live Data Playback for PCP Client Tools Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: Multi-Volume Archive + Live Data Playback for PCP Client Tools Thread-Index: EXzoE+970GjzzjVlo4qdrpMfQkN2YA== X-Barracuda-Connect: mx4-phx2.redhat.com[209.132.183.25] X-Barracuda-Start-Time: 1412223730 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.03 X-Barracuda-Spam-Status: No, SCORE=0.03 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_SA_TO_FROM_DOMAIN_MATCH, THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10116 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... 0.01 BSF_SC0_SA_TO_FROM_DOMAIN_MATCH Sender Domain Matches Recipient Domain Hi guys, ----- Original Message ----- > [...] > Oh, ok. I didn't realize that there was a distinction and I didn't know > that libpcp already supported treating a collection of .0 - .N volumes > as a single archive (didn't get that deep into the code). So part of > what I thought we didn't support is already supported. Great! :) > >> PM_CONTEXT_ARCHIVE could be extended to support more than one archive > >> volume. > I wasn't actually proposing multiple explicit contexts within pmapi > code, but rather extending a single PM_CONTEXT_ARCHIVE to be able to > handle more than one archive. The difficulty there is that pmNewContext(3) can take one path only via its "name" parameter (hence Franks suggestions about quoted globbing and using sub-directories - as these identify multiple archives via a single string). > > This is possible, but causes a potentially tragic amount I/O. What > > would be desirable is to extend the libpcp code for PM_CONTEXT_ARCHIVE > > handling to transparently jump from archive to archive, in much the > > same way it already jumps from volume to volume, as the current "time" > > changes. > Seems doable. The volume-skipping analogy is a good one I think. However, some new kinds of problems arrive - such as having to deal with differences in the metadata between archives, which cannot happen between volumes (as they share a .meta file - e.g., metric pmDesc structures might change in the new world order, name<->PMID mappings, hostnames - these things do not happen in the multi-volume-archive case). Other tricky aspects are the need/desire to have a single __pmContext structure but multiple __pmArchCtl structures ... but this is getting fairly low level, you'll have to ponder that aspect deeply soon enough I guess. You'll also want to think about the data structures you use to represent all the archives, and the timespans they cover. Think about how pmchart has a need to quickly position to anywhere within the time series and start replaying near-immediately (any tool that supports -S/-O uses that actually). That is done using the temporal index, today, of which there is one per archive. There is no across-archives temporal index, yet, of course. So I guess pmNewContext will need to first identify all of the available archives, build in-memory structures for each, and then build some extra data structure that allows quick identification of archives that are near to the given time position (see pmSetMode(3) for the PMAPI interface). So, heh, have no doubt - its a really difficult series of problems that you're tackling here, even just limiting things to archives-only. > >> Streaming live data: > >> [...] I didn't follow the need to talk to pmlogger - what was the rationale there? Why not treat all archives as potentially live? If we don't, it precludes archive updates being written via distributed filesystems (pmlogger is not necessarily local to the host doing archive replay). With Kens pmlogger changes from a few months back, we should be in much better shape there in terms of no longer observing partially-consistent results in libpcp. The auto-switch-to-live-mode you're talking about would be great (later) - that is one of the cornerstones of the original unified context mode RFC (so, that, and this ability to switch between archives that you're looking into pretty much are "unified context" mode). The idea there is to "stack" __pmContext structures (which can be live and/or archive) and have a new __pmUnifyCtl structure (or some good name) - analogous to the __pm{Arch,Host}Ctl structures anyway - that sits atop the stack and holds current positions for sub-contexts. It might be useful for you to think about stacking here too - so, having sub-contexts for each individual archive, and one "special" __pmContext structure at the top of the stack, providing the temporal indexing into the others. The PMAPI clients do not need to know that the top of the stack is "special", for the most part they access the context via handle only (in this scheme, pmNewContext would return a handle identifying the top of the stack, and from there we cascade down to individual archives within libpcp). Also, be aware that __pmContext, __pmArchCtl, __pmLogCtl and friends are part of the ABI, so we need to be sensitive to any changes in these data structures (extending at the end only, etc). > > Lots of good ideas in there, but how about we leave this part until > > later? > Why wait to start developing ideas? Don't wait, and don't limit your thinking - please continue to put all your ideas out here. Otherwise we'll be restricting our (collective) thinking to a fixed, limited implementation and may not see the bigger picture / looming problems in the next stage(s). Can't wait to see progress though - this is going to be a fantastic step forward! cheers. -- Nathan From nscott@redhat.com Thu Oct 2 02:10:20 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 61B427F60 for ; Thu, 2 Oct 2014 02:10:20 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id F1160AC010 for ; Thu, 2 Oct 2014 00:10:19 -0700 (PDT) X-ASG-Debug-ID: 1412233814-04cb6c50e4657f40001-S8gJnT Received: from mx6-phx2.redhat.com (mx6-phx2.redhat.com [209.132.183.39]) by cuda.sgi.com with ESMTP id XXH3m59UDVZ7OfmJ (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 02 Oct 2014 00:10:15 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.39 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx6-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s927ACR7031377; Thu, 2 Oct 2014 03:10:12 -0400 Date: Thu, 2 Oct 2014 03:10:12 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: "Wu, Liming" Cc: PCP Mailing List Message-ID: <828890275.60182252.1412233812693.JavaMail.zimbra@redhat.com> In-Reply-To: <1856213083.60179703.1412233368793.JavaMail.zimbra@redhat.com> Subject: =?utf-8?Q?Re:_[pcp]_=E7=AD=94=E5=A4=8D:_pcp_Digest,_Vol_74,_Issue_57?= MIME-Version: 1.0 X-ASG-Orig-Subj: =?utf-8?Q?Re:_[pcp]_=E7=AD=94=E5=A4=8D:_pcp_Digest,_Vol_74,_Issue_57?= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: pcp Digest, Vol 74, Issue 57 Thread-Index: +ydE2i4vkDc8cHLE8eoTXTSl4DeM6w== X-Barracuda-Connect: mx6-phx2.redhat.com[209.132.183.39] X-Barracuda-Start-Time: 1412233815 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_MV0113c, THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10120 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... 0.00 BSF_SC0_MV0113c BSF_SC0_MV0113c 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header Hi there, On Fri Sep 26, 2014 "Wu, Liming" wrote: > Excuse me=E3=80=82I want to ask a question=E3=80=82 > If I want to send patches to PCP, how can I do it ? >=20 > wulm Sorry about the delayed reply - I think your earlier mail was not delivered to me (possibly tagged as spam). Feel free to send PCP patches to the list (pcp@oss.sgi.com) at any time. If you have a git tree with your changes committed, that is great too - just send the git URL and someone will get back to you (CC to me if you like, and I'll make sure). cheers. -- Nathan From dsmith@redhat.com Thu Oct 2 09:50:24 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id A45E47F6A for ; Thu, 2 Oct 2014 09:50:24 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 930FC304071 for ; Thu, 2 Oct 2014 07:50:21 -0700 (PDT) X-ASG-Debug-ID: 1412261416-04cbb073046f4930001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 1Szyuw3esC8Bw4UC (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 02 Oct 2014 07:50:17 -0700 (PDT) X-Barracuda-Envelope-From: dsmith@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s92EoGkA012049 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Thu, 2 Oct 2014 10:50:16 -0400 Received: from t540p.usersys.redhat.com (vpn-60-245.rdu2.redhat.com [10.10.60.245]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s92EoEZh026713 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Thu, 2 Oct 2014 10:50:16 -0400 Message-ID: <542D6626.50703@redhat.com> Date: Thu, 02 Oct 2014 09:50:14 -0500 From: David Smith User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.1 MIME-Version: 1.0 To: pcp Subject: [PATCH] Fix memory corruption in python support Content-Type: text/plain; charset=utf-8 X-ASG-Orig-Subj: [PATCH] Fix memory corruption in python support Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412261417 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Here's a patch that fixes some memory corruption I've seen in pcp's python support. Here's a note from the python documentation (): ==== Note that any Python object references which are provided to the caller [of PyArg_ParseTuple(), PyArg_ParseTupleAndKeywords(), or PyArg_Parse()] are borrowed references; do not decrement their reference count! ==== Unfortunately, pcp's python support was decrementing reference counts in some cases. I went through and audited the calls to PyArg_ParseTuple(), PyArg_ParseTupleAndKeywords(), and PyArg_Parse() and believe I've fixed the reference count issues. This patch fixed the memory corruption I was seeing. --- diff --git a/src/python/pmapi.c b/src/python/pmapi.c index 6886214..13a6dcd 100644 --- a/src/python/pmapi.c +++ b/src/python/pmapi.c @@ -682,6 +682,9 @@ getOptionsFromList(PyObject *self, PyObject *args, PyObject *keywords) PyObject *pyargv = NULL; char *keyword_list[] = {"argv", NULL}; + // Note that PyArg_ParseTupleAndKeywords() returns a "borrowed" + // reference, so there is no need to decrement a reference to + // pyargv. if (!PyArg_ParseTupleAndKeywords(args, keywords, "O:pmGetOptionsFromList", keyword_list, &pyargv)) return NULL; @@ -691,17 +694,14 @@ getOptionsFromList(PyObject *self, PyObject *args, PyObject *keywords) if (!PyList_Check(pyargv)) { PyErr_SetString(PyExc_TypeError, "pmGetOptionsFromList uses a list"); - Py_DECREF(pyargv); return NULL; } if ((argc = PyList_GET_SIZE(pyargv)) <= 0) { - Py_DECREF(pyargv); return Py_BuildValue("i", 0); } if ((argv = malloc(argc * sizeof(char *))) == NULL) { - Py_DECREF(pyargv); return PyErr_NoMemory(); } @@ -718,7 +718,6 @@ getOptionsFromList(PyObject *self, PyObject *args, PyObject *keywords) */ if (i == 0 && (string = strdup(string)) == NULL) { free(argv); - Py_DECREF(pyargv); return PyErr_NoMemory(); } argv[i] = string; diff --git a/src/python/pmda.c b/src/python/pmda.c index 73e3d92..f0c4845 100644 --- a/src/python/pmda.c +++ b/src/python/pmda.c @@ -124,6 +124,11 @@ namespace_refresh(PyObject *self, PyObject *args, PyObject *keywords) "O:namespace_refresh", keyword_list, &pmns_dict)) return NULL; if (pmns_dict) { + // PyArg_ParseTupleAndKeywords() returns a "borrowed" + // reference. Since we're going to keep this object around for + // use later, increase its reference count. + Py_INCREF(pmns_dict); + if (!PyDict_Check(pmns_dict)) { __pmNotifyErr(LOG_ERR, "attempted to refresh namespace with non-dict type"); @@ -153,6 +158,11 @@ pmid_oneline_refresh(PyObject *self, PyObject *args, PyObject *keywords) return NULL; if (pmid_oneline_dict) { + // PyArg_ParseTupleAndKeywords() returns a "borrowed" + // reference. Since we're going to keep this object around for + // use later, increase its reference count. + Py_INCREF(pmid_oneline_dict); + if (!PyDict_Check(pmid_oneline_dict)) { __pmNotifyErr(LOG_ERR, "attempted to refresh pmid oneline help with non-dict type"); @@ -180,6 +190,11 @@ pmid_longtext_refresh(PyObject *self, PyObject *args, PyObject *keywords) return NULL; if (pmid_longtext_dict) { + // PyArg_ParseTupleAndKeywords() returns a "borrowed" + // reference. Since we're going to keep this object around for + // use later, increase its reference count. + Py_INCREF(pmid_longtext_dict); + if (!PyDict_Check(pmid_longtext_dict)) { __pmNotifyErr(LOG_ERR, "attempted to refresh pmid long help with non-dict type"); @@ -207,6 +222,11 @@ indom_oneline_refresh(PyObject *self, PyObject *args, PyObject *keywords) return NULL; if (indom_oneline_dict) { + // PyArg_ParseTupleAndKeywords() returns a "borrowed" + // reference. Since we're going to keep this object around for + // use later, increase its reference count. + Py_INCREF(indom_oneline_dict); + if (!PyDict_Check(indom_oneline_dict)) { __pmNotifyErr(LOG_ERR, "attempted to refresh indom oneline help with non-dict type"); @@ -234,6 +254,11 @@ indom_longtext_refresh(PyObject *self, PyObject *args, PyObject *keywords) return NULL; if (indom_longtext_dict) { + // PyArg_ParseTupleAndKeywords() returns a "borrowed" + // reference. Since we're going to keep this object around for + // use later, increase its reference count. + Py_INCREF(indom_longtext_dict); + if (!PyDict_Check(indom_longtext_dict)) { __pmNotifyErr(LOG_ERR, "attempted to refresh indom long help with non-dict type"); --- -- David Smith dsmith@redhat.com Red Hat http://www.redhat.com 256.217.0141 (direct) 256.837.0057 (fax) From fche@redhat.com Thu Oct 2 09:52:58 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 89A1C7F6A for ; Thu, 2 Oct 2014 09:52:58 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 1BA8EAC013 for ; Thu, 2 Oct 2014 07:52:54 -0700 (PDT) X-ASG-Debug-ID: 1412261573-04cb6c50e7668a70001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id aLnw2KXkdCcXpgrH (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 02 Oct 2014 07:52:53 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s92Eqqi2025224 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Thu, 2 Oct 2014 10:52:52 -0400 Received: from fche.csb (vpn-230-163.phx2.redhat.com [10.3.230.163]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s92EqpZ0003019; Thu, 2 Oct 2014 10:52:51 -0400 Received: by fche.csb (Postfix, from userid 2569) id 2BEB058548; Thu, 2 Oct 2014 10:52:51 -0400 (EDT) To: Nathan Scott Cc: Dave Brolley , pcp@oss.sgi.com Subject: Re: Multi-Volume Archive + Live Data Playback for PCP Client Tools References: <542C21AE.1010504@redhat.com> <542C4EC1.4040407@redhat.com> <408521091.60123629.1412223728964.JavaMail.zimbra@redhat.com> X-ASG-Orig-Subj: Re: Multi-Volume Archive + Live Data Playback for PCP Client Tools From: fche@redhat.com (Frank Ch. Eigler) Date: Thu, 02 Oct 2014 10:52:51 -0400 In-Reply-To: <408521091.60123629.1412223728964.JavaMail.zimbra@redhat.com> (Nathan Scott's message of "Thu, 2 Oct 2014 00:22:08 -0400 (EDT)") Message-ID: User-Agent: Gnus/5.1008 (Gnus v5.10.8) Emacs/21.4 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412261573 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 nathans wrote: > [...] > The volume-skipping analogy is a good one I think. However, some new > kinds of problems arrive - such as having to deal with differences in > the metadata between archives, which cannot happen between volumes (as > they share a .meta file - e.g., metric pmDesc structures might change > in the new world order, name<->PMID mappings, hostnames - these things > do not happen in the multi-volume-archive case). Indeed. Part of this work would be to hide that effect by making those name/pmid/indom mappings stable across multiple archives. > You'll also want to think about the data structures you use to represent > all the archives, and the timespans they cover. [...] "All problems in computer science can be solved by another level of indirection ..." >> >> Streaming live data: >> >> [...] > > I didn't follow the need to talk to pmlogger - what was the rationale > there? In the grand unified world, there will be a need to talk to -something- that has both historical data and live data. Making pmlogger into a network server would make it possible to have libpcp engage in conversation with only a single thing. pmlogger already knows about its own archives as well as the pmcd it's pulling from. No coincidences or client-side magic required, so to speak. > Why not treat all archives as potentially live? [...] In previous context of multi-archive virtual-gluing, it could make the cross-archive indexing structures uncomfortably dynamic. > Also, be aware that __pmContext, __pmArchCtl, __pmLogCtl and friends > are part of the ABI, so we need to be sensitive to any changes in > these data structures (extending at the end only, etc). One way around the leakage of these structs into the ABI is to create new structures and a new shadow-API, all under a separate symbol-versioning epoch. (Or one can bump the SONAME, and condemn prior ABI-user binaries to a compat-pcp-libs package from a museum.) - FChE From brolley@redhat.com Thu Oct 2 10:53:56 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 2E3567F61 for ; Thu, 2 Oct 2014 10:53:56 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 0DC6B304082 for ; Thu, 2 Oct 2014 08:53:53 -0700 (PDT) X-ASG-Debug-ID: 1412265231-04cb6c50e766c240001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id w76zWAKvHLIoybwM (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 02 Oct 2014 08:53:51 -0700 (PDT) X-Barracuda-Envelope-From: brolley@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s92Fro10012022 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Thu, 2 Oct 2014 11:53:50 -0400 Received: from [10.15.16.139] (dhcp-10-15-16-139.yyz.redhat.com [10.15.16.139]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s92FroAd008347; Thu, 2 Oct 2014 11:53:50 -0400 Message-ID: <542D756B.8030906@redhat.com> Date: Thu, 02 Oct 2014 11:55:23 -0400 From: Dave Brolley User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: "Frank Ch. Eigler" , Nathan Scott CC: pcp@oss.sgi.com Subject: Re: Multi-Volume Archive + Live Data Playback for PCP Client Tools References: <542C21AE.1010504@redhat.com> <542C4EC1.4040407@redhat.com> <408521091.60123629.1412223728964.JavaMail.zimbra@redhat.com> X-ASG-Orig-Subj: Re: Multi-Volume Archive + Live Data Playback for PCP Client Tools In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412265231 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On 10/02/2014 10:52 AM, Frank Ch. Eigler wrote: > nathans wrote: >> >> I didn't follow the need to talk to pmlogger - what was the rationale >> there? > In the grand unified world, there will be a need to talk to > -something- that has both historical data and live data. Making > pmlogger into a network server would make it possible to have libpcp > engage in conversation with only a single thing. pmlogger already > knows about its own archives as well as the pmcd it's pulling from. > No coincidences or client-side magic required, so to speak. Yes, that was the idea behind this suggestion. This would only be required for archive-transitioning-to-live situations. Historical-only, which is what I will address initially need not use a pmlogger and, in fact, a pmlogger need not even be running for this case. It just seemed to me that the archive-to-live case implied the existence of one or more running pmloggers and that each was already managing everything needed to make the transition. Dave From fche@redhat.com Thu Oct 2 15:58:15 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 0B99E7F6B for ; Thu, 2 Oct 2014 15:58:15 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id E013C8F8052 for ; Thu, 2 Oct 2014 13:58:14 -0700 (PDT) X-ASG-Debug-ID: 1412283490-04cb6c50e767db10001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id ovmfLPbqAAjrpwbG (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 02 Oct 2014 13:58:11 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s92KwAC5005758 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Thu, 2 Oct 2014 16:58:10 -0400 Received: from fche.csb (vpn-230-163.phx2.redhat.com [10.3.230.163]) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s92Kw9ax005133 for ; Thu, 2 Oct 2014 16:58:10 -0400 Received: by fche.csb (Postfix, from userid 2569) id 7849E58548; Thu, 2 Oct 2014 16:58:09 -0400 (EDT) To: pcp@oss.sgi.com Subject: Re: [pcp-announce] PCP developers conference call References: <901130688.60042744.1412210950889.JavaMail.zimbra@redhat.com> <1968683202.60097085.1412217054636.JavaMail.zimbra@redhat.com> X-ASG-Orig-Subj: Re: [pcp-announce] PCP developers conference call From: fche@redhat.com (Frank Ch. Eigler) Date: Thu, 02 Oct 2014 16:58:09 -0400 In-Reply-To: <1968683202.60097085.1412217054636.JavaMail.zimbra@redhat.com> (Nathan Scott's message of "Wed, 1 Oct 2014 22:30:54 -0400 (EDT)") Message-ID: User-Agent: Gnus/5.1008 (Gnus v5.10.8) Emacs/21.4 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412283491 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 > [...] > - source trees > [...] > - issues with separate trees, pcp web status [fche, nathans] > [...] I would like to discuss the larger issue of the maintenance role, whose centralized nature allowed the current situation to develop. I'd like the community to consider spreading different aspects of maintenance amongst a larger pool of people: - reviewing/merging random community patches - improving infrastructure - developing technical sub-area - testing on an ongoing basis - fixing filed bugs - engineering releases - packaging in distros - providing technical direction - building consensus in community - breaking ties - FChE From nscott@redhat.com Thu Oct 2 17:03:29 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 3E3A57F53 for ; Thu, 2 Oct 2014 17:03:29 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 28E9B30407A for ; Thu, 2 Oct 2014 15:03:26 -0700 (PDT) X-ASG-Debug-ID: 1412287402-04cb6c50e56809f0001-S8gJnT Received: from mx3-phx2.redhat.com (mx3-phx2.redhat.com [209.132.183.24]) by cuda.sgi.com with ESMTP id wTN9C2X9Aij0f4LN (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 02 Oct 2014 15:03:23 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.24 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx3-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s92M3M6k022599; Thu, 2 Oct 2014 18:03:22 -0400 Date: Thu, 2 Oct 2014 18:03:21 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: David Smith Cc: pcp Message-ID: <336785285.60750380.1412287401855.JavaMail.zimbra@redhat.com> In-Reply-To: <542D6626.50703@redhat.com> References: <542D6626.50703@redhat.com> Subject: Re: [pcp] [PATCH] Fix memory corruption in python support MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] [PATCH] Fix memory corruption in python support Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.6] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: Fix memory corruption in python support Thread-Index: 11rim5vu/KCbOdjNGZRCnOvzC4CH2A== X-Barracuda-Connect: mx3-phx2.redhat.com[209.132.183.24] X-Barracuda-Start-Time: 1412287403 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.03 X-Barracuda-Spam-Status: No, SCORE=0.03 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_SA_TO_FROM_DOMAIN_MATCH, THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10141 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... 0.01 BSF_SC0_SA_TO_FROM_DOMAIN_MATCH Sender Domain Matches Recipient Domain Hi David, ----- Original Message ----- > [...] > Unfortunately, pcp's python support was decrementing reference counts in > some cases. I went through and audited the calls to PyArg_ParseTuple(), > PyArg_ParseTupleAndKeywords(), and PyArg_Parse() and believe I've fixed > the reference count issues. Nice work! > This patch fixed the memory corruption I was seeing. Can we distil that memory corruption into a reproducible test case? We have several tests exercising various aspects of the PMDA wrapper but they are not triggering this problem - it'd be good to come up with something (e.g. minimal python script?) that demonstrates the problem and which we can use to verify the fix. thanks! -- Nathan From nscott@redhat.com Thu Oct 2 20:37:02 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 5DB807F55 for ; Thu, 2 Oct 2014 20:37:02 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 1FEDC8F8033 for ; Thu, 2 Oct 2014 18:36:59 -0700 (PDT) X-ASG-Debug-ID: 1412300214-04bdf003a0712500001-S8gJnT Received: from mx3-phx2.redhat.com (mx3-phx2.redhat.com [209.132.183.24]) by cuda.sgi.com with ESMTP id vLMH2GmGFoPPLveU (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 02 Oct 2014 18:36:54 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.24 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx3-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s931ar7g020980; Thu, 2 Oct 2014 21:36:53 -0400 Date: Thu, 2 Oct 2014 21:36:53 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Dave Brolley , "Frank Ch. Eigler" Cc: pcp@oss.sgi.com Message-ID: <1741011764.60812520.1412300213647.JavaMail.zimbra@redhat.com> In-Reply-To: <542D756B.8030906@redhat.com> References: <542C21AE.1010504@redhat.com> <542C4EC1.4040407@redhat.com> <408521091.60123629.1412223728964.JavaMail.zimbra@redhat.com> <542D756B.8030906@redhat.com> Subject: Re: Multi-Volume Archive + Live Data Playback for PCP Client Tools MIME-Version: 1.0 X-ASG-Orig-Subj: Re: Multi-Volume Archive + Live Data Playback for PCP Client Tools Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.6] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: Multi-Volume Archive + Live Data Playback for PCP Client Tools Thread-Index: s/gUPKosUu6ewwsmP7xBvtf6Zlw0kg== X-Barracuda-Connect: mx3-phx2.redhat.com[209.132.183.24] X-Barracuda-Start-Time: 1412300214 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.03 X-Barracuda-Spam-Status: No, SCORE=0.03 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_SA_TO_FROM_DOMAIN_MATCH, THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10141 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... 0.01 BSF_SC0_SA_TO_FROM_DOMAIN_MATCH Sender Domain Matches Recipient Domain ----- Original Message ----- > >> [...] > >> I didn't follow the need to talk to pmlogger - what was the rationale > >> there? > > In the grand unified world, there will be a need to talk to > > -something- that has both historical data and live data. Making If by -something- you mean a daemon, I'm unconvinced. I think an archive (written unbuffered as we now do) would make an excellent -something- (as in, "tail -f" on an archive) along with direct use of either pmcd or local-context DSOs for live data (optionally). > > pmlogger into a network server would make it possible to have libpcp > > engage in conversation with only a single thing. pmlogger already > > knows about its own archives as well as the pmcd it's pulling from. Ah I remember this line of thinking now. As you say, it doesn't help for non-actively-written data, so is not as big a win as it initially seemed. And there was other issues with that approach, like pmlogger having to duplicate all of the pmcd client-facing functionality (and more) - so, namespace/pmDesc/auth/PMID/value lookups, etc - its alot of code. And another "hop" (synchronous round trip for every PDU) whereas a direct pmcd connection is clearly available and simpler. > Yes, that was the idea behind this suggestion. This would only be > required for archive-transitioning-to-live situations. Yeah, this is the case I was curious about the rationale for - there doesn't seem to be an explanation so far as to why a libpcp "tail -f" mode would be insufficient for the actively-written data. > Historical-only, > which is what I will address initially need not use a pmlogger and, in > fact, a pmlogger need not even be running for this case. Yep, understood. BTW, note that the unified context concept requires *zero* daemons (live data is optional and/or local context can be used). This is important, and a major design feature with stacking multiple contexts. Which could also be used to simplify and help preserve backward-compat for you here, Dave, I think - building up using multiple archive contexts, and passing back a handle to the top-level one from pmNewContext(3)? Maybe. > It just seemed to me that the archive-to-live case implied the existence > of one or more running pmloggers and that each was already managing > everything needed to make the transition. I'm not seeing that as implied. To my mind it implies the existence of pmcd (well, actually, not even that - one could live-transition between a PM_CONTEXT_LOCAL context and an archive) but to me it doesn't imply a running pmlogger process. I think the insertion of pmlogger here (or any other new daemon) would vastly complicate an already complicated situation, and something that we should avoid if we can. cheers. -- Nathan From fche@redhat.com Thu Oct 2 20:55:15 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 28FA37F4E for ; Thu, 2 Oct 2014 20:55:15 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id DBA8B8F8035 for ; Thu, 2 Oct 2014 18:55:14 -0700 (PDT) X-ASG-Debug-ID: 1412301313-04bdf003a1712ff0001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id vOIIiPUIChJFYBXy (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 02 Oct 2014 18:55:13 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s931tDVZ024041 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Thu, 2 Oct 2014 21:55:13 -0400 Received: from fche.csb (vpn-230-163.phx2.redhat.com [10.3.230.163]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s931tCge031238; Thu, 2 Oct 2014 21:55:12 -0400 Received: by fche.csb (Postfix, from userid 2569) id 4F9D658548; Thu, 2 Oct 2014 21:55:12 -0400 (EDT) Date: Thu, 2 Oct 2014 21:55:12 -0400 From: "Frank Ch. Eigler" To: Nathan Scott Cc: Dave Brolley , pcp@oss.sgi.com Subject: Re: Multi-Volume Archive + Live Data Playback for PCP Client Tools Message-ID: <20141003015512.GB20566@redhat.com> X-ASG-Orig-Subj: Re: Multi-Volume Archive + Live Data Playback for PCP Client Tools References: <542C21AE.1010504@redhat.com> <542C4EC1.4040407@redhat.com> <408521091.60123629.1412223728964.JavaMail.zimbra@redhat.com> <542D756B.8030906@redhat.com> <1741011764.60812520.1412300213647.JavaMail.zimbra@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1741011764.60812520.1412300213647.JavaMail.zimbra@redhat.com> User-Agent: Mutt/1.4.2.2i X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412301313 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Hi - > [...] If by -something- you mean a daemon, I'm unconvinced. I > think an archive (written unbuffered as we now do) would make an > excellent -something- (as in, "tail -f" on an archive) along with > direct use of either pmcd or local-context DSOs for live data > (optionally). [...] I see what you mean. There are two complications: 1- This requires the client to divine pmcd server connection data for the real-time component from the archives (or vice versa). Or else force the a user to specify both? 2- This makes it impossible to log the real-time collected data uniformly with the other historical data. Data that the mystery pmlogger happens to log would be available; data that the PMAPI client requests from live pmcd would be available; but data that the PMAPI client *recently requested* wouldn't be. If OTOH a pmlogger-intimate server were the intermediary, then it can arrange for the recently-requested data to be logged (alongside whatever static configuration it had). That way, when pmchart refreshes recent data, the really old, the recent, and the current can all come from the same place. (The smarter pmlogger could even speculatively log more data, in anticipation of future requests.) > [...] And there was other issues with that approach, like pmlogger > having to duplicate all of the pmcd client-facing functionality (and > more) - so, namespace/pmDesc/auth/PMID/value lookups, etc - its alot > of code. Sure, luckily that code is already written; just needs to be librarified. > And another "hop" (synchronous round trip for every PDU) True (unless perhaps the new pmlogger can anticipate). > whereas a direct pmcd connection is clearly available and simpler. ... assuming a direct connection -is- available! It may not be. Whatever pmlogger process is writing to the grand-unified archives (consider archives coming via NFS from some faraway box) may have a way to get to its pmcd that the pmapi client doesn't, whether via special authentication, or network topology differences. (IOW, as we still suffer here and there from the contrary assumption, archive-label.hostname != usable PM_CONTEXT_HOST parameter.) - FChE From nscott@redhat.com Thu Oct 2 23:27:16 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id CFDA37F53 for ; Thu, 2 Oct 2014 23:27:16 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id AF1A58F8059 for ; Thu, 2 Oct 2014 21:27:13 -0700 (PDT) X-ASG-Debug-ID: 1412310431-04bdf003a1718450001-S8gJnT Received: from mx3-phx2.redhat.com (mx3-phx2.redhat.com [209.132.183.24]) by cuda.sgi.com with ESMTP id prg6PTvc7JgZIhBy (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 02 Oct 2014 21:27:11 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.24 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx3-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s934RBQY012509 for ; Fri, 3 Oct 2014 00:27:11 -0400 Date: Fri, 3 Oct 2014 00:27:11 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: pcp Message-ID: <1173436883.60847093.1412310431247.JavaMail.zimbra@redhat.com> Subject: pcp updates: pmdads389log MIME-Version: 1.0 X-ASG-Orig-Subj: pcp updates: pmdads389log Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.6] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: pcp updates: pmdads389log Thread-Index: /+rh6iQ/zfL2/OlsTOOYXQFWSUHOUw== X-Barracuda-Connect: mx3-phx2.redhat.com[209.132.183.24] X-Barracuda-Start-Time: 1412310431 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10141 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Changes committed to git://git.pcp.io/pcp.git dev Marko Myllynen (1): ds389log pmda: a new 389 Directory Server log processing PMDA Nathan Scott (1): build, qa: bring the ds389log PMDA along to the party qa/960 | 114 ++++++++++++ qa/960.out | 48 +++++ qa/archives/GNUmakefile | 2 qa/archives/ds390-access.gz |binary qa/group | 1 src/pmdas/GNUmakefile | 2 src/pmdas/ds389log/.gitignore | 4 src/pmdas/ds389log/GNUmakefile | 41 ++++ src/pmdas/ds389log/Install | 32 +++ src/pmdas/ds389log/Remove | 23 ++ src/pmdas/ds389log/pmdads389log.1 | 81 ++++++++ src/pmdas/ds389log/pmdads389log.pl | 350 +++++++++++++++++++++++++++---------- src/pmns/stdpmid.pcp | 1 13 files changed, 611 insertions(+), 88 deletions(-) From nscott@redhat.com Thu Oct 2 23:27:47 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 3FB037F53 for ; Thu, 2 Oct 2014 23:27:47 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 1F395304048 for ; Thu, 2 Oct 2014 21:27:44 -0700 (PDT) X-ASG-Debug-ID: 1412310457-04cbb0730271beb0001-S8gJnT Received: from mx4-phx2.redhat.com (mx4-phx2.redhat.com [209.132.183.25]) by cuda.sgi.com with ESMTP id PBMAFQsyVVfWSWQ0 (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 02 Oct 2014 21:27:38 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.25 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx4-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s934Rbfw013100; Fri, 3 Oct 2014 00:27:37 -0400 Date: Fri, 3 Oct 2014 00:27:37 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: myllynen@redhat.com Cc: pcp@oss.sgi.com, Rich Megginson Message-ID: <1172063229.60847097.1412310457413.JavaMail.zimbra@redhat.com> In-Reply-To: <542424C2.6080900@redhat.com> References: <542424C2.6080900@redhat.com> Subject: Re: [pcp] [PATCH] 389 DS Log PCP PMDA MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] [PATCH] 389 DS Log PCP PMDA Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.6] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: 389 DS Log PCP PMDA Thread-Index: Kh5ydKjr/QFYVtQpudwYZmERDy0L3g== X-Barracuda-Connect: mx4-phx2.redhat.com[209.132.183.25] X-Barracuda-Start-Time: 1412310458 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.03 X-Barracuda-Spam-Status: No, SCORE=0.03 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_SA_TO_FROM_DOMAIN_MATCH, THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10141 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.01 BSF_SC0_SA_TO_FROM_DOMAIN_MATCH Sender Domain Matches Recipient Domain Hi Marko, ----- Original Message ----- > [...] > Running logconv.pl on an access log on a busy server without defining the > start/end time can be a long, CPU consuming task. Thus, we always ask > metrics for a certain time slice only and calculate the actual metrics since > the PMDA was started to minimize resource usage and to avoid pmcd(1) > dropping the PMDA (see http://oss.sgi.com/bugzilla/show_bug.cgi?id=1036). In > few cases the code could perhaps a bit more Perlish but OTOH TIMTOWTDI. Looks good to me. Can you take a look over the pcp dev branch code, and verify everything is in order? > There is, however, one minor issue currently with the PMDA which I think I > should mention - it doesn't work :) Details, details. ;) I'll assume there's nothing we can do about this and it'll get fixed independently of any PCP use of the tool. > > To test this locally one would ideally have 389 DS running but testing > against a previously captured access log (e.g. from the above RHBZ) should > help to do basic sanity checking. > Test qa/960 exercises the PMDA, optionally using this log now if no other is available. It all seems to be working nicely for me. cheers. -- Nathan From nscott@redhat.com Thu Oct 2 23:29:43 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 19FD97F53 for ; Thu, 2 Oct 2014 23:29:43 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 066C6304048 for ; Thu, 2 Oct 2014 21:29:42 -0700 (PDT) X-ASG-Debug-ID: 1412310581-04cb6c50e468f4e0001-S8gJnT Received: from mx3-phx2.redhat.com (mx3-phx2.redhat.com [209.132.183.24]) by cuda.sgi.com with ESMTP id py8W35nXAFhU5rEs (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 02 Oct 2014 21:29:41 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.24 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx3-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s934Tf4h012585; Fri, 3 Oct 2014 00:29:41 -0400 Date: Fri, 3 Oct 2014 00:29:41 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: "Frank Ch. Eigler" , Dave Brolley Cc: pcp@oss.sgi.com Message-ID: <992354017.60847300.1412310581028.JavaMail.zimbra@redhat.com> In-Reply-To: <20141003015512.GB20566@redhat.com> References: <542C21AE.1010504@redhat.com> <542C4EC1.4040407@redhat.com> <408521091.60123629.1412223728964.JavaMail.zimbra@redhat.com> <542D756B.8030906@redhat.com> <1741011764.60812520.1412300213647.JavaMail.zimbra@redhat.com> <20141003015512.GB20566@redhat.com> Subject: Re: Multi-Volume Archive + Live Data Playback for PCP Client Tools MIME-Version: 1.0 X-ASG-Orig-Subj: Re: Multi-Volume Archive + Live Data Playback for PCP Client Tools Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.6] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: Multi-Volume Archive + Live Data Playback for PCP Client Tools Thread-Index: FTo4oCwzXbK0sUCoRt0A7YTPUN1mKA== X-Barracuda-Connect: mx3-phx2.redhat.com[209.132.183.24] X-Barracuda-Start-Time: 1412310581 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.03 X-Barracuda-Spam-Status: No, SCORE=0.03 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_SA_TO_FROM_DOMAIN_MATCH, THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10141 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... 0.01 BSF_SC0_SA_TO_FROM_DOMAIN_MATCH Sender Domain Matches Recipient Domain (Heh, we've wandered a fair way from multi-archive support now, and we're not helping Dave here anymore, I suspect - this will be my last mail on this topic for now) ----- Original Message ----- > > 1- This requires the client to divine pmcd server connection data for > the real-time component from the archives (or vice versa). Or else > force the a user to specify both? It starts from a host spec (just like pmlogger does), as described in the earlier unified context discussions. > requests from live pmcd would be available; but data that the PMAPI > client *recently requested* wouldn't be. If it needs to persist, pmlogger needs to write it. End of story. Keep it simple. As described in the earlier discussions. This scheme where clients have to force all data to be written to disk, via a system daemon no less, before they can see it - that's often not going to be appropriate, sorry. Many tools definitely do not want that - we simply cannot mandate that behaviour. > > And another "hop" (synchronous round trip for every PDU) > > True (unless perhaps the new pmlogger can anticipate). I don't see how, and certainly not using a 1-to-1 "librarified" pmcd client protocol. So we'd invariably be increasing latency across all PCP clients in one fell swoop. I just cannot see this approach being feasible, sorry - beyond just the poor performance, there's protocol deadlock issues, reconnect becomes complex - all error handling, in fact, is complicated by the multiple hops - there's a raft of issues. > [...] > as we still suffer here and there from the contrary assumption, > archive-label.hostname != usable PM_CONTEXT_HOST parameter.) That's not being assumed here. cheers. -- Nathan From nscott@redhat.com Thu Oct 2 23:32:19 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 0E5477F53 for ; Thu, 2 Oct 2014 23:32:19 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 9E266AC001 for ; Thu, 2 Oct 2014 21:32:18 -0700 (PDT) X-ASG-Debug-ID: 1412310737-04bdf003a0718750001-S8gJnT Received: from mx6-phx2.redhat.com (mx6-phx2.redhat.com [209.132.183.39]) by cuda.sgi.com with ESMTP id pKjjTLQd4vBWMF5v (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 02 Oct 2014 21:32:17 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.39 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx6-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s934WGux020126; Fri, 3 Oct 2014 00:32:16 -0400 Date: Fri, 3 Oct 2014 00:32:16 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: "Frank Ch. Eigler" Cc: pcp@oss.sgi.com Message-ID: <1279214536.60847858.1412310736778.JavaMail.zimbra@redhat.com> In-Reply-To: References: <901130688.60042744.1412210950889.JavaMail.zimbra@redhat.com> <1968683202.60097085.1412217054636.JavaMail.zimbra@redhat.com> Subject: Re: [pcp] [pcp-announce] PCP developers conference call MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] [pcp-announce] PCP developers conference call Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.6] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: PCP developers conference call Thread-Index: I+GUFgx3iAjtIIqxIEopoJkEJ5Dhcw== X-Barracuda-Connect: mx6-phx2.redhat.com[209.132.183.39] X-Barracuda-Start-Time: 1412310737 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.03 X-Barracuda-Spam-Status: No, SCORE=0.03 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_SA_TO_FROM_DOMAIN_MATCH, THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10141 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.01 BSF_SC0_SA_TO_FROM_DOMAIN_MATCH Sender Domain Matches Recipient Domain Hi Frank, ----- Original Message ----- > > [...] > > - source trees > > [...] > > - issues with separate trees, pcp web status [fche, nathans] > > [...] > > I would like to discuss the larger issue of the maintenance role, Absolutely - I had assumed this'd be discussed in the above topic, but lets indeed make this a separate topic of its own. I have two other topics from Michele to throw into the mix too - I'll send an updated agenda out early next week. > whose centralized nature allowed the current situation to develop. This assertion is pure conjecture, however. Many other successful open source projects follow a similar model. Let us instead state only facts: this "current situation" is one more in the series of disagreements involving you and I about some of your contributions (this is, of course, not the first time we have discussed separate trees for new components you are contributing). In the 15+ years I've been involved with PCP - with the last ~7 years as the primary maintainer-and-release-guy - I've not seen similar situations arise with contributions from anyone else. Not laying blame here, rather I'm saying you and I have, at times, bumped into some very-hard-to-resolve issues around code - we often both believe we're right and we both have strong-willed personality types, I guess. Anyway, facts only please - conjecture and emotive words ("cleaving" source trees, "extirpation" of source code, "FUD") - not so much. > I'd like the community to consider spreading different aspects > of maintenance amongst a larger pool of people: > > - reviewing/merging random community patches > - improving infrastructure > - developing technical sub-area > - testing on an ongoing basis > - fixing filed bugs > - engineering releases > - packaging in distros > - providing technical direction > - building consensus in community > - breaking ties Sounds good to me - lets keep it open, honest and respectful. Thanks for raising these topics. cheers. -- Nathan From myllynen@redhat.com Fri Oct 3 03:13:36 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 78C657F53 for ; Fri, 3 Oct 2014 03:13:36 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 24EDFAC001 for ; Fri, 3 Oct 2014 01:13:32 -0700 (PDT) X-ASG-Debug-ID: 1412324008-04bdf003a171f8a0001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id ClIgKHCEzDiKTud9 (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Fri, 03 Oct 2014 01:13:29 -0700 (PDT) X-Barracuda-Envelope-From: myllynen@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s938DSEq004909 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Fri, 3 Oct 2014 04:13:28 -0400 Received: from mmyllyne.csb (vpn1-6-56.ams2.redhat.com [10.36.6.56]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s938DQvr027144; Fri, 3 Oct 2014 04:13:27 -0400 Message-ID: <542E5AA5.7090202@redhat.com> Date: Fri, 03 Oct 2014 11:13:25 +0300 From: Marko Myllynen Reply-To: myllynen@redhat.com Organization: Red Hat User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Nathan Scott CC: pcp@oss.sgi.com, Rich Megginson Subject: Re: [pcp] [PATCH] 389 DS Log PCP PMDA References: <542424C2.6080900@redhat.com> <1172063229.60847097.1412310457413.JavaMail.zimbra@redhat.com> X-ASG-Orig-Subj: Re: [pcp] [PATCH] 389 DS Log PCP PMDA In-Reply-To: <1172063229.60847097.1412310457413.JavaMail.zimbra@redhat.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412324009 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Hi, On 2014-10-03 07:27, Nathan Scott wrote: > > Looks good to me. Can you take a look over the pcp dev branch code, and > verify everything is in order? yes, looks good, although I wonder should Install/Remove scripts have execution bit set in the repo already all will it be set when building / packaging? >> There is, however, one minor issue currently with the PMDA which I think I >> should mention - it doesn't work :) > > Details, details. ;) I'll assume there's nothing we can do about this > and it'll get fixed independently of any PCP use of the tool. Yes, this is now being addressed in upstream: https://fedorahosted.org/389/ticket/47910 >> To test this locally one would ideally have 389 DS running but testing >> against a previously captured access log (e.g. from the above RHBZ) should >> help to do basic sanity checking. > > Test qa/960 exercises the PMDA, optionally using this log now if no other > is available. It all seems to be working nicely for me. Thanks for the finishing touches. I'll certainly report if any additional changes are needed once logconv.pl support is complete. Cheers, -- Marko Myllynen From fche@redhat.com Fri Oct 3 09:05:08 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 73F5D7F37 for ; Fri, 3 Oct 2014 09:05:08 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 372908F8054 for ; Fri, 3 Oct 2014 07:05:08 -0700 (PDT) X-ASG-Debug-ID: 1412345103-04bdf0039f72b670001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 6hUqoeqRCTeC79f3 (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Fri, 03 Oct 2014 07:05:04 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s93E53o8004120 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Fri, 3 Oct 2014 10:05:03 -0400 Received: from fche.csb (vpn-233-63.phx2.redhat.com [10.3.233.63]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s93E53Yr008856; Fri, 3 Oct 2014 10:05:03 -0400 Received: by fche.csb (Postfix, from userid 2569) id C857258114; Fri, 3 Oct 2014 10:05:02 -0400 (EDT) To: Nathan Scott Cc: Dave Brolley , pcp@oss.sgi.com Subject: Re: Multi-Volume Archive + Live Data Playback for PCP Client Tools References: <542C21AE.1010504@redhat.com> <542C4EC1.4040407@redhat.com> <408521091.60123629.1412223728964.JavaMail.zimbra@redhat.com> <542D756B.8030906@redhat.com> <1741011764.60812520.1412300213647.JavaMail.zimbra@redhat.com> <20141003015512.GB20566@redhat.com> <992354017.60847300.1412310581028.JavaMail.zimbra@redhat.com> X-ASG-Orig-Subj: Re: Multi-Volume Archive + Live Data Playback for PCP Client Tools From: fche@redhat.com (Frank Ch. Eigler) Date: Fri, 03 Oct 2014 10:05:02 -0400 In-Reply-To: <992354017.60847300.1412310581028.JavaMail.zimbra@redhat.com> (Nathan Scott's message of "Fri, 3 Oct 2014 00:29:41 -0400 (EDT)") Message-ID: User-Agent: Gnus/5.1008 (Gnus v5.10.8) Emacs/21.4 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412345104 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 nathans wrote: >> 1- This requires the client to divine pmcd server connection data for >> the real-time component from the archives (or vice versa). Or else >> force the a user to specify both? > > It starts from a host spec (just like pmlogger does), as described > in the earlier unified context discussions. That's not the complete story. There are one or more archives, zero or more pmloggers, and a pmcd server. The client would need information about all those, not just "a host spec". (The attraction of using a pmlogger-like thing as an intermediary is that -it- would already have info about all those things.) >> requests from live pmcd would be available; but data that the PMAPI >> client *recently requested* wouldn't be. > > If it needs to persist, pmlogger needs to write it. [...] Exactly, but we don't draw the same conclusions from that. > This scheme where clients have to force all data to be written to > disk, via a system daemon no less [...] No one said it'd have to be a *system* daemon. For example, pmlogger is fully usable as transient & personal program. > [...] Many tools definitely do not want that - we simply cannot > mandate that behaviour. Tools that don't need "grand unification" don't need to use this stuff at all. Tools that just want to listen passively to archives and actively to pmcds of their choice can already do so today: just open two pcp contexts (an archive and a host). (Tools that want to monitor -growing- archives will need to wait for archive-tail-f type functionality in libpcp and make some minor changes in their own code, as we outlined back in January. [1]) [1] http://oss.sgi.com/pipermail/pcp/2014-January/004260.html By the way, in your mental model, do you plan to address the needs of tools that would like simply remote access to archive data (not pmcd, and without locally reachable archive -files-)? The pmlogger-proxy idea can do that. (Or is that niche to be reserved for pmwebd?) > [...] there's a raft of issues. Certainly. - FChE From brolley@redhat.com Fri Oct 3 14:36:29 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id EF0537F37 for ; Fri, 3 Oct 2014 14:36:29 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id CEA4830404E for ; Fri, 3 Oct 2014 12:36:26 -0700 (PDT) X-ASG-Debug-ID: 1412364984-04cbb0730173ff50001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id BfoTAdmN2T1cPlnY (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Fri, 03 Oct 2014 12:36:25 -0700 (PDT) X-Barracuda-Envelope-From: brolley@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s93JaNoK011268 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Fri, 3 Oct 2014 15:36:24 -0400 Received: from [10.10.62.136] (vpn-62-136.rdu2.redhat.com [10.10.62.136]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s93JaM2Y022237 for ; Fri, 3 Oct 2014 15:36:23 -0400 Message-ID: <542EFB14.6020208@redhat.com> Date: Fri, 03 Oct 2014 15:37:56 -0400 From: Dave Brolley User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: Re: [pcp] Multi-Volume Archive + Live Data Playback for PCP Client Tools References: <542C21AE.1010504@redhat.com> X-ASG-Orig-Subj: Re: [pcp] Multi-Volume Archive + Live Data Playback for PCP Client Tools In-Reply-To: <542C21AE.1010504@redhat.com> Content-Type: multipart/alternative; boundary="------------080707020703050505080308" X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412364985 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 This is a multi-part message in MIME format. --------------080707020703050505080308 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hmmm, ok, so maybe Frank was right about opening a can of worms before we're ready to eat .... However, I see multi-archive support and (multi-)archive-transitioning-to-live support as related things worth thinking about together. In reading everyone's responses, feedback and ideas (thanks!) I think that the two camps on the "use pmlogger as a server" idea have possibly arisen because of different notions of what we're trying to accomplish here. For my initial, mistaken, idea that we are trying to provide multi-volume support which transitions to live, I had it in my mind that we were talking about an archive directory which was being actively written to by a pmlogger from which we wanted to extract metrics from a time window extending from some time in the past to some time in the future. Hopefully, in that context, it is easier to see why I thought that having clients talk to the active pmlogger as the single source of all of the metrics might make sense. The client (via libpcp) would not have to manage the ever-growing final archive volume, the possible switch to a new ever-growing final archive volume (e.g. if the current one were reach the 2GB size limit), or the possibility that logging stops. The pmlogger would continue to log new data as normal but would also simply relay it back to the client once all data from the past had been sent. This is how I envisioned the "tail -f" support working. It now appears to me (and once again, please correct me if I am wrong!) that the goal is for clients to be able to examine metrics from multiple archives, each of which may have multiple volumes (already supported) and each of which may be arbitrarily distinct (i.e. not necessarily created by the same pmlogger or even on the same machine). It also appears to me that the idea of transitioning to live metrics means transitioning to any arbitrary source (or sources?) of live metrics which could be one (or more?) of: * following an active archive as it grows, similar to 'tail -f' * switching to an arbitrary active pmcd Until we all agree on what the ultimate goal is here (and perhaps I'm the only one who is still not sure), it would seem unlikely that we can agree on an architecture or design. So please, correct me where I am wrong and fill in the blanks where I have missed them. FWIW, I still think that the "pmlogger as a server" idea may still have value as a conceptually simple way of providing 'tail -f' support and, as Frank has pointed out, could also provide a way of accessing archives (with tailing) on remote hosts regardless of how we do it for locally-accessible archives. Dave --------------080707020703050505080308 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hmmm, ok, so maybe Frank was right about opening a can of worms before we're ready to eat ....

However, I see multi-archive support and (multi-)archive-transitioning-to-live support as related things worth thinking about together.

In reading everyone's responses, feedback and ideas (thanks!) I think that the two camps on the "use pmlogger as a server" idea have possibly arisen because of different notions of what we're trying to accomplish here. For my initial, mistaken, idea that we are trying to provide multi-volume support which transitions to live, I had it in my mind that we were talking about an archive directory which was being actively written to by a pmlogger from which we wanted to extract metrics from a time window extending from some time in the past to some time in the future.

Hopefully, in that context, it is easier to see why I thought that having clients talk to the active pmlogger as the single source of all of the metrics might make sense. The client (via libpcp) would not have to manage the ever-growing final archive volume, the possible switch to a new ever-growing final archive volume (e.g. if the current one were reach the 2GB size limit), or the possibility that logging stops. The pmlogger would continue to log new data as normal but would also simply relay it back to the client once all data from the past had been sent. This is how I envisioned the "tail -f" support working.

It now appears to me (and once again, please correct me if I am wrong!) that the goal is for clients to be able to examine metrics from multiple archives, each of which may have multiple volumes (already supported) and each of which may be arbitrarily distinct (i.e. not necessarily created by the same pmlogger or even on the same machine).

It also appears to me that the idea of transitioning to live metrics means transitioning to any arbitrary source (or sources?) of live metrics which could be one (or more?) of:
  • following an active archive as it grows, similar to 'tail -f'
  • switching to an arbitrary active pmcd
Until we all agree on what the ultimate goal is here (and perhaps I'm the only one who is still not sure), it would seem unlikely that we can agree on an architecture or design. So please, correct me where I am wrong and fill in the blanks where I have missed them.

FWIW, I still think that the "pmlogger as a server" idea may still have value as a conceptually simple way of providing 'tail -f' support and, as Frank has pointed out, could also provide a way of accessing archives (with tailing) on remote hosts regardless of how we do it for locally-accessible archives.

Dave
--------------080707020703050505080308-- From fche@redhat.com Fri Oct 3 16:38:15 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id C436A7F37 for ; Fri, 3 Oct 2014 16:38:15 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id A02C2304048 for ; Fri, 3 Oct 2014 14:38:15 -0700 (PDT) X-ASG-Debug-ID: 1412372293-04cb6c50e46b8100001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id R9dE6vMP6Yp7AGH4 (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Fri, 03 Oct 2014 14:38:14 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s93LcDHu029558 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Fri, 3 Oct 2014 17:38:13 -0400 Received: from fche.csb (vpn-233-63.phx2.redhat.com [10.3.233.63]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s93LcCS3005097 for ; Fri, 3 Oct 2014 17:38:13 -0400 Received: by fche.csb (Postfix, from userid 2569) id 5544558114; Fri, 3 Oct 2014 17:38:12 -0400 (EDT) To: pcp developers Subject: Re: PMAPI observations re. converting an app to pcp References: <20140925180407.GA18679@redhat.com> X-ASG-Orig-Subj: Re: PMAPI observations re. converting an app to pcp From: fche@redhat.com (Frank Ch. Eigler) Date: Fri, 03 Oct 2014 17:38:11 -0400 In-Reply-To: <20140925180407.GA18679@redhat.com> (Frank Ch. Eigler's message of "Thu, 25 Sep 2014 14:04:07 -0400") Message-ID: User-Agent: Gnus/5.1008 (Gnus v5.10.8) Emacs/21.4 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412372294 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 I wrote: > - The API is really wordy for basic tasks such as extracting and > scaling some metrics. [...] We need a higher level API that > combines the lookup/indom/fetch/extract/scale steps (doing internal > caching/lookups as necessary), with the goal of making it easy to > interface with an enclosing application. [...] Time to unveil a first sketch at a possible partial solution. (Are those enough qualifiers?) Let's say hello to "fetch groups": a way for a PMAPI client to identify metrics it will want to load, what scaling/conversion it would like, and the place to put the results. The client will then fetch the group one or more times, and consume the data in its preassigned locations. Hello, fetch groups! int pmCreateFetchGroup(); // create a fetchgroup for current pcp context // specify what to fetch int pmExtendFetchGroup_TYPE(int, const char*, const char *, const char *, TYPE *); int pmExtendFetchGroup_indom_TYPE(int, const char*, const char *, char***, TYPE **); // (where TYPE goes from uint32 ... double) int pmExtendFetchGroup_timestamp(int, struct timeval*); // specify where to put timestamp int pmFetchGroup(int); // (re)fetch the group as many times as required int pmDestroyFetchGroup(int); // destroy the fetchgroup All the API magic is clearly in the pmExtendFetchGroup_* functions, so let's take a look at one: int pmExtendFetchGroup_int32( int pmfg, const char *metric, const char *scale, const char *instance, __int32_t *value) Example usage from a dramatically shrunk pmclient.c: pmExtendFetchGroup_uint32(pmfg, "disk.all.total", "count/second", NULL, &info.dkiops); and pmExtendFetchGroup_float(pmfg, "kernel.all.load", NULL, "1 min", &info.load1); The former asks libpcp to prepare for fetching the disk.all.total metric, type-convert, rescale, and rate-convert the data to counts/second, and deposit the result into the given unsigned int. (Rate-conversion would come built-in, as would a reverse of the pmUnitsStr function.) The latter asks libpcp to prepare for fetching the kernel.all.load metric, no scaling, just to pick one named instance from the indom, and again deposit the result into the given float. OK, OK, that's pretty clear. How about pulling whole instance domains? char **cpu_user_name = NULL; float *cpu_user = NULL; pmExtendFetchGroup_indom_float(pmfg, "kernel.percpu.cpu.user", "second/second", &cpu_user_name, &cpu_user); This asks libpcp to prepare for fetching all the instances of the named metric, do conversion to plain unit rates (from millisecond-counter), and deposit the instance names & values in parallel arrays. (libpcp would allocate both those arrays with realloc.) OK, with that one-time preparation out of the way, how would the main loop look? Too simple: if ((sts = pmFetchGroup(pmfg)) < 0) { /* tragedy */ } printf ("%u %f\n", info.dkiops, =info.load1) for (i=0; cpu_user_name[i] != NULL; i++) printf ("%s:%f\n", cpu_user_name[i], cpu_user[i]); ... and I guess do a pmDestroyFetchGroup at the end, for valgrind-correctness. (Internal implementation would involve some data structures stored within libpcp associated with that pmfg integer, including all the pmDesc and pmID goo, prior pmResults for rate conversion, etc. The pmExtendFetchGroup would do name/desc resolution & prep. pmFetchGroup would do a pmFetch(), then a table-driven extraction of the pmResult into the user-specified locations. I haven't started building it out but nothing in there worries me.) Some questions remain. First of all, what do you think of the general flavour? Second, do you endorse the type-safe nature of passing float* etc. destinations for the metric values, tolerating N sibling foo_TYPE() functions, instead of something icky like the pmAtomValue unions or void pointers? Third, I haven't fully sketched out what to do with PM_TYPE_{STRING,AGGREGATE}; probably have the user pass a destination buffer/length, or else have libpcp allocate, or (why not) both as alternatives. Fine? Fourth, I don't know what to do with events. They're fiendishly complicated in the pmValue structures, and would have a whale of a time encoding its recursive subdivision as one function call. Maybe skip it? Maybe some nasty varargs monstrosity? Fifth, it is entirely possible for values to come back absent from a pmFetch, and thus a pmFetchGroup. Encoding the absence of data is probably necessary, but I'm not sure whether better to do it with a formal flag (another bool* argument at the *Extend* functions), or maybe just use sentinel values (0, NaN) on the output. Or both? Sixth, not sure how/whether to represent indom profiles. Maybe not needed; just have the user invoke the non-_indom variant of the *Extend* functions, enumerating the instances and places where to put each's data. Or maybe have a variant of the _indom_ functions where the instance names are already provided? Seventh, several of the above represent optionalities that could lead to a cartesian-product-style explosion of the number of individual functions in the API. I don't mind that FWIW, but someone might. Eigth, aw heck, why not say that *Extend* with pmfg index #0 means "do it right now", i.e., the equivalent of tmp = pmCreateFetchGroup() pmExtendFetchGroup(tmp, .....) pmFetchGroup(tmp) pmDestroyFetchGroup(tmp) What could be simpler for one-shot inquiries? Ninth, this should apply OK to archives too (for consecutive fetches); should we bother wrap pmSetMode, or just let the client call it directly before pmFetchGroup()? Feedback welcome! - FChE From kenj@internode.on.net Sat Oct 4 15:19:58 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_FRT_FOLLOW1 autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id A9CCC7F3F for ; Sat, 4 Oct 2014 15:19:58 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 525E4AC007 for ; Sat, 4 Oct 2014 13:19:57 -0700 (PDT) X-ASG-Debug-ID: 1412453991-04bdf0039f76b0c0001-S8gJnT Received: from ipmail05.adl6.internode.on.net (ipmail05.adl6.internode.on.net [150.101.137.143]) by cuda.sgi.com with ESMTP id K0uBZAEaGndIYMgX for ; Sat, 04 Oct 2014 13:19:52 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.143 Received: from ppp118-209-112-30.lns20.mel4.internode.on.net (HELO bozohorize) ([118.209.112.30]) by ipmail05.adl6.internode.on.net with ESMTP; 05 Oct 2014 06:49:21 +1030 From: "Ken McDonell" To: "'Dave Brolley'" , "'PCP Mailing List'" References: <542C21AE.1010504@redhat.com> In-Reply-To: <542C21AE.1010504@redhat.com> Subject: RE: [pcp] Multi-Volume Archive + Live Data Playback for PCP Client Tools Date: Sun, 5 Oct 2014 07:19:06 +1100 X-ASG-Orig-Subj: RE: [pcp] Multi-Volume Archive + Live Data Playback for PCP Client Tools Message-ID: <007e01cfe010$7867f090$6937d1b0$@internode.on.net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 15.0 Thread-Index: AQHclF1cpR2XxHkXIjv6In84hym83JwFf+EQ Content-Language: en-au X-Barracuda-Connect: ipmail05.adl6.internode.on.net[150.101.137.143] X-Barracuda-Start-Time: 1412453991 X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.01 X-Barracuda-Spam-Status: No, SCORE=0.01 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10205 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== G'day Dave. Looking forward to the next chapter of Dave's Most Excellent Adventure! Apologies for jumping in late, ... this is not my day job! > -----Original Message----- > From: pcp-bounces@oss.sgi.com [mailto:pcp-bounces@oss.sgi.com] On Behalf > Of Dave Brolley > Sent: Thursday, 2 October 2014 1:46 AM > To: PCP Mailing List > Subject: [pcp] Multi-Volume Archive + Live Data Playback for PCP Client Tools > > ... > The current situation as I understand it: > > * PCP archives are created by pmlogger in distinct volumes due to various > constraints, such as a maximum file size of 2GB, the desire to allow organization > of the collected data, the desire to be able to manage data retention (i.e. log > rotation) and, undoubtedly, for other reasons as well. There are 2 things going on here ... - we have multiple archives because each archive accommodates metrics from a single host (this is part of the archives and pmcd are interchangeable sources of metrics, and part because of the difficulty of mixing metadata from different hosts within a single archive format) - we have multiple archives for management reasons - log rotation, backup, etc - within a single archive, we _may_ have multiple volumes - the 32-bit offsets in the index file is the main reason for this (although this is rarely reached in practice because of the multiple archives issues above) and the other reasons were speculative at the time of the original design, but never turned out to be important > * Some multi-volume support exists in the form of archive folios. These can > be created by mkaf(1) but are also created by some other tools, such as > pmchart(1) and pmview(1). Archive volumes in a folio may be examined using > pmafm(1) using its 'replay', 'repeat' or 'run' commands. The latter two > commands allow for repeated application of PCP client tools against one or more > archives in the folio. Folios have nothing really to do with multi-volume. They are motivated by 2 different considerations: 1. For the pmlogger_check/pmlogger_daily managed archives, the name of the archive is a timestamp which is variable and impossible to guess with any certaintly ... the Latest folio alongside these archives creates as constant "symbolic" name that can be used to manipulate the current archive. The Latest folio is created by pmlogger_check and pmnewlog. 2. For the gui tools (pmchart and pmview), there is a File->Record action that creates a standalone PCP archive for whatever metrics are currently being displayed. Now these tools support concurrent display of metrics from multiple hosts, so the "recording" may generate multiple archives, and also the configuration of the tool that launches the "recording" is not fixed, so the folio wraps up all the archives and the configuration of the "recording" tool ... this is sufficient to allow the "replay" operation to be performed after the archives have been created. > * The archive management tool, pmlogextract and indirectly, > pmlogger_daily and pmlogger_merge, provide the ability to extract data from > multiple archives and combine that data into a single archive volume. pmlogreduce is another tool in the same vein, but this one does statistical reduction, while pmlogextract slices and dices in none, one or two of the temporal and metric dimensions. > * Otherwise, PCP client tools are currently restricted to extracting metrics > from a single archive volume via PM_CONTEXT_ARCHIVE (the -a option). A single > archive volume and an option time window is specified, which is applied against > that single archive volume. Some tools (pmie and pmdumptext, for example) can handle multiple archives from different hosts, but restrict themselves to at most one archive per host. > What we would like have is for PCP client tools to have the ability to easily extract > metrics from multiple archive volumes. ... I think the goal is to have the tools easily able to process multiple archives from the _same_ host. Not only must the archives be from the same host, but they need to represent _disjoint_ time intervals (no overlapping time between the archives) so they can be temporally sorted to provide a single time series. > ... Ultimately, we would also like tail-like > following of an active archive volume with seamless transition from archived data > to live data. While this is highly desirable, I suspect it will be a disjoint piece of work from processing a set of temporally ordered and existing archives. > Here are a few ideas for realizing these goals: > > Client/tool interface: > Currently only a single archive volume may be specified by its base name (via > PM_CONTEXT_ARCHIVE or -a). We could allow the specification of multiple > archive specs, each of which could be: > > > * an archive volume file base name -- same as now > * the name of a directory containing a collection of PCP archive volumes > * wildcarded names which resolve to a list of the above items > > > For example, > > pminfo -a 20140930.0 -a 201408*.* -a /some/path/archives -a > /another/path/archive* > > PM_CONTEXT_ARCHIVE could be extended to support more than one archive > volume. As others have pointed out ... each -a gets mapped to a context, so we need some sort of syntax that can name more than one archive in a single command line argument to be used with -a ... so this leads to the following options: - dirname - glob-like , probably not just * but the whole shooting match of ?, [...] and {...,...} - a list, e.g. -a 20141001,20140930 Now this could be handed off to pmNewContext, and the client could use a single PMAPI context as a handle to access this _set_ of archives For this to work, we need some restrictions on the set of archives that can be combined in this way: - all for the same host - non-overlapping time windows If these are not satisfied, pmNewContext needs to return a (new) error code. Then we need to consider the metadata: - timezone could change - this will require some further investigation before a cunning plan can be proposed - PMNS - merge 'em all the while there are no conflicts ... in the case of a conflict (different names map to the same PMID or the same name is assigned more than one PMID) we probably need dynamic remapping ("first one found wins" is probably the right strategy) - metric descriptors - if these change it gets very messy, although is rare in practice - instance domains - should be close to OK, as these are already expected to vary over time ... it would be bad if the semantics of the instance domain members changed between archives, but this is more of a PMDA botch issue than a problem for libpcp to solve One simple solution that might be acceptable for 95% of the cases would be to rule all of the metadata data differences (except instance domains) to be unsupported. So pmNewContext would fail. The user's option for resolving this is to use pmlogrewrite to amend one or more of the archives and remove the differences. I think this is definitely an OK plan. > Extracting multi-volume data: > For PCP tools, one very simple idea for extracting data from multiple existing > archive volumes would be to use pmlogextract(1) ... Probably does not move the game along enough to warrant consideration, and involves a bucket load of I/O ... we can do better than this I believe. > Streaming live data: > ... I suggest deferring this for another discussion ... it really is an independent piece of work that could go before or after or in parallel with the archive "set" changes. Thanks for making the running on this one, Dave. From kenj@internode.on.net Sat Oct 4 15:38:08 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 03EB17F3F for ; Sat, 4 Oct 2014 15:38:08 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 777DEAC001 for ; Sat, 4 Oct 2014 13:38:07 -0700 (PDT) X-ASG-Debug-ID: 1412455084-04cb6c50e46e0b20001-S8gJnT Received: from ipmail07.adl2.internode.on.net (ipmail07.adl2.internode.on.net [150.101.137.131]) by cuda.sgi.com with ESMTP id 6MEam7AoiHYgjUlC for ; Sat, 04 Oct 2014 13:38:05 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.131 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ArgJAMxZMFR20VJb/2dsb2JhbABggw6BK4Iy0EMEAgKBBRcBe4QDAQEBAwEIAh4SHCgHAQMCBgMRAQMBASgHGSANAwYIAgQBEgsFiCYHvyQYkEwGhEUFkXRcjBuDQpERg3cpL4JKAQEB Received: from ppp118-209-82-91.lns20.mel4.internode.on.net (HELO bozohorize) ([118.209.82.91]) by ipmail07.adl2.internode.on.net with ESMTP; 05 Oct 2014 07:08:03 +1030 From: "Ken McDonell" To: "'Dave Brolley'" , References: <542C21AE.1010504@redhat.com> <542EFB14.6020208@redhat.com> In-Reply-To: <542EFB14.6020208@redhat.com> Subject: RE: [pcp] Multi-Volume Archive + Live Data Playback for PCP Client Tools Date: Sun, 5 Oct 2014 07:37:42 +1100 X-ASG-Orig-Subj: RE: [pcp] Multi-Volume Archive + Live Data Playback for PCP Client Tools Message-ID: <009301cfe013$1490d430$3db27c90$@internode.on.net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 15.0 Thread-Index: AQHclF1cpR2XxHkXIjv6In84hym83AGxytyom/lqutA= Content-Language: en-au X-Barracuda-Connect: ipmail07.adl2.internode.on.net[150.101.137.131] X-Barracuda-Start-Time: 1412455084 X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.01 X-Barracuda-Spam-Status: No, SCORE=0.01 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10205 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== > -----Original Message----- > From: pcp-bounces@oss.sgi.com [mailto:pcp-bounces@oss.sgi.com] On > Behalf Of Dave Brolley > Sent: Saturday, 4 October 2014 5:38 AM > To: pcp@oss.sgi.com > Subject: Re: [pcp] Multi-Volume Archive + Live Data Playback for PCP Client > Tools > > ... > It now appears to me (and once again, please correct me if I am wrong!) that > the goal is for clients to be able to examine metrics from multiple archives, > each of which may have multiple volumes (already supported) and each of > which may be arbitrarily distinct (i.e. not necessarily created by the same > pmlogger or even on the same machine). Yep, but all the archives have contain data collected from the same host ... different hosts require different contexts and different archives (or different sets of archives) > It also appears to me that the idea of transitioning to live metrics means > transitioning to any arbitrary source (or sources?) of live metrics which could > be one (or more?) of: > > > * following an active archive as it grows, similar to 'tail -f' > * switching to an arbitrary active pmcd > > Until we all agree on what the ultimate goal is here (and perhaps I'm the only > one who is still not sure), it would seem unlikely that we can agree on an > architecture or design. So please, correct me where I am wrong and fill in the > blanks where I have missed them. > I still see these as disjoint and independent pieces of work. There is a use case for both at the same time, but I think this is likely to be a less common use case. The more common use cases are - historical analysis over a set of archives that are _not_ being actively written - live analysis where you want some (small) history from one archive that is being written to provide context for on-going live reporting Even in the less common case of combining the two, they are still disjoint ... the transition from archive to live involves only ONE archive. So we could proceed with the archive set work which is better understood and not subject to architectural discussion, and independently focus on the architectural design for the transition piece which is where there is less agreement at the moment. From mgoodwin@redhat.com Sun Oct 5 21:27:18 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 2E3A57F4E for ; Sun, 5 Oct 2014 21:27:18 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id A23B4AC003 for ; Sun, 5 Oct 2014 19:27:17 -0700 (PDT) X-ASG-Debug-ID: 1412562432-04cb6c50e7705cf0001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id mfWBmGQ9GFruIHGI (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Sun, 05 Oct 2014 19:27:12 -0700 (PDT) X-Barracuda-Envelope-From: mgoodwin@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s962RBnf023883 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Sun, 5 Oct 2014 22:27:12 -0400 Received: from [10.64.49.225] (vpn1-49-225.bne.redhat.com [10.64.49.225]) by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s962R9BQ008708; Sun, 5 Oct 2014 22:27:10 -0400 Message-ID: <5431FDFD.6020208@redhat.com> Date: Mon, 06 Oct 2014 13:27:09 +1100 From: Mark Goodwin User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: "Frank Ch. Eigler" , pcp developers Subject: Re: [pcp] PMAPI observations re. converting an app to pcp References: <20140925180407.GA18679@redhat.com> X-ASG-Orig-Subj: Re: [pcp] PMAPI observations re. converting an app to pcp In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412562432 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On 10/04/2014 07:38 AM, Frank Ch. Eigler wrote: [...] > First of all, what do you think of the general flavour? Amazingly coincidental choice of names, check out : git://oss.sgi.com/markgw/pcp/pcp-new-open-source.git src/libpcp_mon/src/fetchgroup.[ch] which I wrote way back in 1995 (!) as part of the original pmchart - in response to exactly the same API wordiness issues that you've encountered. We separated out the fetchGroup API and put it in libpcp_mon. You can probably reuse some or all of that code, it's really well debugged. The original concept is similar to your new proposal in that it juggles multiple contexts for you (multiple live or archive, including multiple archives for the same host, but not live and archive simultaneously), and plonks values directly into your application's data structures on each fetch, using pointers supplied during setup of each fetch group. > Second, do you endorse the type-safe nature of passing float* > etc. destinations for the metric values, tolerating N sibling > foo_TYPE() functions, instead of something icky like the pmAtomValue > unions or void pointers? yes definitely, though you could take it further and canonicalize on just double (or even long double for better precision), string and aggregate (and event?). > Third, I haven't fully sketched out what to do with > PM_TYPE_{STRING,AGGREGATE}; probably have the user pass a destination > buffer/length, or else have libpcp allocate, or (why not) both as > alternatives. Fine? yep, that would work just fine as well, provided everyone is clear on where the alloc and free need to occur. > Fourth, I don't know what to do with events. They're fiendishly > complicated in the pmValue structures, and would have a whale of a > time encoding its recursive subdivision as one function call. Maybe > skip it? Maybe some nasty varargs monstrosity? hmm, we didn't have that problem back in 1995 :) Just treat it in a similar way - copy the event into a user supplied buffer .. something like that, basically a special aggregate type. > Fifth, it is entirely possible for values to come back absent from a > pmFetch, and thus a pmFetchGroup. Encoding the absence of data is > probably necessary, but I'm not sure whether better to do it with a > formal flag (another bool* argument at the *Extend* functions), or > maybe just use sentinel values (0, NaN) on the output. Or both? the original fetchGroupFetch has integrated error handling, with tri-state error results. See comments in the code. This worked well, but did require the application to check for errors before dereferencing the returned values. > Sixth, not sure how/whether to represent indom profiles. Maybe not > needed; just have the user invoke the non-_indom variant of the > *Extend* functions, enumerating the instances and places where to put > each's data. Or maybe have a variant of the _indom_ functions where > the instance names are already provided? indom profiles are optimized via the optFetch API that Ken wrote, circa the same time period. Some regex instance name matching in the API could be handy too though ... > Seventh, several of the above represent optionalities that could lead > to a cartesian-product-style explosion of the number of individual > functions in the API. I don't mind that FWIW, but someone might. as above, canonicalize on double, string and aggregate; after all, most stuff we're interested in will be rate converted anyway, and and so end up as a double result for the application to report. The exception could be metrics with discrete semantics, maybe. > > Eigth, aw heck, why not say that *Extend* with pmfg index #0 means > "do it right now", i.e., the equivalent of > tmp = pmCreateFetchGroup() > pmExtendFetchGroup(tmp, .....) > pmFetchGroup(tmp) > pmDestroyFetchGroup(tmp) > What could be simpler for one-shot inquiries? wrap that up in a function, e.g. ncpu = fetchGroupGetValue(source, time, "hinv.ncpu") where time is NOW (live) or some archive position. > > Ninth, this should apply OK to archives too (for consecutive fetches); > should we bother wrap pmSetMode, or just let the client call it > directly before pmFetchGroup()? pmFetchGroupArchiveMode(int mode, const struct timeval *when, int interval) in the old implementation seemed to work well - it walks all contexts and sets the current time (aka position), sampling interval, replay direction etc. Similarly for archive bounds, so you could have multiple archives open and set the bounds you're interested in for all of them. The API then picks which archive context to fetch from based on the current time, source, metric and instance. > > Feedback welcome! there you go, and half the work's already done :) Cheers -- Mark From nscott@redhat.com Mon Oct 6 00:52:21 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 928C37F4E for ; Mon, 6 Oct 2014 00:52:21 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 62C2D30404E for ; Sun, 5 Oct 2014 22:52:18 -0700 (PDT) X-ASG-Debug-ID: 1412574733-04bdf003a1794a40001-S8gJnT Received: from mx6-phx2.redhat.com (mx6-phx2.redhat.com [209.132.183.39]) by cuda.sgi.com with ESMTP id BdEQ2zFamcQ0N5Zc (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Sun, 05 Oct 2014 22:52:13 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.39 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx6-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s965q9F9013239; Mon, 6 Oct 2014 01:52:09 -0400 Date: Mon, 6 Oct 2014 01:52:08 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Ken McDonell , Dave Brolley Cc: pcp@oss.sgi.com Message-ID: <510673095.61694614.1412574728270.JavaMail.zimbra@redhat.com> In-Reply-To: <009301cfe013$1490d430$3db27c90$@internode.on.net> References: <542C21AE.1010504@redhat.com> <542EFB14.6020208@redhat.com> <009301cfe013$1490d430$3db27c90$@internode.on.net> Subject: Re: [pcp] Multi-Volume Archive + Live Data Playback for PCP Client Tools MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] Multi-Volume Archive + Live Data Playback for PCP Client Tools Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.6] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: Multi-Volume Archive + Live Data Playback for PCP Client Tools Thread-Index: AQHclF1cpR2XxHkXIjv6In84hym83AGxytyom/lqutBAbNOMYQ== X-Barracuda-Connect: mx6-phx2.redhat.com[209.132.183.39] X-Barracuda-Start-Time: 1412574733 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10251 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... ----- Original Message ----- > > -----Original Message----- > > [...] > > It now appears to me (and once again, please correct me if I am wrong!) > that > > the goal is for clients to be able to examine metrics from multiple > archives, > > each of which may have multiple volumes (already supported) and each of > > which may be arbitrarily distinct (i.e. not necessarily created by the > same > > pmlogger or even on the same machine). > > Yep, but all the archives have contain data collected from the same host ... > different hosts require different contexts and different archives (or > different sets of archives) *nod* > > Until we all agree on what the ultimate goal is here (and perhaps I'm the > only > > one who is still not sure), it would seem unlikely that we can agree on an I think there's still lots of uncertainty in this area, you're definitely not the only one who is unsure Dave. > I still see these as disjoint and independent pieces of work. There is a > use case for both at the same time, but I think this is likely to be a less > common use case. The more common use cases are > - historical analysis over a set of archives that are _not_ being actively > written > - live analysis where you want some (small) history from one archive that is > being written to provide context for on-going live reporting *nod* - those are consistent with the requirements I've seen in production use. But, I think Frank has a good point re auto-logging ... it'd be nice to be able to allow clients to optionally/transparently do this (and doing a better job than pmchart of today). The cost of a turn-pmlogger-into-pmcd approach is too high IMO (refactoring large amounts of pmcd and pmlogger, new protocols, high complexity factor, inefficient-by-design, enforced-use, and so on) - however, I'm sure we can devise more effective ways of reaching that goal. This too I see as an orthogonal, follow-up kind of project - something that is above and beyond even the original unified context concept. Perhaps its more like the caching context concept - an optionally-recorded live context. > Even in the less common case of combining the two, they are still disjoint > ... the transition from archive to live involves only ONE archive. (could be more than one I think - it'd be good to be able to merge system- logged data ala pmlogger_daily(1), and personal-logged data ala pmchart(1) into one stream; both of those sources would be independently growing). > So we could proceed with the archive set work which is better understood and > not subject to architectural discussion, and independently focus on the > architectural design for the transition piece which is where there is less > agreement at the moment. *nod* ... there are of course many unknown aspects to multi-archive support, at this early stage (all the below-the-surface technical aspects). Moving the focus there would be good - but I also agree with Dave's earlier comment that its worth thinking about future paths, to not inadvertently make life more difficult for the next stages (e.g. maybe stacking the archive contexts will help us later too, rather than simply expanding __pmLogCtl/__pmArchCtl at this early stage). cheers. -- Nathan From kenj@internode.on.net Mon Oct 6 05:42:10 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 7FF1D7F4E for ; Mon, 6 Oct 2014 05:42:10 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 6E8838F8035 for ; Mon, 6 Oct 2014 03:42:10 -0700 (PDT) X-ASG-Debug-ID: 1412592124-04cb6c50e4710240001-S8gJnT Received: from ipmail05.adl6.internode.on.net (ipmail05.adl6.internode.on.net [150.101.137.143]) by cuda.sgi.com with ESMTP id ajDe2K04oRlepJ1K for ; Mon, 06 Oct 2014 03:42:05 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.143 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AlAOAE9xMlR20Xt7PGdsb2JhbABfgw6DXYUHyQ+DGwSBCBcBBgEBAQE4OYQKCAIeEhwwBQZiIAoVAQQeBYgtnCykA5BihDUFkXRcojUBgi8pgnkBAQE Received: from ppp118-209-123-123.lns20.mel4.internode.on.net (HELO bozohorize) ([118.209.123.123]) by ipmail05.adl6.internode.on.net with ESMTP; 06 Oct 2014 21:12:02 +1030 From: "Ken McDonell" To: Subject: qa/946 - pmfind question Date: Mon, 6 Oct 2014 21:41:59 +1100 X-ASG-Orig-Subj: qa/946 - pmfind question Message-ID: <001801cfe152$29472510$7bd56f30$@internode.on.net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 15.0 Thread-Index: Ac/hUaE+sxBTtDioRei7WOw36Y7xYA== Content-Language: en-au X-Barracuda-Connect: ipmail05.adl6.internode.on.net[150.101.137.143] X-Barracuda-Start-Time: 1412592124 X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.01 X-Barracuda-Spam-Status: No, SCORE=0.01 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10256 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== This is a Centos 7 system, with avahi-daemon running kenj@vm08:~/src/pcp/qa$ ps -ef | grep avahi avahi 1020 1 0 21:35 ? 00:00:00 avahi-daemon: running [vm08.local] avahi 1021 1020 0 21:35 ? 00:00:00 avahi-daemon: chroot helper kenj 1104 31976 0 21:38 pts/1 00:00:00 grep --color=auto avahi and pmconfig -L shows service discovery is enabled kenj@vm08:~/src/pcp/qa$ pmconfig -L | grep discovery service_discovery=true pmcd is up kenj@vm08:~/src/pcp/qa$ pcp Performance Co-Pilot configuration on vm08.localdomain: platform: Linux vm08.localdomain 3.10.0-123.6.3.el7.x86_64 #1 SMP Wed Aug 6 21:12:36 UTC 2014 x86_64 hardware: 1 cpu, 1 disk, 1 node, 994MB RAM timezone: AEDT-11 services: pmcd pmcd: Version 3.10.0-5, 9 agents, 1 client pmda: pmcd proc xfs sample sampledso linux mmv jbd2 simple pmlogger: primary logger: /var/log/pcp/pmlogger/vm08.localdomain/20141006.19.43 but pmfind -m avahi fails, and so does qa/946 kenj@vm08:~/src/pcp/qa$ pmfind -m avahi No pmcd servers discovered No pmproxy servers discovered No pmwebd servers discovered I'd appreciate any clues that would help debug or explain this. From fche@redhat.com Mon Oct 6 09:54:31 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 3A1EC7F50 for ; Mon, 6 Oct 2014 09:54:31 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id C8A2AAC001 for ; Mon, 6 Oct 2014 07:54:30 -0700 (PDT) X-ASG-Debug-ID: 1412607268-04cbb073047a7590001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id xmkGVYpYTbdlWsSF (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Mon, 06 Oct 2014 07:54:29 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s96EsOOx023792 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 6 Oct 2014 10:54:25 -0400 Received: from fche.csb (vpn-233-63.phx2.redhat.com [10.3.233.63]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s96EsNaL007570; Mon, 6 Oct 2014 10:54:23 -0400 Received: by fche.csb (Postfix, from userid 2569) id DB76058197; Mon, 6 Oct 2014 10:54:22 -0400 (EDT) To: "Ken McDonell" Cc: Subject: Re: qa/946 - pmfind question References: <001801cfe152$29472510$7bd56f30$@internode.on.net> X-ASG-Orig-Subj: Re: qa/946 - pmfind question From: fche@redhat.com (Frank Ch. Eigler) Date: Mon, 06 Oct 2014 10:54:22 -0400 In-Reply-To: <001801cfe152$29472510$7bd56f30$@internode.on.net> (Ken McDonell's message of "Mon, 6 Oct 2014 21:41:59 +1100") Message-ID: User-Agent: Gnus/5.1008 (Gnus v5.10.8) Emacs/21.4 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412607269 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 kenj wrote: > This is a Centos 7 system, with avahi-daemon running > [...] > kenj@vm08:~/src/pcp/qa$ pmconfig -L | grep discovery > service_discovery=true > [...] > but pmfind -m avahi fails, and so does qa/946 > [...] > I'd appreciate any clues that would help debug or explain this. OK, a few things to check: % ... the pmcd.log file % avahi-browse -r -t _pmcd._tcp # service avahi-daemon restart # service pmcd restart % avahi-browse -r -t _pmcd._tcp % avahi-browse -a -t # journalctl UNIT=avahi-daemon.service - FChE From brolley@redhat.com Mon Oct 6 09:58:08 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 6EF2A7F50 for ; Mon, 6 Oct 2014 09:58:08 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 3EF8E8F8035 for ; Mon, 6 Oct 2014 07:58:05 -0700 (PDT) X-ASG-Debug-ID: 1412607483-04bdf0039f7a7470001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id T5cfStKPEcSVGAG7 (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Mon, 06 Oct 2014 07:58:04 -0700 (PDT) X-Barracuda-Envelope-From: brolley@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s96Ew0bF016964 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 6 Oct 2014 10:58:01 -0400 Received: from [10.15.16.139] (dhcp-10-15-16-139.yyz.redhat.com [10.15.16.139]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s96Ew0Wu010606; Mon, 6 Oct 2014 10:58:00 -0400 Message-ID: <5432AE59.8070309@redhat.com> Date: Mon, 06 Oct 2014 10:59:37 -0400 From: Dave Brolley User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: Ken McDonell , pcp@oss.sgi.com Subject: Re: [pcp] qa/946 - pmfind question References: <001801cfe152$29472510$7bd56f30$@internode.on.net> X-ASG-Orig-Subj: Re: [pcp] qa/946 - pmfind question In-Reply-To: <001801cfe152$29472510$7bd56f30$@internode.on.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412607484 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On 10/06/2014 06:41 AM, Ken McDonell wrote: > This is a Centos 7 system, with avahi-daemon running [ ... ] > pmcd is up [ ... ] > but pmfind -m avahi fails, and so does qa/946 > > kenj@vm08:~/src/pcp/qa$ pmfind -m avahi > No pmcd servers discovered > No pmproxy servers discovered > No pmwebd servers discovered > > I'd appreciate any clues that would help debug or explain this. > First possible problem is that avahi-devel was not available when PCP was configured/built. Check your configure logs for problems finding the needed avahi components. However, I would expect pmfind(1) report that avahi is not supported in this case. Next place to look is the pmcd log. If it failed to register itself with avahi-daemon, then there should be some messages in the log. The next, and most common cause of avahi-related problems is that the mDNS port, udp 5353, is not open in the firewall. Try avahi-browse -rt _pmcd._tcp If that works, then pmfind should also work with avahi. avahi-browse is part of the avahi-tools package. I it looks like qa/646 checks that avahi-daemon is ok and checks that service discovery is supported. We should probably create another pmconfig(1) category, something like 'service_discovery_avahi' which could be used to catch the first two problems above. I'm not sure how to portably check that a port is open in the firewall and I'm not sure that we can depend on avahi-tools being installed for checeking with avahi-browse. I hope this helps, Dave From fche@redhat.com Mon Oct 6 12:38:48 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 62F957F50 for ; Mon, 6 Oct 2014 12:38:48 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 346508F8050 for ; Mon, 6 Oct 2014 10:38:44 -0700 (PDT) X-ASG-Debug-ID: 1412617120-04bdf003a17b1360001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id PVfxecpiapVIMnhJ (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Mon, 06 Oct 2014 10:38:40 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s96Hcev2023213 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Mon, 6 Oct 2014 13:38:40 -0400 Received: from fche.csb (vpn-233-63.phx2.redhat.com [10.3.233.63]) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s96Hcedb007823; Mon, 6 Oct 2014 13:38:40 -0400 Received: by fche.csb (Postfix, from userid 2569) id 7F9EE58197; Mon, 6 Oct 2014 13:38:39 -0400 (EDT) To: Mark Goodwin Cc: pcp developers Subject: Re: PMAPI observations re. converting an app to pcp References: <20140925180407.GA18679@redhat.com> <5431FDFD.6020208@redhat.com> X-ASG-Orig-Subj: Re: PMAPI observations re. converting an app to pcp From: fche@redhat.com (Frank Ch. Eigler) Date: Mon, 06 Oct 2014 13:38:39 -0400 In-Reply-To: <5431FDFD.6020208@redhat.com> (Mark Goodwin's message of "Mon, 06 Oct 2014 13:27:09 +1100") Message-ID: User-Agent: Gnus/5.1008 (Gnus v5.10.8) Emacs/21.4 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412617120 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Hi, Mark - mgoodwin wrote: > [...] > Amazingly coincidental choice of names, check out : > git://oss.sgi.com/markgw/pcp/pcp-new-open-source.git > src/libpcp_mon/src/fetchgroup.[ch] > > which I wrote way back in 1995 (!) as part of the original pmchart - in > response to exactly the same API wordiness issues that you've encountered. Funny not-entirely-coincidence. Thanks for the pointer to the code. >> Second, do you endorse the type-safe nature of passing float* >> etc. destinations for the metric values [...] > > yes definitely, though you could take it further and canonicalize > on just double (or even long double for better precision), string > and aggregate (and event?). Yeah, that would make sense (because representing moderate-sized integers in float/doubles is lossless). > [...] >> Fourth, I don't know what to do with events. They're fiendishly >> complicated in the pmValue structures, and would have a whale of a >> time encoding its recursive subdivision as one function call. Maybe >> skip it? Maybe some nasty varargs monstrosity? > > hmm, we didn't have that problem back in 1995 :) Just treat it in a > similar way - copy the event into a user supplied buffer .. something > like that, basically a special aggregate type. It's not that easy though, since fields of an event tuple are N individual atom values with distinct types/etc. I'm leaning to punting for now. > [...] > there you go, and half the work's already done :) Thanks, I'll study your code and steal lots of bits. I'm leaning toward a little simpler solution in some ways. For example, how about just use one pcp context per "fetch-group"? That would make it less necessary to have special time-setting or profile-manipulation wrappers, since it would compose with the normal PMAPI functions. - FChE From kenj@internode.on.net Mon Oct 6 14:48:52 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id D06B77F51 for ; Mon, 6 Oct 2014 14:48:52 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 5E872AC004 for ; Mon, 6 Oct 2014 12:48:49 -0700 (PDT) X-ASG-Debug-ID: 1412624923-04cbb073047ba000001-S8gJnT Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by cuda.sgi.com with ESMTP id EMPVLeOZQn9L7SsN for ; Mon, 06 Oct 2014 12:48:43 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.145 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgUDALzxMlR20Xt7PGdsb2JhbAANUotyyReDIQKBHAEGAQEBATiEPQEBBHgBEAsYCRYPCQMCAQIBMRQGDQEHAQG0DZVbAReQRQeESwEEj1qnW4MiAQEB Received: from ppp118-209-123-123.lns20.mel4.internode.on.net (HELO [192.168.1.100]) ([118.209.123.123]) by ipmail06.adl6.internode.on.net with ESMTP; 07 Oct 2014 06:18:13 +1030 Message-ID: <5432F213.2080209@internode.on.net> Date: Tue, 07 Oct 2014 06:48:35 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: "Frank Ch. Eigler" CC: pcp@oss.sgi.com Subject: Re: qa/946 - pmfind question References: <001801cfe152$29472510$7bd56f30$@internode.on.net> X-ASG-Orig-Subj: Re: qa/946 - pmfind question In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl6.internode.on.net[150.101.137.145] X-Barracuda-Start-Time: 1412624923 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10271 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header Thanks Frank. Info you requested is below, but I'm no closer. On 07/10/14 01:54, Frank Ch. Eigler wrote: > > OK, a few things to check: > > % ... the pmcd.log file grep -i avahi reveals ... [Mon Oct 6 21:35:47] pmcd(25510) Error: Avahi client failure: Daemon connection failed but I don't know what that means. > % avahi-browse -r -t _pmcd._tcp avahi-browse is not installed, punt on this being in the avahi-tools package, install it, run it, no output > # service avahi-daemon restart > # service pmcd restart > % avahi-browse -r -t _pmcd._tcp > % avahi-browse -a -t still no output > # journalctl UNIT=avahi-daemon.service -- Logs begin at Tue 2014-10-07 02:05:15 AEDT, end at Tue 2014-10-07 06:46:52 AE Oct 07 02:05:32 vm08.localdomain systemd[1]: Starting Avahi mDNS/DNS-SD Stack... Oct 07 02:05:39 vm08.localdomain systemd[1]: Started Avahi mDNS/DNS-SD Stack. Oct 06 21:35:47 vm08.localdomain systemd[1]: Stopping Avahi mDNS/DNS-SD Stack... Oct 06 21:35:47 vm08.localdomain systemd[1]: Starting Avahi mDNS/DNS-SD Stack... Oct 06 21:35:47 vm08.localdomain systemd[1]: Starting Avahi mDNS/DNS-SD Stack... Oct 06 21:35:47 vm08.localdomain systemd[1]: Started Avahi mDNS/DNS-SD Stack. Oct 06 21:36:07 vm08.localdomain systemd[1]: Started Avahi mDNS/DNS-SD Stack. Oct 07 06:46:11 vm08.localdomain systemd[1]: Stopping Avahi mDNS/DNS-SD Stack... Oct 07 06:46:11 vm08.localdomain systemd[1]: Starting Avahi mDNS/DNS-SD Stack... Oct 07 06:46:11 vm08.localdomain systemd[1]: Started Avahi mDNS/DNS-SD Stack. From dsmith@redhat.com Mon Oct 6 14:50:29 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id C7A707F51 for ; Mon, 6 Oct 2014 14:50:29 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 54CBBAC004 for ; Mon, 6 Oct 2014 12:50:29 -0700 (PDT) X-ASG-Debug-ID: 1412625027-04cbb073037ba170001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id aaVNp11ambKU5GTF (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Mon, 06 Oct 2014 12:50:28 -0700 (PDT) X-Barracuda-Envelope-From: dsmith@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s96JoR3M005744 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Mon, 6 Oct 2014 15:50:27 -0400 Received: from t540p.usersys.redhat.com (vpn-59-145.rdu2.redhat.com [10.10.59.145]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s96JoOs4014723 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Mon, 6 Oct 2014 15:50:27 -0400 Message-ID: <5432F280.20008@redhat.com> Date: Mon, 06 Oct 2014 14:50:24 -0500 From: David Smith User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.1 MIME-Version: 1.0 To: Nathan Scott CC: pcp Subject: Re: [pcp] [PATCH] Fix memory corruption in python support References: <542D6626.50703@redhat.com> <336785285.60750380.1412287401855.JavaMail.zimbra@redhat.com> X-ASG-Orig-Subj: Re: [pcp] [PATCH] Fix memory corruption in python support In-Reply-To: <336785285.60750380.1412287401855.JavaMail.zimbra@redhat.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412625028 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On 10/02/2014 05:03 PM, Nathan Scott wrote: > Hi David, > > Can we distil that memory corruption into a reproducible test case? We > have several tests exercising various aspects of the PMDA wrapper but they > are not triggering this problem - it'd be good to come up with something > (e.g. minimal python script?) that demonstrates the problem and which we > can use to verify the fix. On the new pcpfans/dsmith/dev branch, there are 2 new commits: - 44cb24d: This adds a new test (called '843') that can catch this bug. Note that I based this test on an existing test, but I'm still not 100% sure I updated the testsuite machinery properly. Also note that this new test only tests the changes to src/python/pmda.c. I"m not really sure how to test the changes to src/python/pmapi.c. - 8e62465: This is the the python fix itself (that I posted to the list earlier). -- David Smith dsmith@redhat.com Red Hat http://www.redhat.com 256.217.0141 (direct) 256.837.0057 (fax) From kenj@internode.on.net Mon Oct 6 14:56:25 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 8398B7F51 for ; Mon, 6 Oct 2014 14:56:25 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 70B5F8F8035 for ; Mon, 6 Oct 2014 12:56:25 -0700 (PDT) X-ASG-Debug-ID: 1412625382-04cb6c50e772dae0001-S8gJnT Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by cuda.sgi.com with ESMTP id FcgpXPulc1zaFkat for ; Mon, 06 Oct 2014 12:56:22 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.145 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgYDAPHyMlR20Xt7PGdsb2JhbAANUotyyReDIQKBHAEGAQEBATiEPAEBAQMBOEAGCwsYCRYPCQMCAQIBMRQGAQwIAQGIMqtblVoBF490AQFWhEsBBJ5rkFSHdoFngTsBAQE Received: from ppp118-209-123-123.lns20.mel4.internode.on.net (HELO [192.168.1.100]) ([118.209.123.123]) by ipmail06.adl6.internode.on.net with ESMTP; 07 Oct 2014 06:26:21 +1030 Message-ID: <5432F3FC.2060304@internode.on.net> Date: Tue, 07 Oct 2014 06:56:44 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: Dave Brolley , pcp@oss.sgi.com Subject: Re: [pcp] qa/946 - pmfind question References: <001801cfe152$29472510$7bd56f30$@internode.on.net> <5432AE59.8070309@redhat.com> X-ASG-Orig-Subj: Re: [pcp] qa/946 - pmfind question In-Reply-To: <5432AE59.8070309@redhat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl6.internode.on.net[150.101.137.145] X-Barracuda-Start-Time: 1412625382 X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10271 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- On 07/10/14 01:59, Dave Brolley wrote: > ... > First possible problem is that avahi-devel was not available when PCP > was configured/built. Check your configure logs for problems finding the > needed avahi components. However, I would expect pmfind(1) report that > avahi is not supported in this case. avahi-devel was and is installed. > Next place to look is the pmcd log. If it failed to register itself with > avahi-daemon, then there should be some messages in the log. As per last mail to Frank, the registration apparently did not work, but I don't know why ... more strangely, there is no avahi message in pmcd's log file after I restarted pmcd. > The next, and most common cause of avahi-related problems is that the > mDNS port, udp 5353, is not open in the firewall. Try > > avahi-browse -rt _pmcd._tcp That returns nothing. But there may be a disconnection of understanding here. I am expecting that pmfind will find pmcd on the local host which should not involve the firewall (or rather the firewall should not get in the way). If pmfind is only looking externally, then the QA test is pathologically broken ... it will only work in an environment where some _other_ host is also avahi-enabled and accessible via multi-cast udp packets. > If that works, then pmfind should also work with avahi. avahi-browse is > part of the avahi-tools package. > > I it looks like qa/646 checks that avahi-daemon is ok and checks that > service discovery is supported. We should probably create another > pmconfig(1) category, something like 'service_discovery_avahi' which > could be used to catch the first two problems above. I'm not sure how to > portably check that a port is open in the firewall and I'm not sure that > we can depend on avahi-tools being installed for checeking with > avahi-browse. > > I hope this helps, It helps move my understanding along ... it does not help getting the qa test passing I'm afraid. From fche@redhat.com Mon Oct 6 15:03:02 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 040177F51 for ; Mon, 6 Oct 2014 15:03:02 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 8678CAC002 for ; Mon, 6 Oct 2014 13:02:58 -0700 (PDT) X-ASG-Debug-ID: 1412625776-04cbb073037bad50001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id GmceiEyJYc0e3kth (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Mon, 06 Oct 2014 13:02:57 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s96K2rlu013340 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 6 Oct 2014 16:02:53 -0400 Received: from fche.csb (vpn-233-63.phx2.redhat.com [10.3.233.63]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s96K2rQ1020718; Mon, 6 Oct 2014 16:02:53 -0400 Received: by fche.csb (Postfix, from userid 2569) id 9280958197; Mon, 6 Oct 2014 16:02:52 -0400 (EDT) Date: Mon, 6 Oct 2014 16:02:52 -0400 From: "Frank Ch. Eigler" To: Ken McDonell Cc: pcp@oss.sgi.com Subject: Re: qa/946 - pmfind question Message-ID: <20141006200252.GD3226@redhat.com> X-ASG-Orig-Subj: Re: qa/946 - pmfind question References: <001801cfe152$29472510$7bd56f30$@internode.on.net> <5432F213.2080209@internode.on.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5432F213.2080209@internode.on.net> User-Agent: Mutt/1.4.2.2i X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412625777 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Hi, Ken - > Info you requested is below, but I'm no closer. OK, thanks. > > % ... the pmcd.log file > > grep -i avahi reveals ... > [Mon Oct 6 21:35:47] pmcd(25510) Error: Avahi client failure: Daemon connection failed > but I don't know what that means. That means that pmcd (an avahi client) was unable to announce its existence (by communicating with the avahi daemon). (I believe the client library uses dbus to talk to the local daemon ... has the machine suffered OOM or something lately? Considered a reboot?) > > # service avahi-daemon restart > > # service pmcd restart > > % avahi-browse -r -t _pmcd._tcp > > % avahi-browse -a -t > > still no output And a new copy of the daemon-connection-failed message in pmcd.log? Something must be wrong with the local avahi-daemon server installation. ("avahi-browse -a -t" should list some items for the local host, not just pmcd.) - FChE From kenj@internode.on.net Mon Oct 6 16:17:58 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 234277F4E for ; Mon, 6 Oct 2014 16:17:58 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id E7FBF304032 for ; Mon, 6 Oct 2014 14:17:57 -0700 (PDT) X-ASG-Debug-ID: 1412630272-04bdf003a07c03d0001-S8gJnT Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by cuda.sgi.com with ESMTP id C717htApfbG1cJjB for ; Mon, 06 Oct 2014 14:17:52 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.145 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgYDAOQFM1R20Xt7PGdsb2JhbAANUotyyReDIQKBHQEGAQEBATiEPAEBAQMBOEABBQsLGAkWDwkDAgECATEUBg0BBwEBiDKrapVqAReQRQeESwEEtzWDIgEBAQ Received: from ppp118-209-123-123.lns20.mel4.internode.on.net (HELO [192.168.1.100]) ([118.209.123.123]) by ipmail06.adl6.internode.on.net with ESMTP; 07 Oct 2014 07:47:52 +1030 Message-ID: <54330715.8050401@internode.on.net> Date: Tue, 07 Oct 2014 08:18:13 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: "Frank Ch. Eigler" CC: pcp@oss.sgi.com Subject: Re: qa/946 - pmfind question References: <001801cfe152$29472510$7bd56f30$@internode.on.net> <5432F213.2080209@internode.on.net> <20141006200252.GD3226@redhat.com> X-ASG-Orig-Subj: Re: qa/946 - pmfind question In-Reply-To: <20141006200252.GD3226@redhat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl6.internode.on.net[150.101.137.145] X-Barracuda-Start-Time: 1412630272 X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10273 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On 07/10/14 07:02, Frank Ch. Eigler wrote: > ... has the > machine suffered OOM or something lately? Considered a reboot?) No. And reboot does not change anything that I can see. > >>> # service avahi-daemon restart >>> # service pmcd restart >>> % avahi-browse -r -t _pmcd._tcp >>> % avahi-browse -a -t >> >> still no output > > And a new copy of the daemon-connection-failed message in pmcd.log? No. I saw the pmcd.log message once only. > Something must be wrong with the local avahi-daemon server > installation. ("avahi-browse -a -t" should list some items for the > local host, not just pmcd.) Nope, nothing listed. A quick diversion into googleland suggests that avahi is very picky about the DNS setup (the domain .local seems magic) ... I am giving up on this for now unless someone can suggest a quick fix. This test is passing for me on other RH-based systems (like FC) without any special or different setup, so I assume that avahi out of the box is just broken on Centos 7. From wwwrun@oss.sgi.com Mon Oct 6 16:29:05 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE,NO_RELAYS autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: by oss.sgi.com (Postfix, from userid 30) id 6CA9D7F52; Mon, 6 Oct 2014 16:29:05 -0500 (CDT) From: bugzilla-daemon@oss.sgi.com To: pcp@oss.sgi.com Subject: [Bug 1068] New: pmUnitsStr_r can clobber memory Date: Mon, 06 Oct 2014 21:29:05 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Classification: Unclassified X-Bugzilla-Product: pcp X-Bugzilla-Component: pcp X-Bugzilla-Keywords: X-Bugzilla-Severity: major X-Bugzilla-Who: fche@redhat.com X-Bugzilla-Status: NEW X-Bugzilla-Priority: P5 X-Bugzilla-Assigned-To: pcp@kenj.com.au X-Bugzilla-Target-Milestone: --- X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter cc classification Message-ID: Content-Type: multipart/alternative; boundary="1412630945.72Dce1.1019"; charset="us-ascii" X-Bugzilla-URL: http://oss.sgi.com/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 --1412630945.72Dce1.1019 Date: Mon, 6 Oct 2014 16:29:05 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" http://oss.sgi.com/bugzilla/show_bug.cgi?id=1068 Bug ID: 1068 Summary: pmUnitsStr_r can clobber memory Product: pcp Version: unspecified Hardware: All OS: Linux Status: NEW Severity: major Priority: P5 Component: pcp Assignee: pcp@kenj.com.au Reporter: fche@redhat.com CC: pcp@oss.sgi.com Classification: Unclassified The way pmUnitsStr_r accumulates text fragments to create the complete units text string ignores the incoming string's buffer length to some extent and can slightly smash the stack. % cat foo.c #include #include #ifndef SIZE #define SIZE 1 #endif char buffer[SIZE]; void main() { pmUnits foo = {.dimSpace=7, .dimTime=-8, .dimCount=7, .scaleSpace=PM_SPACE_TBYTE, .scaleTime=PM_TIME_USEC, .scaleCount=7 }; (void) pmUnitsStr_r(& foo, buffer, sizeof(buffer)); printf ("%*s\n", sizeof(buffer), buffer); } % gcc foo.c -lpcp % ./a.out / microsec^8 % valgring ./a.out *** does not complain (on my f19 x86-64 box) (just for reference:) % gcc foo.c -lpcp -DSIZE=40 % ./a.out Tbyte^7 count x 10^7^7 / microsec^8 (that ^7^7 part looks fishy too, btw). -- You are receiving this mail because: You are on the CC list for the bug. --1412630945.72Dce1.1019 Date: Mon, 6 Oct 2014 16:29:05 -0500 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8"
Bug ID 1068
Summary pmUnitsStr_r can clobber memory
Product pcp
Version unspecified
Hardware All
OS Linux
Status NEW
Severity major
Priority P5
Component pcp
Assignee pcp@kenj.com.au
Reporter fche@redhat.com
CC pcp@oss.sgi.com
Classification Unclassified 

The way pmUnitsStr_r accumulates text fragments to create
the complete units text string ignores the incoming
string's buffer length to some extent and can
slightly smash the stack.

% cat foo.c

#include <pcp/pmapi.h>
#include <stdio.h>
#ifndef SIZE
#define SIZE 1
#endif
char buffer[SIZE];

void main() {
  pmUnits foo = {.dimSpace=7, .dimTime=-8, .dimCount=7,
                 .scaleSpace=PM_SPACE_TBYTE, .scaleTime=PM_TIME_USEC,
.scaleCount=7 };
  (void) pmUnitsStr_r(& foo, buffer, sizeof(buffer));
  printf ("%*s\n", sizeof(buffer), buffer);
}

% gcc foo.c -lpcp
% ./a.out
  / microsec^8
% valgring ./a.out
*** does not complain (on my f19 x86-64 box)

(just for reference:)
% gcc foo.c -lpcp -DSIZE=40
% ./a.out
Tbyte^7 count x 10^7^7 / microsec^8

(that ^7^7 part looks fishy too, btw).

You are receiving this mail because:
  • You are on the CC list for the bug.
--1412630945.72Dce1.1019-- From brolley@redhat.com Mon Oct 6 16:56:18 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 9FA647F4E for ; Mon, 6 Oct 2014 16:56:18 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 8BCB5304039 for ; Mon, 6 Oct 2014 14:56:18 -0700 (PDT) X-ASG-Debug-ID: 1412632577-04cb6c50e5734780001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id u0095q60bee6rAiB (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Mon, 06 Oct 2014 14:56:17 -0700 (PDT) X-Barracuda-Envelope-From: brolley@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s96LuC8t018874 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 6 Oct 2014 17:56:12 -0400 Received: from [10.15.16.139] (dhcp-10-15-16-139.yyz.redhat.com [10.15.16.139]) by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s96LuB6K011659; Mon, 6 Oct 2014 17:56:11 -0400 Message-ID: <5433105D.7080302@redhat.com> Date: Mon, 06 Oct 2014 17:57:49 -0400 From: Dave Brolley User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: Ken McDonell , pcp@oss.sgi.com Subject: Re: [pcp] qa/946 - pmfind question References: <001801cfe152$29472510$7bd56f30$@internode.on.net> <5432AE59.8070309@redhat.com> <5432F3FC.2060304@internode.on.net> X-ASG-Orig-Subj: Re: [pcp] qa/946 - pmfind question In-Reply-To: <5432F3FC.2060304@internode.on.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412632577 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On 10/06/2014 03:56 PM, Ken McDonell wrote: > >> The next, and most common cause of avahi-related problems is that the >> mDNS port, udp 5353, is not open in the firewall. Try >> >> avahi-browse -rt _pmcd._tcp > > That returns nothing. But there may be a disconnection of > understanding here. I am expecting that pmfind will find pmcd on the > local host which should not involve the firewall (or rather the > firewall should not get in the way). If pmfind is only looking > externally, then the QA test is pathologically broken ... it will only > work in an environment where some _other_ host is also avahi-enabled > and accessible via multi-cast udp packets. No, the test is expecting to find a local pmcd. No external one needed. I just suggested looking at the firewall in case it does cause a problem for local lookups on your platform. Dave From kenj@internode.on.net Mon Oct 6 17:33:31 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 685B17F4E for ; Mon, 6 Oct 2014 17:33:31 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id EB759AC008 for ; Mon, 6 Oct 2014 15:33:27 -0700 (PDT) X-ASG-Debug-ID: 1412634802-04cbb073047c34b0001-S8gJnT Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by cuda.sgi.com with ESMTP id p21HDp1UolsMKIB5 for ; Mon, 06 Oct 2014 15:33:22 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.145 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgUDAD8XM1R20Xt7PGdsb2JhbAANUotyyRqDIQKBHgEGAQEBATiEPQEBBDhAEQsYCRYPCQMCAQIBMRQTCAEBtA2VdRiQTBaENQEEj1qnW4MiAQEB Received: from ppp118-209-123-123.lns20.mel4.internode.on.net (HELO [192.168.1.100]) ([118.209.123.123]) by ipmail06.adl6.internode.on.net with ESMTP; 07 Oct 2014 09:03:21 +1030 Message-ID: <543318C7.3030604@internode.on.net> Date: Tue, 07 Oct 2014 09:33:43 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: Re: [pcp] [PATCH] Fix memory corruption in python support References: <542D6626.50703@redhat.com> <336785285.60750380.1412287401855.JavaMail.zimbra@redhat.com> <5432F280.20008@redhat.com> X-ASG-Orig-Subj: Re: [pcp] [PATCH] Fix memory corruption in python support In-Reply-To: <5432F280.20008@redhat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl6.internode.on.net[150.101.137.145] X-Barracuda-Start-Time: 1412634802 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10277 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- On 07/10/14 06:50, David Smith wrote: >... > > On the new pcpfans/dsmith/dev branch, there are 2 new commits: > > - 44cb24d: This adds a new test (called '843') that can catch this bug. > Note that I based this test on an existing test, but I'm still not 100% > sure I updated the testsuite machinery properly. Also note that this new > test only tests the changes to src/python/pmda.c. I"m not really sure > how to test the changes to src/python/pmapi.c. > > - 8e62465: This is the the python fix itself (that I posted to the list > earlier). Thanks David. I've cherry picked and confirmed ... - qa/843 fails before the code change - after the code change, debian builds are ok - after the code change, qa/843 passes - after the code change qa -g python passes Integration of qa pieces looks OK to me also, except 843 probably should be in the python group as well (in qa/group) and I'll fix this in my next batch of commits. Cheers, Ken. From kenj@internode.on.net Mon Oct 6 17:40:03 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id B6FEA7F4E for ; Mon, 6 Oct 2014 17:40:03 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id A57E1304039 for ; Mon, 6 Oct 2014 15:40:00 -0700 (PDT) X-ASG-Debug-ID: 1412635198-04cbb073047c3950001-S8gJnT Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by cuda.sgi.com with ESMTP id BRFwKHVXR98OSEmB for ; Mon, 06 Oct 2014 15:39:58 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.145 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgYDAAUaM1R20Xt7PGdsb2JhbAANUotyyRqDIQKBHgEGAQEBATiEPAEBAQMBOEAGCwsYCRYPCQMCAQIBMRQGAQwIAQGIMqtelXUBF5BMhEsBBI9ajxGQVId2gyIBAQE Received: from ppp118-209-123-123.lns20.mel4.internode.on.net (HELO [192.168.1.100]) ([118.209.123.123]) by ipmail06.adl6.internode.on.net with ESMTP; 07 Oct 2014 09:09:58 +1030 Message-ID: <54331A54.7030301@internode.on.net> Date: Tue, 07 Oct 2014 09:40:20 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: Dave Brolley , pcp@oss.sgi.com Subject: Re: [pcp] qa/946 - pmfind question References: <001801cfe152$29472510$7bd56f30$@internode.on.net> <5432AE59.8070309@redhat.com> <5432F3FC.2060304@internode.on.net> <5433105D.7080302@redhat.com> X-ASG-Orig-Subj: Re: [pcp] qa/946 - pmfind question In-Reply-To: <5433105D.7080302@redhat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl6.internode.on.net[150.101.137.145] X-Barracuda-Start-Time: 1412635198 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10277 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- On 07/10/14 08:57, Dave Brolley wrote: > ... > No, the test is expecting to find a local pmcd. No external one needed. OK ... I'm just exposing my ignorance in public ... 8^)> > I just suggested looking at the firewall in case it does cause a problem > for local lookups on your platform. Excellent catch. Opened 5353/udp on firewalld and everything avahi-related (including qa/946) started to work. Thanks. From kenj@internode.on.net Mon Oct 6 19:37:11 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 6DE487F4E for ; Mon, 6 Oct 2014 19:37:11 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id F1A43AC002 for ; Mon, 6 Oct 2014 17:37:10 -0700 (PDT) X-ASG-Debug-ID: 1412642223-04cbb073017c8ac0001-S8gJnT Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by cuda.sgi.com with ESMTP id pg74LsM4mGis3RZZ for ; Mon, 06 Oct 2014 17:37:04 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.145 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgkDAMc0M1R20Xt7PGdsb2JhbAANUoNhWIMChDfEa4h1AQYBAQEBOIRmBFEwBgIFFgsCCwMCAQIBMQ4ZBgIBAYhHq0t4lR+BLJIXgVQFljChBViCSgEBAQ Received: from ppp118-209-123-123.lns20.mel4.internode.on.net (HELO [192.168.1.100]) ([118.209.123.123]) by ipmail06.adl6.internode.on.net with ESMTP; 07 Oct 2014 11:06:27 +1030 Message-ID: <543335A0.6070302@internode.on.net> Date: Tue, 07 Oct 2014 11:36:48 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: pcp updates Content-Type: text/plain; charset=utf-8 X-ASG-Orig-Subj: pcp updates Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl6.internode.on.net[150.101.137.145] X-Barracuda-Start-Time: 1412642223 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10279 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Changes committed to git://git.performancecopilot.org/kenj/pcp.git dev man/man1/sar2pcp.1 | 2 qa/022 | 59 ++++++++------- qa/359 | 1 qa/709 | 4 - qa/843 | 53 +++++++++++++ qa/843.out | 28 +++++++ qa/admin/check-vm | 2 qa/admin/pcp-daily | 3 qa/admin/pcp-qa-summary | 7 + qa/admin/show-me-all | 9 ++ qa/group | 3 qa/pmdas/GNUmakefile | 2 qa/pmdas/GNUmakefile.install | 2 qa/pmdas/memory_python/.gitignore | 3 qa/pmdas/memory_python/GNUmakefile | 38 +++++++++ qa/pmdas/memory_python/GNUmakefile.install | 5 + qa/pmdas/memory_python/Install | 28 +++++++ qa/pmdas/memory_python/Remove | 26 ++++++ qa/pmdas/memory_python/pmdamemory_python.python | 92 ++++++++++++++++++++++++ qa/src/GNUlocaldefs | 41 ++++------ src/pmns/stdpmid.pcp | 1 src/python/pmapi.c | 7 - src/python/pmda.c | 25 ++++++ 23 files changed, 379 insertions(+), 62 deletions(-) commit 78ae32254c3029fa1f5ffc0f6f31f5614ebc896b Author: Ken McDonell Date: Tue Oct 7 09:30:09 2014 +1100 qa/709 - fix typo so the test will run when pmcollectl installed commit 71bcadab77d3c32c29edd517629ef881d9dd38af Author: Ken McDonell Date: Tue Oct 7 09:29:46 2014 +1100 qa/group - add 843 to python group commit e88d697c8c7a99d15a7314405fcd1ab3fe800c2c Author: David Smith Date: Mon Oct 6 14:32:39 2014 -0500 Fix python PMDA bug by handling python object reference counts correctly. * src/python/pmda.c (namespace_refresh): Increment the python object reference count when needed. (pmid_oneline_refresh): Ditto. (pmid_longtext_refresh): Ditto. (indom_oneline_refresh): Ditto. (indom_longtext_refresh): Ditto. * src/python/pmapi.c (getOptionsFromList): Remove decrementing python object reference when it wasn't needed. commit 0f5d1565c1f04a073ef3f9e5215b617f844dd662 Author: David Smith Date: Mon Oct 6 14:27:12 2014 -0500 Added test for python PMDA memory corruption bug. * qa/843.out: Ditto. * qa/group: Added test 843. * qa/pmdas/GNUmakefile: Added the memory_python directory. * qa/pmdas/GNUmakefile.install: Ditto. * qa/pmdas/memory_python/.gitignore: New file. * qa/pmdas/memory_python/GNUmakefile: Ditto. * qa/pmdas/memory_python/GNUmakefile.install: Ditto. * qa/pmdas/memory_python/Install: Ditto. * qa/pmdas/memory_python/Remove: Ditto. * qa/pmdas/memory_python/pmdamemory_python.python: Ditto. * src/pmns/stdpmid.pcp: Added 'memory_python' define. commit 9ec35e3f6e32ef6172dff9376f7c693e345a0c10 Author: Ken McDonell Date: Mon Oct 6 21:29:03 2014 +1100 qa/admin/show-me-all - map N -> 00N and NN -> 0NN (like show-me) commit 2d234c8444f739cfeb04b9e6244aa2a25a2c5e70 Author: Ken McDonell Date: Mon Oct 6 21:28:03 2014 +1100 qa/admin/pcp-qa-summary - tweak output formatting commit fc89c0c6bd8269920026982ee0353eb141adb9e6 Author: Ken McDonell Date: Mon Oct 6 21:27:40 2014 +1100 qa/admin/pcp-daily - don't need make clean all the time commit 465308c56e1eb03a1d11237c4aab4416d95b7cb0 Author: Ken McDonell Date: Mon Oct 6 21:25:23 2014 +1100 qa/admin/check-vm - check $HOME is searchable by others Needed for some QA tests so user pcp (aka pmcd) can find QA PMDAs. commit bc80b270c3f6174d4a057734856c9c092fb78dc5 Author: Ken McDonell Date: Mon Oct 6 21:24:18 2014 +1100 qa/src/GNUlocaldefs - pmlogger and pmlogextract don't need special pathnames Now binaries are in /usr/bin. commit 94fffc66c7173c911116e4aeb478179d1332dab4 Author: Ken McDonell Date: Mon Oct 6 15:01:29 2014 +1100 qa/022 - dodge newlines in proc metric instance names Come from our own awk scripts in QA! commit 35aa8c1dfd6e7e9454a992f9a20e27ae135c0c29 Author: Ken McDonell Date: Mon Oct 6 07:26:21 2014 +1100 qa/359 - more cgroups filtering required commit 02d18c8f2bb329cf35039ea23545a1799366cf6e Author: Ken McDonell Date: Fri Oct 3 06:30:44 2014 +1000 sar2pcp man page - correct NAME entry Was collectl2pcp (!) ... which caused an installation failure on FreeBSD (don't even ask). From nscott@redhat.com Mon Oct 6 22:22:16 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 7FB8D7F4E for ; Mon, 6 Oct 2014 22:22:16 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 419DC8F8050 for ; Mon, 6 Oct 2014 20:22:15 -0700 (PDT) X-ASG-Debug-ID: 1412652129-04bdf003a07d06a0001-S8gJnT Received: from mx4-phx2.redhat.com (mx4-phx2.redhat.com [209.132.183.25]) by cuda.sgi.com with ESMTP id TbeyFsC2R4uBC45O (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Mon, 06 Oct 2014 20:22:09 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.25 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx4-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s973M9N5008995 for ; Mon, 6 Oct 2014 23:22:09 -0400 Date: Mon, 6 Oct 2014 23:22:09 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: pcp Message-ID: <116165838.62701212.1412652129112.JavaMail.zimbra@redhat.com> In-Reply-To: <635266388.62701022.1412652062518.JavaMail.zimbra@redhat.com> Subject: pcp updates: qa, docs, python MIME-Version: 1.0 X-ASG-Orig-Subj: pcp updates: qa, docs, python Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: pcp updates: qa, docs, python Thread-Index: k1tdJFTWUYTvpULXuvnqnkGuFoWePQ== X-Barracuda-Connect: mx4-phx2.redhat.com[209.132.183.25] X-Barracuda-Start-Time: 1412652129 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10283 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Changes committed to git://git.pcp.io/pcp.git dev Ken McDonell (10): sar2pcp man page - correct NAME entry qa/359 - more cgroups filtering required qa/022 - dodge newlines in proc metric instance names qa/src/GNUlocaldefs - pmlogger and pmlogextract don't need special pathnames qa/admin/check-vm - check $HOME is searchable by others qa/admin/pcp-daily - don't need make clean all the time qa/admin/pcp-qa-summary - tweak output formatting qa/admin/show-me-all - map N -> 00N and NN -> 0NN (like show-me) qa/group - add 843 to python group qa/709 - fix typo so the test will run when pmcollectl installed David Smith (2): Added test for python PMDA memory corruption bug. Fix python PMDA bug by handling python object reference counts correctly. Nathan Scott (2): man pages: migrate perl web-related PMDA man pages to new scheme man pages: migrate perl rsyslog, postfix man pages to new scheme man/man1/sar2pcp.1 | 2 qa/022 | 59 ++++++++------- qa/359 | 1 qa/709 | 4 - qa/843 | 53 +++++++++++++ qa/843.out | 28 +++++++ qa/admin/check-vm | 2 qa/admin/pcp-daily | 3 qa/admin/pcp-qa-summary | 7 + qa/admin/show-me-all | 9 ++ qa/group | 3 qa/pmdas/GNUmakefile | 2 qa/pmdas/GNUmakefile.install | 2 qa/pmdas/memory_python/.gitignore | 3 qa/pmdas/memory_python/GNUmakefile | 38 +++++++++ qa/pmdas/memory_python/GNUmakefile.install | 5 + qa/pmdas/memory_python/Install | 28 +++++++ qa/pmdas/memory_python/Remove | 26 ++++++ qa/pmdas/memory_python/pmdamemory_python.python | 92 ++++++++++++++++++++++++ qa/src/GNUlocaldefs | 41 ++++------ src/pmdas/elasticsearch/.gitignore | 1 src/pmdas/elasticsearch/GNUmakefile | 28 ++----- src/pmdas/elasticsearch/pmdaelasticsearch.1 | 57 ++++++++++++++ src/pmdas/elasticsearch/pmdaelasticsearch.pl | 60 --------------- src/pmdas/memcache/.gitignore | 1 src/pmdas/memcache/GNUmakefile | 16 ---- src/pmdas/memcache/pmdamemcache.1 | 69 ++++++++++++++++++ src/pmdas/memcache/pmdamemcache.pl | 67 ----------------- src/pmdas/nginx/.gitignore | 1 src/pmdas/nginx/GNUmakefile | 15 --- src/pmdas/nginx/pmdanginx.1 | 57 ++++++++++++++ src/pmdas/nginx/pmdanginx.pl | 58 --------------- src/pmdas/postfix/.gitignore | 1 src/pmdas/postfix/GNUmakefile | 10 -- src/pmdas/postfix/pmdapostfix.1 | 56 ++++++++++++++ src/pmdas/postfix/pmdapostfix.pl | 51 ------------- src/pmdas/rsyslog/.gitignore | 1 src/pmdas/rsyslog/GNUmakefile | 10 -- src/pmdas/rsyslog/pmdarsyslog.1 | 75 +++++++++++++++++++ src/pmdas/rsyslog/pmdarsyslog.pl | 68 ----------------- src/pmns/stdpmid.pcp | 1 src/python/pmapi.c | 7 - src/python/pmda.c | 25 ++++++ 43 files changed, 709 insertions(+), 434 deletions(-) From wwwrun@oss.sgi.com Mon Oct 6 22:42:22 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE,NO_RELAYS autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: by oss.sgi.com (Postfix, from userid 30) id 13B647F53; Mon, 6 Oct 2014 22:42:22 -0500 (CDT) From: bugzilla-daemon@oss.sgi.com To: pcp@oss.sgi.com Subject: [Bug 1068] pmUnitsStr_r can clobber memory Date: Tue, 07 Oct 2014 03:42:21 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Classification: Unclassified X-Bugzilla-Product: pcp X-Bugzilla-Component: pcp X-Bugzilla-Keywords: X-Bugzilla-Severity: major X-Bugzilla-Who: kenj@internode.on.net X-Bugzilla-Status: NEW X-Bugzilla-Priority: P5 X-Bugzilla-Assigned-To: kenj@internode.on.net X-Bugzilla-Target-Milestone: --- X-Bugzilla-Changed-Fields: cc assigned_to Message-ID: In-Reply-To: References: Content-Type: multipart/alternative; boundary="1412653342.D6Ec8Cf3.6693"; charset="us-ascii" X-Bugzilla-URL: http://oss.sgi.com/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 --1412653342.D6Ec8Cf3.6693 Date: Mon, 6 Oct 2014 22:42:22 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" http://oss.sgi.com/bugzilla/show_bug.cgi?id=1068 Ken McDonell changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kenj@internode.on.net Assignee|pcp@kenj.com.au |kenj@internode.on.net -- You are receiving this mail because: You are on the CC list for the bug. --1412653342.D6Ec8Cf3.6693 Date: Mon, 6 Oct 2014 22:42:22 -0500 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" changed bug 1068
What Removed Added
CC   kenj@internode.on.net
Assignee pcp@kenj.com.au kenj@internode.on.net

You are receiving this mail because:
  • You are on the CC list for the bug.
--1412653342.D6Ec8Cf3.6693-- From nscott@redhat.com Mon Oct 6 23:57:54 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 9773A7F4E for ; Mon, 6 Oct 2014 23:57:54 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 5ACD6304032 for ; Mon, 6 Oct 2014 21:57:51 -0700 (PDT) X-ASG-Debug-ID: 1412657866-04bdf0039f7d47d0001-S8gJnT Received: from mx3-phx2.redhat.com (mx3-phx2.redhat.com [209.132.183.24]) by cuda.sgi.com with ESMTP id Hbrb7cfnBRKYnnQf (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Mon, 06 Oct 2014 21:57:46 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.24 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx3-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s974viuA021037; Tue, 7 Oct 2014 00:57:44 -0400 Date: Tue, 7 Oct 2014 00:57:44 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Martins Innus Cc: pcp@oss.sgi.com Message-ID: <905561536.62723741.1412657864635.JavaMail.zimbra@redhat.com> In-Reply-To: <54230FAF.2080201@buffalo.edu> References: <536D28B4.6010504@buffalo.edu> <1139662762.4765310.1399862104653.JavaMail.zimbra@redhat.com> <54230FAF.2080201@buffalo.edu> Subject: Re: [pcp] hotproc rfc MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] hotproc rfc Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: hotproc rfc Thread-Index: l304GiZy4X2wg4OcO46VLl60gM5Z6A== X-Barracuda-Connect: mx3-phx2.redhat.com[209.132.183.24] X-Barracuda-Start-Time: 1412657866 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10285 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Hi Martins, I've had time to go through the code in detail now - this looks like a good step forward, and its much simpler than I expected. ----- Original Message ----- > [...] > This version works, *but* likely has more cleanup to go if this is > an ok way forward. This is on my hotproc branch on github. I've also > attached a cumulative patch and git log since over the course of > development there were a bunch of changes and refactors and the linear > history might be confusing. > > ******* > github.com/ubccr/pcp/tree/hotproc > ******* Thanks Martins. I've extracted the diff from there, some review notes follow below. If I can help out with hacking on any of this with you, just lemme know (especially dynamic metrics, which are a bit quirky). > Some notes on this implementation: > > 1. I tried to keep as much functionality as I could from the sgi version > but its been quite a while since we ran it and haven't had this hardware > for quite some time! There are a few things missing that I discuss below. OK. > 2. If you recall, the original hotproc pmda used a series of scripts to > duplicate the proc pmda and then overlay the hotproc functionality as a > separate pmda. Instead of having a separate pmda, I merged it into > linux_proc for ease of maintenance. I modified the root_proc file by > hand for now, but that could likely be done with an awk script as before > as part of the build process. I think the approach of merging the two PMDAs has worked well. Since hotproc was originally written, the dynamic metrics concept has come along and I think it'll be a better match here than any namespace file script approach. Some more notes in the review - but, if I can help with coding this aspect, just let me know (I have some of this fresh in-mind from other recent hacking). > 3. What should the default state of hotproc be given that proc is > usually always running? Should there be some default config file to It should be available but "off" I think. There is already a pmStore interface to allow dynamic changes (we should add some access controls to those, see "have_access" in proc_store - use of that is missing for the new metrics). In this case, checking for zero uid attribute would be a good option, I think (in the absence of any more involved ACLs, which could be added later). > have some metrics available or should it be disabled (timer doesn't run, > etc) > if the user does not provide a config file? I have provided a Yes, disabled and no-values by default I think. > sample that is a simplification of what we are currently using. That should be named sample.conf and hotproc.conf should not exist by default, since we have no way of knowing up-front what any individual site might want to use. pmdashping does something like this, btw, might provide a reference - oh, the old pmdahotproc Install script implements the same logic too actually. This could be added into the pmdaproc Install. > 4. Many of the changes are actually pretty self contained. The biggest Yeah, its neat - I was a bit surprised at how easily it slotted in. > change was in proc_pid where I needed to change the assumption that > there was only one list of processes that we cared about. This resulted > in having to change a bunch of the functions to take a process list to > operate on. *nod* > 5. Some stats I couldn't find per process yet: syscalls (systemtap seems > to be the only way to get these?) , and schedwait (doesn't seem to be > available at all). Yeah, IRIX had a fair bit more per-process instrumentation. > 6. idletime is provided by the linux pmda. I just re-implemented this > small part of code. Is there an alternative way to do inter pmda fetches? Looks fine to me, other than the minor comments below. > 7.Some cleanups and error checking still to do. The original code used > lots of "externs" and I started to factor those out, but then saw use > in other pmdas so maybe thats not a concern? Its good to reduce them where it makes sense. Also, the calls to exit(1) are a concern, from the original PMDA - we should refactor those. > 8. I realize there is a lot to cleanup yet: commented out old code, > debugging left on, maybe some memory leaks, etc but wanted to make sure > there were no show stoppers before I kept going. Also, some of the data > types might might not be correct, i.e converting some of the long and > long long types in the original code. Its progressing nicely, I think, please keep moving ahead with it. > 9. I'm open to any and all suggestions as having this available makes > logging proc much more manageable for us. I have it running in a couple > of VMs right now and will be deploying further over the next week. Good stuff. Other missing things would be docs and QA testing. The original man page for pmdahotproc(1) is currently over in man/retired in the pcp git tree - maybe go through that and pull out the relevant bits (config format, pmstore interface, etc)? >From a QA POV, I'm pretty sure there were some hotproc tests originally - but, I cannot find any currently in the tree. I've sent a note to our friends at SGI to see if they can help us out there. I'll send through details once I hear back, but in general we should start thinking about automated tests now that we have the basics in place, to support the next rounds of development (and ongoing, into the future, of course). OK, here's my review notes - let me know if/where I can help out, or if anything makes no sense. diff --git a/src/pmdas/linux_proc/GNUmakefile b/src/pmdas/linux_proc/GNUmakefile index c259bda..4ec9438 100644 --- a/src/pmdas/linux_proc/GNUmakefile +++ b/src/pmdas/linux_proc/GNUmakefile +LSRCFILES = help root linux_proc_migrate.conf $(SCRIPTS) (no need for LSRCFILES anymore) +root_proc: root_proc.in root_hotproc.in + ./build_root_proc.sh I think it'll be simpler to use dynamic metrics - simplifies the above step, and... diff --git a/src/pmdas/linux_proc/build_root_proc.sh b/src/pmdas/linux_proc/build_root_proc.sh new file mode 100755 (removes the need for this) diff --git a/src/pmdas/linux_proc/create_hot_pmns.awk b/src/pmdas/linux_proc/create_hot_pmns.awk new file mode 100644 (and this) diff --git a/src/pmdas/linux_proc/insert_into_root_pmns.awk b/src/pmdas/linux_proc/insert_into_root_pmns.awk new file mode 100644 (and this) diff --git a/src/pmdas/linux_proc/config.c b/src/pmdas/linux_proc/config.c new file mode 100644 index 0000000..498b5f9 --- /dev/null +++ b/src/pmdas/linux_proc/config.c @@ -0,0 +1,561 @@ [...] + (void)fprintf(stderr, "%s: Unable to open configuration file \"%s\": %s\n", + pmProgname, hotproc_configfile, osstrerror()); + exit(1); This, and quite a few other cases of calling exit() here, should be reworked to not fail quite so drastically. +read_test_var(char *line, config_vars *vars) There's helper code here for testing, we should look into how to automate its use. diff --git a/src/pmdas/linux_proc/pmda.c b/src/pmdas/linux_proc/pmda.c index 1efaabd..ab3710f 100644 --- a/src/pmdas/linux_proc/pmda.c +++ b/src/pmdas/linux_proc/pmda.c @@ -63,6 +68,11 @@ char *proc_statspath = ""; /* optional path prefix for all stats files */ static pmdaIndom indomtab[NUM_INDOMS]; /* + * Real metric tab that will be used after we add in hotprocs + */ +static pmdaMetric *fullmetrictab; OK, I think the idea here (fullmetrictab, proc_init_hotproc, and createHotprocMetric - and the namespace file scripting you talked about earlier) are duplicating concepts from the dynamic namespace support already used in the Linux pmdaproc. It seems like we can fit this in better ... if the [hot]proc.{psinfo,id,memory,io,fd, schedstat} subtrees become dynamic, they'd be able to share single definition, help text, and so on. + case CLUSTER_HOTPROC_PID_STAT: + active_proc_pid = &hotproc_pid; case CLUSTER_PID_STAT: (This is cunning - I like this piggy-backing approach!) @@ -1702,43 +1932,85 @@ proc_store(pmResult *result, pmdaExt *pmda) [...] + case CLUSTER_CONTROL: [...] + case CLUSTER_HOTPROC_GLOBAL: + switch(idp->item){ + case 1: /* Update interval */ + if ((sts = pmExtractValue(vsp->valfmt, &vsp->vlist[0], + PM_TYPE_U32, &av, PM_TYPE_U32)) >= 0) { + hotproc_update_interval.tv_sec = av.ul; + reset_hotproc_timer(); + } + break; + case 8: /* CONFIG */ + { + bool_node *tree = NULL; + char *savebuffer; + if ((sts = pmExtractValue(vsp->valfmt, &vsp->vlist[0], + PM_TYPE_STRING, &av, PM_TYPE_STRING)) >= 0) { The above is the answer to one of your earlier questions about how to enable this dynamically, when no initial configuration file exists? It'd be good to use descriptive macros instead of "1" and "8" here. +void +proc_init_hotproc(){ + + nmetrics = sizeof(metrictab)/sizeof(metrictab[0]); + + fullmetrictab = (pmdaMetric *) malloc(nmetrics * 2 * sizeof(pmdaMetric)); + /* memcopy, then add hotproc, then set nmetrics for use below, then update all to use fullmetrictab */ Yeah, this is more of the code I mentioned earlier re dynamic metrics and duplicating concepts implemented there already. @@ -1871,6 +2249,21 @@ proc_init(pmdaInterface *dp) indomtab[CGROUP_MOUNTS_INDOM].it_indom = CGROUP_MOUNTS_INDOM; proc_pid.indom = &indomtab[PROC_INDOM]; + + indomtab[HOTPROC_INDOM].it_indom = HOTPROC_INDOM; + hotproc_pid.indom = &indomtab[HOTPROC_INDOM]; + + char h_configfile[MAXPATHLEN]; + + snprintf(h_configfile, sizeof(h_configfile), "%s%c" "proc" "%c" "hotproc.conf", + pmGetConfig("PCP_PMDAS_DIR"), sep, sep); + + //Send this all to a hotproc init config function? TODO (yep, like cgroups_init I s'pose) diff --git a/src/pmdas/linux_proc/proc_pid.c b/src/pmdas/linux_proc/proc_pid.c index c0bab14..6a7e4ec 100644 --- a/src/pmdas/linux_proc/proc_pid.c +++ b/src/pmdas/linux_proc/proc_pid.c (there's several different coding styles in this file, makes it a bit jarring to read for me) +/* The idea of this is copied from linux/proc_stat.c */ +static unsigned long long +get_idle_time(){ (this is fine - its a trivial code snippet) + FILE * fp = NULL; /* kept open until exit() */ (this comment is incorrect - doesn't match the code) + char *linux_statspath = ""; + if ((envpath = getenv("LINUX_STATSPATH")) != NULL) + linux_statspath = envpath; (this will need to be PROC_STATSPATH in this PMDA) + static long hz = -1; + + if (hz == -1){ + hz = sysconf(_SC_CLK_TCK); + num_cpus = sysconf(_SC_NPROCESSORS_ONLN); + } This dup's code in proc_fetchCallBack - could move this to PMDA init time, and share "hz" to avoid duplicating the calls. + /* How do we deal with errors here??? */ Issue __pmNotifyErr warnings and move on is the best option I guess. + /* Just copied the old Diff code. Is there a new way to do this? */ Nope, no new way exists - this looks fine to me. + bzero(&newnode->preds, sizeof(newnode->preds)); We tend to use memset-to-zero everywhere else in PCP nowadays. + double hptime = (ts.tv_sec - p_timestamp.tv_sec) + (ts.tv_usec - p_timestamp.tv_usec)/1000000.0; + //fprintf(stderr, "Hotproc Update took %f time\n", hptime); Hide this handy info inside a pmDebug diagnostic branch (uncommented). cheers. -- Nathan From wwwrun@oss.sgi.com Tue Oct 7 00:33:23 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE,NO_RELAYS autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: by oss.sgi.com (Postfix, from userid 30) id 488017F50; Tue, 7 Oct 2014 00:33:23 -0500 (CDT) From: bugzilla-daemon@oss.sgi.com To: pcp@oss.sgi.com Subject: [Bug 1068] pmUnitsStr_r can clobber memory Date: Tue, 07 Oct 2014 05:33:22 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Classification: Unclassified X-Bugzilla-Product: pcp X-Bugzilla-Component: pcp X-Bugzilla-Keywords: X-Bugzilla-Severity: major X-Bugzilla-Who: kenj@internode.on.net X-Bugzilla-Status: RESOLVED X-Bugzilla-Priority: P5 X-Bugzilla-Assigned-To: kenj@internode.on.net X-Bugzilla-Target-Milestone: --- X-Bugzilla-Changed-Fields: bug_status resolution Message-ID: In-Reply-To: References: Content-Type: multipart/alternative; boundary="1412660003.e21f28Aa2.17083"; charset="us-ascii" X-Bugzilla-URL: http://oss.sgi.com/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 --1412660003.e21f28Aa2.17083 Date: Tue, 7 Oct 2014 00:33:23 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" http://oss.sgi.com/bugzilla/show_bug.cgi?id=1068 Ken McDonell changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |FIXED --- Comment #1 from Ken McDonell --- Fix is in commit d57df37 in my tree. ^7^7 is not fishy ... the strange smell is from the test program that specifies the dimension of count to be 7, and also specifies the scale of count to be 7 ... so the units are correct. -- You are receiving this mail because: You are on the CC list for the bug. --1412660003.e21f28Aa2.17083 Date: Tue, 7 Oct 2014 00:33:23 -0500 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" changed bug 1068
What Removed Added
Status NEW RESOLVED
Resolution --- FIXED

Comment # 1 on bug 1068 from 
Fix is in commit d57df37 in my tree.

^7^7 is not fishy ... the strange smell is from the test program that specifies
the dimension of count to be 7, and also specifies the scale of count to be 7
... so the units are correct.

You are receiving this mail because:
  • You are on the CC list for the bug.
--1412660003.e21f28Aa2.17083-- From kenj@internode.on.net Tue Oct 7 00:35:24 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 483D87F3F for ; Tue, 7 Oct 2014 00:35:24 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 2690B304039 for ; Mon, 6 Oct 2014 22:35:24 -0700 (PDT) X-ASG-Debug-ID: 1412660117-04cbb073027d5860001-S8gJnT Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id QKyrImI7jbQfSYoO for ; Mon, 06 Oct 2014 22:35:18 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AsEBABJ7M1R20Xt7PGdsb2JhbAANUoNhWIMCyUuIcQEGAQEBATiEZlUwBgIFFgsCCwMCAQIBMScGAgEBiEerIniVH4EsgTWOAYJhgVQFj1qGVpNgjSVYAYJJAQEB Received: from ppp118-209-123-123.lns20.mel4.internode.on.net (HELO [192.168.1.100]) ([118.209.123.123]) by ipmail04.adl6.internode.on.net with ESMTP; 07 Oct 2014 16:04:33 +1030 Message-ID: <54337B7F.7000803@internode.on.net> Date: Tue, 07 Oct 2014 16:34:55 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: pcp updates - pmUnitsStr_r fix and timezone shift strangeness in QAland. Content-Type: text/plain; charset=utf-8 X-ASG-Orig-Subj: pcp updates - pmUnitsStr_r fix and timezone shift strangeness in QAland. Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1412660117 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10286 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Changes committed to git://git.performancecopilot.org/kenj/pcp.git dev man/man3/pmunitsstr.3 | 9 +++++++++ qa/151 | 13 +++++++++++++ qa/746 | 29 +++++++++++++++++++++++++++++ qa/746.out | 14 ++++++++++++++ qa/group | 1 + src/libpcp/src/units.c | 7 +++++++ 6 files changed, 73 insertions(+) commit d57df37c5c8a83fde57476592b73d51bd62e8092 Author: Ken McDonell Date: Tue Oct 7 16:28:29 2014 +1100 libpcp - fix for pmUnitsStr_r buffer overrun Since the definition of pmUnitsStr_r() dictates that buflen be at least 60, enforce this on entry. There is a small ABI change here that in the event that buflen is too short, pmUnitsStr_r will now return NULL instead of buf. Updated the man page. Added qa/746 to tickle this buglet and prove it is fixed. Addresses http://oss.sgi.com/bugzilla/show_bug.cgi?id=1068 commit 0eb6867db4eabf5774b6324da3f7e1748b335436 Author: Ken McDonell Date: Tue Oct 7 12:09:05 2014 +1100 qa/151 - don't run for 6 days after a timezone change Strange but true! pmlogger_daily is doing the right thing, but this QA test is confused ... simply skip it until things settle down (the 6 days comes from the test, it is not make believe). From nscott@redhat.com Tue Oct 7 00:41:20 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id E28097F51 for ; Tue, 7 Oct 2014 00:41:20 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 79E9FAC002 for ; Mon, 6 Oct 2014 22:41:17 -0700 (PDT) X-ASG-Debug-ID: 1412660475-04bdf003a07d62b0001-S8gJnT Received: from mx6-phx2.redhat.com (mx6-phx2.redhat.com [209.132.183.39]) by cuda.sgi.com with ESMTP id GCHye89vP7haOKgT (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Mon, 06 Oct 2014 22:41:16 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.39 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx6-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s975fCRC029341; Tue, 7 Oct 2014 01:41:12 -0400 Date: Tue, 7 Oct 2014 01:41:12 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Ken McDonell Cc: pcp@oss.sgi.com Message-ID: <1755509564.62732609.1412660472166.JavaMail.zimbra@redhat.com> In-Reply-To: <54337B7F.7000803@internode.on.net> References: <54337B7F.7000803@internode.on.net> Subject: Re: [pcp] pcp updates - pmUnitsStr_r fix and timezone shift strangeness in QAland. MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] pcp updates - pmUnitsStr_r fix and timezone shift strangeness in QAland. Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: pcp updates - pmUnitsStr_r fix and timezone shift strangeness in QAland. Thread-Index: CFa1lEE4BExmCng6vGkagGczjR0I2Q== X-Barracuda-Connect: mx6-phx2.redhat.com[209.132.183.39] X-Barracuda-Start-Time: 1412660475 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10286 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... ----- Original Message ----- > [...] > Author: Ken McDonell > Date: Tue Oct 7 12:09:05 2014 +1100 > > qa/151 - don't run for 6 days after a timezone change > > Strange but true! pmlogger_daily is doing the right thing, but > this QA test is confused ... simply skip it until things settle > down (the 6 days comes from the test, it is not make believe). > [18%] 149 0s ... [19%] 150 0s ... [19%] 151 1s ... - output mismatch (see 151.out.bad) [19%] 152 5s ... [19%] 153 0s ... Thanks!!! Was on my list to follow up with, so, much time -> saved. cheers. -- Nathan From kenj@internode.on.net Tue Oct 7 00:44:21 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 09DBC7F51 for ; Tue, 7 Oct 2014 00:44:21 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id DCDFB8F804C for ; Mon, 6 Oct 2014 22:44:17 -0700 (PDT) X-ASG-Debug-ID: 1412660655-04cbb073017d5e60001-S8gJnT Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id x0urSDta7rqmZNnM for ; Mon, 06 Oct 2014 22:44:16 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AiMCADB9M1R20Xt7PGdsb2JhbAANUoNhWIMChDfFEoh2AQYBAQEBOIRmBFEwBgIFFgsCCwMCAQIBMScGAgEBiEerIXiVHoEsjzaCYYFUBZYwoQVYgkoBAQE Received: from ppp118-209-123-123.lns20.mel4.internode.on.net (HELO [192.168.1.100]) ([118.209.123.123]) by ipmail04.adl6.internode.on.net with ESMTP; 07 Oct 2014 16:13:51 +1030 Message-ID: <54337DAE.6000107@internode.on.net> Date: Tue, 07 Oct 2014 16:44:14 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: pcp updates - nfs4 request metrics Content-Type: text/plain; charset=utf-8 X-ASG-Orig-Subj: pcp updates - nfs4 request metrics Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1412660655 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10286 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- qa/232 was failing on Centos 7. There are NR_RPC4_SVR_COUNTERS instances, numbered 1 (not 0) to NR_RPC4_SVR_COUNTERS, but the values are stored in [0] ... [NR_RPC4_SVR_COUNTERS-1] of proc_net_rpc->server.reqcounts4[]. It would be _really_ good to have someone who knows the nfs code in the linux pmda to review these changes, and if they are correct to check that similar mismatches are not happening for the other nfs/rpc metrics. Changes committed to git://git.performancecopilot.org/kenj/pcp.git dev src/pmdas/linux/pmda.c | 6 +++--- src/pmdas/linux/proc_net_rpc.c | 15 ++++++++++++--- 2 files changed, 15 insertions(+), 6 deletions(-) commit 56281b7d3e56bd6b8fdfcb819bc08dea9b8fbde1 Author: Ken McDonell Date: Tue Oct 7 16:35:14 2014 +1100 linux pmda - nfs4 metrics The indom for server requests for NFSv4 was not being handled correctly. When nfsd was started and supporting NFSv4, qa/232 was failing because pminfo -f was not returning the last member of the instance domain. From kenj@internode.on.net Tue Oct 7 01:08:03 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id D2CCD7F3F for ; Tue, 7 Oct 2014 01:08:03 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id C0A67304039 for ; Mon, 6 Oct 2014 23:08:00 -0700 (PDT) X-ASG-Debug-ID: 1412662078-04cbb073047d6d00001-S8gJnT Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id HtRZ8e5GZt2VtSvU for ; Mon, 06 Oct 2014 23:07:59 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AnUNAB6CM1R20Xt7PGdsb2JhbAANUotysU2TAoR7gyECgSIBBgEBAQE4hD0BAQR4EQsYCRYPCQMCAQIBMRQTCAEBs2mVfRiQTBaENQEEnmuGbJFegV4kgSABAQE Received: from ppp118-209-123-123.lns20.mel4.internode.on.net (HELO [192.168.1.100]) ([118.209.123.123]) by ipmail04.adl6.internode.on.net with ESMTP; 07 Oct 2014 16:37:58 +1030 Message-ID: <54338354.2070507@internode.on.net> Date: Tue, 07 Oct 2014 17:08:20 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: Re: [pcp] qa/652 - systemd pmda References: <5418A165.5070809@internode.on.net> X-ASG-Orig-Subj: Re: [pcp] qa/652 - systemd pmda In-Reply-To: <5418A165.5070809@internode.on.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1412662078 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10287 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- On 17/09/14 06:45, Ken McDonell wrote: > Is failing and I can't see why ... no indication in the systemd pmda log > file. > > .out.bad and .full files attached. Frank helped with this a little, but we did not resolve it and dismissed it as an old platform issue. Well, the problem is back on Centos 7 ... so I'm guessing it is worth understanding what's going wrong ... and for that I need some help as I know less than zippo about systemd or the PMDA. Now it looks like the systemd pmda is running with the wrong uid ... on this system it is adm but strace shows open("/run/log/journal/6b092c3ed31ed3412f8508b0df478269/system.journal", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied) and sure enough /run/log/.../system.journal is owned by root kenj@vm08:~/src/pcp/qa$ ls -l /run/log/journal/6b092c3ed31ed3412f8508b0df478269/system.journal -rw-r-----. 1 root root 6516736 Oct 7 17:01 /run/log/journal/6b092c3ed31ed3412f8508b0df478269/system.journal Forcing the PMDA to run as root did not really help, although the problem changed ... 8^)> ... the PMDA now fails like this ... select(6, [0 5], NULL, NULL, {48, 373602}) = 1 (in [5], left {43, 698763}) clock_gettime(CLOCK_MONOTONIC, {32506, 307799717}) = 0 read(5, "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 4112) = 32 read(5, 0x7fff7744e8a0, 4112) = -1 EAGAIN (Resource temporarily unavailable) clock_gettime(CLOCK_MONOTONIC, {32506, 307913495}) = 0 read(-1, 0x7fff7744e8a0, 4112) = -1 EBADF (Bad file descriptor) gettimeofday({1412661985, 179860}, NULL) = 0 select(6, [0 5], NULL, NULL, {60, 0}) = 1 (in [5], left {59, 968625}) clock_gettime(CLOCK_MONOTONIC, {32506, 340349457}) = 0 read(5, "\2\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0system.journal\0\0", 4112) = 32 read(5, 0x7fff7744e8a0, 4112) = -1 EAGAIN (Resource temporarily unavailable) clock_gettime(CLOCK_MONOTONIC, {32506, 340704185}) = 0 read(-1, 0x7fff7744e8a0, 4112) = -1 EBADF (Bad file descriptor) gettimeofday({1412661985, 212670}, NULL) = 0 select(6, [0 5], NULL, NULL, {60, 0} From nscott@redhat.com Tue Oct 7 01:44:59 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 9555A7F3F for ; Tue, 7 Oct 2014 01:44:59 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 757C18F804C for ; Mon, 6 Oct 2014 23:44:56 -0700 (PDT) X-ASG-Debug-ID: 1412664294-04bdf003a17d8e10001-S8gJnT Received: from mx3-phx2.redhat.com (mx3-phx2.redhat.com [209.132.183.24]) by cuda.sgi.com with ESMTP id S5AotsNkM5XiYCpL (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Mon, 06 Oct 2014 23:44:54 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.24 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx3-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s976ip8M004422; Tue, 7 Oct 2014 02:44:51 -0400 Date: Tue, 7 Oct 2014 02:44:51 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Ken McDonell Cc: pcp@oss.sgi.com Message-ID: <1851472641.62780119.1412664291586.JavaMail.zimbra@redhat.com> In-Reply-To: <54337DAE.6000107@internode.on.net> References: <54337DAE.6000107@internode.on.net> Subject: Re: [pcp] pcp updates - nfs4 request metrics MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] pcp updates - nfs4 request metrics Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: pcp updates - nfs4 request metrics Thread-Index: wgdO+r9soEgiRPvkblaWT8phMGq7SA== X-Barracuda-Connect: mx3-phx2.redhat.com[209.132.183.24] X-Barracuda-Start-Time: 1412664294 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10287 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Hi Ken, ----- Original Message ----- > qa/232 was failing on Centos 7. > > There are NR_RPC4_SVR_COUNTERS instances, numbered 1 (not 0) to > NR_RPC4_SVR_COUNTERS, but the values are stored in [0] ... > [NR_RPC4_SVR_COUNTERS-1] of proc_net_rpc->server.reqcounts4[]. > > It would be _really_ good to have someone who knows the nfs code in the linux > pmda to review these changes, and if they are correct to check that similar > mismatches are not happening for the other nfs/rpc metrics. I recently spent several hours pouring over the NFS kernel code to get a handle on these metrics (adding the nfs4.1 ops support), as well as the userspace code in nfsstat(1). Is the test failing with latest dev? (its not failing here - can you send through the .bad file you saw? thanks!) Was this commit in place for this QA run: commit b690dcb8049963a6c7516cc5c89312dd9095840a Author: Nathan Scott Date: Fri Sep 26 10:44:01 2014 +1000 Update qa/232 to handle recent nfs4.1 indom changes The odd non-zero indexing is a quirk of the NFS v4 code, and does not affect the earlier NFS variants. I was fairly confident that the pmdalinux code was correct when I left it, so I'm keen to get more of a handle on things from that .bad file. cheers. -- Nathan From kenj@internode.on.net Tue Oct 7 02:51:10 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 478DE7F3F for ; Tue, 7 Oct 2014 02:51:10 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 345ED304053 for ; Tue, 7 Oct 2014 00:51:09 -0700 (PDT) X-ASG-Debug-ID: 1412668267-04cb6c50e774c360001-S8gJnT Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id 9CHLsPZ95VVHgFIe for ; Tue, 07 Oct 2014 00:51:07 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AiQCAH+aM1R20Xt7PGdsb2JhbAANUoQ5gwKEN8x4AoEiAQYBAQEBOIQ8AQEBBCMEUQEMBAsUBAICBRYLAgIJAwIBAgExFAYNAQUCAQGzc3iVDQEXgSyPGQeCd4FUBZ5rhmyGVosIWIEGB4E9AQEB Received: from ppp118-209-123-123.lns20.mel4.internode.on.net (HELO [192.168.1.100]) ([118.209.123.123]) by ipmail04.adl6.internode.on.net with ESMTP; 07 Oct 2014 18:21:05 +1030 Message-ID: <54339B80.5010607@internode.on.net> Date: Tue, 07 Oct 2014 18:51:28 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: Nathan Scott CC: pcp@oss.sgi.com Subject: Re: [pcp] pcp updates - nfs4 request metrics References: <54337DAE.6000107@internode.on.net> <1851472641.62780119.1412664291586.JavaMail.zimbra@redhat.com> X-ASG-Orig-Subj: Re: [pcp] pcp updates - nfs4 request metrics In-Reply-To: <1851472641.62780119.1412664291586.JavaMail.zimbra@redhat.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1412668267 X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10289 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- On 07/10/14 17:44, Nathan Scott wrote: > Hi Ken, > > ----- Original Message ----- >> qa/232 was failing on Centos 7. >> >> There are NR_RPC4_SVR_COUNTERS instances, numbered 1 (not 0) to >> NR_RPC4_SVR_COUNTERS, but the values are stored in [0] ... >> [NR_RPC4_SVR_COUNTERS-1] of proc_net_rpc->server.reqcounts4[]. >> >> It would be _really_ good to have someone who knows the nfs code in the linux >> pmda to review these changes, and if they are correct to check that similar >> mismatches are not happening for the other nfs/rpc metrics. > > I recently spent several hours pouring over the NFS kernel code to > get a handle on these metrics (adding the nfs4.1 ops support), as > well as the userspace code in nfsstat(1). Is the test failing with > latest dev? (its not failing here - can you send through the .bad > file you saw? thanks!) Was this commit in place for this QA run: > > commit b690dcb8049963a6c7516cc5c89312dd9095840a > Author: Nathan Scott > Date: Fri Sep 26 10:44:01 2014 +1000 > > Update qa/232 to handle recent nfs4.1 indom changes > > > The odd non-zero indexing is a quirk of the NFS v4 code, and does > not affect the earlier NFS variants. I was fairly confident that > the pmdalinux code was correct when I left it, so I'm keen to get > more of a handle on things from that .bad file. first line of output below appears in 232.out.bad but more telling is that the 59th member of the indom "[58] 59 reclaim_comp" is not returned by pmFetch with no profile. kenj@vm08:~/src/pcp/qa$ src/torture_indom -v nfs4.server.reqs number of instances from unprofiled fetch (58) != that for pmGetInDom (59) unprofiled fetch: [0] 1 op0-unused [1] 2 op1-unused [2] 3 op2-future [3] 4 access [4] 5 close [5] 6 commit [6] 7 create [7] 8 delegpurge [8] 9 delegreturn [9] 10 getattr [10] 11 getfh [11] 12 link [12] 13 lock [13] 14 lockt [14] 15 locku [15] 16 lookup [16] 17 lookup_root [17] 18 nverify [18] 19 open [19] 20 openattr [20] 21 open_conf [21] 22 open_dgrd [22] 23 putfh [23] 24 putpubfh [24] 25 putrootfh [25] 26 read [26] 27 readdir [27] 28 readlink [28] 29 remove [29] 30 rename [30] 31 renew [31] 32 restorefh [32] 33 savefh [33] 34 secinfo [34] 35 setattr [35] 36 setcltid [36] 37 setcltidconf [37] 38 verify [38] 39 write [39] 40 rellockowner [40] 41 bc_ctl [41] 42 bind_conn [42] 43 exchange_id [43] 44 create_ses [44] 45 destroy_ses [45] 46 free_stateid [46] 47 getdirdeleg [47] 48 getdevinfo [48] 49 getdevlist [49] 50 layoutcommit [50] 51 layoutget [51] 52 layoutreturn [52] 53 secinfononam [53] 54 sequence [54] 55 set_ssv [55] 56 test_stateid [56] 57 want_deleg [57] 58 destroy_clid pmGetInDom: [0] 1 op0-unused [1] 2 op1-unused [2] 3 op2-future [3] 4 access [4] 5 close [5] 6 commit [6] 7 create [7] 8 delegpurge [8] 9 delegreturn [9] 10 getattr [10] 11 getfh [11] 12 link [12] 13 lock [13] 14 lockt [14] 15 locku [15] 16 lookup [16] 17 lookup_root [17] 18 nverify [18] 19 open [19] 20 openattr [20] 21 open_conf [21] 22 open_dgrd [22] 23 putfh [23] 24 putpubfh [24] 25 putrootfh [25] 26 read [26] 27 readdir [27] 28 readlink [28] 29 remove [29] 30 rename [30] 31 renew [31] 32 restorefh [32] 33 savefh [33] 34 secinfo [34] 35 setattr [35] 36 setcltid [36] 37 setcltidconf [37] 38 verify [38] 39 write [39] 40 rellockowner [40] 41 bc_ctl [41] 42 bind_conn [42] 43 exchange_id [43] 44 create_ses [44] 45 destroy_ses [45] 46 free_stateid [46] 47 getdirdeleg [47] 48 getdevinfo [48] 49 getdevlist [49] 50 layoutcommit [50] 51 layoutget [51] 52 layoutreturn [52] 53 secinfononam [53] 54 sequence [54] 55 set_ssv [55] 56 test_stateid [56] 57 want_deleg [57] 58 destroy_clid [58] 59 reclaim_comp From mgoodwin@redhat.com Tue Oct 7 06:45:48 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id C4E0B7F3F for ; Tue, 7 Oct 2014 06:45:48 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id A3E1D304032 for ; Tue, 7 Oct 2014 04:45:45 -0700 (PDT) X-ASG-Debug-ID: 1412682344-04bdf0039f7e3d50001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 8ullYZFALqv5MmkY (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 07 Oct 2014 04:45:44 -0700 (PDT) X-Barracuda-Envelope-From: mgoodwin@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s97BjhE4005531 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Tue, 7 Oct 2014 07:45:44 -0400 Received: from [10.64.50.5] (vpn1-50-5.bne.redhat.com [10.64.50.5]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s97BjgB4027184; Tue, 7 Oct 2014 07:45:42 -0400 Message-ID: <5433D265.7030002@redhat.com> Date: Tue, 07 Oct 2014 22:45:41 +1100 From: Mark Goodwin User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: "Frank Ch. Eigler" CC: pcp developers Subject: Re: PMAPI observations re. converting an app to pcp References: <20140925180407.GA18679@redhat.com> <5431FDFD.6020208@redhat.com> X-ASG-Orig-Subj: Re: PMAPI observations re. converting an app to pcp In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412682344 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On 10/07/2014 04:38 AM, Frank Ch. Eigler wrote: [..] > > Thanks, I'll study your code and steal lots of bits. I'm leaning > toward a little simpler solution in some ways. For example, how about > just use one pcp context per "fetch-group"? IIRC, the optFetch API that fetchGroup uses decides how many contexts to use based (in part) on the instance profiles. > That would make it less > necessary to have special time-setting or profile-manipulation > wrappers, since it would compose with the normal PMAPI functions. well, you'll still need the pmSetMode() wrapper for multiple contexts because multiple hosts or archives are supported. So __pmFetchGroupArchiveMode just walks all of the contexts in each of the fetch groups, vis : void __pmFetchGroupArchiveMode(int mode, const struct timeval *when, int interval) { int fg; int c; fetchGroup *group; int curcontext = pmWhichContext(); for (fg=0; fg < nfetchGroups; fg++) { group = &fetchGroups[fg]; for (c=0; c < group->ncontexts; c++) { if (group->contexts[c] >= 0) { pmUseContext(group->contexts[c]); pmSetMode(mode, when, interval); } } } if (curcontext >= 0) pmUseContext(curcontext); } BTW, I doubt this code is thread safe ... From pcp-announce-bounces@oss.sgi.com Tue Oct 7 17:35:24 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from oss.sgi.com (localhost [IPv6:::1]) by oss.sgi.com (Postfix) with ESMTP id AAD717F4E; Tue, 7 Oct 2014 17:35:24 -0500 (CDT) X-Original-To: pcp-announce@oss.sgi.com Delivered-To: pcp-announce@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 73F9F7F3F for ; Tue, 7 Oct 2014 17:35:23 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 548DE304059 for ; Tue, 7 Oct 2014 15:35:23 -0700 (PDT) X-ASG-Debug-ID: 1412721317-04cbb018ae112f0001-87ZIJf Received: from mx5-phx2.redhat.com (mx5-phx2.redhat.com [209.132.183.37]) by cuda.sgi.com with ESMTP id zrRfUxLwxd6jBz5k (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 07 Oct 2014 15:35:18 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.37 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx5-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s97MZHiK030389 for ; Tue, 7 Oct 2014 18:35:17 -0400 Date: Tue, 7 Oct 2014 18:35:17 -0400 (EDT) From: Nathan Scott To: pcp-announce Message-ID: <384242147.63419978.1412721317488.JavaMail.zimbra@redhat.com> In-Reply-To: <1968683202.60097085.1412217054636.JavaMail.zimbra@redhat.com> References: <1968683202.60097085.1412217054636.JavaMail.zimbra@redhat.com> MIME-Version: 1.0 X-ASG-Orig-Subj: Re: PCP developers conference call - updated agenda X-Originating-IP: [10.5.82.6] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: PCP developers conference call Thread-Index: is+tBAM7Gw61JXdO4KtrNRZTS9utJeZVS6vr X-Barracuda-Connect: mx5-phx2.redhat.com[209.132.183.37] X-Barracuda-Start-Time: 1412721318 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10310 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Subject: Re: [pcp-announce] PCP developers conference call - updated agenda X-BeenThere: pcp-announce@oss.sgi.com X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Nathan Scott List-Id: Performance Co-Pilot announcements List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pcp-announce-bounces@oss.sgi.com Sender: pcp-announce-bounces@oss.sgi.com Hi all, Quick reminder - this call is tomorrow, updated agenda below. Expected attendees - michele (if not asleep), kenj, mgoodwin, nathans, lberk, brolley and fche. If anyone else is planning to call in, feel free to lemme know and we'll wait until you are on - else, catch up via the recording or written notes. ----- Original Message ----- > [...] > Details for the call: > > - Starting at 7am on Thursday 9th October in Melbourne, track > your local start date/time here: > http://www.timeanddate.com/worldclock/fixedtime.html?msg=PCP+Developers+Meeting&iso=20141009T07&p1=152&ah=2 > > - The phone number to call from your location is here: > https://www.intercallonline.com/listNumbersByCode.action?confCode=6409801839 > - When prompted, enter the conference code: 6409801839 Agenda: - welcome - any general news - website revamp request/options [nathans, michele] - easy hacks reference, ala libreoffice [nathans, michele] https://wiki.documentfoundation.org/Development/Easy_Hacks - quality assurance - status [kenj, nathans] - pcp culture and zen of QA [nathans] - non-intrusive QA testing for pcp [nathans, fche, kenj] - source trees - code encumbrance concerns, a history [nathans] - issues with separate trees, pcp web status [fche, nathans] - discussion around the maintenance role [fche] - spreading different aspects amongst a larger pool of people: - reviewing/merging random community patches - improving infrastructure - developing technical sub-area - testing on an ongoing basis - fixing filed bugs - engineering releases - packaging in distros - providing technical direction - building consensus in community - breaking ties - WIP / future work - local-context pmlogger [mgoodwin] - multi-archive libpcp support [brolley] - generic json pmda, python pmda module work [dsmith?, fche] - smaller packages for containers [nathans] - simplifying PMAPI access (new APIs, caching) [fche, mgoodwin, kenj?] - jvm instrumentation planning [nathans] - ... any other hacking in progress? - anyone need help with anything? - any other topics? cheers. -- Nathan _______________________________________________ pcp-announce mailing list pcp-announce@oss.sgi.com http://oss.sgi.com/mailman/listinfo/pcp-announce From nscott@redhat.com Tue Oct 7 20:35:08 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 141327F3F for ; Tue, 7 Oct 2014 20:35:08 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 96573AC005 for ; Tue, 7 Oct 2014 18:35:04 -0700 (PDT) X-ASG-Debug-ID: 1412732102-04cb6c7707182a0001-S8gJnT Received: from mx5-phx2.redhat.com (mx5-phx2.redhat.com [209.132.183.37]) by cuda.sgi.com with ESMTP id aPgKo2FrXJYzqZM0 (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 07 Oct 2014 18:35:03 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.37 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx5-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s981Z0f3029383; Tue, 7 Oct 2014 21:35:00 -0400 Date: Tue, 7 Oct 2014 21:35:00 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Ken McDonell Cc: pcp@oss.sgi.com Message-ID: <326810754.63465628.1412732100431.JavaMail.zimbra@redhat.com> In-Reply-To: <54339B80.5010607@internode.on.net> References: <54337DAE.6000107@internode.on.net> <1851472641.62780119.1412664291586.JavaMail.zimbra@redhat.com> <54339B80.5010607@internode.on.net> Subject: Re: [pcp] pcp updates - nfs4 request metrics MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] pcp updates - nfs4 request metrics Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: pcp updates - nfs4 request metrics Thread-Index: j7ziw/TyteFOqZbKW8zyt6Ug2L7vBA== X-Barracuda-Connect: mx5-phx2.redhat.com[209.132.183.37] X-Barracuda-Start-Time: 1412732102 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10317 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... ----- Original Message ----- > [...] > but more telling is that the 59th member of the indom "[58] 59 reclaim_comp" > is not returned by pmFetch with no profile. > > kenj@vm08:~/src/pcp/qa$ src/torture_indom -v nfs4.server.reqs > number of instances from unprofiled fetch (58) != that for pmGetInDom (59) Yep, its a real bug in my recent changes - thanks!!! I've extended your earlier fix slightly, and added more QA testing to also verify the value at the final inst array element (via test-data-injection in qa/731). cheers. -- Nathan From nscott@redhat.com Tue Oct 7 20:38:35 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 0B6AC7F3F for ; Tue, 7 Oct 2014 20:38:35 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id DFBDA8F8050 for ; Tue, 7 Oct 2014 18:38:31 -0700 (PDT) X-ASG-Debug-ID: 1412732309-04cb6c770518480001-S8gJnT Received: from mx3-phx2.redhat.com (mx3-phx2.redhat.com [209.132.183.24]) by cuda.sgi.com with ESMTP id iq9C8BIfUvYyYsHP (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 07 Oct 2014 18:38:30 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.24 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx3-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s981cTXR008009 for ; Tue, 7 Oct 2014 21:38:29 -0400 Date: Tue, 7 Oct 2014 21:38:29 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: pcp Message-ID: <1374760692.63468002.1412732309065.JavaMail.zimbra@redhat.com> Subject: pcp updates: qa, bug fixes MIME-Version: 1.0 X-ASG-Orig-Subj: pcp updates: qa, bug fixes Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: pcp updates: qa, bug fixes Thread-Index: l2wIY/as50uWRyqfOH38JAq8IT80zA== X-Barracuda-Connect: mx3-phx2.redhat.com[209.132.183.24] X-Barracuda-Start-Time: 1412732309 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10317 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Changes committed to git://git.pcp.io/pcp.git dev Nathan Scott (4): Update pmdads389 man transition after Marko reviewed Reinstate dropped prefix setting in qa/361, thanks Ken Revert "linux pmda - nfs4 metrics" pmdalinux: nfsv4 operations metrics fix reworked Ken McDonell (3): qa/151 - don't run for 6 days after a timezone change libpcp - fix for pmUnitsStr_r buffer overrun linux pmda - nfs4 metrics man/man3/pmunitsstr.3 | 9 ++++ qa/151 | 13 ++++++ qa/361 | 1 qa/732.out | 5 +- qa/746 | 29 +++++++++++++ qa/746.out | 14 ++++++ qa/group | 1 qa/linux/nfsrpc-root-001.tgz |binary src/libpcp/src/units.c | 7 +++ src/pmdas/ds389/pmdads389.pl | 87 ----------------------------------------- src/pmdas/linux/pmda.c | 16 +++---- src/pmdas/linux/proc_net_rpc.c | 34 ++++++++-------- src/pmdas/linux/proc_net_rpc.h | 4 + From fche@redhat.com Tue Oct 7 22:21:34 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id C9DDD7F3F for ; Tue, 7 Oct 2014 22:21:34 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id A93678F8049 for ; Tue, 7 Oct 2014 20:21:34 -0700 (PDT) X-ASG-Debug-ID: 1412738489-04bdf028761c910001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id gG0Jux9cd42ddHwe (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 07 Oct 2014 20:21:30 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s983LQK1029018 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Tue, 7 Oct 2014 23:21:26 -0400 Received: from fche.csb (vpn-233-63.phx2.redhat.com [10.3.233.63]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s983LPvl019023; Tue, 7 Oct 2014 23:21:25 -0400 Received: by fche.csb (Postfix, from userid 2569) id 3F12C5853E; Tue, 7 Oct 2014 23:21:25 -0400 (EDT) To: Ken McDonell Cc: pcp@oss.sgi.com Subject: Re: qa/652 - systemd pmda References: <5418A165.5070809@internode.on.net> <54338354.2070507@internode.on.net> X-ASG-Orig-Subj: Re: qa/652 - systemd pmda From: fche@redhat.com (Frank Ch. Eigler) Date: Tue, 07 Oct 2014 23:21:25 -0400 In-Reply-To: <54338354.2070507@internode.on.net> (Ken McDonell's message of "Tue, 07 Oct 2014 17:08:20 +1100") Message-ID: User-Agent: Gnus/5.1008 (Gnus v5.10.8) Emacs/21.4 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412738490 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 kenj wrote: > [...] Now it looks like the systemd pmda is running with the wrong > uid ... on this system it is adm but strace shows > > open("/run/log/journal/6b092c3ed31ed3412f8508b0df478269/system.journal", > O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied) I wonder if strace is misleading here, as though it interfered with the setuid process. systemd marks its journal files with POSIX ACLs in order to permit a variety of users/groups to get at the data. On my RHEL7 VM, # getfacl /var/log/journal/.../system.journal [...] group:adm:r-x [...] is included, and indeed the systemd pmda running as adm:adm (3:4) can get at the data without -EACCES, and the 652 test passes nicely. > [...] Forcing the PMDA to run as root did not really help, although > the problem changed ... 8^)> ... the PMDA now fails like this ... > [...] I'm afraid nothing jumps out at me in there, except perhaps that bad read(fd=-1), which I'm also seeing on my working machine. ISTR having some success using systemtap to trace the internal activity of the pmda. Interested in giving that a try? - FChE From kenj@internode.on.net Tue Oct 7 22:53:40 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id A87857F3F for ; Tue, 7 Oct 2014 22:53:40 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 887ED8F8040 for ; Tue, 7 Oct 2014 20:53:40 -0700 (PDT) X-ASG-Debug-ID: 1412740415-04cb6c77051cd20001-S8gJnT Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by cuda.sgi.com with ESMTP id 2VwnqRqCvmB0kFLZ for ; Tue, 07 Oct 2014 20:53:35 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.145 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AuUCAK60NFR20YErPGdsb2JhbAANUoc9hDfMZQEGAQEBATiEZlUuCAIFFgsCCwMCAQIBMQYUDQgBAbRLeJUhgSySFoFUBbdBgyIBAQE Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail06.adl6.internode.on.net with ESMTP; 08 Oct 2014 14:23:24 +1030 Message-ID: <5434B54D.1030401@internode.on.net> Date: Wed, 08 Oct 2014 14:53:49 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: PCP Subject: compilation warnings on Mac OS X Content-Type: text/plain; charset=utf-8 X-ASG-Orig-Subj: compilation warnings on Mac OS X Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl6.internode.on.net[150.101.137.145] X-Barracuda-Start-Time: 1412740415 X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=FB_WORD1_END_DOLLAR X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10321 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 FB_WORD1_END_DOLLAR BODY: Looks like a word ending with a $ I'm just getting my steam powered iMac back into QA harness. Does this look familiar to anyone? fuji:pcp kenj$ grep -i warning Logs/pcp ld: warning: directory '../libpcp_qed/src/build/release' following -L not found ld: warning: directory '../libpcp_qmc/src/build/release' following -L not found ld: warning: directory '../libpcp_qwt/src/build/release' following -L not found ld: warning: directory '../libpcp_qwt/src/build/release' following -L not found ld: warning: directory '../libpcp_qmc/src/build/release' following -L not found ld: warning: in ../../src/libpcp/src/libpcp.dylib, missing required architecture ppc in file ld: warning: in ../../src/libpcp_pmda/src/libpcp_pmda.dylib, missing required architecture ppc in file ld: warning: in ../../src/libpcp/src/libpcp.dylib, missing required architecture ppc in file ld: warning: in ../../src/libpcp_gui/src/libpcp_gui.dylib, missing required architecture ppc in file ld: warning: in ../../src/libpcp_import/src/libpcp_import.dylib, missing required architecture ppc in file ld: warning: in ../../src/libpcp_mmv/src/libpcp_mmv.dylib, missing required architecture ppc in file ld: warning: directory '../../../src/libpcp_qmc/src/build/release' following -L not found ld: warning: directory '../../../src/libpcp_qmc/src/build/release' following -L not found ld: warning: directory '../../../src/libpcp_qmc/src/build/release' following -L not found ld: warning: directory '../../../src/libpcp_qmc/src/build/release' following -L not found ld: warning: directory '../../../src/libpcp_qmc/src/build/Default' following -L not found ld: warning: directory '../../../src/libpcp_qmc/src/build/release' following -L not found ld: warning: directory '../../../src/libpcp_qmc/src/build/release' following -L not found ld: warning: directory '../../../src/libpcp_qmc/src/build/release' following -L not found ld: warning: directory '../../../src/libpcp_qmc/src/build/release' following -L not found ld: warning: directory '../../../src/libpcp_qmc/src/build/release' following -L not found From nscott@redhat.com Tue Oct 7 23:02:41 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 3A1147F3F for ; Tue, 7 Oct 2014 23:02:41 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 0A3D38F804C for ; Tue, 7 Oct 2014 21:02:37 -0700 (PDT) X-ASG-Debug-ID: 1412740956-04bdf028791dff0001-S8gJnT Received: from mx4-phx2.redhat.com (mx4-phx2.redhat.com [209.132.183.25]) by cuda.sgi.com with ESMTP id yfeouSH9rsTE015h (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 07 Oct 2014 21:02:36 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.25 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx4-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s9842WeQ029519; Wed, 8 Oct 2014 00:02:32 -0400 Date: Wed, 8 Oct 2014 00:02:32 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Ken McDonell Cc: PCP Message-ID: <1611886799.63506627.1412740952483.JavaMail.zimbra@redhat.com> In-Reply-To: <5434B54D.1030401@internode.on.net> References: <5434B54D.1030401@internode.on.net> Subject: Re: [pcp] compilation warnings on Mac OS X MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] compilation warnings on Mac OS X Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: compilation warnings on Mac OS X Thread-Index: AoyEeu8cT16UyfDJdvB/OzP1Pdg/Nw== X-Barracuda-Connect: mx4-phx2.redhat.com[209.132.183.25] X-Barracuda-Start-Time: 1412740956 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=FB_WORD1_END_DOLLAR, THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10321 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... 0.00 FB_WORD1_END_DOLLAR BODY: Looks like a word ending with a $ ----- Original Message ----- > I'm just getting my steam powered iMac back into QA harness. (is that an old PowerPC or any x86 model - the latter I think?) > Does this look familiar to anyone? > Yep... > fuji:pcp kenj$ grep -i warning Logs/pcp > ld: warning: directory '../libpcp_qed/src/build/release' following -L not > found These ones should be safe to ignore; they flow from platform differences between the generated makefiles, and the various debug/release builds on each being in different places IIRC. > ld: warning: in ../../src/libpcp/src/libpcp.dylib, missing required > architecture ppc in file This will be qmake trying to build "universal binaries" for the GUI tools (i.e. containing both PPC and x86 code in the one binary) but we no longer build libpcp that way - tossed sometime back, and we build only x86 now. There may be qmake mechanisms to make it build GUI tools only for x86...? Or we could start building universal binaries in PCP again (IIRC, that'd involve using multiple -arch invocations in CFLAGS - but I don't think we have any users on PowerPC Macs anymore). cheers. -- Nathan From kenj@internode.on.net Wed Oct 8 01:04:40 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id C340C7F3F for ; Wed, 8 Oct 2014 01:04:40 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id B1FDC8F8035 for ; Tue, 7 Oct 2014 23:04:37 -0700 (PDT) X-ASG-Debug-ID: 1412748271-04cb6c770621040001-S8gJnT Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by cuda.sgi.com with ESMTP id kW3o2xmkNCIOowVi for ; Tue, 07 Oct 2014 23:04:32 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.145 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Am0DAGPTNFR20YErPGdsb2JhbAANUoNhWIMEhDfDe4hxAQYBAQEBOIRmgQUGAgUhAhECMg4ZBgIBAYhHrAx4lRaBLI81gmGBVAWGK5AKoRBYgkoBAQE Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail06.adl6.internode.on.net with ESMTP; 08 Oct 2014 16:34:13 +1030 Message-ID: <5434D3F6.30303@internode.on.net> Date: Wed, 08 Oct 2014 17:04:38 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: pcp updates Content-Type: text/plain; charset=utf-8 X-ASG-Orig-Subj: pcp updates Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl6.internode.on.net[150.101.137.145] X-Barracuda-Start-Time: 1412748271 X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10323 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Changes committed to git://git.performancecopilot.org/kenj/pcp.git dev qa/260 | 19 ++++------- qa/260.out | 46 ++++++++++++++++++++++++++ qa/555 | 1 qa/753 | 16 +++++---- qa/753.out | 16 ++++----- qa/src/GNUlocaldefs | 2 - src/pmdas/sample/help | 40 +++++++++++++++++------ src/pmdas/sample/src/sample.c | 72 ++++++++++++++++++++++++++++++------------ 8 files changed, 155 insertions(+), 57 deletions(-) commit 0d9a1094b32bee79c4c8f0998cc36a88ef703913 Author: Ken McDonell Date: Wed Oct 8 12:23:34 2014 +1100 qa makefile - missed change when qa/746 added New QA program, badUnitsStr_r.c commit 6558d65bc7f6b967b07818b7547ea9b782297d5b Author: Ken McDonell Date: Wed Oct 8 12:22:47 2014 +1100 qa/555 - more diags to try and track down failures commit 36eaf683c6e4fe01e48ae32b43e03cf2bdcbc21b Author: Ken McDonell Date: Wed Oct 8 12:19:58 2014 +1100 qa/753 - changes to improve test stability and determinism - use sample.colour instead of kernel.all.load - reset sample.colour before any tests are done - add diagnostics Previously, if the load average was updated (and changed) during the test, the results were non-deterministic. commit 9adbaf01fb50ab564d2ba409c8c217cdacc6aef1 Author: Ken McDonell Date: Wed Oct 8 12:16:48 2014 +1100 sample pmda - add pmStore support for some metrics These metrics share the same underlying (internal) counter: sample.colour sample.mirage sample.mirage_longlong pmStore into any of them will reset the underlying counter to the value supplied to pmStore. Useful for QA to reset the metrics and produce a deterministic sequence of values. commit 14819716243fc13be297a401998adcaf52390b3b Author: Ken McDonell Date: Wed Oct 8 09:01:34 2014 +1100 qa/260 - improve determinism - up sample interval from 0.2sec to 0.5sec - use pmcd.pdu_in.fetch and pmcd.pdu_out.result instead of pmcd.pdu_in.total and pmcd.pdu_out.total From makc@iinet.net.au Wed Oct 8 01:32:47 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 606227F3F for ; Wed, 8 Oct 2014 01:32:47 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 4D9358F8040 for ; Tue, 7 Oct 2014 23:32:47 -0700 (PDT) X-ASG-Debug-ID: 1412749961-04cbb018ad21ff0001-S8gJnT Received: from icp-osb-irony-out4.external.iinet.net.au (icp-osb-irony-out4.external.iinet.net.au [203.59.1.220]) by cuda.sgi.com with ESMTP id AcZahhiZE7WyvRAD for ; Tue, 07 Oct 2014 23:32:42 -0700 (PDT) X-Barracuda-Envelope-From: makc@iinet.net.au X-Barracuda-Apparent-Source-IP: 203.59.1.220 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvoKADnZNFQYBzP9/2dsb2JhbABfgw5TWII0tTYGkyaIXRYBe4QEAQEEOj8QCw0nEiwrBog8ARTCJ4YgjnYFi1yKWYcPiAuNf4QDHS+CSgEBAQ X-IronPort-AV: E=Sophos;i="5.04,675,1406563200"; d="scan'208";a="369043511" Received: from unknown (HELO emma.crabbed.net) ([24.7.51.253]) by icp-osb-irony-out4.iinet.net.au with ESMTP; 08 Oct 2014 14:32:39 +0800 Received: by emma.crabbed.net (Postfix, from userid 16314) id 52596AAEE25; Tue, 7 Oct 2014 23:32:35 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <21556.55939.297115.606184@iinet.net.au> Date: Tue, 7 Oct 2014 23:32:35 -0700 From: Max Matveev To: Ken McDonell Cc: PCP Subject: Re: [pcp] compilation warnings on Mac OS X In-Reply-To: <5434B54D.1030401@internode.on.net> X-ASG-Orig-Subj: Re: [pcp] compilation warnings on Mac OS X References: <5434B54D.1030401@internode.on.net> X-Mailer: VM 7.17 under 21.4 (patch 19) "Constant Variable" XEmacs Lucid X-Barracuda-Connect: icp-osb-irony-out4.external.iinet.net.au[203.59.1.220] X-Barracuda-Start-Time: 1412749961 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10324 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- On Wed, 08 Oct 2014 14:53:49 +1100, Ken McDonell wrote: ken> I'm just getting my steam powered iMac back into QA harness. ken> Does this look familiar to anyone? This is highly dependent on the version of Mac OS X and on the compiler/linker combination: I've had to tweak builds for 10.8 and 10.9. BTW, question for Nathan - where should I get QT for Mac OSX from and which version? I'm trying to debug pmchart stutter in live mode when monitoring local host: archive monitoring is OK, live mode shows big gaps in graphs, almost like it takes too long to fetch simple metrics like kernel.all.cpu.{user,sys}. pmchart is from the pre-built 3.9.9 package from oss.sgi.com. max From nscott@redhat.com Wed Oct 8 01:40:11 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 669547F3F for ; Wed, 8 Oct 2014 01:40:11 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 53A24304053 for ; Tue, 7 Oct 2014 23:40:11 -0700 (PDT) X-ASG-Debug-ID: 1412750409-04cb6c770522510001-S8gJnT Received: from mx4-phx2.redhat.com (mx4-phx2.redhat.com [209.132.183.25]) by cuda.sgi.com with ESMTP id WsJCbDGBfuUinRBN (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 07 Oct 2014 23:40:09 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.25 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx4-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s986e1HU019943; Wed, 8 Oct 2014 02:40:01 -0400 Date: Wed, 8 Oct 2014 02:40:01 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Max Matveev Cc: Ken McDonell , PCP Message-ID: <33917552.64048263.1412750401328.JavaMail.zimbra@redhat.com> In-Reply-To: <21556.55939.297115.606184@iinet.net.au> References: <5434B54D.1030401@internode.on.net> <21556.55939.297115.606184@iinet.net.au> Subject: Re: [pcp] compilation warnings on Mac OS X MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] compilation warnings on Mac OS X Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: compilation warnings on Mac OS X Thread-Index: zCOrpR9GIlJPPrMQpDCgG19d2jtOoA== X-Barracuda-Connect: mx4-phx2.redhat.com[209.132.183.25] X-Barracuda-Start-Time: 1412750409 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10324 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Hi Max, ----- Original Message ----- > On Wed, 08 Oct 2014 14:53:49 +1100, Ken McDonell wrote: > > ken> I'm just getting my steam powered iMac back into QA harness. > ken> Does this look familiar to anyone? > > This is highly dependent on the version of Mac OS X and on the > compiler/linker combination: I've had to tweak builds for 10.8 and > 10.9. Ugh. :( > BTW, question for Nathan - where should I get QT for Mac OSX from and > which version? I've not setup a new Mac build machine for awhile - I guess the best bet would be to go to www.qt.io and follow the download links. > I'm trying to debug pmchart stutter in live mode when > monitoring local host: archive monitoring is OK, live mode shows big > gaps in graphs, almost like it takes too long to fetch simple metrics > like kernel.all.cpu.{user,sys}. pmchart is from the pre-built 3.9.9 > package from oss.sgi.com. FWLIW, I've not seen that behaviour here (neither on Mac nor Linux). cheers. -- Nathan From mgoodwin@redhat.com Wed Oct 8 01:40:56 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: ** X-Spam-Status: No, score=3.0 required=5.0 tests=TVD_SUBJ_NUM_OBFU_MINFP autolearn=no version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 0DC597F3F for ; Wed, 8 Oct 2014 01:40:56 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id F0B7D30405F for ; Tue, 7 Oct 2014 23:40:55 -0700 (PDT) X-ASG-Debug-ID: 1412750454-04cbb018ab22550001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id cTKp8CVGNkFLyEdG (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 07 Oct 2014 23:40:55 -0700 (PDT) X-Barracuda-Envelope-From: mgoodwin@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s986esHC020288 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Wed, 8 Oct 2014 02:40:54 -0400 Received: from [10.64.50.29] (vpn1-50-29.bne.redhat.com [10.64.50.29]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s986epQv031112; Wed, 8 Oct 2014 02:40:52 -0400 Message-ID: <5434DC73.4080400@redhat.com> Date: Wed, 08 Oct 2014 17:40:51 +1100 From: Mark Goodwin User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: pcp CC: Bud Brown , Laurence Oberman Subject: iostat2pcp broken for iostat in RHEL/fedora Content-Type: text/plain; charset=ISO-8859-1; format=flowed X-ASG-Orig-Subj: iostat2pcp broken for iostat in RHEL/fedora Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412750455 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 iostat2pcp doesn't seem to like the iostat output on various RHEL and Fedora flavours - this was reported by Bud Brown here at Red Hat (cc:'d). Any idea what's wrong anyone? qa/373 works but it's using output captured from some other iostat version, not the locally installed iostat - the latter would be a better test no doubt ... $ iostat -t -x 2 10 > iostat.10 $ iostat2pcp iostat.10 iostat.pcp [26] Time: 03:49:59 PM Device: number of values? expected 12, found 3 $ more iostat.10 Linux 2.6.18-371.9.1.el5 (somehost.redhat.com) 10/07/2014 Time: 03:49:57 PM avg-cpu: %user %nice %system %iowait %steal %idle 19.09 0.04 8.10 1.06 0.00 71.71 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.17 27.04 4.82 4.64 63.77 254.59 33.67 0.14 15.26 3.45 3.26 sda1 0.00 0.00 0.00 0.00 0.00 0.00 13.34 0.00 1.27 1.25 0.00 sda2 0.00 0.00 0.00 0.00 0.00 0.00 14.72 0.00 2.01 2.01 0.00 sda3 0.00 0.00 0.00 0.00 0.00 0.00 13.40 0.00 2.34 2.34 0.00 sda4 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 2.00 2.00 0.00 sda5 0.17 27.04 4.82 4.64 63.77 254.59 33.67 0.14 15.26 3.45 3.26 sda6 0.00 0.00 0.00 0.00 0.00 0.00 19.97 0.00 9.25 9.13 0.00 sdb 0.24 84.28 0.90 66.04 34.03 1202.56 18.47 0.04 0.60 0.21 1.44 sdb1 0.00 0.00 0.00 0.00 1.11 0.01 868.25 0.00 40.23 37.23 0.00 sdb2 0.16 0.34 0.05 0.02 9.69 2.88 177.91 0.00 61.21 4.63 0.03 sdb3 0.08 83.92 0.73 66.02 22.29 1199.46 18.30 0.03 0.51 0.21 1.37 sdb4 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 6.42 6.42 0.00 sdb5 0.00 0.02 0.12 0.01 0.94 0.22 9.36 0.00 9.94 3.71 0.05 dm-0 0.00 0.00 0.12 0.03 0.94 0.22 8.00 0.00 17.80 3.18 0.05 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 7.99 0.00 5.83 2.42 0.00 dm-2 0.00 0.00 4.40 30.36 49.75 242.86 8.42 0.15 4.43 0.94 3.27 dm-3 0.00 0.00 0.59 1.34 14.01 11.73 13.32 0.26 136.46 0.28 0.05 Time: 03:49:59 PM avg-cpu: %user %nice %system %iowait %steal %idle 14.62 0.00 10.88 0.00 0.00 74.50 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 From nscott@redhat.com Wed Oct 8 01:57:50 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 8213F7F3F for ; Wed, 8 Oct 2014 01:57:50 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 5F9428F8040 for ; Tue, 7 Oct 2014 23:57:50 -0700 (PDT) X-ASG-Debug-ID: 1412751466-04cb6c770522e70001-S8gJnT Received: from mx4-phx2.redhat.com (mx4-phx2.redhat.com [209.132.183.25]) by cuda.sgi.com with ESMTP id 6O43ry5UPG4lutgz (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 07 Oct 2014 23:57:46 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.25 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx4-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s986vkNm022339 for ; Wed, 8 Oct 2014 02:57:46 -0400 Date: Wed, 8 Oct 2014 02:57:46 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: PCP Message-ID: <2096519073.64053413.1412751466090.JavaMail.zimbra@redhat.com> Subject: pcp updates: python3 build/packaging MIME-Version: 1.0 X-ASG-Orig-Subj: pcp updates: python3 build/packaging Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: pcp updates: python3 build/packaging Thread-Index: vtnseZD/UhDTUFGPTNyzQVjvuJGdIg== X-Barracuda-Connect: mx4-phx2.redhat.com[209.132.183.25] X-Barracuda-Start-Time: 1412751466 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10325 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Changes committed to git://git.pcp.io/pcp.git dev Nathan Scott (3): build: python3 configure/make/packaging support qa: additional hosts in my local mix build: use shortform URLs in local rpm packaging too GNUmakefile | 5 +- build/rpm/GNUmakefile | 1 build/rpm/fedora.spec | 32 +++++++++++++-- build/rpm/pcp.spec.in | 67 +++++++++++++++++++++---------- configure | 99 ++++++++++++++++++++++++++++++++++++++++++++--- configure.ac | 48 +++++++++++++++++++--- debian/rules | 2 qa/qa_hosts.master | 4 + src/include/builddefs.in | 20 +++++++++ src/python/.gitignore | 1 src/python/GNUmakefile | 28 ++++++++++--- From kenj@internode.on.net Wed Oct 8 06:04:22 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_FRT_PROFILE1 autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 57C027F3F for ; Wed, 8 Oct 2014 06:04:22 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id E7154AC008 for ; Wed, 8 Oct 2014 04:04:18 -0700 (PDT) X-ASG-Debug-ID: 1412766252-04cb6c77052b390001-S8gJnT Received: from ipmail06.adl2.internode.on.net (ipmail06.adl2.internode.on.net [150.101.137.129]) by cuda.sgi.com with ESMTP id mq82Vnnm5dFLi623 for ; Wed, 08 Oct 2014 04:04:13 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.129 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AmoDAAoZNVR20YErPGdsb2JhbAANUoc9hDfIFYMhAoEfAQYBAQEBOIQ8AQEBAwEjFUABDAQLEgYCAgUWCwICCQMCAQIBMQYOBg0BBwEBiDKsI3iVBQEXgSyONWMHgneBVAEEt0WDIgEBAQ Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail06.adl2.internode.on.net with ESMTP; 08 Oct 2014 21:34:11 +1030 Message-ID: <54351A44.5040601@internode.on.net> Date: Wed, 08 Oct 2014 22:04:36 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: Nathan Scott CC: PCP Subject: Re: [pcp] compilation warnings on Mac OS X References: <5434B54D.1030401@internode.on.net> <1611886799.63506627.1412740952483.JavaMail.zimbra@redhat.com> X-ASG-Orig-Subj: Re: [pcp] compilation warnings on Mac OS X In-Reply-To: <1611886799.63506627.1412740952483.JavaMail.zimbra@redhat.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl2.internode.on.net[150.101.137.129] X-Barracuda-Start-Time: 1412766252 X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=FB_WORD1_END_DOLLAR X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10330 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 FB_WORD1_END_DOLLAR BODY: Looks like a word ending with a $ On 08/10/14 15:02, Nathan Scott wrote: > > > ----- Original Message ----- >> I'm just getting my steam powered iMac back into QA harness. > > (is that an old PowerPC or any x86 model - the latter I think?) Yep, ... x86 ... I said "steam powered", not "horse drawn". >> Does this look familiar to anyone? >> > > Yep... > >> fuji:pcp kenj$ grep -i warning Logs/pcp >> ld: warning: directory '../libpcp_qed/src/build/release' following -L not >> found > > These ones should be safe to ignore; they flow from platform differences > between the generated makefiles, and the various debug/release builds on > each being in different places IIRC. The .pro files where the warnings come from contain lines like release:DESTDIR = build/debug debug:DESTDIR = build/release but the .pro files in the libraries that are being referenced do not contain these lines. Adding these lines to the latter .pro files removes these warnings and (after fixing a typo in the pmdumptext .pro file and one of the qa .pro files) runs the build to completion, but I'm not sure if that is going to be correct on other platforms. >> ld: warning: in ../../src/libpcp/src/libpcp.dylib, missing required >> architecture ppc in file > > This will be qmake trying to build "universal binaries" for the GUI tools > (i.e. containing both PPC and x86 code in the one binary) but we no longer > build libpcp that way - tossed sometime back, and we build only x86 now. No these are coming from the python code. From kenj@internode.on.net Wed Oct 8 14:34:44 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 8BFE97F3F for ; Wed, 8 Oct 2014 14:34:44 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 6AEAD30408F for ; Wed, 8 Oct 2014 12:34:41 -0700 (PDT) X-ASG-Debug-ID: 1412796878-04bdf02878478b0001-S8gJnT Received: from ipmail06.adl2.internode.on.net (ipmail06.adl2.internode.on.net [150.101.137.129]) by cuda.sgi.com with ESMTP id vGXhHa4bnATQAtDu for ; Wed, 08 Oct 2014 12:34:39 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.129 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AuYCAP2QNVR20YErPGdsb2JhbAANUpINwWiDIQKBIAEGAQEBATiEPQEBBDhAEQsSBgkWDwkDAgECATEGDhMIAQG4Z5VRGJBLFoQ1AQSPXadsgyIBAQE Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail06.adl2.internode.on.net with ESMTP; 09 Oct 2014 06:04:37 +1030 Message-ID: <543591E8.8060903@internode.on.net> Date: Thu, 09 Oct 2014 06:35:04 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: Re: [pcp] compilation warnings on Mac OS X References: <5434B54D.1030401@internode.on.net> <1611886799.63506627.1412740952483.JavaMail.zimbra@redhat.com> <54351A44.5040601@internode.on.net> X-ASG-Orig-Subj: Re: [pcp] compilation warnings on Mac OS X In-Reply-To: <54351A44.5040601@internode.on.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl2.internode.on.net[150.101.137.129] X-Barracuda-Start-Time: 1412796879 X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10345 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- On 08/10/14 22:04, Ken McDonell wrote: > ... >>> ld: warning: in ../../src/libpcp/src/libpcp.dylib, missing required >>> architecture ppc in file >> > ... > No these are coming from the python code. I have a fix for this. From minnus@buffalo.edu Wed Oct 8 14:36:44 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 2B4BD7F3F for ; Wed, 8 Oct 2014 14:36:44 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id CB385AC001 for ; Wed, 8 Oct 2014 12:36:40 -0700 (PDT) X-ASG-Debug-ID: 1412796998-04bdf02876479d0001-S8gJnT Received: from mtareserve1.acsu.buffalo.edu (mtareserve6.acsu.buffalo.edu [128.205.6.4]) by cuda.sgi.com with ESMTP id sPQEdCMKaTNnW2eE for ; Wed, 08 Oct 2014 12:36:39 -0700 (PDT) X-Barracuda-Envelope-From: minnus@buffalo.edu X-Barracuda-Apparent-Source-IP: 128.205.6.4 Received: from localmailA.acsu.buffalo.edu (localmaila.acsu.buffalo.edu [128.205.5.196]) by mtareserve1.acsu.buffalo.edu (Postfix) with ESMTP id 9D248659F; Wed, 8 Oct 2014 15:36:38 -0400 (EDT) Received: from localmailA.acsu.buffalo.edu (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id 98E5E506D; Wed, 8 Oct 2014 15:36:38 -0400 (EDT) Received: from localmailA.acsu.buffalo.edu (localhost [127.0.0.1]) by localmailA.acsu.buffalo.edu (Postfix) with ESMTP id E59DC5067; Wed, 8 Oct 2014 15:36:37 -0400 (EDT) Received: from smtp.buffalo.edu (smtp3.acsu.buffalo.edu [128.205.5.226]) by localmailA.acsu.buffalo.edu (Prefixe) with ESMTP id D7C815066; Wed, 8 Oct 2014 15:36:37 -0400 (EDT) Received: from prince.ccr.buffalo.edu (prince.ccr.buffalo.edu [128.205.40.45]) (Authenticated sender: minnus@buffalo.edu) by smtp.buffalo.edu (Postfix) with ESMTPSA id CCBB02F79; Wed, 8 Oct 2014 15:36:37 -0400 (EDT) Message-ID: <5435926F.6030004@buffalo.edu> Date: Wed, 08 Oct 2014 15:37:19 -0400 From: Martins Innus User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: Nathan Scott CC: pcp@oss.sgi.com Subject: Re: [pcp] hotproc rfc References: <536D28B4.6010504@buffalo.edu> <1139662762.4765310.1399862104653.JavaMail.zimbra@redhat.com> <54230FAF.2080201@buffalo.edu> <905561536.62723741.1412657864635.JavaMail.zimbra@redhat.com> X-ASG-Orig-Subj: Re: [pcp] hotproc rfc In-Reply-To: <905561536.62723741.1412657864635.JavaMail.zimbra@redhat.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-PM-EL-Spam-Prob: : 8% X-Barracuda-Connect: mtareserve6.acsu.buffalo.edu[128.205.6.4] X-Barracuda-Start-Time: 1412796999 X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10345 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Nathan, > Hi Martins, > > I've had time to go through the code in detail now - this looks like a > good step forward, and its much simpler than I expected. > Thanks for the review. > I think the approach of merging the two PMDAs has worked well. Since > hotproc was originally written, the dynamic metrics concept has come > along and I think it'll be a better match here than any namespace file > script approach. Some more notes in the review - but, if I can help > with coding this aspect, just let me know (I have some of this fresh > in-mind from other recent hacking). I'm going to focus on the dynamic metrics changes first and leave all the rest until I make sure I have this straight. More to follow on the rest of the comments. In general, any guidance on fleshing out the dynamic metrics framework would be great. > > > OK, I think the idea here (fullmetrictab, proc_init_hotproc, and > createHotprocMetric - and the namespace file scripting you talked > about earlier) are duplicating concepts from the dynamic namespace > support already used in the Linux pmdaproc. It seems like we can > fit this in better ... if the [hot]proc.{psinfo,id,memory,io,fd, > schedstat} subtrees become dynamic, they'd be able to share single > definition, help text, and so on. > > > So if I understand you correctly, the {psinfo,id,memory,io,fd, schedstat} metrics become dynamic both in proc and hotproc? For implementation, the steps would be something like?: 1. Clean out root_proc of metrics we want to make dynamic 2. Replicate the logic from cgroups.c and adapt to [hot]proc to create these on the fly 3. Even though these are dynamic, we reuse existing cluster and id numbers so as not to confuse client tools. It seems like the first step would be to convert the existing pmda to handle dynamic metrics for the appropriate clusters that already exist before we worry about hotproc? Do you want me to have a go at that or do you have a specific plan in mind already? Thanks Martins From kenj@internode.on.net Wed Oct 8 18:33:22 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 814277F3F for ; Wed, 8 Oct 2014 18:33:22 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 61AE48F8033 for ; Wed, 8 Oct 2014 16:33:19 -0700 (PDT) X-ASG-Debug-ID: 1412811193-04cbb018ab4e710001-S8gJnT Received: from ipmail07.adl2.internode.on.net (ipmail07.adl2.internode.on.net [150.101.137.131]) by cuda.sgi.com with ESMTP id pKtN8J7mxo51gpHp for ; Wed, 08 Oct 2014 16:33:13 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.131 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApQBAI/INVR20YEr/2dsb2JhbAANUoNhg1zID4hvAYUoFUAwBgIFFgsCCwMCAQIBPxkGAgEBiEewJ3iUd4EsjzWCYYFUBZY1oRRYgkoBAQE Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail07.adl2.internode.on.net with ESMTP; 09 Oct 2014 10:03:12 +1030 Message-ID: <5435C9C7.5050407@internode.on.net> Date: Thu, 09 Oct 2014 10:33:27 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: pcp updates Content-Type: text/plain; charset=utf-8; format=flowed X-ASG-Orig-Subj: pcp updates Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail07.adl2.internode.on.net[150.101.137.131] X-Barracuda-Start-Time: 1412811193 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10353 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Changes committed to git://git.performancecopilot.org/kenj/pcp.git dev qa/994 | 4 ++-- qa/src/.gitignore | 1 + qa/src/badUnitsStr_r.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ src/python/GNUmakefile | 4 ++++ 4 files changed, 52 insertions(+), 2 deletions(-) commit 8ba42b6ae9196bea3a3cdf5711c53870d556139d Author: Ken McDonell Date: Thu Oct 9 10:32:35 2014 +1100 qa/994 - more Mac OS X tweaks commit ecc30187ef3286593764841eb20e55911e037c74 Author: Ken McDonell Date: Thu Oct 9 06:35:24 2014 +1100 python bits - don't build for PPC on Mac OS X commit 33d92e0e6f6d86e7bff0bac737ef17cd2a76b9e6 Author: Ken McDonell Date: Wed Oct 8 17:23:15 2014 +1100 qa/746 - third attemot to get all the bits checked in Sigh. badUnitsStr_r.c this time. From nscott@redhat.com Wed Oct 8 18:59:37 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 363EB7F3F for ; Wed, 8 Oct 2014 18:59:37 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 24943304059 for ; Wed, 8 Oct 2014 16:59:34 -0700 (PDT) X-ASG-Debug-ID: 1412812771-04cbb018ac4f850001-S8gJnT Received: from mx4-phx2.redhat.com (mx4-phx2.redhat.com [209.132.183.25]) by cuda.sgi.com with ESMTP id LWCanzqgzG9WKXpo (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 08 Oct 2014 16:59:31 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.25 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx4-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s98NxUcF007318 for ; Wed, 8 Oct 2014 19:59:30 -0400 Date: Wed, 8 Oct 2014 19:59:30 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: PCP Message-ID: <484349544.64611249.1412812770826.JavaMail.zimbra@redhat.com> In-Reply-To: <42448173.302141412808160359.JavaMail.sa_dps@linux1697> References: <42448173.302141412808160359.JavaMail.sa_dps@linux1697> Subject: PCP Developers Conference Call Playback Instructions MIME-Version: 1.0 X-ASG-Orig-Subj: PCP Developers Conference Call Playback Instructions Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: PCP Developers Conference Call Playback Instructions Thread-Index: iHucm/7jF2CwFSOtvNz8QINCMhBDqw== X-Barracuda-Connect: mx4-phx2.redhat.com[209.132.183.25] X-Barracuda-Start-Time: 1412812771 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10354 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Hi all, Playback instructions for todays call follow. I'll be collating notes (thanks Mark!) and distributing a written summary soon as well. cheers. ----- Forwarded Message ----- Sent: Thursday, October 9, 2014 9:42:40 AM Subject: Conference Playback Instructions for Reservationless-Plus Record and Playback Your recent conference was recorded and archived and is now available for playback. Conference Date: 9/10/2014 Conference Time: 03:56 BCU Playback ID: 143695281 Conference Topic: Leader: Nathan Scott Archived through: 7/11/2014 To listen to your conference playback via the Internet, refer to the following instructions: Playback via the Internet 1. Click on the following link or paste the entire URL into your browser: http://www2.reservationless-plus.net/moderator/presentation/Playback?id=0de44b8e-2ef8-4f3e-b170-11867c3e3d90.rpm 2. At the prompt, enter your name and email address. 3. Click "Submit". The playback will begin. From nscott@redhat.com Wed Oct 8 19:15:37 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 4B7B07F3F for ; Wed, 8 Oct 2014 19:15:37 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 3A2AE30405F for ; Wed, 8 Oct 2014 17:15:37 -0700 (PDT) X-ASG-Debug-ID: 1412813732-04cb6c770453d90001-S8gJnT Received: from mx5-phx2.redhat.com (mx5-phx2.redhat.com [209.132.183.37]) by cuda.sgi.com with ESMTP id xRxCKwGVQgDhGsLb (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 08 Oct 2014 17:15:32 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.37 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx5-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s990FUsQ002874; Wed, 8 Oct 2014 20:15:30 -0400 Date: Wed, 8 Oct 2014 20:15:30 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Martins Innus Cc: pcp@oss.sgi.com Message-ID: <265957919.64613233.1412813730718.JavaMail.zimbra@redhat.com> In-Reply-To: <5435926F.6030004@buffalo.edu> References: <536D28B4.6010504@buffalo.edu> <1139662762.4765310.1399862104653.JavaMail.zimbra@redhat.com> <54230FAF.2080201@buffalo.edu> <905561536.62723741.1412657864635.JavaMail.zimbra@redhat.com> <5435926F.6030004@buffalo.edu> Subject: Re: [pcp] hotproc rfc MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] hotproc rfc Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: hotproc rfc Thread-Index: jldAX4NJCNPW2ZE8YmQZ848/VPV5hg== X-Barracuda-Connect: mx5-phx2.redhat.com[209.132.183.37] X-Barracuda-Start-Time: 1412813732 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10354 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Hi Martins, ----- Original Message ----- > > [...] > > I've had time to go through the code in detail now - this looks like a > > good step forward, and its much simpler than I expected. > > > Thanks for the review. No problem. > So if I understand you correctly, the {psinfo,id,memory,io,fd, > schedstat} metrics become dynamic both in proc and hotproc? For > implementation, the steps would be something like?: > > 1. Clean out root_proc of metrics we want to make dynamic > 2. Replicate the logic from cgroups.c and adapt to [hot]proc to create > these on the fly > 3. Even though these are dynamic, we reuse existing cluster and id > numbers so as not to confuse client tools. Spot on, yes; keeping the existing PMIDs for those proc metrics that become dynamic and making the hotproc variants just the same (but with non-conflicting cluster IDs of course). > It seems like the first step would be to convert the existing pmda to > handle dynamic metrics for the appropriate clusters that already exist > before we worry about hotproc? Yep. I believe that will work nicely - of course there may be issues I've not anticipated (lemme know early on, if so), but I'm pretty sure that will work out well. > Do you want me to have a go at that or do you have a specific plan in > mind already? It'd be great if you could take that on - please go right ahead. cheers. -- Nathan From nscott@redhat.com Thu Oct 9 00:29:59 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 610297F47 for ; Thu, 9 Oct 2014 00:29:59 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 4E9AB8F8035 for ; Wed, 8 Oct 2014 22:29:59 -0700 (PDT) X-ASG-Debug-ID: 1412832579-04cbb018ac5b2e0001-S8gJnT Received: from mx5-phx2.redhat.com (mx5-phx2.redhat.com [209.132.183.37]) by cuda.sgi.com with ESMTP id HMzoq7g3tBZpoSE3 (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 08 Oct 2014 22:29:39 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.37 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx5-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s995Tc4O024404 for ; Thu, 9 Oct 2014 01:29:38 -0400 Date: Thu, 9 Oct 2014 01:29:38 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: PCP Message-ID: <879356941.64676761.1412832578835.JavaMail.zimbra@redhat.com> In-Reply-To: <1337819626.64674757.1412831161274.JavaMail.zimbra@redhat.com> Subject: PCP Developers Meeting Minutes MIME-Version: 1.0 X-ASG-Orig-Subj: PCP Developers Meeting Minutes Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: PCP Developers Meeting Minutes Thread-Index: Y1Xbq2oyn59Xh+yICSEZkFqs54Rs0A== X-Barracuda-Connect: mx5-phx2.redhat.com[209.132.183.37] X-Barracuda-Start-Time: 1412832579 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Barracuda-BRTS-Status: 1 X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10363 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Hi all, Here's a collation of notes that Mark and I made from today's call. I've stuck with the first-person style that Mark used - if I have missed or misrepresented anything, please send an update; thanks! - Welcome - Any general news? Nathan: python3 packaging is available from latest dev now, please kick the tires and let me know how it goes. - Website revamp request/options [nathans, michele] Nathan (paraphrasing Michele): would be good to update the website, with a more modern look Nathan: any takers or anyone know UI folks inside RH, please contact me (website is all committed in a git tree, easily updated - so mostly needs folks with more/betterer UI skills than myself). - Easy hacks reference, ala libreoffice [nathans, michele] https://wiki.documentfoundation.org/Development/Easy_Hacks Nathan: overview of what "easy hacks" is on other projects. Frank: suggest simply pointing people toward bugzilla may be best. Ken: can be good to have an environment without the possibility for feeling foolish in public, for not knowing stuff about things. Mark: newbie syndrome, 1:1 mentoring isn't very productive Mark: need to identify some projects and get these guys involved (e.g. some RH GSS folks) Nathan: anyone have potential projects? small projects - possibly things like PCP variants of other simple tools, like vmstat, mpstat, etc. Nathan: if anyone else has suggestions and are prepared to act as mentors please send me details and I'll start up a page. - Quality assurance - Status [kenj, nathans] Ken: recent daylight savings change causing some problems 2% failure rate at the moment (1% is typical) approx 10 failures on all machines out of 700 or so tests fixes can take 4 days to make their way thru (run/scheduling delays) time conversion issues on some platforms (getdate stuff) Nathan: ACK, also seeing that intermittently, ditch unstable times around the epoch in testing? (not useful) Ken: agree. Bit more noise than couple weeks back, at the moment Nathan/Ken: permission checking stuff - needs some tweaking - warning from test 000. Nathan/Ken: we're testing predominantly upgrades (not new installs) Nathan: RH QE always do a fresh install - packages tend to be closer to our product releases than what we'd be doing upstream of course. Nathan: they run sanity group as smoke test (expect 100% pass), and then run full suite in local mode, but no remote tests. Ken: good to know Ken: remote hosts are non-deterministic (like it this way, wants it passing as is!) Nathan: starting to look into scripts for QA farm setup, ran into hard-code hosts Ken: qahosts.master, need to think about how to make this go away somehow (infrastructure) Frank: wcohen testing large host counts, not wanting to expose host details Ken/Nathan: no valuable host details being exposed (local setups only, for convenience), easy to externalise this config if others want to start using it and are concerned. Ken/Nathan: for remote testing its good for talking to non-homogeneous and down-revision-pcp remote hosts Ken: grundy.sgi.com needs to be expunged (and all old sgi.com references) Nathan: most unused hosts are gone, after recent cleanup - just grundy left. - PCP culture and Zen of QA [nathans] Nathan: increasingly, experienced people are not writing QA tests - phrase "hand-testing" is becoming frequent. Nathan: Testing by-hand isn't really good enough - we need automated regression tests - usually just shell scripts doing that same testing anyway! Nathan: pmdapapi case, where contrary advice given to not write tests just prior to release and focus on coding instead; caused me pain, had to drop things. Frank: there was more to it than that, code not in state fit for release anyway. Nathan: looked fine to me, test I ended up writing worked (and exposed issues, but they were readily resolved and test passed prior to release). Frank: disagree. Nathan: another example - untested pmwebd back-compat (http://goo.gl/kdMJBI) Nathan: these are not isolated incidents. Nathan: am I being too harsh here? Ken: Not being harsh enough! Also unhappy with those who are perfectly capable of producing good test cases just reporting bugs and moving on, not really making an appropriate level of effort to help others with fixing. Nathan: PCP testsuite has been invaluable over the past decade, especially for Dave/Ken/myself doing deep surgery in core areas (IPv6, NSS, auth, etc) which potentially affects so many areas of the code base. Nathan: generally encourage people to always write tests, please. Frank: sometimes hand-testing is the only way, e.g. pmcd and pmda under valgrind Ken: dbpmda? Generally - "hand testing" when a regression test could be written isn't good enough for experienced PCP developers. Ken: QA needs constant care and feeding, QA tests an intimate part of the code Ken: histogram of people making qa/ contributions to illustrate frustration: Ken and Nathan [many hundreds each] ... Dave [~100] then Frank [~30], then it's onesies-and-twosies all the way down. Nathan: running QA is a good way to learn and identify issues - makes for a valuable contribution to the project and your own learning as to what makes a good test, how others have tackled different aspects of testing, etc. Ken: would be happier if there was more QA activity in general - Non-intrusive QA testing for pcp [nathans, fche, kenj] Nathan: non-invasive testing - e.g. running along-side a production pmcd, using freshly built (not installed) PCP bits. Ken: what improvement will this make? Testing was designed to run on dedicated systems, and assumption has been baked in for many many years now Frank: making it easy to run on build systems, no dedicated systems needed Nathan: full QA run takes two hours, impractical to add to builds, enough value? Frank: making it easier to set up and run could increase QA activity Ken: there are small parts of QA that could be tested in an isolated environment e.g. perhaps libpcp related stuff (kenj+fche to chat more) Nathan: core pcp testing uses dedicated infrastructure by design, prefer a layered products approach to introduce new testing styles (Java, Web) as a pragmatic approach. Mark: discussion around when to write a new test vs modify existing test Ken: can use the "reserved" tag in group file or just add to the test and let it fail until the fix is checked-in Nathan: later is just fine, helps around release time to know issue is unfixed; points out this has not happened so far (i.e. that qa changes only are sent in with no accompanying fix) - but welcomes it. Ken: yes please! - Source trees - Code encumbrance concerns, a history [nathans] Nathan: overview of personal experiences in the XFS encumbrance review process (circa 2000) - noted surprise at level of code copying that had happened, unexpectedly (legally of course). - its built-in to programmers to not reinvent, and they'll use what is available to them (or what they've written before). which is OK as long as cross-contamination cannot happen - but it can and does in some situations, from this experience. Nathan: discussed the later litigation that unexpectedly followed a few years later against the Linux kernel - companies with developers and large end users of the kernel (Red Hat & other distros customers were targetted), noted effects on Linux kernel processes (sign off) that were allowing legally questionable contributions. Nathan: concerns about volumes of code being requested pushed into core PCP that we didn't author and know nothing about, esp with the potential legal risks associated with that. Ken: there are boundaries we should not cross with included code for the core PCP product - Issues with separate trees, pcp web status [fche, nathans] Nathan: gives interpretation of state of web tree and how it got there, and what I believe are the many good reasons for separating core from web functionality. Frank: re encumbrance, its all totally irrelevent to the issue at hand. Nathan: believe its a genuine concern, was initial case for refactoring before many of the other good reasons came to light. Frank: disagree, its all nonsense, and we could've fixed this one problem in other ways Frank: gives interpretation of state of web tree and how it got there, and questions code that has been in PCP for years now moving. Nathan: "years"? nothing here earlier than this year. Nathan: generally disagree with your disagree. Complex issues, many factors. Frank: disagree. Mark: talked with RH middleware guys, webasset bundling is the way javascript infrastructure is shipped Mark: cannot be solved with install/build deps (maybe one day, but not today) Nathan: agreed - AIUI thats why Frank went that way - doesn't need to be in core PCP sources though - release separately. Frank: graphite and jsquery are mature enough that encumbrance is not an issue Frank: inconceivable that PCP will get blasted for licensing issues, especially with apache or MIT license Nathan: not simply the license thats the issue - developers in some 3rd party stuff also work on proprietary code (see extjs). Pollution can happen. And it is not a valid legal argument to say "others do it, so its fine". Frank: that is a risk for that company, not the end-user. Nathan: it is also a risk for the end users (and developers), that was one of the points of the earlier discussion. Frank: the licensing and bundling issues are separate issues to the code/tree separation of pmwebd. Frank: handling webassets could have been solved without as much "trauma" Nathan: "trauma" .. is an over-reaction, why is this approach traumatic? seems a helpful approach to solving a complex array of issues. Frank: disagree with not not a trauma. Question why not to do this is separate question to why to do it. Nathan: much more comfortable with separate tree. Why the concern? Its working fine, and others take this approach too - solves many problems here. Frank: what are the benefits of the split? Ken: This is not helping the PCP project. Suggest we take this topic off the agenda, moderate a resolution in an offline forum. Resolution will not be achieved in a public forum. Nathan: sure, either way - thanks for the offer. Frank: prefer to have the community decide something now. Nathan: points toward the lack of interest on the list too, someone had to make a decision and so moved forward with what is the most mutually agreeable path. Frank: disagree. Nathan/Frank/Ken: agree to take this offline with Ken moderating on how to move the web tree forward. - discussion around the maintenance role [fche] Frank: discussed spreading maintenance aspects amongst more people in general Ken: are there enough people involved in the project to warrant spreading the maintenance role? Frank: yes you are quite right - enough bodies is not a requirement - some of these jobs need more than one person involved. constitutional questions - policy decisions, consensus type manner driving gatekeeper role Ken: agree .. start with some uncontentious things, guide behaviour and policy decisions Nathan: have been trying to spread the load a bit, e.g. asking for people to do code reviews. no requirement for me to do all code reviews, e.g. Marco's stuff recently - anyone could have helped review & help test. noone did. Ken: if you're willing to contribute code then you should be willing to review other people's code Nathan: discussed the libcontainer project source tree, which explicitly splits "maintainer" and "contributor" roles (https://github.com/docker/libcontainer) Nathan: we effectively have two "maintainers" for PCP at the moment (myself & Ken) Keen to add more people - if they are experienced, reviewing, testing, fixing bugs, doing the mundane maintenance stuff. Other folks are "contributors" - also doing highly valuable work, but the roles and responsibilities differ a great deal. - WIP / future work - local-context pmlogger [mgoodwin] Mark: many enterprise sites wont install anything - breaks their qualification so aim for minimal logger service, installed and enabled by default some sites will install, if their perf issues are bad enough (had a few recently) Mark: minimize daemons/services required for a minimal logger deployment. Mark: no pmcd daemon - pmlogger service only, in local context mode Mark: do not listen on any internet domain sockets (unix domain only or nothing for pmlc) Mark: local context mode only supports DSO PMDAS Mark: only really need linux_pmda, proc and pmcd PMDAs, others would be good too though - pmcd PMDA uses extern symbols from pmcd itself, needs fixing - some other PMDAs don't have DSO versions - alternate uid, need args, etc - PMDA_INTERFACE version that supports args to foo_init() or foo_args() Nathan: to tackle the elevated privilege aspects, we could replace the pmsocks field in control file, rev version of control file - other notes above. (and deprecate pmsocks - it's ancient and bit-rotting) Ken: this is doable Mark: will make time to work on it some more - multi-archive libpcp support [brolley] Dave: pushing along .... as per email list discussions .. surprised this was not there to start with! Early days. - generic json pmda, python pmda module work [fche] Frank: progressing well, will use split metadata/values json files. Frank: current work is extending the python PMDA wrapper with more of the C API for adding metrics in a more dynamic fashion. - smaller packages for containers [nathans] Nathan: minimal dependencies (pcp-libs only), need a package with pmcd and Linux and core PMDAs, reduced footprint, reduce duplication of daemon PMDAs and DSO PMDAs. Split the base package. Ken: would pmlogger be in there? Nathan: Jeremy wanting a minimal install in a container, so thinking not pmlogger. Discussion about being able to monitor containers from outside vs inside. Frank: people inexperienced with container use cases as yet, proc filesystem information leakage an issue - probably only want to monitor from base. Ken: in the VM case, need a lot more. Nathan: unclear if we'll need this small pmcd-focussed package then, maybe defer? Frank: think its still generally useful. Ken: worth looking into what it'd take to get PMDA installation in there too (Install scripts - pminfo/pmprobe, etc - maybe not a huge addition?) - simplifying PMAPI access (new APIs, caching) [fche, mgoodwin, kenj?] Frank: investigating, on-going work. caching possibilities, but not the focus at this stage. Ken/Mark: what version of the "metrics class" is this? (jokingly, from SGI days where this was tackled many times, never quite right for everyone it seems) - JVM instrumentation planning [nathans] Nathan: wanting to instrument middleware more easily, something that can be switched on easily without custom code (but supporting that too). Plan to iterate on current Parfait and Metrics, basically. Mark: will connect nathans with RH middleware guys working in MW support, and also dealing with kernel issues (when JVM goes beserk). Nathan: thanks, agreed - sweet spot for PCP if we can progress JVM side. - ... any other hacking in progress? - anyone need help with anything? Paul: working on ensuring coverage for RH supported filesystems (GFS2, ext[3-4], XFS, NFS, CEPH/block storage, Samba, CIFS instrumentation) Nathan: Great to hear! Mark & I hacking in device-mapper instrumentation area recently - if that comes under this banner we can certaingly help. Nathan: Good coverage on XFS, extN instrumentation (mainly JBD2, not much else) - CIFS kernel code available but we're not exporting it yet - so definitely work needed there. Ken: discussion around working with Tridge years ago to add instrumentation into Samba, for PCP (with PMDA), which is available if Samba is built with it switched on. - any other topics? All done for today - thanks all! cheers. -- Nathan From brolley@redhat.com Thu Oct 9 10:11:41 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id DE7E07F3F for ; Thu, 9 Oct 2014 10:11:41 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id CCE348F8033 for ; Thu, 9 Oct 2014 08:11:38 -0700 (PDT) X-ASG-Debug-ID: 1412867496-04cb6c77047b260001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id hGzHllR8GAXvCJWF (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 09 Oct 2014 08:11:37 -0700 (PDT) X-Barracuda-Envelope-From: brolley@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s99FBZhQ025204 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Thu, 9 Oct 2014 11:11:35 -0400 Received: from [10.15.16.139] (dhcp-10-15-16-139.yyz.redhat.com [10.15.16.139]) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s99FBYtR018966 for ; Thu, 9 Oct 2014 11:11:34 -0400 Message-ID: <5436A60B.9090308@redhat.com> Date: Thu, 09 Oct 2014 11:13:15 -0400 From: Dave Brolley User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: PCP Mailing List Subject: Python 3 Build Problem Content-Type: text/plain; charset=ISO-8859-1; format=flowed X-ASG-Orig-Subj: Python 3 Build Problem Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412867497 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Hi, I got the following while doing a clean configure+build. Installing the python3-devel package solved the problem.Some kind of CHECK_HEADERS for python3.3m/Python.h is probably needed in configure.ac, but I'm unsure about hard coding the path and not sure about how to derive the correct path. Probably something similar should be added in the python2 section as well. Dave === python === python setup.py build_ext --include-dirs=../../src/include --library-dirs=../../src/libpcp/src:../../src/libpcp_pmda/src:../../src/libpcp_gui/src:../../src/libpcp_import/src:../../src/libpcp_mmv/src python3 setup.py build_ext --include-dirs=../../src/include --library-dirs=../../src/libpcp/src:../../src/libpcp_pmda/src:../../src/libpcp_gui/src:../../src/libpcp_import/src:../../src/libpcp_mmv/src running build_ext building 'cpmapi' extension creating build creating build/temp.linux-x86_64-2.7 gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I../../src/include -I/usr/include/python2.7 -c pmapi.c -o build/temp.linux-x86_64-2.7/pmapi.o running build_ext building 'cpmapi' extension creating build/temp.linux-x86_64-3.3 gcc -pthread -Wno-unused-result -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I../../src/include -I/usr/include/python3.3m -c pmapi.c -o build/temp.linux-x86_64-3.3/pmapi.o pmapi.c:27:20: fatal error: Python.h: No such file or directory #include ^ compilation terminated. error: command 'gcc' failed with exit status 1 make[2]: *** [build_python3] Error 1 From kenj@internode.on.net Thu Oct 9 16:31:52 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_FRT_PROFILE1 autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 200677F47 for ; Thu, 9 Oct 2014 16:31:52 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 0EC6F304071 for ; Thu, 9 Oct 2014 14:31:48 -0700 (PDT) X-ASG-Debug-ID: 1412890304-04cbb018ad16e200001-S8gJnT Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id w9DFtSaJ86v04RaO for ; Thu, 09 Oct 2014 14:31:45 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AloCAO39NlR20YErPGdsb2JhbAANU4NhWIMGhDfDWIh0AQYBAQEBOIRmVTAGAgUWCwILAwIBAgExJwYCAQGIR61neJRIgSyPNYJhgVQFljuIepgeWIJKAQEB Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail04.adl6.internode.on.net with ESMTP; 10 Oct 2014 08:01:44 +1030 Message-ID: <5436FEDC.3060002@internode.on.net> Date: Fri, 10 Oct 2014 08:32:12 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: pcp updates - qt bits Content-Type: text/plain; charset=utf-8 X-ASG-Orig-Subj: pcp updates - qt bits Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1412890304 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10387 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Changes committed to git://git.performancecopilot.org/kenj/pcp.git dev qa/qt/qmc_format/qmc_format.pro | 2 +- src/libpcp_qed/src/libpcp_qed.pro | 3 ++- src/libpcp_qmc/src/libpcp_qmc.pro | 2 ++ src/libpcp_qwt/src/libpcp_qwt.pro | 2 ++ src/pmdumptext/pmdumptext.pro | 2 +- 5 files changed, 8 insertions(+), 3 deletions(-) commit 7c58e2c759c8ecb8180f1ceec5625902c6f873cd Author: Ken McDonell Date: Fri Oct 10 08:29:19 2014 +1100 Makefile changes for qt bits When Makefile is generated by qmake from a *.pro file, be consistent about where the build artefacts are placed. These changes make sure the libpcp_q* libraries and all the apps that use those libraries are consistent. From nscott@redhat.com Thu Oct 9 16:43:28 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 63A077F3F for ; Thu, 9 Oct 2014 16:43:28 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 523B1304053 for ; Thu, 9 Oct 2014 14:43:28 -0700 (PDT) X-ASG-Debug-ID: 1412891002-04cb6c7704131770001-S8gJnT Received: from mx4-phx2.redhat.com (mx4-phx2.redhat.com [209.132.183.25]) by cuda.sgi.com with ESMTP id sL9e154olDfwXM9e (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 09 Oct 2014 14:43:23 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.25 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx4-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s99LhMCY032716; Thu, 9 Oct 2014 17:43:22 -0400 Date: Thu, 9 Oct 2014 17:43:22 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Dave Brolley Cc: PCP Mailing List Message-ID: <265631943.65344126.1412891002061.JavaMail.zimbra@redhat.com> In-Reply-To: <5436A60B.9090308@redhat.com> References: <5436A60B.9090308@redhat.com> Subject: Re: [pcp] Python 3 Build Problem MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] Python 3 Build Problem Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: Python 3 Build Problem Thread-Index: QD9qHY8Uq381BBJDOBGvqGp4ILEgNQ== X-Barracuda-Connect: mx4-phx2.redhat.com[209.132.183.25] X-Barracuda-Start-Time: 1412891003 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.03 X-Barracuda-Spam-Status: No, SCORE=0.03 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_SA_TO_FROM_DOMAIN_MATCH, THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10387 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... 0.01 BSF_SC0_SA_TO_FROM_DOMAIN_MATCH Sender Domain Matches Recipient Domain ----- Original Message ----- > Hi, > > I got the following while doing a clean configure+build. Installing the > python3-devel package solved the problem.Some kind of CHECK_HEADERS for > python3.3m/Python.h is probably needed in configure.ac, but I'm unsure > about hard coding the path and not sure about how to derive the correct > path. Probably something similar should be added in the python2 section > as well. Yep - thanks Dave, I'll add those checks today. -- Nathan From nscott@redhat.com Thu Oct 9 20:35:19 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 006787F4E for ; Thu, 9 Oct 2014 20:35:19 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id D4A8A8F8068 for ; Thu, 9 Oct 2014 18:35:18 -0700 (PDT) X-ASG-Debug-ID: 1412904914-04bdf028761d9650001-S8gJnT Received: from mx4-phx2.redhat.com (mx4-phx2.redhat.com [209.132.183.25]) by cuda.sgi.com with ESMTP id IXOpZCFfOdC7KIAH (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 09 Oct 2014 18:35:14 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.25 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx4-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s9A1ZEPS001720; Thu, 9 Oct 2014 21:35:14 -0400 Date: Thu, 9 Oct 2014 21:35:13 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Mark Goodwin Cc: pcp , Laurence Oberman , Bud Brown Message-ID: <1192069605.65384998.1412904913945.JavaMail.zimbra@redhat.com> In-Reply-To: <5434DC73.4080400@redhat.com> References: <5434DC73.4080400@redhat.com> Subject: Re: [pcp] iostat2pcp broken for iostat in RHEL/fedora MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] iostat2pcp broken for iostat in RHEL/fedora Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: iostat2pcp broken for iostat in RHEL/fedora Thread-Index: bR4n1/iB2ef1sx3GpZKOkhcPCY+5Xw== X-Barracuda-Connect: mx4-phx2.redhat.com[209.132.183.25] X-Barracuda-Start-Time: 1412904914 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.03 X-Barracuda-Spam-Status: No, SCORE=0.03 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_SA_TO_FROM_DOMAIN_MATCH, THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10394 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... 0.01 BSF_SC0_SA_TO_FROM_DOMAIN_MATCH Sender Domain Matches Recipient Domain Hi guys, ----- Original Message ----- > iostat2pcp doesn't seem to like the iostat output on various > RHEL and Fedora flavours - this was reported by Bud Brown here > at Red Hat (cc:'d). Any idea what's wrong anyone? Yep... > qa/373 works but it's using output captured from some other iostat > version, not the locally installed iostat - the latter would be > a better test no doubt ... > > $ iostat -t -x 2 10 > iostat.10 > $ iostat2pcp iostat.10 iostat.pcp > [26] Time: 03:49:59 PM > Device: number of values? expected 12, found 3 It looks like iostat2pcp doesn't understand this timestamp format, and the parser is attempting to treat the line "Time: 03:49:59 PM" as a set of device values. Is LC_TIME set in the environment when iostat is being run? And which version of iostat being used? (rhel5 in output - is it an old version?) I think we can attempt to cater for this output form, but it's worth reading the iostat2pcp(1) man page, from the section starting: "The best results are obtained when iostat(1) was run with [...]" cheers. -- Nathan From mgoodwin@redhat.com Thu Oct 9 22:47:38 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: ** X-Spam-Status: No, score=3.0 required=5.0 tests=TVD_SUBJ_NUM_OBFU_MINFP autolearn=no version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 2DE3B7F3F for ; Thu, 9 Oct 2014 22:47:38 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 1C95B30404E for ; Thu, 9 Oct 2014 20:47:34 -0700 (PDT) X-ASG-Debug-ID: 1412912853-04cb6c770527db60001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id DLBAC8BpnkMzA8E2 (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 09 Oct 2014 20:47:33 -0700 (PDT) X-Barracuda-Envelope-From: mgoodwin@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9A3lW5G004916 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Thu, 9 Oct 2014 23:47:32 -0400 Received: from [10.64.50.11] (vpn1-50-11.bne.redhat.com [10.64.50.11]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9A3lUY0006977; Thu, 9 Oct 2014 23:47:31 -0400 Message-ID: <543756D2.2020606@redhat.com> Date: Fri, 10 Oct 2014 14:47:30 +1100 From: Mark Goodwin User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: Nathan Scott CC: pcp , Laurence Oberman , Bud Brown Subject: Re: [pcp] iostat2pcp broken for iostat in RHEL/fedora References: <5434DC73.4080400@redhat.com> <1192069605.65384998.1412904913945.JavaMail.zimbra@redhat.com> X-ASG-Orig-Subj: Re: [pcp] iostat2pcp broken for iostat in RHEL/fedora In-Reply-To: <1192069605.65384998.1412904913945.JavaMail.zimbra@redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412912853 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On 10/10/2014 12:35 PM, Nathan Scott wrote: [ ... ] >> $ iostat -t -x 2 10 > iostat.10 >> $ iostat2pcp iostat.10 iostat.pcp >> [26] Time: 03:49:59 PM >> Device: number of values? expected 12, found 3 > > It looks like iostat2pcp doesn't understand this timestamp format, > and the parser is attempting to treat the line "Time: 03:49:59 PM" > as a set of device values. Is LC_TIME set in the environment when > iostat is being run? no. it fails on f19 too, with iostat v10, which is probably the same version Bud is running. > And which version of iostat being used? (rhel5 in output - is it > an old version?) I think we can attempt to cater for this output > form, but it's worth reading the iostat2pcp(1) man page, from the > section starting: > > "The best results are obtained when iostat(1) was run with [...]" # S_TIME_FORMAT=ISO iostat -t -x 2 10 > iostat.out this gives more parseable timestamps, e.g. "2014-10-10T14:28:15+1100" and we get past the timestamp problem, but then hit the next problem : # iostat2pcp iostat.out iostat.pcp [20] sda 0.00 0.00 0.50 0.00 2.00 0.00 8.00 0.04 87.00 87.00 0.00 87.00 4.35 Device: number of values? expected 12, found 14 version 10 and later iostat -x has await split into await, r_await and w_await, e.g. 2014-10-10T14:28:17+1100 avg-cpu: %user %nice %system %iowait %steal %idle 15.51 0.00 1.45 1.18 0.00 81.87 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.00 0.50 0.00 2.00 0.00 8.00 0.04 87.00 87.00 0.00 87.00 4.35 Once you get past that the resulting pcp archive also needs to include disk.dev.read_rawactive and disk.dev.write_rawactive so pmiostat can replay it, which is what I suspect Bud was going to try next ... I'll try and find some time over the w/e to fix it. Regards - Mark From kenj@kenj.com.au Thu Oct 9 22:58:16 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 2D7DF7F3F for ; Thu, 9 Oct 2014 22:58:16 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 1AEEB30404E for ; Thu, 9 Oct 2014 20:58:15 -0700 (PDT) X-ASG-Debug-ID: 1412913487-04cbb018ad2368e0001-S8gJnT Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id q41SFevzSiZTlppb for ; Thu, 09 Oct 2014 20:58:08 -0700 (PDT) X-Barracuda-Envelope-From: kenj@kenj.com.au X-Barracuda-Apparent-Source-IP: 150.101.137.141 Received: from ppp118-209-62-212.lns20.mel4.internode.on.net (HELO bozo-vm.localdomain) ([118.209.62.212]) by ipmail04.adl6.internode.on.net with ESMTP; 10 Oct 2014 14:28:07 +1030 Received: by bozo-vm.localdomain (Postfix, from userid 1000) id E7F08A358F; Fri, 10 Oct 2014 14:57:49 +1100 (EST) To: pcp@oss.sgi.com Subject: pcp updates - qa and sample pmda Message-Id: <20141010035749.E7F08A358F@bozo-vm.localdomain> X-ASG-Orig-Subj: pcp updates - qa and sample pmda Date: Fri, 10 Oct 2014 14:57:49 +1100 (EST) From: kenj@kenj.com.au (Ken McDonell) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1412913487 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10397 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Changes committed to git://git.performancecopilot.org/kenj/pcp.git dev qa/025.out | 18 ++++++++++++++---- qa/026.out | 33 +++++++++++++++++++++++++++------ qa/043.out | 2 +- qa/273.out | 40 ++++++++++++++++++++++++++++++---------- qa/449.out | 2 +- 5 files changed, 73 insertions(+), 22 deletions(-) commit 283d8b62a479f8da1e2982cf23e2b54c56c317f7 Author: Ken McDonell Date: Fri Oct 10 14:55:23 2014 +1100 qa assorted - new .out files after recent sample pmda changes In particular, more/new help text and pmStore() now OK for sample.colour. From kenj@internode.on.net Fri Oct 10 01:44:19 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 9BC547F3F for ; Fri, 10 Oct 2014 01:44:19 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 6C75A8F8054 for ; Thu, 9 Oct 2014 23:44:19 -0700 (PDT) X-ASG-Debug-ID: 1412923453-04bdf0287827c9d0001-S8gJnT Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id GCIkxu9R7lClze0m for ; Thu, 09 Oct 2014 23:44:13 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AtcMAIt/N1R20T7UPGdsb2JhbABggw6BK4I2hQfLKAQCAoEGFwEGAQEBATg5hAMBAQEECAIeEhwjDAEDAgYDDgMEAQEBJwcZIAoDCQgCBAESCwWILcI1AReQRAcGhEUFkXldjCODRpUUKS+CSgEBAQ Received: from ppp118-209-62-212.lns20.mel4.internode.on.net (HELO bozohorize) ([118.209.62.212]) by ipmail04.adl6.internode.on.net with ESMTP; 10 Oct 2014 17:14:12 +1030 From: "Ken McDonell" To: "'Mark Goodwin'" , "'Nathan Scott'" Cc: "'Bud Brown'" , "'Laurence Oberman'" , "'pcp'" References: <5434DC73.4080400@redhat.com> <1192069605.65384998.1412904913945.JavaMail.zimbra@redhat.com> <543756D2.2020606@redhat.com> In-Reply-To: <543756D2.2020606@redhat.com> Subject: RE: [pcp] iostat2pcp broken for iostat in RHEL/fedora Date: Fri, 10 Oct 2014 17:44:09 +1100 X-ASG-Orig-Subj: RE: [pcp] iostat2pcp broken for iostat in RHEL/fedora Message-ID: <003d01cfe455$9a187180$ce495480$@internode.on.net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 15.0 Thread-Index: AQHv78qT6GTizPCAZUlbfjlz7HgHLwFhLbWoApTfuU2byRo+UA== Content-Language: en-au X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1412923453 X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.01 X-Barracuda-Spam-Status: No, SCORE=0.01 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, THREAD_INDEX X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10400 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header When you've got to the bottom of this, could we please circle back to understand why QA did not find this one? There are 4 "iostat" QA tests: 366, 373, 536 and 842. All are passing on all platforms in my QA Farm (which includes lots of CentOS and Fedora variants). > -----Original Message----- > From: pcp-bounces@oss.sgi.com [mailto:pcp-bounces@oss.sgi.com] On > Behalf Of Mark Goodwin > Sent: Friday, 10 October 2014 2:48 PM > To: Nathan Scott > Cc: Bud Brown; Laurence Oberman; pcp > Subject: Re: [pcp] iostat2pcp broken for iostat in RHEL/fedora > > On 10/10/2014 12:35 PM, Nathan Scott wrote: > [ ... ] > >> $ iostat -t -x 2 10 > iostat.10 > >> $ iostat2pcp iostat.10 iostat.pcp > >> [26] Time: 03:49:59 PM > >> Device: number of values? expected 12, found 3 > > > > It looks like iostat2pcp doesn't understand this timestamp format, and > > the parser is attempting to treat the line "Time: 03:49:59 PM" > > as a set of device values. Is LC_TIME set in the environment when > > iostat is being run? > > no. it fails on f19 too, with iostat v10, which is probably the same version Bud > is running. > >... From nscott@redhat.com Fri Oct 10 01:57:41 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id B8ED07F3F for ; Fri, 10 Oct 2014 01:57:41 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id A7027304053 for ; Thu, 9 Oct 2014 23:57:38 -0700 (PDT) X-ASG-Debug-ID: 1412924254-04cbb018ac289810001-S8gJnT Received: from mx4-phx2.redhat.com (mx4-phx2.redhat.com [209.132.183.25]) by cuda.sgi.com with ESMTP id yLudUbGvSfDBSSjt (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 09 Oct 2014 23:57:34 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.25 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx4-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s9A6vXkD015037 for ; Fri, 10 Oct 2014 02:57:33 -0400 Date: Fri, 10 Oct 2014 02:57:33 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: pcp Message-ID: <1412389614.65453108.1412924253743.JavaMail.zimbra@redhat.com> In-Reply-To: <1331090682.65452560.1412924194500.JavaMail.zimbra@redhat.com> Subject: pcp updates: kenj qa, python3 MIME-Version: 1.0 X-ASG-Orig-Subj: pcp updates: kenj qa, python3 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: pcp updates: kenj qa, python3 Thread-Index: Eoq7xlQNmlEXbJLgDq/jpFlIoR6mqQ== X-Barracuda-Connect: mx4-phx2.redhat.com[209.132.183.25] X-Barracuda-Start-Time: 1412924254 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10401 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Changes committed to git://git.pcp.io/pcp.git dev Ken McDonell (10): qa/260 - improve determinism sample pmda - add pmStore support for some metrics qa/753 - changes to improve test stability and determinism qa/555 - more diags to try and track down failures qa makefile - missed change when qa/746 added qa/746 - third attemot to get all the bits checked in python bits - don't build for PPC on Mac OS X qa/994 - more Mac OS X tweaks Makefile changes for qt bits qa assorted - new .out files after recent sample pmda changes Nathan Scott (2): build: improve python configure checking for required headers python3: several module updates and pmiostat running configure | 56 +++++- configure.ac | 22 +- qa/025.out | 18 +- qa/026.out | 33 +++ qa/043.out | 2 qa/260 | 19 -- qa/260.out | 46 +++++ qa/273.out | 40 +++- qa/449.out | 2 qa/555 | 1 qa/753 | 16 + qa/753.out | 16 - qa/994 | 4 qa/qt/qmc_format/qmc_format.pro | 2 qa/src/.gitignore | 1 qa/src/GNUlocaldefs | 2 qa/src/badUnitsStr_r.c | 45 +++++ src/libpcp_qed/src/libpcp_qed.pro | 3 src/libpcp_qmc/src/libpcp_qmc.pro | 2 src/libpcp_qwt/src/libpcp_qwt.pro | 2 src/pmdas/sample/help | 40 +++- src/pmdas/sample/src/sample.c | 72 ++++++-- src/pmdumptext/pmdumptext.pro | 2 src/pmiostat/pmiostat.py | 323 +++++++++++++++++++------------------- src/python/GNUmakefile | 4 src/python/pcp/mmv.py | 6 src/python/pcp/pmapi.py | 54 ++++-- src/python/pcp/pmcc.py | 16 + src/python/pcp/pmda.py | 6 src/python/pcp/pmgui.py | 2 src/python/pcp/pmsubsys.py | 23 +- 31 files changed, 581 insertions(+), 299 deletions(-) From mgoodwin@redhat.com Fri Oct 10 02:08:24 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: ** X-Spam-Status: No, score=3.0 required=5.0 tests=TVD_SUBJ_NUM_OBFU_MINFP autolearn=no version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id B563A7F3F for ; Fri, 10 Oct 2014 02:08:24 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id A37548F8050 for ; Fri, 10 Oct 2014 00:08:21 -0700 (PDT) X-ASG-Debug-ID: 1412924899-04cbb018ac28ab90001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id zqeuIL8T3TtY0UYL (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Fri, 10 Oct 2014 00:08:20 -0700 (PDT) X-Barracuda-Envelope-From: mgoodwin@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9A78Eu1012401 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 10 Oct 2014 03:08:15 -0400 Received: from [10.64.50.11] (vpn1-50-11.bne.redhat.com [10.64.50.11]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9A780dA024416; Fri, 10 Oct 2014 03:08:06 -0400 Message-ID: <543785D0.5040006@redhat.com> Date: Fri, 10 Oct 2014 18:08:00 +1100 From: Mark Goodwin User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: Ken McDonell , "'Nathan Scott'" CC: "'Bud Brown'" , "'Laurence Oberman'" , "'pcp'" Subject: Re: [pcp] iostat2pcp broken for iostat in RHEL/fedora References: <5434DC73.4080400@redhat.com> <1192069605.65384998.1412904913945.JavaMail.zimbra@redhat.com> <543756D2.2020606@redhat.com> <003d01cfe455$9a187180$ce495480$@internode.on.net> X-ASG-Orig-Subj: Re: [pcp] iostat2pcp broken for iostat in RHEL/fedora In-Reply-To: <003d01cfe455$9a187180$ce495480$@internode.on.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412924900 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On 10/10/2014 05:44 PM, Ken McDonell wrote: > When you've got to the bottom of this, could we please circle back to > understand why QA did not find this one? > > There are 4 "iostat" QA tests: 366, 373, 536 and 842. All are passing on > all platforms in my QA Farm (which includes lots of CentOS and Fedora > variants). Hi Ken, yes ok will do. I suspect only qa/373 actually runs iostat2pcp but it uses pre-captured iostat output (qa/src/iostat*). So it looks like we're not testing iostat2pcp against the iostat that's actually installed on each platform. Cheers - Mark From myllynen@redhat.com Fri Oct 10 07:05:16 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id E998C7F3F for ; Fri, 10 Oct 2014 07:05:16 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id BAF8A30407B for ; Fri, 10 Oct 2014 05:05:16 -0700 (PDT) X-ASG-Debug-ID: 1412942712-04bdf02877295a30001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id RdZc65rzYiyoPcuQ (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Fri, 10 Oct 2014 05:05:12 -0700 (PDT) X-Barracuda-Envelope-From: myllynen@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9AC5Ch4017351 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Fri, 10 Oct 2014 08:05:12 -0400 Received: from mmyllyne.csb (vpn1-7-165.ams2.redhat.com [10.36.7.165]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9AC53K9003267 for ; Fri, 10 Oct 2014 08:05:11 -0400 Message-ID: <5437CB6F.8040706@redhat.com> Date: Fri, 10 Oct 2014 15:05:03 +0300 From: Marko Myllynen Reply-To: myllynen@redhat.com Organization: Red Hat User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: PCP Subject: deb packaging Content-Type: text/plain; charset=ISO-8859-1 X-ASG-Orig-Subj: deb packaging Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412942712 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Hi, while updating the PCP Quick Guide I happened to notice that at least on Ubuntu "apt-get install pcp" brings in also pcp-doc and pcp-gui and their dependencies (like Qt, which doesn't show prominently below because the screenshot was taken on a desktop install): test@ubuntu-test:~$ sudo apt-get install pcp Reading package lists... Building dependency tree... Reading state information... The following extra packages will be installed: gawk libpcp-gui2 libpcp-import-perl libpcp-import1 libpcp-pmda-perl libpcp-pmda3 libpcp-trace2 libpcp3 libsigsegv2 pcp-conf pcp-doc pcp-gui python-pcp Suggested packages: gawk-doc tsocks xvfb qt4-dev-tools The following NEW packages will be installed: gawk libpcp-gui2 libpcp-import-perl libpcp-import1 libpcp-pmda-perl libpcp-pmda3 libpcp-trace2 libpcp3 libsigsegv2 pcp pcp-conf pcp-doc pcp-gui python-pcp 0 upgraded, 14 newly installed, 0 to remove and 0 not upgraded. Need to get 5 764 kB of archives. After this operation, 18,9 MB of additional disk space will be used. Do you want to continue? [Y/n] y [...] test@ubuntu-test:~$ sudo apt-cache depends pcp pcp Depends: libc6 Depends: libpcp-gui2 Depends: libpcp-pmda3 Depends: libpcp-trace2 Depends: libpcp3 Depends: libreadline6 Depends: gawk gawk:i386 Depends: procps procps:i386 Depends: libpcp-pmda-perl Depends: python-pcp Depends: python Suggests: tsocks Recommends: pcp-gui Recommends: libpcp-import-perl Conflicts: pgpool2 Conflicts: pgpool2:i386 Conflicts: pcp:i386 test@Ubuntu-Test:~$ This is on Ubuntu 14.10 Beta 2 + updates. PCP is 3.9.8. Thanks, -- Marko Myllynen From fche@redhat.com Fri Oct 10 14:06:51 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 9CD147F55 for ; Fri, 10 Oct 2014 14:06:51 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 7C80C8F8054 for ; Fri, 10 Oct 2014 12:06:48 -0700 (PDT) X-ASG-Debug-ID: 1412968003-04cb6c77062f7b90001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 1iXZ64cuvR95wnj9 (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Fri, 10 Oct 2014 12:06:44 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9AJ6hsZ032419 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Fri, 10 Oct 2014 15:06:43 -0400 Received: from fche.csb (vpn-234-10.phx2.redhat.com [10.3.234.10]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9AJ6hmK023925; Fri, 10 Oct 2014 15:06:43 -0400 Received: by fche.csb (Postfix, from userid 2569) id 7E4535823F; Fri, 10 Oct 2014 15:06:42 -0400 (EDT) Date: Fri, 10 Oct 2014 15:06:42 -0400 From: "Frank Ch. Eigler" To: lberk@redhat.com Cc: pcp developers Subject: another round of review for lberk/papi Message-ID: <20141010190642.GD26999@redhat.com> X-ASG-Orig-Subj: another round of review for lberk/papi Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.2i X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1412968004 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Hi - My usual channel for this was unavailable today, so here goes in plain email. Took another look at pcpfans.git lberk/papi (up to commit 05c56e4f18deaa9). Many good fixes from the previous reviews, thanks! At least one more correctness issue is still present in the code, by inspection: the way the add_metric & remove_metric functions (and subsidiaries) use the papi_info[].position number both as an index (into values[]) and as the "user wishes this counter activated" flag. This fails if the set-of-activated-counters is manipulated via pmStore in a non-LIFO way. It's because the remove path largely preserves the previous .position numbers even though it recreates the EventSet in increasing-index order (thus invalidating those position#'s). Any time after that the values[] will be mis-ordered. What the papi_info[] struct needs is an added simple on/off flag that the .enable/.disable callback can set/clear, and a single common papi-eventset-refresh function that shuts everything down, saves counter values from former values[.position]'s, and *recalculates* brand a new .position for all surviving events (in whatever order they're being added to the new eventset). A small memory-corruption bug with the new status_string[]: strdup() it and PMDA_FETCH_DYNAMIC as the return code, or else make it a static char[] and keep PMDA_FETCH_STATIC. (PMDA_FETCH_STATIC on a stack local would ask libpcp_pmda to access memory from a stack frame that's already been unwound.) I've suggested nuking the hard-coded-papi-event parts of metrictab[], but it's clear now why you can't quite do that yet: there is no other source of pmDesc type data for the metrics. You'll need to intercept the pmda dispatch->version.any.desc field with a callback function that fills in the given pmDesc*, very much the same way as your papi_text() function returns the helptext stuff. Then metrictab[] can shrink to the bare minimum. Some ideas regarding testing. First of all, the existing test cases (qa/914 and qa/799) simply must stop relying on the -L local context this way. As we discussed on IRC last week, -L makes no sense at all for papi (since it's so stateful; each "pmCLIENT -L ..." loads the pmda.so+papi.so to form a new blank slate, does one thing, and immediately dies). So no such test will ever see counter numbers, or results other than those related to exactly one initialization-type operation -- plus any __pmNotifyErr noise that would've gone to a pmcd/papi.log file. In fact, the only way anyone could have ever seen the pmda work and produce counter *numbers*, was by -not- using the -L mode, which means its core was only ever "hand tested" and qa/914 was to that extent a talisman. Anyway, as we were saying on IRC, one plausible way of using -L is via dbpmda, instead of a PMAPI client. That could work ... but OTOH might be tricky to feed it a fixed dbpmda command script (like a few other qa/NNN tests do) and get meaningful answers out, at least because the available counters=metrics & runtime environment can so easily vary between machines. A simple testcase driver is probably not up to the job. If you agree a more complex driver seems needed, we could try something using plain default "-h local:" rather than "-L", chooses a bunch of counters, fetches some samples, waits awhile to generate load, fetches more samples (sanity checking that they are growing at reasonable rates), turning some off / on in various orderings, fetching them, checking that the survivors are still growing at reasonable rates. It might be a job for a python script rather than bash. (When writing tests for pmwebd, I found it helpful to compile the pmda with the gcc test-coverage options. It made it possible to see that all the working parts of the code were in fact touched by the testsuite's execution. You might find it useful here too, considering the tricky state manipulation required.) - FChE From nscott@redhat.com Sun Oct 12 19:26:55 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id C8C217F4E for ; Sun, 12 Oct 2014 19:26:55 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 66E2AAC004 for ; Sun, 12 Oct 2014 17:26:52 -0700 (PDT) X-ASG-Debug-ID: 1413160007-04bdf02879335f40001-S8gJnT Received: from mx5-phx2.redhat.com (mx5-phx2.redhat.com [209.132.183.37]) by cuda.sgi.com with ESMTP id 0o7NrGk8GJkjDGyu (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Sun, 12 Oct 2014 17:26:47 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.37 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx5-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9D0QlHp009383; Sun, 12 Oct 2014 20:26:47 -0400 Date: Sun, 12 Oct 2014 20:26:46 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: myllynen@redhat.com Cc: PCP Message-ID: <1393785185.66474760.1413160006299.JavaMail.zimbra@redhat.com> In-Reply-To: <5437CB6F.8040706@redhat.com> References: <5437CB6F.8040706@redhat.com> Subject: Re: [pcp] deb packaging MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] deb packaging Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: deb packaging Thread-Index: GcwHLE4ZG8yHORjH9bL0Fbd37dLHaw== X-Barracuda-Connect: mx5-phx2.redhat.com[209.132.183.37] X-Barracuda-Start-Time: 1413160007 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.03 X-Barracuda-Spam-Status: No, SCORE=0.03 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_SA_TO_FROM_DOMAIN_MATCH, THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10496 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.01 BSF_SC0_SA_TO_FROM_DOMAIN_MATCH Sender Domain Matches Recipient Domain ----- Original Message ----- > Hi, > > while updating the PCP Quick Guide I happened to notice that at least on > Ubuntu "apt-get install pcp" brings in also pcp-doc and pcp-gui and > their dependencies Hmm, that's unexpected. It seems pcp should be only "Suggest"ing pcp-gui, not "Recommend"ing it ... will look into it further, thanks Marco. cheers. -- Nathan From kenj@internode.on.net Sun Oct 12 19:57:20 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 3B7BF7F4E for ; Sun, 12 Oct 2014 19:57:20 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 1798A304064 for ; Sun, 12 Oct 2014 17:57:19 -0700 (PDT) X-ASG-Debug-ID: 1413161836-04cb6c77053564c0001-S8gJnT Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by cuda.sgi.com with ESMTP id Ibe8xFx1ybCS4ekY for ; Sun, 12 Oct 2014 17:57:17 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.145 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AscBACoiO1R20YErPGdsb2JhbAANToNhWIMGhDfDRYh0AQYBAQEBOIRnVTAGAgUWCwILAwIBAgExJwYCAQGIR61OeJUEgSyPNoJhgVQFhi2QDqEsWIJKAQEB Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail06.adl6.internode.on.net with ESMTP; 13 Oct 2014 11:27:15 +1030 Message-ID: <543B238F.6000601@internode.on.net> Date: Mon, 13 Oct 2014 11:57:51 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: pcp updates - libpcp lock race and qa Content-Type: text/plain; charset=utf-8 X-ASG-Orig-Subj: pcp updates - libpcp lock race and qa Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl6.internode.on.net[150.101.137.145] X-Barracuda-Start-Time: 1413161836 X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10497 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Changes committed to git://git.performancecopilot.org/kenj/pcp.git dev qa/734 | 4 ++++ qa/752 | 29 ++++++++++++++++++++++------- qa/752.out | 36 ++++++++++++++++++------------------ qa/943 | 7 ++++++- qa/qa_hosts.master | 3 --- src/libpcp/src/AF.c | 8 ++++++++ 6 files changed, 58 insertions(+), 29 deletions(-) commit dac7ff002d708ee4b0df07b31a2f83e01158ce1d Author: Ken McDonell Date: Mon Oct 13 11:53:22 2014 +1100 libpcp/AF.c - fix lock initialization race Sometimes seen in qa/134 (and others), a pmlogger -c /dev/null -L would hang and not allow pmlc to connect. Tracked down to a nested __pmInitLocks() call because the timer alarm went off in the middle of setting up. This fix forces __pmInitLocks() to be called early on when the __pmAF* routines are called. commit 96c6df35fe640e4badd7997a12626e81123fe6ed Author: Ken McDonell Date: Mon Oct 13 11:50:10 2014 +1100 qa/752 - fix problems with timezones and certain times of day This test has been notoriously flakey. Rework the filtering to try and improve stability. commit 2a8721a30977dbcc29f2faf61e9e93f2d404faea Author: Ken McDonell Date: Mon Oct 13 11:49:11 2014 +1100 qa/734 - fix for longer hostnames Truncate the hostname in the QA test to match what a correctly behaving pmstat would report. commit d06c5c4ab233e25a4f6a2512f0b27bc659462b9a Merge: b4c0f8a bd733e5 Author: Ken McDonell Date: Sun Oct 12 21:34:37 2014 +1100 Merge branch 'dev' of git://git.performancecopilot.org/kenj/pcp into dev commit bd733e55adc3b6f6ad1e924be54027c89e9ee5d1 Author: Ken McDonell Date: Sat Oct 11 20:14:40 2014 +1100 qa/qa_hosts.master - tweak kenj's entries commit b4c0f8a7084c2a76b95becc254d9b03cf644bbe1 Author: Ken McDonell Date: Fri Oct 10 15:05:55 2014 +1100 qa/943 - dodge error from proc.psinfo.cgroups On some platforms we're seeing proc.psinfo.cgroups -61 No data available just pretend this is OK. From nscott@redhat.com Sun Oct 12 20:11:58 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id D159B7F4E for ; Sun, 12 Oct 2014 20:11:57 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 624C9AC006 for ; Sun, 12 Oct 2014 18:11:53 -0700 (PDT) X-ASG-Debug-ID: 1413162711-04bdf02879336d70001-S8gJnT Received: from mx3-phx2.redhat.com (mx3-phx2.redhat.com [209.132.183.24]) by cuda.sgi.com with ESMTP id quXJfJ94dgDe6ROM (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Sun, 12 Oct 2014 18:11:51 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.24 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx3-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s9D1BoAd026570; Sun, 12 Oct 2014 21:11:50 -0400 Date: Sun, 12 Oct 2014 21:11:50 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: "Frank Ch. Eigler" , lberk@redhat.com Cc: pcp developers Message-ID: <479818101.66478026.1413162710461.JavaMail.zimbra@redhat.com> In-Reply-To: <20141010190642.GD26999@redhat.com> References: <20141010190642.GD26999@redhat.com> Subject: Re: [pcp] another round of review for lberk/papi MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] another round of review for lberk/papi Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: another round of review for lberk/papi Thread-Index: JUWuALJ8JL5KGhlT3DT4FDWXLnLrNQ== X-Barracuda-Connect: mx3-phx2.redhat.com[209.132.183.24] X-Barracuda-Start-Time: 1413162711 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.03 X-Barracuda-Spam-Status: No, SCORE=0.03 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_SA_TO_FROM_DOMAIN_MATCH, THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10498 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... 0.01 BSF_SC0_SA_TO_FROM_DOMAIN_MATCH Sender Domain Matches Recipient Domain ----- Original Message ----- > Hi - > > My usual channel for this was unavailable today, so here goes in plain > email. Took another look at pcpfans.git lberk/papi (up to commit > 05c56e4f18deaa9). Many good fixes from the previous reviews, thanks! Thanks for sending this out, its great that these reviews can become publicly visible (even a summary to the list after internal chat is highly valuable, if you prefer that channel initially). > [...] > First of all, the existing test cases (qa/914 and qa/799) simply must (I think you mean qa/903 and qa/914) > stop relying on the -L local context this way. As we discussed on IRC Add new tests please, as discussed. As stated at the time they were added, these are simple, directed tests (since we needed *some* form of test coverage) and the tests exercise specific things (and at no point did they attempt to verify the values themselves, as of course they cannot). They exposed bugs in the authentication mechanism, exercise the install and remove mechanisms, and are well-suited to some basic pmda-valgrind exercising, so please keep these in place and now extend with new tests ... thanks. Having one huge test that attempts to exercise *everything* about any individual component is usually the wrong way to go - it makes that one test much slower, and more difficult to debug the test/component when it does fail (both because its slower *and* because its attempting to do everything - so, harder to focus in on the one/two aspects its testing, and its slower when iterating, as one works towards a fix). > last week, -L makes no sense at all for papi (since it's so stateful; > each "pmCLIENT -L ..." loads the pmda.so+papi.so to form a new blank > slate, does one thing, and immediately dies). So no such test will > ever see counter numbers, or results other than those related to > exactly one initialization-type operation -- plus any __pmNotifyErr > noise that would've gone to a pmcd/papi.log file. That's correct, and well known (they still cover a large amount of the PMDA functionality). What's your point?... > In fact, the only way anyone could have ever seen the pmda work and produce > counter *numbers*, was by -not- using the -L mode, which means its core > was only ever "hand tested" and qa/914 was to that extent a talisman. Heh, I see, seeking opportunity to grind the "hand testing" axe. Contrary to what you say here, the tests exposed a number of bugs & will continue to do so going forward. They are also small, fast smoke tests for initial sanity and will be handy for exposing PAPI problems related to enabling every possible hardware counter at once (i.e. the notorious short-lived client case). As discussed awhile back, I added these tests as just a start on testing, so that we had some basic initial coverage. So, please leave these tests as-is and build on this base with new tests. > Anyway, as we were saying on IRC, one plausible way of using -L is via > dbpmda, instead of a PMAPI client. That could work ... but OTOH might > be tricky to feed it a fixed dbpmda command script (like a few other > qa/NNN tests do) and get meaningful answers out, at least because the > available counters=metrics & runtime environment can so easily vary > between machines. A simple testcase driver is probably not up to the job. > I think you should be able to get by with dbpmda/pmcd and a helper PAPI test program (ala qa/src/*.c) - you can then check the values the PMDA gives (via dbpmda/pmcd) against the values a helper test program gives. As mentioned on IRC, dbpmda and local context opens up valgrind testing opportunities (highly recommended for this PMDA). For checking actual values (again, only a tiny part of the the testing needed), using either pmcd/dbpmda will suit I'm sure. cheers. -- Nathan From chandana@desilva.id.au Sun Oct 12 20:41:59 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 15E047F4E for ; Sun, 12 Oct 2014 20:41:59 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id D9D4A304048 for ; Sun, 12 Oct 2014 18:41:55 -0700 (PDT) X-ASG-Debug-ID: 1413164510-04cb6c77043571c0001-S8gJnT Received: from mho-01-ewr.mailhop.org (mho-03-ewr.mailhop.org [204.13.248.66]) by cuda.sgi.com with ESMTP id zAE4t9LqmPjakMWU (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Sun, 12 Oct 2014 18:41:51 -0700 (PDT) X-Barracuda-Envelope-From: chandana@desilva.id.au X-Barracuda-Apparent-Source-IP: 204.13.248.66 Received: from ec2-54-252-74-219.ap-southeast-2.compute.amazonaws.com ([54.252.74.219] helo=mail.desilva.id.au) by mho-01-ewr.mailhop.org with esmtpa (Exim 4.72) (envelope-from ) id 1XdUdy-000LIk-Bt for pcp@oss.sgi.com; Mon, 13 Oct 2014 01:41:50 +0000 Received: from [192.168.19.83] (unknown [175.45.83.34]) by mail.desilva.id.au (Postfix) with ESMTPSA id 06C0524A64 for ; Mon, 13 Oct 2014 01:41:48 +0000 (UTC) X-Mail-Handler: Dyn Standard SMTP by Dyn X-Originating-IP: 54.252.74.219 X-Report-Abuse-To: abuse@dyndns.com (see http://www.dyndns.com/services/sendlabs/outbound_abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX19goawv+c2mvFyBFk0lkSmtFcVi38IVb8Y= Message-ID: <1413164506.6110.15.camel@tardis> Subject: Possible Bug in Linux PMDA From: Chandana De Silva X-ASG-Orig-Subj: Possible Bug in Linux PMDA Reply-To: chandana@desilva.id.au To: pcp@oss.sgi.com Date: Mon, 13 Oct 2014 12:41:46 +1100 Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.10.4 (3.10.4-4.fc20) Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: mho-03-ewr.mailhop.org[204.13.248.66] X-Barracuda-Start-Time: 1413164510 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10498 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- $ pminfo -d -t disk.all.avactive disk.all.avactive [total count of active time, summed for all disks] Data Type: 64-bit unsigned int InDom: PM_INDOM_NULL 0xffffffff Semantics: counter Units: millisec Should the semantics be "instantaneous" ? Chandana From kenj@internode.on.net Sun Oct 12 21:49:39 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 4C24D7F4E for ; Sun, 12 Oct 2014 21:49:39 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 2D991304059 for ; Sun, 12 Oct 2014 19:49:36 -0700 (PDT) X-ASG-Debug-ID: 1413168570-04cbb018ae32ac30001-S8gJnT Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by cuda.sgi.com with ESMTP id 7J0o9SmYU5IY0u8j for ; Sun, 12 Oct 2014 19:49:31 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.145 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AsYBAC89O1R20YErPGdsb2JhbAANTot2x3eDIQKBJAEGAQEBATiEPgEBBDhAEQsYCRYPCQMCAQIBMRQTCAEBthWVUwEBCAIBH5BMFoQ1AQS3Z4MiAQEB Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail06.adl6.internode.on.net with ESMTP; 13 Oct 2014 13:19:12 +1030 Message-ID: <543B3DCB.7070401@internode.on.net> Date: Mon, 13 Oct 2014 13:49:47 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: Re: [pcp] Possible Bug in Linux PMDA References: <1413164506.6110.15.camel@tardis> X-ASG-Orig-Subj: Re: [pcp] Possible Bug in Linux PMDA In-Reply-To: <1413164506.6110.15.camel@tardis> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl6.internode.on.net[150.101.137.145] X-Barracuda-Start-Time: 1413168570 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10502 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- On 13/10/14 12:41, Chandana De Silva wrote: > $ pminfo -d -t disk.all.avactive > > disk.all.avactive [total count of active time, summed for all disks] > Data Type: 64-bit unsigned int InDom: PM_INDOM_NULL 0xffffffff > Semantics: counter Units: millisec > > > Should the semantics be "instantaneous" ? G'day Chandana. Nope, this one is a counter. If you sample it with pmval or pmchart or pmie then the "value" will be turned into a time utilization in the range 0 (no activity) to N.0 (for N (== hinv.ndisk) disks in the system, all devices are busy all of the time during the sample). For practical purposes, one would probably choose disk.dev.avactive for the per disk utilization (in the range 0.0 to 1.0), else combine in pmie and divide by hinv.ndisk (to get the average disk utilization in the range 0.0 to 1.0). From kenj@internode.on.net Sun Oct 12 23:05:42 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: ** X-Spam-Status: No, score=3.0 required=5.0 tests=TVD_SUBJ_NUM_OBFU_MINFP autolearn=no version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 80DF97F4E for ; Sun, 12 Oct 2014 23:05:42 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 55C35304064 for ; Sun, 12 Oct 2014 21:05:39 -0700 (PDT) X-ASG-Debug-ID: 1413173136-04bdf0287733a320001-S8gJnT Received: from ipmail06.adl2.internode.on.net (ipmail06.adl2.internode.on.net [150.101.137.129]) by cuda.sgi.com with ESMTP id HUgHYG8PAMr5u0nh for ; Sun, 12 Oct 2014 21:05:37 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.129 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Au0DAAtOO1QBmK/lPGdsb2JhbAANToc/hDewCQabAAEIAoElAQYBAQEBOIQ+AQEEI1YQCw0BDAImAgJDFAYBtjF4lGIBAQEBAQEBAQEBAQEBAQEBARuBLIR0iiUHgneBVAWlWpINgyIBAQE Received: from unknown (HELO [10.144.54.134]) ([1.152.175.229]) by ipmail06.adl2.internode.on.net with ESMTP; 13 Oct 2014 14:35:35 +1030 Date: Mon, 13 Oct 2014 15:05:34 +1100 From: Kem McDonell To: Mark Goodwin ,Nathan Scott Cc: Bud Brown ,Laurence Oberman ,pcp Message-ID: <55260de5-5779-4baa-a152-6fd020ad79a5.maildroid@localhost> In-Reply-To: <1192069605.65384998.1412904913945.JavaMail.zimbra@redhat.com> References: <5434DC73.4080400@redhat.com> <1192069605.65384998.1412904913945.JavaMail.zimbra@redhat.com> Subject: Re: [pcp] iostat2pcp broken for iostat in RHEL/fedora MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] iostat2pcp broken for iostat in RHEL/fedora Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl2.internode.on.net[150.101.137.129] X-Barracuda-Start-Time: 1413173136 X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10503 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Can we do the same as for sadc? If I recall we checkin the output files for each version we know about and force the test to fail if we see a version that we have not tested? From kenj@internode.on.net Mon Oct 13 01:46:32 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id B679C7F4E for ; Mon, 13 Oct 2014 01:46:32 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 9CD358F8049 for ; Sun, 12 Oct 2014 23:46:29 -0700 (PDT) X-ASG-Debug-ID: 1413182783-04cbb018ac32ff70001-S8gJnT Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by cuda.sgi.com with ESMTP id DmRVUpgYTpjuApmZ for ; Sun, 12 Oct 2014 23:46:23 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.145 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AscBAIF0O1R20YErPGdsb2JhbAANToNhWIMGhDfDTIh8AQYBAQEBOIRnVTAGAgUWCwILAwIBAgExJwYCAQGIR61peJUZgSyPNoJhgVQFhi2QDqEsWIJKAQEB Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail06.adl6.internode.on.net with ESMTP; 13 Oct 2014 17:16:22 +1030 Message-ID: <543B7562.6040907@internode.on.net> Date: Mon, 13 Oct 2014 17:46:58 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: pcp updates - qa Content-Type: text/plain; charset=utf-8 X-ASG-Orig-Subj: pcp updates - qa Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl6.internode.on.net[150.101.137.145] X-Barracuda-Start-Time: 1413182783 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10506 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Changes committed to git://git.performancecopilot.org/kenj/pcp.git dev qa/823 | 47 +++++++++++++++++++++++++++++++---------------- qa/README | 5 ++++- qa/check | 1 + 3 files changed, 36 insertions(+), 17 deletions(-) commit 2e71f95769bd38c55f4a983513de5f41dc234e73 Author: Ken McDonell Date: Mon Oct 13 17:46:24 2014 +1100 qa/README - add notes about firewalls commit 3661fd1ca6aba35297305f037034c968ea63f505 Author: Ken McDonell Date: Mon Oct 13 17:46:01 2014 +1100 qa/823 - lots more diagnostics to help debug this one commit 128e1cfea8ebf36e74cf82380e6dbfe6e3285c71 Author: Ken McDonell Date: Mon Oct 13 17:45:24 2014 +1100 qa/check - add one more \n when a failure occurs before the diff output From nscott@redhat.com Mon Oct 13 01:58:13 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id C1CF47F4E for ; Mon, 13 Oct 2014 01:58:13 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 45E85AC005 for ; Sun, 12 Oct 2014 23:58:13 -0700 (PDT) X-ASG-Debug-ID: 1413183490-04cb6c770535c890001-S8gJnT Received: from mx3-phx2.redhat.com (mx3-phx2.redhat.com [209.132.183.24]) by cuda.sgi.com with ESMTP id 1alAD2US14VLjUyS (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Sun, 12 Oct 2014 23:58:10 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.24 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx3-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s9D6wA4S009386 for ; Mon, 13 Oct 2014 02:58:10 -0400 Date: Mon, 13 Oct 2014 02:58:10 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: pcp developers Message-ID: <4643669.66587324.1413183490073.JavaMail.zimbra@redhat.com> In-Reply-To: <1373667104.66586691.1413183331563.JavaMail.zimbra@redhat.com> Subject: pcp updates: qa, libpcp, debs, iostat2pcp MIME-Version: 1.0 X-ASG-Orig-Subj: pcp updates: qa, libpcp, debs, iostat2pcp Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: pcp updates: qa, libpcp, debs, iostat2pcp Thread-Index: SHf7d33ffB+wy6beEHlsEIeSunlCxg== X-Barracuda-Connect: mx3-phx2.redhat.com[209.132.183.24] X-Barracuda-Start-Time: 1413183490 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10507 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Changes committed to git://git.pcp.io/pcp.git dev Ken McDonell (9): qa/943 - dodge error from proc.psinfo.cgroups qa/qa_hosts.master - tweak kenj's entries qa/734 - fix for longer hostnames qa/752 - fix problems with timezones and certain times of day libpcp/AF.c - fix lock initialization race qa/752 - reorder filter lines to dodge ambiguity qa/check - add one more \n when a failure occurs before the diff output qa/823 - lots more diagnostics to help debug this one qa/README - add notes about firewalls Nathan Scott (2): iostat2pcp: cater for changes to iostat output formats packaging: update debian suggests vs recommends usage debian/control | 5 debian/control.master | 4 debian/control.pcpgui | 3 man/man1/iostat2pcp.1 | 2 qa/373 | 3 qa/373.out | 287 ++++++++++++++++++++++++++++++++++++++++++++++ qa/734 | 4 qa/752 | 34 ++++- qa/752.out | 36 ++--- qa/823 | 47 ++++--- qa/943 | 7 - qa/README | 5 qa/check | 1 qa/qa_hosts.master | 3 qa/src/iostat-t-x-bud | 62 +++++++++ src/iostat2pcp/iostat2pcp | 75 ++++++++++-- src/libpcp/src/AF.c | 8 + 17 files changed, 517 insertions(+), 69 deletions(-) From fche@redhat.com Mon Oct 13 06:44:53 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id B04C27F4E for ; Mon, 13 Oct 2014 06:44:53 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 91700304048 for ; Mon, 13 Oct 2014 04:44:53 -0700 (PDT) X-ASG-Debug-ID: 1413200692-04cbb018ae3386f0001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id PD0xuMZ9PlvqbIE0 (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Mon, 13 Oct 2014 04:44:52 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9DBiplh012130 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Mon, 13 Oct 2014 07:44:51 -0400 Received: from fche.csb (vpn-232-103.phx2.redhat.com [10.3.232.103]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9DBippe006414; Mon, 13 Oct 2014 07:44:51 -0400 Received: by fche.csb (Postfix, from userid 2569) id EDBD358127; Mon, 13 Oct 2014 07:44:50 -0400 (EDT) Date: Mon, 13 Oct 2014 07:44:50 -0400 From: "Frank Ch. Eigler" To: Nathan Scott Cc: lberk@redhat.com, pcp developers Subject: Re: [pcp] another round of review for lberk/papi Message-ID: <20141013114450.GA4170@redhat.com> X-ASG-Orig-Subj: Re: [pcp] another round of review for lberk/papi References: <20141010190642.GD26999@redhat.com> <479818101.66478026.1413162710461.JavaMail.zimbra@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <479818101.66478026.1413162710461.JavaMail.zimbra@redhat.com> User-Agent: Mutt/1.4.2.2i X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1413200692 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Hi - > > First of all, the existing test cases (qa/914 and qa/799) simply must > > (I think you mean qa/903 and qa/914) No, in the lberk/papi branch, a qa/799 exists. 903 appears only to run the pmda Install/Remove scripts. > > stop relying on the -L local context this way. As we discussed on IRC > > [...] > They exposed bugs in the authentication mechanism, exercise the install > and remove mechanisms Only barely so (see below): and not the "remove" mechanisms (since there is nothing to remove from an -L blank slate). > [...] > > In fact, the only way anyone could have ever seen the pmda work and produce > > counter *numbers*, was by -not- using the -L mode, which means its core > > was only ever "hand tested" and qa/914 was to that extent a talisman. > > Heh, I see, seeking opportunity to grind the "hand testing" axe. You drafted the term into service -as- an axe. > Contrary to what you say here, the tests exposed a number of bugs & will > continue to do so going forward. It is no surprise at all that the very early snapshot of any code has a number of bugs. The fact that trivial tests can tickle only one or two entry points and fail is only evidence that the code was not ready. It is not evidence of the thoroughness of QA, nor that the code was working. It did not "assure quality", just "show incompleteness". (At several points, not even hand-tests worked in the merged code.) > They are also small, fast smoke tests for initial sanity and will be > handy for exposing PAPI problems related to enabling every possible > hardware counter at once (i.e. the notorious short-lived client > case). No, they're not actually effective at even that. With the pmStore-many-names model, as foreseen, there's no way for the user to know which of multiple counters requested was erroneous, and which counters were activated. And with the -L testing mode, you can't see the aftereffects either. > As discussed awhile back, I added these tests as just a start on > testing, so that we had some basic initial coverage. So, please > leave these tests as-is and build on this base with new tests. > [...] Simply switching them from -L to the default -h would actually get a bit of new coverage of the core code. What's the disadvantage? - FChE From wwwrun@oss.sgi.com Mon Oct 13 09:18:51 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE,NO_RELAYS autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: by oss.sgi.com (Postfix, from userid 30) id 1CDD97F51; Mon, 13 Oct 2014 09:18:51 -0500 (CDT) From: bugzilla-daemon@oss.sgi.com To: pcp@oss.sgi.com Subject: [Bug 1069] New: libpcp AF functionality has posix-signal-unsafe elements Date: Mon, 13 Oct 2014 14:18:50 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Classification: Unclassified X-Bugzilla-Product: pcp X-Bugzilla-Component: pcp X-Bugzilla-Keywords: X-Bugzilla-Severity: major X-Bugzilla-Who: fche@redhat.com X-Bugzilla-Status: NEW X-Bugzilla-Priority: P5 X-Bugzilla-Assigned-To: pcp@kenj.com.au X-Bugzilla-Target-Milestone: --- X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter cc classification Message-ID: Content-Type: multipart/alternative; boundary="1413209931.500a11.19936"; charset="us-ascii" X-Bugzilla-URL: http://oss.sgi.com/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 --1413209931.500a11.19936 Date: Mon, 13 Oct 2014 09:18:51 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" http://oss.sgi.com/bugzilla/show_bug.cgi?id=1069 Bug ID: 1069 Summary: libpcp AF functionality has posix-signal-unsafe elements Product: pcp Version: unspecified Hardware: All OS: Linux Status: NEW Severity: major Priority: P5 Component: pcp Assignee: pcp@kenj.com.au Reporter: fche@redhat.com CC: pcp@oss.sgi.com Classification: Unclassified An inspection of the libpcp/src/AF.c code indicates that it relies on unsafe mechanisms that can cause heisencrashes. The gist of the problem is that from within a SIGALRM signal handler, it is not safe to invoke general libc/application functions. A poorly timed callback can corrupt e.g. libc malloc/stdio or libpcp internals, or result in hangs. (These have been observed in the wild, just not necessarily in the context of pcp.) Some general references on async-signal safety, which applies to the whole transitive callchain of signal handlers: http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html https://www.gnu.org/software/libc/manual/html_node/POSIX-Safety-Concepts.html The specific problems include: AF.c:onalarm() - calling free (heap operations) - calling stdio printf/putc (e.g. in pmDebug case) - pmprintf() and pmflush() AF.c:enqueue() (called both from onalarm and __pmAFregister): - doing list manipulation with limited (AFhold) reentrancy controls, which could be broken if e.g. __pmAFregister is interrupted by some other signal wherein __pmAF* functions are called. The lack of documented constraints on __pmAFregister callback functions in pmaf.3 leads clients to do risky things: pmlogger.c:run_done_callback - more stdio pmlogger.c:vol_switch_callback - more stdio, including fopen/fclose - more racy AF manpulation callback.c:log_callback - heap operations - many general LIBPCP ops perl/PMDA/PMDA.xs:timer_callback - call into general perl interpreter It may be possible to trigger some example problems with numerous/rapid/unsafe-content __pmAFregister callbacks. I got a toy program to show some malloc corruption, but some of the race windows are short enough that auditing rather than simple tests may be necessary. (Lengthening some of the race windows by inserting usleep() here and there might help.) The longevity of this code testifies that these races & corruption are infrequent, so fixing the problems is not urgent. One possible thorough approach for an eventual fix would be to move away from timers/signal handlers, and manage events/timing at (say) the exit of PMAPI functions, or with a more formal application-main-loop mechanism. Some of the races may be shrunk with more aggressive __pmAFblock, and/or nestedness counting for AF.c:block. -- You are receiving this mail because: You are on the CC list for the bug. --1413209931.500a11.19936 Date: Mon, 13 Oct 2014 09:18:51 -0500 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8"
Bug ID 1069
Summary libpcp AF functionality has posix-signal-unsafe elements
Product pcp
Version unspecified
Hardware All
OS Linux
Status NEW
Severity major
Priority P5
Component pcp
Assignee pcp@kenj.com.au
Reporter fche@redhat.com
CC pcp@oss.sgi.com
Classification Unclassified 

An inspection of the libpcp/src/AF.c code indicates that it relies on
unsafe mechanisms that can cause heisencrashes.  The gist of the
problem is that from within a SIGALRM signal handler, it is not safe
to invoke general libc/application functions.  A poorly timed callback
can corrupt e.g. libc malloc/stdio or libpcp internals, or result in
hangs.  (These have been observed in the wild, just not necessarily
in the context of pcp.)

Some general references on async-signal safety, which applies to the
whole transitive callchain of signal handlers:

http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html
https://www.gnu.org/software/libc/manual/html_node/POSIX-Safety-Concepts.html

The specific problems include:

AF.c:onalarm()
- calling free (heap operations)
- calling stdio printf/putc (e.g. in pmDebug case)
- pmprintf() and pmflush()

AF.c:enqueue() (called both from onalarm and __pmAFregister):
- doing list manipulation with limited (AFhold) reentrancy controls,
  which could be broken if e.g. __pmAFregister is interrupted by
  some other signal wherein __pmAF* functions are called.

The lack of documented constraints on __pmAFregister callback
functions in pmaf.3 leads clients to do risky things:

pmlogger.c:run_done_callback
- more stdio
pmlogger.c:vol_switch_callback
- more stdio, including fopen/fclose
- more racy AF manpulation
callback.c:log_callback
- heap operations
- many general LIBPCP ops

perl/PMDA/PMDA.xs:timer_callback
- call into general perl interpreter


It may be possible to trigger some example problems with
numerous/rapid/unsafe-content __pmAFregister callbacks.  I got a toy
program to show some malloc corruption, but some of the race windows
are short enough that auditing rather than simple tests may be
necessary.  (Lengthening some of the race windows by inserting
usleep() here and there might help.)


The longevity of this code testifies that these races & corruption are
infrequent, so fixing the problems is not urgent.  One possible
thorough approach for an eventual fix would be to move away from
timers/signal handlers, and manage events/timing at (say) the exit of
PMAPI functions, or with a more formal application-main-loop
mechanism.  Some of the races may be shrunk with more aggressive
__pmAFblock, and/or nestedness counting for AF.c:block.

You are receiving this mail because:
  • You are on the CC list for the bug.
--1413209931.500a11.19936-- From michele@acksyn.org Mon Oct 13 13:43:23 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id E0BDC7F4E for ; Mon, 13 Oct 2014 13:43:22 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 60094AC00E for ; Mon, 13 Oct 2014 11:43:19 -0700 (PDT) X-ASG-Debug-ID: 1413225792-04cbb018ac353540001-S8gJnT Received: from palahniuk.acksyn.org (palahniuk.acksyn.org [5.9.7.26]) by cuda.sgi.com with ESMTP id QHK5aPLU2ChMMeOo for ; Mon, 13 Oct 2014 11:43:13 -0700 (PDT) X-Barracuda-Envelope-From: michele@acksyn.org X-Barracuda-Apparent-Source-IP: 5.9.7.26 Received: from localhost (localhost [127.0.0.1]) by palahniuk.acksyn.org (Postfix) with ESMTP id 52C0828DF2 for ; Mon, 13 Oct 2014 14:43:12 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=acksyn.org; h= user-agent:content-disposition:content-type:content-type :mime-version:message-id:subject:subject:from:from:date:date :received:received; s=2010; t=1413225791; bh=GIuHZMtzizkLP1WyMdG a9zwwVzHXOW33P1mOuumYGmg=; b=i4SpeN1KU7OK+WZg+2rwDvFgPEmgrEMhXKL Rmgq8oxrd3LmQS977ap9r+OTwpTEsmEYqJMDVJu9uKJ34oL1itUC+yWDxUP+cMr+ izDu+Q/PPkB8Kn45QwjwmKFQetvlg4c/PcnMM9h3UKgybTHFKLvJEREP7q/v8f5T rPRljFew= Received: from palahniuk.acksyn.org ([127.0.0.1]) by localhost (mail.acksyn.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id sb6jrqyjD_a4 for ; Mon, 13 Oct 2014 14:43:11 -0400 (EDT) Received: from localhost (host41-190-dynamic.24-79-r.retail.telecomitalia.it [79.24.190.41]) by palahniuk.acksyn.org (Postfix) with ESMTPSA id 4604324CF6 for ; Mon, 13 Oct 2014 14:43:11 -0400 (EDT) Date: Mon, 13 Oct 2014 20:43:10 +0200 From: Michele Baldessari To: pcp@oss.sgi.com Subject: pmwebd oddity? Message-ID: <20141013184310.GE19405@marquez.int.rhx> X-ASG-Orig-Subj: pmwebd oddity? MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2012-12-30) X-Barracuda-Connect: palahniuk.acksyn.org[5.9.7.26] X-Barracuda-Start-Time: 1413225793 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-BRTS-Evidence: acksyn.org X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10525 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature Hi all, it's likely something silly I am doing, but at the moment I am just not seeing it. I am printing out my ADSL bandwith using pmwebd/json and an Arduino [1]. Now I wanted to add latency as well and I thought about using a pretty horrendous but very quick hack via the shping PMDA: I add a simple script which just outputs the latency via status error. >From shping's sample.conf: acksyn.org /var/lib/pcp/pmdas/shping/ping.sh ping.sh being: """ #!/bin/sh # This hack will break if latency > 254 ;) ret=`ping -c 2 -n -q 8.8.8.8 | grep ^rtt | cut -f5 -d\/ | cut -f1 -d\.` exit $ret """ Now, leaving aside for a second how fragile and broken by design the whole thing is, everything works via pminfo but not via pmwebapi: - pminfo -f -m -M shping.error shping.error PMID: 19.0.10 = 79691786 = 0x4c0000a inst [0 or "8.8.8.8"] value 52 - pmwebd returns empty values : {u'timestamp': {u's': 1413224620, u'us': 977986}, u'values': []} On the network I see the following exchange: """ GET /pmapi/747499891/_fetch?pmids=79691786 HTTP/1.1 Host: fw:44323 Accept-Encoding: gzip, deflate Accept: */* User-Agent: python-requests/2.3.0 CPython/2.7.8 Linux/3.17.0-301.fc21.x86_64 HTTP/1.1 200 OK Connection: Keep-Alive Content-Length: 63 Access-Control-Allow-Origin: * Content-Type: application/json Date: Mon, 13 Oct 2014 18:19:30 GMT { "timestamp": { "s":1413224370, "us":801625 }, "values": [ ] } """ I've tried restarting both pmcd and pmwebd. Do I need to do anything special to get shping.error exported via JSON? I have also tried adding -v to the options but I only see context creation or expiration messages. Anything I missed? (I assume that without specifying -K it should take all PMDAs?) Thanks, Michele [1] http://acksyn.org/blog/2014/10/09/performance-co-pilot-and-arduino/ -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D From fche@redhat.com Mon Oct 13 14:02:25 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 9A8AA7F4E for ; Mon, 13 Oct 2014 14:02:25 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 4590CAC00C for ; Mon, 13 Oct 2014 12:02:22 -0700 (PDT) X-ASG-Debug-ID: 1413226937-04bdf0287735a250001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id HLSk3zdG0RCNt3CC (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Mon, 13 Oct 2014 12:02:18 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9DJ2FWZ005729 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 13 Oct 2014 15:02:15 -0400 Received: from fche.csb (vpn-232-103.phx2.redhat.com [10.3.232.103]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9DJ2EuH029526; Mon, 13 Oct 2014 15:02:15 -0400 Received: by fche.csb (Postfix, from userid 2569) id 54F9158127; Mon, 13 Oct 2014 15:02:14 -0400 (EDT) To: Michele Baldessari Cc: pcp@oss.sgi.com Subject: Re: pmwebd oddity? References: <20141013184310.GE19405@marquez.int.rhx> X-ASG-Orig-Subj: Re: pmwebd oddity? From: fche@redhat.com (Frank Ch. Eigler) Date: Mon, 13 Oct 2014 15:02:14 -0400 In-Reply-To: <20141013184310.GE19405@marquez.int.rhx> (Michele Baldessari's message of "Mon, 13 Oct 2014 20:43:10 +0200") Message-ID: User-Agent: Gnus/5.1008 (Gnus v5.10.8) Emacs/21.4 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1413226938 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Hi, Michele - > it's likely something silly I am doing, but at the moment I am just not > seeing it. I am printing out my ADSL bandwith using pmwebd/json and an > Arduino [1]. Yup, saw it, cute! > [...] > Now, leaving aside for a second how fragile and broken by design the whole thing is, > everything works via pminfo but not via pmwebapi: > - pminfo -f -m -M shping.error > shping.error PMID: 19.0.10 = 79691786 = 0x4c0000a > inst [0 or "8.8.8.8"] value 52 Right, note that pminfo defaults to "-h local:". > - pmwebd returns empty values : > GET /pmapi/747499891/_fetch?pmids=79691786 HTTP/1.1 > [...] > { "timestamp": { "s":1413224370, "us":801625 }, > "values": [ > ] } How was that context 747499891 created? /pmapi/context?local=ANYTHING is not the same thing; use /pmapi/context?hostname=local: instead. Perhaps pmwebd should default to -h local: (and support an unparametrized /pmapi/context?) the same way pm* clients do. Opinions? If that's not the problem - if you're using -h local: context via pmwebd also, then it could be time for libpcp debugging type work, namely invoking "pmwebd -Dall ..." and gaze at the trace data. - FChE From michele@acksyn.org Mon Oct 13 15:12:54 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 696607F4E for ; Mon, 13 Oct 2014 15:12:54 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id DD91CAC00D for ; Mon, 13 Oct 2014 13:12:50 -0700 (PDT) X-ASG-Debug-ID: 1413231165-04cb6c770638f8c0001-S8gJnT Received: from palahniuk.acksyn.org (palahniuk.acksyn.org [5.9.7.26]) by cuda.sgi.com with ESMTP id uB6IATgar9mwCOEf for ; Mon, 13 Oct 2014 13:12:45 -0700 (PDT) X-Barracuda-Envelope-From: michele@acksyn.org X-Barracuda-Apparent-Source-IP: 5.9.7.26 Received: from localhost (localhost [127.0.0.1]) by palahniuk.acksyn.org (Postfix) with ESMTP id 39E6324A02; Mon, 13 Oct 2014 16:12:45 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=acksyn.org; h= user-agent:in-reply-to:content-disposition:content-type :content-type:mime-version:references:message-id:subject:subject :from:from:date:date:received:received; s=2010; t=1413231164; bh=hCQFRFDcxbZDQOi6ylCqvteOaYoYm5LLe5IIOyDMV1o=; b=XtIg7c6n6ihr Df04jYNzuruEZFYfQZpvO5kULSTHiI64Lp7Bdha0EkYTkFajl5vFWyj+hRpM0adv pRLHGnve1GqjHdRSNM46kGWFbicU76Q3KFg1nLcA7ZGAf+l77WekUD/o/ihlbBqU RM82IfzfuRZXMX0TtunW97i+Ds24v9I= Received: from palahniuk.acksyn.org ([127.0.0.1]) by localhost (mail.acksyn.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id m7EMg-G1aiNB; Mon, 13 Oct 2014 16:12:44 -0400 (EDT) Received: from localhost (host41-190-dynamic.24-79-r.retail.telecomitalia.it [79.24.190.41]) by palahniuk.acksyn.org (Postfix) with ESMTPSA id 3389022977; Mon, 13 Oct 2014 16:12:44 -0400 (EDT) Date: Mon, 13 Oct 2014 22:12:43 +0200 From: Michele Baldessari To: "Frank Ch. Eigler" Cc: pcp@oss.sgi.com Subject: Re: pmwebd oddity? Message-ID: <20141013201243.GG19405@marquez.int.rhx> X-ASG-Orig-Subj: Re: pmwebd oddity? References: <20141013184310.GE19405@marquez.int.rhx> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="+HP7ph2BbKc20aGI" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2012-12-30) X-Barracuda-Connect: palahniuk.acksyn.org[5.9.7.26] X-Barracuda-Start-Time: 1413231165 X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10528 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature --+HP7ph2BbKc20aGI Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi Frank, On Mon, Oct 13, 2014 at 03:02:14PM -0400, Frank Ch. Eigler wrote: > > - pmwebd returns empty values : > > > GET /pmapi/747499891/_fetch?pmids=79691786 HTTP/1.1 > > [...] > > { "timestamp": { "s":1413224370, "us":801625 }, > > "values": [ > > ] } > > How was that context 747499891 created? /pmapi/context?local=ANYTHING > is not the same thing; use /pmapi/context?hostname=local: instead. > Perhaps pmwebd should default to -h local: (and support an unparametrized > /pmapi/context?) the same way pm* clients do. Opinions? Ah there you go. I am attaching a patch to the man page, to make it slightly more obvious and to cleanse away my API sins. I failed to notice the difference between the two ;) Personally, I think -h local: is a bit more obvious and expected, but as long as it's somehwat clearer in the pmwebapi man page, I'm fine with the status quo. Thanks again, Michele -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D --+HP7ph2BbKc20aGI Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="0001-Clarify-the-limits-of-PM_CONTEXT_LOCAL-when-using-pm.patch" >From a72c1c716ac4306a3ba307ccb3d3ad4dba8b11c6 Mon Sep 17 00:00:00 2001 From: Michele Baldessari Date: Mon, 13 Oct 2014 22:08:43 +0200 Subject: [PATCH] Clarify the limits of PM_CONTEXT_LOCAL when using pmwebd --- man/man3/pmwebapi.3 | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/man/man3/pmwebapi.3 b/man/man3/pmwebapi.3 index e4ad6f4639e0..7a6d0abf0062 100644 --- a/man/man3/pmwebapi.3 +++ b/man/man3/pmwebapi.3 @@ -49,7 +49,10 @@ header is added to all JSON responses. To create a new web context identifier, a web client invokes: .TP .B /pmapi/context?local=ANYTHING -Creates a PM_CONTEXT_LOCAL PMAPI context. +Creates a PM_CONTEXT_LOCAL PMAPI context. Note that when using this context the range of accessible performance metrics +is constrained to those from the operating system. See +.BR pmNewContext (3) +for more information. .TP .B /pmapi/context?hostname=STRING .TP -- 2.1.0 --+HP7ph2BbKc20aGI-- From fche@redhat.com Mon Oct 13 15:44:14 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id E9E857F4E for ; Mon, 13 Oct 2014 15:44:13 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 8408FAC00D for ; Mon, 13 Oct 2014 13:44:13 -0700 (PDT) X-ASG-Debug-ID: 1413233052-04bdf0287635ef90001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id Gj0BN0VWowhVvVgN (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Mon, 13 Oct 2014 13:44:12 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9DKiAUc006456 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 13 Oct 2014 16:44:10 -0400 Received: from fche.csb (vpn-232-103.phx2.redhat.com [10.3.232.103]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9DKi99q018624; Mon, 13 Oct 2014 16:44:10 -0400 Received: by fche.csb (Postfix, from userid 2569) id 5DC0058127; Mon, 13 Oct 2014 16:44:09 -0400 (EDT) Date: Mon, 13 Oct 2014 16:44:09 -0400 From: "Frank Ch. Eigler" To: Michele Baldessari Cc: pcp@oss.sgi.com Subject: Re: pmwebd oddity? Message-ID: <20141013204409.GA5645@redhat.com> X-ASG-Orig-Subj: Re: pmwebd oddity? References: <20141013184310.GE19405@marquez.int.rhx> <20141013201243.GG19405@marquez.int.rhx> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20141013201243.GG19405@marquez.int.rhx> User-Agent: Mutt/1.4.2.2i X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1413233052 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Hi, Michele - > Ah there you go. I am attaching a patch to the man page, to make it slightly > more obvious and to cleanse away my API sins. I failed to notice the > difference between the two ;) OK. > Personally, I think -h local: is a bit more obvious and expected, but as > long as it's somehwat clearer in the pmwebapi man page, I'm fine with > the status quo. OK. Your patch and a followup amplification are merged into the git://sourceware.org/git/pcpfans.git pcpfans-dev branch, which is a traditional pcp+pmwebd+etc. unified branch that otherwise tracks upstream dev/. - FChE From michele@acksyn.org Mon Oct 13 16:00:10 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id AD9CE7F4E for ; Mon, 13 Oct 2014 16:00:10 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 476D7AC00E for ; Mon, 13 Oct 2014 14:00:10 -0700 (PDT) X-ASG-Debug-ID: 1413234008-04bdf0287635fc00001-S8gJnT Received: from palahniuk.acksyn.org (palahniuk.acksyn.org [5.9.7.26]) by cuda.sgi.com with ESMTP id pqCwBnWvYzaiUJ1Q for ; Mon, 13 Oct 2014 14:00:08 -0700 (PDT) X-Barracuda-Envelope-From: michele@acksyn.org X-Barracuda-Apparent-Source-IP: 5.9.7.26 Received: from localhost (localhost [127.0.0.1]) by palahniuk.acksyn.org (Postfix) with ESMTP id 17D4C28E26; Mon, 13 Oct 2014 17:00:08 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=acksyn.org; h= user-agent:in-reply-to:content-disposition:content-type :content-type:mime-version:references:message-id:subject:subject :from:from:date:date:received:received; s=2010; t=1413234007; bh=G7acHQSNPzK8lNukQ8qRL8s6SMZqWnu2ru8fgE7Us48=; b=CBLTofyPL27K IJ3t+6D0RJTfhCCqv76rTM+AD3cApOta3YV0A71lH/n+z4zxl3Fn0a1KDkFc6AdL aGnxXiEhIxpfRFUx1HY6E7b9Yt/kEwIvzHwKDUrLfeoAdzBOjoSqOagL4g4/qdFv dfyS9EJg8r3y8V1pirjKpvUuwwA3+ow= Received: from palahniuk.acksyn.org ([127.0.0.1]) by localhost (mail.acksyn.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id tW5wvvrNbpZY; Mon, 13 Oct 2014 17:00:07 -0400 (EDT) Received: from localhost (host41-190-dynamic.24-79-r.retail.telecomitalia.it [79.24.190.41]) by palahniuk.acksyn.org (Postfix) with ESMTPSA id 4820328D7F; Mon, 13 Oct 2014 17:00:07 -0400 (EDT) Date: Mon, 13 Oct 2014 23:00:05 +0200 From: Michele Baldessari To: "Frank Ch. Eigler" Cc: pcp@oss.sgi.com Subject: Re: pmwebd oddity? Message-ID: <20141013210005.GH19405@marquez.int.rhx> X-ASG-Orig-Subj: Re: pmwebd oddity? References: <20141013184310.GE19405@marquez.int.rhx> <20141013201243.GG19405@marquez.int.rhx> <20141013204409.GA5645@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20141013204409.GA5645@redhat.com> User-Agent: Mutt/1.5.21 (2012-12-30) X-Barracuda-Connect: palahniuk.acksyn.org[5.9.7.26] X-Barracuda-Start-Time: 1413234008 X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10531 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature Hi Frank, On Mon, Oct 13, 2014 at 04:44:09PM -0400, Frank Ch. Eigler wrote: > OK. Your patch and a followup amplification are merged into the > git://sourceware.org/git/pcpfans.git pcpfans-dev branch, which is a > traditional pcp+pmwebd+etc. unified branch that otherwise tracks > upstream dev/. thanks a lot, much appreciated. regards, Michele -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D From nscott@redhat.com Mon Oct 13 16:08:40 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 3014F7F4E for ; Mon, 13 Oct 2014 16:08:40 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 10141304043 for ; Mon, 13 Oct 2014 14:08:36 -0700 (PDT) X-ASG-Debug-ID: 1413234515-04cb6c7705392120001-S8gJnT Received: from mx5-phx2.redhat.com (mx5-phx2.redhat.com [209.132.183.37]) by cuda.sgi.com with ESMTP id d7vmzujQPjyC1Os7 (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Mon, 13 Oct 2014 14:08:35 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.37 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx5-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9DL8Y39023876; Mon, 13 Oct 2014 17:08:34 -0400 Date: Mon, 13 Oct 2014 17:08:34 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: "Frank Ch. Eigler" , lberk@redhat.com Cc: pcp developers Message-ID: <716991402.67309175.1413234514511.JavaMail.zimbra@redhat.com> In-Reply-To: <20141013114450.GA4170@redhat.com> References: <20141010190642.GD26999@redhat.com> <479818101.66478026.1413162710461.JavaMail.zimbra@redhat.com> <20141013114450.GA4170@redhat.com> Subject: Re: [pcp] another round of review for lberk/papi MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] another round of review for lberk/papi Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: another round of review for lberk/papi Thread-Index: kSCHhtVcYSIwi5tBLspFfXpCpDN+0A== X-Barracuda-Connect: mx5-phx2.redhat.com[209.132.183.37] X-Barracuda-Start-Time: 1413234515 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.03 X-Barracuda-Spam-Status: No, SCORE=0.03 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_SA_TO_FROM_DOMAIN_MATCH, THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10530 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... 0.01 BSF_SC0_SA_TO_FROM_DOMAIN_MATCH Sender Domain Matches Recipient Domain ----- Original Message ----- > > They exposed bugs in the authentication mechanism, exercise the install > > and remove mechanisms > > Only barely so (see below): and not the "remove" mechanisms (since there > is nothing to remove from an -L blank slate). > You've misunderstood - as in, Install and Remove (scripts), which of course is (shell) code, needing testing too. > > As discussed awhile back, I added these tests as just a start on > > testing, so that we had some basic initial coverage. So, please > > leave these tests as-is and build on this base with new tests. > > [...] > > Simply switching them from -L to the default -h would actually get a > bit of new coverage of the core code. What's the disadvantage? That could be done, your earlier mail advocated a more intrusive set of changes though. However, it'd make test 914 significantly slower (reducing its utility as a quick smoke test, & removing all coverage of the DSO mode functionality) since... If this conversion to -h is done, then the test will need to become more like 903 where PMDA Install/Remove is performed - tests cannot assume optional PMDAs are installed at the start, and need to clean 'em up at the end. Writing new tests still seems a good way to go here, so either leave these two as is, or extend 903 (which already has the install/remove logic) with some fetch/store additions like 914. cheers. -- Nathan From nscott@redhat.com Tue Oct 14 02:04:45 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 61FFE7F3F for ; Tue, 14 Oct 2014 02:04:45 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 4158C8F8049 for ; Tue, 14 Oct 2014 00:04:42 -0700 (PDT) X-ASG-Debug-ID: 1413270277-04cbb018ac38a800001-S8gJnT Received: from mx4-phx2.redhat.com (mx4-phx2.redhat.com [209.132.183.25]) by cuda.sgi.com with ESMTP id ihzzTYVqksrUlvle (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 14 Oct 2014 00:04:37 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.25 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx4-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s9E74arB026494 for ; Tue, 14 Oct 2014 03:04:37 -0400 Date: Tue, 14 Oct 2014 03:04:36 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: pcp developers Message-ID: <340618230.67549958.1413270276781.JavaMail.zimbra@redhat.com> In-Reply-To: <740525330.67539082.1413268939421.JavaMail.zimbra@redhat.com> Subject: Pushing the pcp-3.10.0 release back slightly MIME-Version: 1.0 X-ASG-Orig-Subj: Pushing the pcp-3.10.0 release back slightly Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: Pushing the pcp-3.10.0 release back slightly Thread-Index: 2il19jDFiUC3/smwrx7+gyAh9f2TXQ== X-Barracuda-Connect: mx4-phx2.redhat.com[209.132.183.25] X-Barracuda-Start-Time: 1413270277 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10548 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Hi all, We're scheduled for our usual monthly point release tomorrow but it seems we're not quite ready yet. We need to, at least: - finish iostat2pcp QA; - do further work around python3 pcp module stability; - do a pmdapapi/QA update, if Lukas believes its warranted; - sort out where we're heading with the web functionality, as best we can. Chatting to Ken, there's also some remaining QA issues (and that includes regressions around the web tree work) that he is observing too, which will need to be sorted out. Finally, many of the Red Hat crew will be travelling most of next week - so I'm planning to delay pcp-3.10.0 for two weeks while we sort through these outstanding issues, and aim for 1st November instead. Please let me know if this slip will cause issues for anyone. Thanks! cheers. -- Nathan From fche@redhat.com Tue Oct 14 14:30:54 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 5E6A37F3F for ; Tue, 14 Oct 2014 14:30:54 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 306098F8040 for ; Tue, 14 Oct 2014 12:30:51 -0700 (PDT) X-ASG-Debug-ID: 1413315047-04bdf062f909ab0001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 4a0UczA1Y2K25sJp (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 14 Oct 2014 12:30:47 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9EJUkcu024646 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Tue, 14 Oct 2014 15:30:47 -0400 Received: from fche.csb (vpn-232-103.phx2.redhat.com [10.3.232.103]) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9EJUk8a013206 for ; Tue, 14 Oct 2014 15:30:46 -0400 Received: by fche.csb (Postfix, from userid 2569) id E60A358127; Tue, 14 Oct 2014 15:30:45 -0400 (EDT) Date: Tue, 14 Oct 2014 15:30:45 -0400 From: "Frank Ch. Eigler" To: pcp developers Subject: RFC: Makepkgs improvement for RH platforms Message-ID: <20141014193045.GB18476@redhat.com> X-ASG-Orig-Subj: RFC: Makepkgs improvement for RH platforms Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.2i X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1413315047 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 >From pcpfans commit #b171d777de6d: Author: Frank Ch. Eigler Date: Tue Oct 14 15:24:43 2014 -0400 Makepkgs: on RH platforms, use rpm --eval ... for fuller ./configure The basic set of configure defaults in Makepkgs doesn't account for platform specifics, like where 64-bit libraries go. This can result in packages that install parts in directories that mismatch the platform rules, and can cause accidental conflicts later. So, to future-proof this part, on RH platforms we transcribe a portion of the %configure macro into Makepkgs, so e.g. --libdir and many other friends are set. It is likely that other RPM-based platforms could use a similar hack to make Makepkgs base-os-compatible. Tested on F19 x86-64 and RHEL5 i686. From lberk@redhat.com Tue Oct 14 14:59:12 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id A81AB7F3F for ; Tue, 14 Oct 2014 14:59:12 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 7A596304039 for ; Tue, 14 Oct 2014 12:59:09 -0700 (PDT) X-ASG-Debug-ID: 1413316748-04bdf062fa0b130001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id ACr2POufrdoHlFEx (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 14 Oct 2014 12:59:08 -0700 (PDT) X-Barracuda-Envelope-From: lberk@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9EJx76r006675 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Tue, 14 Oct 2014 15:59:08 -0400 Received: from toium (dhcp-10-15-16-117.yyz.redhat.com [10.15.16.117]) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9EJx7Cw005436 (version=TLSv1/SSLv3 cipher=AES128-GCM-SHA256 bits=128 verify=NO); Tue, 14 Oct 2014 15:59:07 -0400 From: Lukas Berk To: "Frank Ch. Eigler" Cc: pcp developers Subject: Re: another round of review for lberk/papi References: <20141010190642.GD26999@redhat.com> X-ASG-Orig-Subj: Re: another round of review for lberk/papi Date: Tue, 14 Oct 2014 15:59:06 -0400 In-Reply-To: <20141010190642.GD26999@redhat.com> (Frank Ch. Eigler's message of "Fri, 10 Oct 2014 15:06:42 -0400") Message-ID: <87ppdu7791.fsf@redhat.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1413316748 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Hey, Thanks for the review, comments in-line. "Frank Ch. Eigler" writes: [...] > At least one more correctness issue is still present in the code, by > inspection: the way the add_metric & remove_metric functions (and > subsidiaries) use the papi_info[].position number both as an index > (into values[]) and as the "user wishes this counter activated" flag. > This fails if the set-of-activated-counters is manipulated via pmStore > in a non-LIFO way. It's because the remove path largely preserves the > previous .position numbers even though it recreates the EventSet in > increasing-index order (thus invalidating those position#'s). Any > time after that the values[] will be mis-ordered. Ah good catch, thanks. > What the papi_info[] struct needs is an added simple on/off flag that > the .enable/.disable callback can set/clear, and a single common > papi-eventset-refresh function that shuts everything down, saves > counter values from former values[.position]'s, and *recalculates* > brand a new .position for all surviving events (in whatever order > they're being added to the new eventset). I've fixed this in commit 324ec32f62a8. I implemented your suggestion of a 'metric_enabled' variable in papi_info. This way, the 'position' variable can serve its single purpose, and we can first assign it once in the add_metric function, and then reassign as needed (when we're recreating the eventset) in remove_metric. I plan on writing a testcase for this as well. > A small memory-corruption bug with the new status_string[]: strdup() > it and PMDA_FETCH_DYNAMIC as the return code, or else make it a static > char[] and keep PMDA_FETCH_STATIC. (PMDA_FETCH_STATIC on a stack > local would ask libpcp_pmda to access memory from a stack frame that's > already been unwound.) Fixed in commit 7cf0569611c. > I've suggested nuking the hard-coded-papi-event parts of metrictab[], > but it's clear now why you can't quite do that yet: there is no other > source of pmDesc type data for the metrics. You'll need to intercept > the pmda dispatch->version.any.desc field with a callback function > that fills in the given pmDesc*, very much the same way as your > papi_text() function returns the helptext stuff. Then metrictab[] can > shrink to the bare minimum. Yes, you've mentioned this before and I still have it marked as TODO. With the previous two items fixed it's my on my plate now. > Some ideas regarding testing. I notice there have been several emails about this already, just going to sum up my thoughts here. > First of all, the existing test cases (qa/914 and qa/799) simply must > stop relying on the -L local context this way. As we discussed on IRC > last week, -L makes no sense at all for papi (since it's so stateful; > each "pmCLIENT -L ..." loads the pmda.so+papi.so to form a new blank > slate, does one thing, and immediately dies). So no such test will > ever see counter numbers, or results other than those related to > exactly one initialization-type operation -- plus any __pmNotifyErr > noise that would've gone to a pmcd/papi.log file. I didn't write qa/914 so I won't speak to that case. In the case of qa/799, Nathan pointed out that trying to add conflicting metrics should be tested. When I was trying to reproduce the condition, it found a bug in the way I handled that case. Even if fixed now, it needs a testcase imo and this is the goal of qa/799. Only reason it was commited was so I could continue to build pcp with Makepkgs. This is a situation where (as Frank has mentioned) state of the pmda and papi counters is required to have a meaningful test, so, aiui, using -L as currently in the testcase will not work. If the answer is a combination of dbpmda/pmcd and a helper PAPI test program, then so be it, and I'll be happy to work on an approach that way! As somebody new to PCP and to the inner workings of its testsuite, I intend to make testcases a priority when introducing new functionality, and will make every effort to do so. If I may humbly inquire, would it not make sense to test how the pmda reacts when "papi proper" (ie, not the test papi) returns, say, an PAPI_ECNFLCT error? This could even be in _addition_ to the test against the qa/src/*.c papi. I imagine this would only increase the level of test coverage the testsuite offers? [...] > Anyway, as we were saying on IRC, one plausible way of using -L is via > dbpmda, instead of a PMAPI client. That could work ... but OTOH might > be tricky to feed it a fixed dbpmda command script (like a few other > qa/NNN tests do) and get meaningful answers out, at least because the > available counters=metrics & runtime environment can so easily vary > between machines. A simple testcase driver is probably not up to the job. After looking over qa/617 as a quick example of using dbpmda (yes, I'm aware there are some additional intricacies when linking a test papi). I think that we could properly script a case where the test papi.c returns a PAPI_ECNFLCT error after a set number of metrics are added, I assume this is mainly in writing a proper qa/src/*.c file which throws PAPI_ECNFLCT at a 'synthetic' point (ie, adding TOT_CYC and TOT_INS metrics may never actually cause an ECNFLCT error, but our test papi could throw it). > If you agree a more complex driver seems needed, we could try something > using plain default "-h local:" rather than "-L", chooses a bunch of > counters, fetches some samples, waits awhile to generate load, fetches > more samples (sanity checking that they are growing at reasonable > rates), turning some off / on in various orderings, fetching them, > checking that the survivors are still growing at reasonable rates. > It might be a job for a python script rather than bash. I think an approach like this could make sense, and looking over the further emails, adding *additional* tests that do this, while keeping the older ones in place seems agreeable. In this case I think the hard(er?) part is reliably finding a case that could cause an ECNFLCT, aiui, that changes based on the hardware. My initial thought is to find a way to iterate over the available metrics, until we reach a limit, and then adding another to set us 'over'. I'm also aware I need to pay attention to installing and removing the pmda as part of the test. In regards to the "sanity checking that they are growing at reasonable rates". Considering that this will be run on different hardware, potentially with vastly different counter performance and capabilities, do you have any thoughts on how we could estimate what a 'reasonable rate' would be? My first thought is to perhaps compare rates/ratios of different, specific, metrics. However, I'm not sure how much those vary per architecture. Thoughts? > (When writing tests for pmwebd, I found it helpful to compile the pmda > with the gcc test-coverage options. It made it possible to see that > all the working parts of the code were in fact touched by the > testsuite's execution. You might find it useful here too, considering > the tricky state manipulation required.) Definitely something I'll look at, thanks. Cheers, Lukas From minnus@buffalo.edu Tue Oct 14 15:09:08 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 2B3DB7F3F for ; Tue, 14 Oct 2014 15:09:08 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 1909B30404E for ; Tue, 14 Oct 2014 13:09:07 -0700 (PDT) X-ASG-Debug-ID: 1413317342-04cbb01fc00b310001-S8gJnT Received: from mtareserve1.acsu.buffalo.edu (mtareserve6.acsu.buffalo.edu [128.205.6.4]) by cuda.sgi.com with ESMTP id LLHFg1bUx7dooqAn for ; Tue, 14 Oct 2014 13:09:02 -0700 (PDT) X-Barracuda-Envelope-From: minnus@buffalo.edu X-Barracuda-Apparent-Source-IP: 128.205.6.4 Received: from localmailA.acsu.buffalo.edu (localmaila.acsu.buffalo.edu [128.205.5.196]) by mtareserve1.acsu.buffalo.edu (Postfix) with ESMTP id 0371961C2; Tue, 14 Oct 2014 16:09:01 -0400 (EDT) Received: from localmailA.acsu.buffalo.edu (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id E01CCE99B; Tue, 14 Oct 2014 16:09:01 -0400 (EDT) Received: from localmailA.acsu.buffalo.edu (localhost [127.0.0.1]) by localmailA.acsu.buffalo.edu (Postfix) with ESMTP id B7950E98D; Tue, 14 Oct 2014 16:08:59 -0400 (EDT) Received: from smtp.buffalo.edu (smtp2.acsu.buffalo.edu [128.205.5.254]) by localmailA.acsu.buffalo.edu (Prefixe) with ESMTP id 98BC2E98B; Tue, 14 Oct 2014 16:08:59 -0400 (EDT) Received: from prince.ccr.buffalo.edu (prince.ccr.buffalo.edu [128.205.40.45]) (Authenticated sender: minnus@buffalo.edu) by smtp.buffalo.edu (Postfix) with ESMTPSA id 88A6A2554; Tue, 14 Oct 2014 16:08:59 -0400 (EDT) Message-ID: <543D8325.4040707@buffalo.edu> Date: Tue, 14 Oct 2014 16:10:13 -0400 From: Martins Innus User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: Nathan Scott CC: pcp@oss.sgi.com Subject: Re: [pcp] hotproc rfc References: <536D28B4.6010504@buffalo.edu> <1139662762.4765310.1399862104653.JavaMail.zimbra@redhat.com> <54230FAF.2080201@buffalo.edu> <905561536.62723741.1412657864635.JavaMail.zimbra@redhat.com> <5435926F.6030004@buffalo.edu> <265957919.64613233.1412813730718.JavaMail.zimbra@redhat.com> X-ASG-Orig-Subj: Re: [pcp] hotproc rfc In-Reply-To: <265957919.64613233.1412813730718.JavaMail.zimbra@redhat.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-PM-EL-Spam-Prob: : 8% X-Barracuda-Connect: mtareserve6.acsu.buffalo.edu[128.205.6.4] X-Barracuda-Start-Time: 1413317342 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10567 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Nathan, On 10/8/14 8:15 PM, Nathan Scott wrote: > Spot on, yes; keeping the existing PMIDs for those proc metrics that > become dynamic and making the hotproc variants just the same (but with > non-conflicting cluster IDs of course). > >> It seems like the first step would be to convert the existing pmda to >> handle dynamic metrics for the appropriate clusters that already exist >> before we worry about hotproc? > Yep. I believe that will work nicely - of course there may be issues > I've not anticipated (lemme know early on, if so), but I'm pretty sure > that will work out well. > I've made some progress on this but wanted to clarify a couple things. This seems a little more involved than I initially thought, unless I'm missing some points. I think we want the metrics to be named as I had in my current implementation: hotproc.psinfo.pid and proc.psinfo.pid ...etc.... but we are still going to have proc metrics that don't map to hotproc variants like: proc.runq.* proc.control.* (Although I suppose a case could be made that someone might want the aggregated runq information for hotproc processes) And certainly there are hotproc metrics that have no corresponding proc metrics, and the control metrics are different between the two. But in general, how would root_proc look for a mix of dynamic and static metrics under a single tree? Like this where we would duplicate the dynamic elements for both subtrees? **************** root { cgroup proc hotproc } proc { nprocs PROC:8:99 psinfo PROC:*:* memory PROC:*:* runq id PROC:*:* io PROC:*:* schedstat PROC:*:* fd PROC:*:* control } hotproc { nprocs PROC:52:99 psinfo PROC:*:* memory PROC:*:* id PROC:*:* io PROC:*:* schedstat PROC:*:* fd PROC:*:* control total predicate } ****************** If this is the case, how do the calls to "pmdaDynamicPMNS" work? We would need at least 2, correct:? pmdaDynamicPMNS("proc",..... pmdaDynamicPMNS("hotproc",..... Or would we need one for each proc.foo entity since we are mixing dynamic and static items in a tree? My confusion mostly stems from the fact that I haven't yet figured out if there is a "right way" to do this mixing. i.e. do I just skip over any static metrics in the "refresh_*" calls? Or should everything under proc be dynamic? Thus, I would just do: *********************** root { cgroup proc PROC:*:* hotproc PROC:*:* } *********************** But then, again, I need some way of handling static metrics that don't overlap. The second observation is that namespace grouping are not 1:1 wrt cluster groupings. For example, proc.psinfo contains metrics from 4 different clusters. From my reading of existing code, I don't think this will be a problem, but just wanted to check. Finally, I looked through various pmdas and some (mmv and sample) handle dynamic metrics in a more direct way, without the infrastructure provided by pmdaDynamicPMNS adding to my confusion on the right way to do this. Thanks for any prodding in the right direction. Martins From fche@redhat.com Tue Oct 14 15:22:26 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 21BBC7F3F for ; Tue, 14 Oct 2014 15:22:26 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 974EBAC011 for ; Tue, 14 Oct 2014 13:22:22 -0700 (PDT) X-ASG-Debug-ID: 1413318140-04cb6c6faf0cdd0001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id E22aLYIAhgeCn6aX (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 14 Oct 2014 13:22:21 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9EKMKI5010354 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Tue, 14 Oct 2014 16:22:20 -0400 Received: from fche.csb (vpn-232-103.phx2.redhat.com [10.3.232.103]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9EKMJ9u013406; Tue, 14 Oct 2014 16:22:20 -0400 Received: by fche.csb (Postfix, from userid 2569) id 32A2B58127; Tue, 14 Oct 2014 16:22:19 -0400 (EDT) To: Lukas Berk Cc: pcp@oss.sgi.com Subject: Re: another round of review for lberk/papi References: <20141010190642.GD26999@redhat.com> <87ppdu7791.fsf@redhat.com> X-ASG-Orig-Subj: Re: another round of review for lberk/papi From: fche@redhat.com (Frank Ch. Eigler) Date: Tue, 14 Oct 2014 16:22:19 -0400 In-Reply-To: <87ppdu7791.fsf@redhat.com> (Lukas Berk's message of "Tue, 14 Oct 2014 15:59:06 -0400") Message-ID: User-Agent: Gnus/5.1008 (Gnus v5.10.8) Emacs/21.4 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1413318141 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Hi, Lukas - > [...] This is a situation where (as Frank has mentioned) state of > the pmda and papi counters is required to have a meaningful test, > so, aiui, using -L as currently in the testcase will not work. If > the answer is a combination of dbpmda/pmcd and a helper PAPI test > program, then so be it, and I'll be happy to work on an approach > that way! A quicker approach to that testing just s/-L...// in the test case; that way your qa script gets to test your code as an end-user would. > As somebody new to PCP and to the inner workings of its testsuite, I > intend to make testcases a priority when introducing new > functionality, and will make every effort to do so. If I may humbly > inquire, would it not make sense to test how the pmda reacts when > "papi proper" (ie, not the test papi) returns, say, an PAPI_ECNFLCT > error? [...] Certainly. IMHO, testing against the real PAPI library is worth its weight in complexity. Testing against a synthetic-PAPI would be a necessary evil at best, if we are unable to evoke desired corner cases on demand from the real thing. (For one thing, testing primarily against a synthetic-PAPI could easily generate a false sense of confidence. For another thing, a nontrivial synthetic-PAPI would itself be a piece of software in need of testing...) > I think an approach like this could make sense, and looking over the > further emails, adding *additional* tests that do this, while > keeping the older ones in place seems agreeable. Yes. I did not mean to come across as suggesting that the current test cases should be dropped - just that those that use -L in a futile manner switch to -hlocal:. > In this case I think the hard(er?) part is reliably finding a case > that could cause an ECNFLCT, aiui, that changes based on the > hardware. My initial thought is to find a way to iterate over the > available metrics, until we reach a limit, and then adding another > to set us 'over'. Yup. Now that the code enables multiplexing by default, it might take a few iterations to get to that saturation point, but one will be reached at -some- point. > I'm also aware I need to pay attention to installing and removing > the pmda as part of the test. That part's taken care of as qa/903. > In regards to the "sanity checking that they are growing at > reasonable rates". Considering that this will be run on different > hardware, potentially with vastly different counter performance and > capabilities, do you have any thoughts on how we could estimate what > a 'reasonable rate' would be? My first thought is to perhaps > compare rates/ratios of different, specific, metrics. However, I'm > not sure how much those vary per architecture. Thoughts? Yeah, it's tricky. I was indeed thinking of selecting a few generally available metrics like TOT_CYC that counts regularly and fast, and one of the L?_?CM (cache miss) counters that should increment on an ongoing basis even on an idlish system, but relatively slowly. The test code might need to do some searching to pick one, and estimate its growth rate. (Then start enabling/disabling, flipping around, etc.) - FChE From nscott@redhat.com Tue Oct 14 18:03:58 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id A55577F47 for ; Tue, 14 Oct 2014 18:03:58 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 7598C304053 for ; Tue, 14 Oct 2014 16:03:57 -0700 (PDT) X-ASG-Debug-ID: 1413327832-04bdf062f714330001-S8gJnT Received: from mx5-phx2.redhat.com (mx5-phx2.redhat.com [209.132.183.37]) by cuda.sgi.com with ESMTP id WFNfNLhnDU768UGg (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 14 Oct 2014 16:03:52 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.37 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx5-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9EN3n35006326; Tue, 14 Oct 2014 19:03:49 -0400 Date: Tue, 14 Oct 2014 19:03:49 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Martins Innus Cc: pcp@oss.sgi.com Message-ID: <905157356.68569240.1413327829218.JavaMail.zimbra@redhat.com> In-Reply-To: <543D8325.4040707@buffalo.edu> References: <536D28B4.6010504@buffalo.edu> <1139662762.4765310.1399862104653.JavaMail.zimbra@redhat.com> <54230FAF.2080201@buffalo.edu> <905561536.62723741.1412657864635.JavaMail.zimbra@redhat.com> <5435926F.6030004@buffalo.edu> <265957919.64613233.1412813730718.JavaMail.zimbra@redhat.com> <543D8325.4040707@buffalo.edu> Subject: Re: [pcp] hotproc rfc MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] hotproc rfc Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: hotproc rfc Thread-Index: QVGrPdH7BHqVTjmaEb5gM6kJpjoRWw== X-Barracuda-Connect: mx5-phx2.redhat.com[209.132.183.37] X-Barracuda-Start-Time: 1413327832 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10574 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Hi Martins, ----- Original Message ----- > Nathan, > > On 10/8/14 8:15 PM, Nathan Scott wrote: > > Spot on, yes; keeping the existing PMIDs for those proc metrics that > > become dynamic and making the hotproc variants just the same (but with > > non-conflicting cluster IDs of course). > > > >> It seems like the first step would be to convert the existing pmda to > >> handle dynamic metrics for the appropriate clusters that already exist > >> before we worry about hotproc? > > Yep. I believe that will work nicely - of course there may be issues > > I've not anticipated (lemme know early on, if so), but I'm pretty sure > > that will work out well. > > > I've made some progress on this but wanted to clarify a couple things. > This seems a little more involved than I initially thought, unless I'm > missing some points. I think we want the metrics to be named as I had > in my current implementation: > > hotproc.psinfo.pid > and > proc.psinfo.pid > > ...etc.... > > but we are still going to have proc metrics that don't map to hotproc > variants like: > > proc.runq.* > proc.control.* Yep. > And certainly there are hotproc metrics that have no corresponding proc > metrics, and the control metrics are different between the two. Right. > But in general, how would root_proc look for a mix of dynamic and static > metrics under a single tree? Like this where we would duplicate the > dynamic elements for both subtrees? > > **************** > [...] > **************** > Yes, that's how I'd imagined the namespace file would look. > > If this is the case, how do the calls to "pmdaDynamicPMNS" work? We > would need at least 2, correct:? Correct, with the way the pmdaDynamicPMNS call works, I think it'd have to be one per cluster in this case. > Or would we need one for each proc.foo entity since we are mixing > dynamic and static items in a tree? Yes, one per [hot]proc.foo entity will be needed I think. > pmdaDynamicPMNS("proc",..... > pmdaDynamicPMNS("hotproc",..... (so one call per top level subtree) pmdaDynamicPMNS("proc.psinfo",..... pmdaDynamicPMNS("proc.memory",..... etc. The functions in the "....." above would very likely be shared, and the namespace "refresh" concept is far, far simpler than the cgroups case (which is a *very* dynamic namespace, potentially changing even between fetches) - our case for proc/hotproc is simpler - an example closer to this case here would be src/pmdas/linux/interrupts.c. > My confusion mostly stems from the fact that I haven't yet figured out > if there is a "right way" to do this mixing. i.e. do I just skip over > any static metrics in the "refresh_*" calls? > > Or should everything under proc be dynamic? Thus, I would just do: > *********************** > root { > cgroup > proc PROC:*:* > hotproc PROC:*:* > } > *********************** As per discussion earlier - nope, not this latter approach, cos... > > But then, again, I need some way of handling static metrics that don't > overlap. *nod* - the above isn't what I was imagining - there's lots of static proc metrics we just want to leave as is, our aim here is to share the metrics which are being duplicated for hotproc (and not only sharing the names, but also the descriptors and help text). > > The second observation is that namespace grouping are not 1:1 wrt > cluster groupings. For example, proc.psinfo contains metrics from 4 > different clusters. From my reading of existing code, I don't think > this will be a problem, but just wanted to check. Agreed - I don't think this will be a problem. I suspect you'll need a series of calls to pmdaDynamicPMNS(), for each [hot]proc.foo subtree and each with just a single cluster in the "set" (2nd parameter). > Finally, I looked through various pmdas and some (mmv and sample) handle > dynamic metrics in a more direct way, without the infrastructure > provided by pmdaDynamicPMNS adding to my confusion on the right way to > do this. > > Thanks for any prodding in the right direction. The interrupts example above might give further ideas around tackling it using pmdaDynamicPMNS. But, if you're finding that to get in the way by all means go for an MMV-style or pmdasample-style approach without using that interface. cheers. -- Nathan From nscott@redhat.com Tue Oct 14 18:25:07 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id EA2D87F47 for ; Tue, 14 Oct 2014 18:25:06 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 661A9AC00F for ; Tue, 14 Oct 2014 16:25:03 -0700 (PDT) X-ASG-Debug-ID: 1413329100-04cbb01fbf13af0001-S8gJnT Received: from mx6-phx2.redhat.com (mx6-phx2.redhat.com [209.132.183.39]) by cuda.sgi.com with ESMTP id lOEAIbobaxBIvuZP (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 14 Oct 2014 16:25:01 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.39 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx6-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9ENP0aO021481; Tue, 14 Oct 2014 19:25:00 -0400 Date: Tue, 14 Oct 2014 19:25:00 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Lukas Berk Cc: "Frank Ch. Eigler" , pcp developers Message-ID: <159065747.68574462.1413329100557.JavaMail.zimbra@redhat.com> In-Reply-To: <87ppdu7791.fsf@redhat.com> References: <20141010190642.GD26999@redhat.com> <87ppdu7791.fsf@redhat.com> Subject: Re: [pcp] another round of review for lberk/papi MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] another round of review for lberk/papi Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: another round of review for lberk/papi Thread-Index: jwVf8Qe/O1gsg37dlwTG/VDTsympXw== X-Barracuda-Connect: mx6-phx2.redhat.com[209.132.183.39] X-Barracuda-Start-Time: 1413329101 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.03 X-Barracuda-Spam-Status: No, SCORE=0.03 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_SA_TO_FROM_DOMAIN_MATCH, THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10575 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... 0.01 BSF_SC0_SA_TO_FROM_DOMAIN_MATCH Sender Domain Matches Recipient Domain ----- Original Message ----- > [...] > Even if fixed now, it needs a testcase imo *nod* > and this is the goal of qa/799. Only reason it was commited was so I > could continue to build pcp with Makepkgs. Ah, this is almost certainly because we drive parts of the build (esp. the source tarball generation) from git now instead of the way we used to (which was via explicitly listing all source files via makefiles). Anyway, it should be enough to "git add" your new test, for Makepkgs to be able to find it - and not go all the way through with git commit until you're good and ready. Not a problem that the test is committed though, of course. > This is a situation where (as > Frank has mentioned) state of the pmda and papi counters is required to > have a meaningful test, so, aiui, using -L as currently in the testcase > will not work. If the answer is a combination of dbpmda/pmcd and a > helper PAPI test program, then so be it, and I'll be happy to work on an > approach that way! Sounds good to me. > As somebody new to PCP and to the inner workings of its testsuite, I > intend to make testcases a priority when introducing new functionality, > and will make every effort to do so. You will fly high and far in PCP-land with that kind of attitude. :) > not make sense to test how the pmda reacts when "papi proper" (ie, not the > test papi) returns, say, an PAPI_ECNFLCT error? This could even be in > _addition_ to the test against the qa/src/*.c papi. I imagine this would > only increase the level of test coverage the testsuite offers? *nod* > I think an approach like this could make sense, and looking over the > further emails, adding *additional* tests that do this, while keeping > the older ones in place seems agreeable. In this case I think the > hard(er?) part is reliably finding a case that could cause an ECNFLCT, > aiui, that changes based on the hardware. My initial thought is to find > a way to iterate over the available metrics, until we reach a limit, and > then adding another to set us 'over'. If that does the trick, awesome. It might also pay dividends to wander through the PAPI code a bit, to see if there are situations where it'll return this ECNFLCT code in a fashion that is deterministic independent of underlying hardware (and/or chat to Will too - maybe he knows ways to trigger that off the top of his head). > In regards to the "sanity checking that they are growing at reasonable > rates". Considering that this will be run on different hardware, > potentially with vastly different counter performance and capabilities, > do you have any thoughts on how we could estimate what a 'reasonable > rate' would be? My first thought is to perhaps compare rates/ratios of > different, specific, metrics. However, I'm not sure how much those vary > per architecture. Thoughts? My only thought there is to keep in mind we don't have to test PAPI itself here - we can and should stop at the point that we have confidence that the pmdapapi metrics provide an accurate reflection of the values returned by PAPI for individual counters. And they don't have to match exactly, of course, sampling time differences will introduce some value fluctuations - the qa/common.check code provides a _within_tolerance() function that may prove handy. cheers. -- Nathan From nscott@redhat.com Tue Oct 14 18:45:27 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 3F3087F47 for ; Tue, 14 Oct 2014 18:45:27 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id BE9B3AC00F for ; Tue, 14 Oct 2014 16:45:26 -0700 (PDT) X-ASG-Debug-ID: 1413330324-04cbb01fbe14560001-S8gJnT Received: from mx5-phx2.redhat.com (mx5-phx2.redhat.com [209.132.183.37]) by cuda.sgi.com with ESMTP id g5hgiCIDkTOGJdsN (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 14 Oct 2014 16:45:25 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.37 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx5-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9ENjOfv012796; Tue, 14 Oct 2014 19:45:24 -0400 Date: Tue, 14 Oct 2014 19:45:24 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: "Frank Ch. Eigler" Cc: pcp developers Message-ID: <1075762793.68582102.1413330324196.JavaMail.zimbra@redhat.com> In-Reply-To: <20141014193045.GB18476@redhat.com> References: <20141014193045.GB18476@redhat.com> Subject: Re: [pcp] RFC: Makepkgs improvement for RH platforms MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] RFC: Makepkgs improvement for RH platforms Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: Makepkgs improvement for RH platforms Thread-Index: MWPQwgCPGB+/nwIwHjm/U2Dpw9GGNg== X-Barracuda-Connect: mx5-phx2.redhat.com[209.132.183.37] X-Barracuda-Start-Time: 1413330325 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.03 X-Barracuda-Spam-Status: No, SCORE=0.03 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_SA_TO_FROM_DOMAIN_MATCH, THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10575 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.01 BSF_SC0_SA_TO_FROM_DOMAIN_MATCH Sender Domain Matches Recipient Domain ----- Original Message ----- > > From pcpfans commit #b171d777de6d: > > Author: Frank Ch. Eigler > Date: Tue Oct 14 15:24:43 2014 -0400 > > Makepkgs: on RH platforms, use rpm --eval ... for fuller ./configure Looks good to me and resolves a long-standing issue I've had to workaround - thanks! I don't follow the --program-prefix=%{?_program_prefix} part at the start of the rpm invocation though - how does that bit work? Does that assume a _program_prefix can be passed into rpm, which (I think?) cannot be, in our use here? (and, do we use _program_prefix anywhere? what's its relation to --prefix?). Its very likely I'm missing some subtlety there, sorry. Later on in Makepkgs, we drive the setting of "nonrpm" based on presence of /usr/bin/rpmbuild (IOW, if that's found, and not explicitly asked to do another type of packaging, we *will* choose rpm packaging)... it'd be good to use the rpm-based settings always in that situation and not only if its a Red Hat system (I think this is what you mean by the "and other?" comment in the code?). The script would need some refactoring to deal properly with its command line arguments - so I wouldn't worry about it for now, but maybe we can revisit that later as a more general improvement. cheers. -- Nathan From fche@redhat.com Tue Oct 14 19:52:59 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id F0AFE7F47 for ; Tue, 14 Oct 2014 19:52:58 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 7F8BDAC003 for ; Tue, 14 Oct 2014 17:52:55 -0700 (PDT) X-ASG-Debug-ID: 1413334370-04cbb01fbe16c70001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id RwisGVm6JzbL9AEG (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 14 Oct 2014 17:52:51 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9F0qo8P006519 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Tue, 14 Oct 2014 20:52:50 -0400 Received: from fche.csb (vpn-232-103.phx2.redhat.com [10.3.232.103]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9F0qnhA026670; Tue, 14 Oct 2014 20:52:50 -0400 Received: by fche.csb (Postfix, from userid 2569) id 4B62D58127; Tue, 14 Oct 2014 20:52:49 -0400 (EDT) Date: Tue, 14 Oct 2014 20:52:49 -0400 From: "Frank Ch. Eigler" To: Nathan Scott Cc: pcp developers Subject: Re: [pcp] RFC: Makepkgs improvement for RH platforms Message-ID: <20141015005249.GD18476@redhat.com> X-ASG-Orig-Subj: Re: [pcp] RFC: Makepkgs improvement for RH platforms References: <20141014193045.GB18476@redhat.com> <1075762793.68582102.1413330324196.JavaMail.zimbra@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1075762793.68582102.1413330324196.JavaMail.zimbra@redhat.com> User-Agent: Mutt/1.4.2.2i X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1413334371 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Hi, Nathan - > > Makepkgs: on RH platforms, use rpm --eval ... for fuller ./configure > > Looks good to me and resolves a long-standing issue I've had to workaround > - thanks! NP! > I don't follow the --program-prefix=%{?_program_prefix} part at the > start of the rpm invocation though - how does that bit work? [...] That is for an auto*tool facility whereby binaries can be automagically prefixed by a string, so as to avoid conflict with system binaries. It's used in some GNU toolchain programs, but not for PCP, so that part's not really needed here. (I put it in there for completness only.) > Later on in Makepkgs, we drive the setting of "nonrpm" based on > presence of /usr/bin/rpmbuild [...] it'd be good to use the > rpm-based settings always in that situation and not only if its a > Red Hat system (I think this is what you mean by the "and other?" > comment in the code?). [...] Indeed, that's just require access-to/testing-on other linux distros to confirm their favorite flavors of rpm --eval. - FChE From nscott@redhat.com Wed Oct 15 01:45:37 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 70F147F3F for ; Wed, 15 Oct 2014 01:45:37 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 507AC304032 for ; Tue, 14 Oct 2014 23:45:34 -0700 (PDT) X-ASG-Debug-ID: 1413355532-04bdf062f923310001-S8gJnT Received: from mx6-phx2.redhat.com (mx6-phx2.redhat.com [209.132.183.39]) by cuda.sgi.com with ESMTP id cDaoWSwDS1pqGnsv (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 14 Oct 2014 23:45:32 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.39 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx6-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9F6jWLE032349 for ; Wed, 15 Oct 2014 02:45:32 -0400 Date: Wed, 15 Oct 2014 02:45:32 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: pcp@oss.sgi.com Message-ID: <661090896.68665525.1413355532106.JavaMail.zimbra@redhat.com> Subject: pcp updates: fche merge, qa MIME-Version: 1.0 X-ASG-Orig-Subj: pcp updates: fche merge, qa Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: pcp updates: fche merge, qa Thread-Index: pDTMYNx7KtTz0xcO0+EhJ9OOUYZobQ== X-Barracuda-Connect: mx6-phx2.redhat.com[209.132.183.39] X-Barracuda-Start-Time: 1413355532 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10586 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Changes committed to git://git.pcp.io/pcp.git dev Nathan Scott (2): qa: allow alternate python versions to be used for testing qa: add qa/975 to perform local iostat vs iostat2pcp checking Frank Ch. Eigler (1): Makepkgs: on RH platforms, use rpm --eval ... for fuller ./configure Makepkgs | 9 ++ qa/553 | 21 ++---- qa/702 | 4 - qa/707 | 4 - qa/708 | 10 --- qa/709 | 10 +-- qa/710 | 10 --- qa/717 | 8 -- qa/718 | 10 --- qa/722 | 10 +-- qa/729 | 12 +-- qa/737 | 11 --- qa/739 | 24 +++---- qa/741 | 8 +- qa/742 | 1 qa/743 | 1 qa/829 | 6 - qa/842 | 19 ++--- qa/843 | 5 - qa/975 | 149 ++++++++++++++++++++++++++++++++++++++++++++++ qa/975.out | 7 ++ qa/979 | 4 - qa/980 | 4 - qa/991 | 1 qa/common.python | 3 src/iostat2pcp/iostat2pcp | 8 +- 26 files changed, 243 insertions(+), 116 deletions(-) From karthikeyan.ps@zohocorp.com Wed Oct 15 04:43:33 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 822D17F3F for ; Wed, 15 Oct 2014 04:43:33 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 70DC7304032 for ; Wed, 15 Oct 2014 02:43:33 -0700 (PDT) X-ASG-Debug-ID: 1413366207-04cbb01fbe258b0001-S8gJnT Received: from sender1.zohomail.com (sender1.zohomail.com [74.201.84.154]) by cuda.sgi.com with ESMTP id iU4dpO9ufz2LzLEJ (version=TLSv1 cipher=RC4-MD5 bits=128 verify=NO) for ; Wed, 15 Oct 2014 02:43:28 -0700 (PDT) X-Barracuda-Envelope-From: karthikeyan.ps@zohocorp.com X-Barracuda-Apparent-Source-IP: 74.201.84.154 Received: from mail.zoho.com by mx.zohomail.com with SMTP id 141336618611993.05372730633985; Wed, 15 Oct 2014 02:43:06 -0700 (PDT) Date: Wed, 15 Oct 2014 15:13:06 +0530 From: "karthikeyan.ps" To: Message-ID: <149132de64a.fc32bc2f31382.7164760145933637595@zohocorp.com> Subject: reg install pcp in freebsd MIME-Version: 1.0 X-ASG-Orig-Subj: reg install pcp in freebsd Content-Type: multipart/alternative; boundary="----=_Part_42279_973266567.1413366186064" X-Priority: MEDIUM User-Agent: Zoho Mail X-Mailer: Zoho Mail X-Barracuda-Connect: sender1.zohomail.com[74.201.84.154] X-Barracuda-Start-Time: 1413366208 X-Barracuda-Encrypted: RC4-MD5 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=HTML_MESSAGE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10590 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 HTML_MESSAGE BODY: HTML included in message ------=_Part_42279_973266567.1413366186064 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit hi, I am karthikeyan i have planned to install pcp in freebsd 9.2. I tried from source but unable to install pcp. If you don't mind can you suggest procedure to install pcp in freebsd. I already tried all the given procedure in your sides but no use. Thanks & Regards Karthikeyan Saravanan ------=_Part_42279_973266567.1413366186064 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: 7bit
hi,

I am karthikeyan i have planned to install pcp in freebsd 9.2. I tried from source but unable to install pcp. If you don't mind can you suggest procedure to install pcp in freebsd. I already tried all the given procedure in your sides but no use. 

Thanks & Regards
Karthikeyan Saravanan

------=_Part_42279_973266567.1413366186064-- From nscott@redhat.com Wed Oct 15 17:35:50 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 1CC307F55 for ; Wed, 15 Oct 2014 17:35:50 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id F07BB8F8033 for ; Wed, 15 Oct 2014 15:35:46 -0700 (PDT) X-ASG-Debug-ID: 1413412545-04bdf038d00bb70001-S8gJnT Received: from mx3-phx2.redhat.com (mx3-phx2.redhat.com [209.132.183.24]) by cuda.sgi.com with ESMTP id CsGDRRAS1tGNmJ84 (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 15 Oct 2014 15:35:45 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.24 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx3-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s9FMZiL7016519; Wed, 15 Oct 2014 18:35:45 -0400 Date: Wed, 15 Oct 2014 18:35:44 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: "karthikeyan.ps" Cc: pcp@oss.sgi.com Message-ID: <368290425.69451807.1413412544884.JavaMail.zimbra@redhat.com> In-Reply-To: <149132de64a.fc32bc2f31382.7164760145933637595@zohocorp.com> References: <149132de64a.fc32bc2f31382.7164760145933637595@zohocorp.com> Subject: Re: [pcp] reg install pcp in freebsd MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] reg install pcp in freebsd Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: reg install pcp in freebsd Thread-Index: Rx9FseiVtD8k1rObuG4cRlCPwFiCYw== X-Barracuda-Connect: mx3-phx2.redhat.com[209.132.183.24] X-Barracuda-Start-Time: 1413412545 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10611 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header Hi Karthikeyan, ----- Original Message ----- > hi, > > I am karthikeyan i have planned to install pcp in freebsd 9.2. I tried from > source but unable to install pcp. If you don't mind can you suggest > procedure to install pcp in freebsd. I already tried all the given procedure > in your sides but no use. When you tried to build the source, what was the failure you saw? I recently built PCP on FreeBSD 9.1, so we should be able to resolve the issue you saw fairly quickly I expect. cheers. -- Nathan From kenj@internode.on.net Thu Oct 16 17:16:21 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id AF7AF7F3F for ; Thu, 16 Oct 2014 17:16:21 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 4BFC8AC003 for ; Thu, 16 Oct 2014 15:16:20 -0700 (PDT) X-ASG-Debug-ID: 1413497775-04bdf038cf42490001-S8gJnT Received: from ipmail07.adl2.internode.on.net (ipmail07.adl2.internode.on.net [150.101.137.131]) by cuda.sgi.com with ESMTP id XjMVZO9SZtu888Gw for ; Thu, 16 Oct 2014 15:16:16 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.131 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApQBAIBCQFR20YEr/2dsb2JhbAANToNhzHSHVQKBKwGFAAEBBDhAEQsYCRYPCQMCAQIBRQYBDAgBAb5HlgcBAQEBBgEBAQEBHZBUhEsBBJZFiEI8kDKICViCSgEBAQ Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail07.adl2.internode.on.net with ESMTP; 17 Oct 2014 08:46:15 +1030 Message-ID: <544043CF.8060309@internode.on.net> Date: Fri, 17 Oct 2014 09:16:47 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: "karthikeyan.ps" , pcp@oss.sgi.com Subject: Re: [pcp] reg install pcp in freebsd References: <149132de64a.fc32bc2f31382.7164760145933637595@zohocorp.com> X-ASG-Orig-Subj: Re: [pcp] reg install pcp in freebsd In-Reply-To: <149132de64a.fc32bc2f31382.7164760145933637595@zohocorp.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail07.adl2.internode.on.net[150.101.137.131] X-Barracuda-Start-Time: 1413497775 X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10651 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- On 15/10/14 20:43, karthikeyan.ps wrote: > hi, > > I am karthikeyan i have planned to install pcp in freebsd 9.2. I tried > from source but unable to install pcp. If you don't mind can you suggest > procedure to install pcp in freebsd. I already tried all the given > procedure in your sides but no use. Karthikeyan, I (well, strictly speaking cron, not I) regularly build and install PCP on lots of platforms, including some FreeBSD ones. We don't have real *BSD packages (no one on the project has that skill set, volunteers would be most welcome), so we're using a tarball and some helper scripts. This is the recipe (extracted from qa/admin/pcp-daily) $ git pull $ ./Makepkgs This should produce a tarball in pcp-/build/tar $ . ./VERSION.pcp $ buildversion=$PACKAGE_MAJOR.$PACKAGE_MINOR.$PACKAGE_REVISION $ here=`pwd` $ tarball=$here/pcp-$buildversion/build/tar/pcp-[0-9]*[0-9].tar.gz $ cd pcp-$buildversion/build/tar $ sudo ./preinstall $ cd / $ sudo tar -zxpf $tarball $ cd $here $ cd pcp-$buildversion/build/tar $ sudo ./postinstall That's the end of the installation steps. Now start pmcd. $ . /etc/pcp.env $ sudo $PCP_RC_DIR/pcp start From nscott@redhat.com Thu Oct 16 18:10:59 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id C19807F3F for ; Thu, 16 Oct 2014 18:10:59 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 5F1BCAC001 for ; Thu, 16 Oct 2014 16:10:56 -0700 (PDT) X-ASG-Debug-ID: 1413501054-04bdf038d244850001-S8gJnT Received: from mx6-phx2.redhat.com (mx6-phx2.redhat.com [209.132.183.39]) by cuda.sgi.com with ESMTP id mW6iqPMPACQCI8DI (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 16 Oct 2014 16:10:55 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.39 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx6-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9GNAp7J028164; Thu, 16 Oct 2014 19:10:51 -0400 Date: Thu, 16 Oct 2014 19:10:50 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Ken McDonell Cc: pcp developers Message-ID: <1013731158.70294119.1413501050948.JavaMail.zimbra@redhat.com> In-Reply-To: <1298580313.70293290.1413500705032.JavaMail.zimbra@redhat.com> Subject: Test qa/645 failing MIME-Version: 1.0 X-ASG-Orig-Subj: Test qa/645 failing Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: Test qa/645 failing Thread-Index: lTKJQ55TvwoVIvsZPoy0T6wUTKKt4Q== X-Barracuda-Connect: mx6-phx2.redhat.com[209.132.183.39] X-Barracuda-Start-Time: 1413501055 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-BRTS-Evidence: bigpondsitehelp.com X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10652 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Hi Ken, I'm seeing a failure in test 645 on a newly added QA machine. Its on a separate network which resolves differently to other machines here, and thats part of the problem ... Telstra have decided to resolve more dodgey hostnames for us. :) PING nosuchhost (199.101.28.130) 56(84) bytes of data. 64 bytes from www.bigpondsitehelp.com (199.101.28.130): icmp_seq=1 ttl=45 time=391 ms Looking at the test, it seems we have lost part of the original intent too "pmlogger config with hyphens in hostname - #828416" - we no longer actually have hostnames with hyphens in the test which simply uses: # real QA test starts here for host in nosuchhost no.such.host.pcp.io ... any thoughts? No matter what shortname I change nosuchhost to, it will now resolve (!!!) ... and it'd be good to add back hyphen in there while I'm at it - thoughts? cheers. -- Nathan From kenj@internode.on.net Thu Oct 16 18:21:15 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 8B5B47F3F for ; Thu, 16 Oct 2014 18:21:15 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 798C08F8040 for ; Thu, 16 Oct 2014 16:21:15 -0700 (PDT) X-ASG-Debug-ID: 1413501669-04cbb070c742a80001-S8gJnT Received: from ipmail07.adl2.internode.on.net (ipmail07.adl2.internode.on.net [150.101.137.131]) by cuda.sgi.com with ESMTP id reRWbhxDesfbseY8 for ; Thu, 16 Oct 2014 16:21:10 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.131 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApUBAPZRQFR20YEr/2dsb2JhbAANToNhg16FW8M/h1ECgScBhQABAQQjVQEQCyEWCwICAgcDAgECAUUGDQEHAQG+SniVEQEBAQEBAQEBAQEBAQEBAQEBAQEBAReQMhsHCYJugVQBBJQNgVBoiH6YO1gBgkkBAQE Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail07.adl2.internode.on.net with ESMTP; 17 Oct 2014 09:51:09 +1030 Message-ID: <54405305.6040400@internode.on.net> Date: Fri, 17 Oct 2014 10:21:41 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: Nathan Scott CC: pcp developers Subject: Re: Test qa/645 failing References: <1013731158.70294119.1413501050948.JavaMail.zimbra@redhat.com> X-ASG-Orig-Subj: Re: Test qa/645 failing In-Reply-To: <1013731158.70294119.1413501050948.JavaMail.zimbra@redhat.com> Content-Type: multipart/mixed; boundary="------------070305000308000800060809" X-Barracuda-Connect: ipmail07.adl2.internode.on.net[150.101.137.131] X-Barracuda-Start-Time: 1413501669 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10653 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- This is a multi-part message in MIME format. --------------070305000308000800060809 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit On 17/10/14 10:10, Nathan Scott wrote: > ... any thoughts? No matter what shortname I change nosuchhost > to, it will now resolve (!!!) ... and it'd be good to add back > hyphen in there while I'm at it - thoughts? I think the attached patch restores the original intent, and may be smart enough to fool the Telstra DNS. --------------070305000308000800060809 Content-Type: text/plain; charset=UTF-8; name="patch.645" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="patch.645" ZGlmZiAtLWdpdCBhL3FhLzY0NSBiL3FhLzY0NQppbmRleCA0YTBmMWM1Li4wMzQ0MjVmIDEw MDc1NQotLS0gYS9xYS82NDUKKysrIGIvcWEvNjQ1CkBAIC0yOCw3ICsyOCw3IEBAIF9maWx0 ZXIoKQogfQogCiAjIHJlYWwgUUEgdGVzdCBzdGFydHMgaGVyZQotZm9yIGhvc3QgaW4gbm9z dWNoaG9zdCBuby5zdWNoLmhvc3QucGNwLmlvCitmb3IgaG9zdCBpbiBub3N1Y2hob3N0LnBj cC5pbyBuby1zdWNoLWhvc3QucGNwLmlvCiBkbwogICAgIGNhdCA8PEVuZC1vZi1GaWxlIHwg cG1sb2dnZXIgLWwgJHRtcC5sb2cgJHRtcAogbG9nIG1hbmRhdG9yeSBvbiBvbmNlIHsgaGlu di5uY3B1IH0K --------------070305000308000800060809-- From nscott@redhat.com Thu Oct 16 18:48:57 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 67A437F3F for ; Thu, 16 Oct 2014 18:48:57 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 4809130404E for ; Thu, 16 Oct 2014 16:48:57 -0700 (PDT) X-ASG-Debug-ID: 1413503332-04cb6c2efa43640001-S8gJnT Received: from mx5-phx2.redhat.com (mx5-phx2.redhat.com [209.132.183.37]) by cuda.sgi.com with ESMTP id OlT8AGQzFNETPUC3 (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 16 Oct 2014 16:48:53 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.37 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx5-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9GNmbjn022708; Thu, 16 Oct 2014 19:48:37 -0400 Date: Thu, 16 Oct 2014 19:48:37 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Ken McDonell Cc: pcp developers Message-ID: <1480759251.70299826.1413503317589.JavaMail.zimbra@redhat.com> In-Reply-To: <54405305.6040400@internode.on.net> References: <1013731158.70294119.1413501050948.JavaMail.zimbra@redhat.com> <54405305.6040400@internode.on.net> Subject: Re: Test qa/645 failing MIME-Version: 1.0 X-ASG-Orig-Subj: Re: Test qa/645 failing Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: Test qa/645 failing Thread-Index: nZPvzRba4wBQt+NVR6TGyecBchQiiw== X-Barracuda-Connect: mx5-phx2.redhat.com[209.132.183.37] X-Barracuda-Start-Time: 1413503332 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10654 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... ----- Original Message ----- > On 17/10/14 10:10, Nathan Scott wrote: > > > ... any thoughts? No matter what shortname I change nosuchhost > > to, it will now resolve (!!!) ... and it'd be good to add back > > hyphen in there while I'm at it - thoughts? > > I think the attached patch restores the original intent, and may be > smart enough to fool the Telstra DNS. > Yep, thats done the trick. Will remake 645.out & commit - thanks. -- Nathan From nscott@redhat.com Thu Oct 16 23:31:27 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id AA78A7F3F for ; Thu, 16 Oct 2014 23:31:27 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 4965EAC003 for ; Thu, 16 Oct 2014 21:31:27 -0700 (PDT) X-ASG-Debug-ID: 1413520281-04cbb070c54c730001-S8gJnT Received: from mx6-phx2.redhat.com (mx6-phx2.redhat.com [209.132.183.39]) by cuda.sgi.com with ESMTP id 5XSa0vrwefKWmAnZ (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 16 Oct 2014 21:31:22 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.39 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx6-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9H4VLhI017952; Fri, 17 Oct 2014 00:31:21 -0400 Date: Fri, 17 Oct 2014 00:31:21 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Lukas Berk Cc: pcp developers Message-ID: <227053328.70372701.1413520281062.JavaMail.zimbra@redhat.com> In-Reply-To: <1953450796.70372138.1413520090797.JavaMail.zimbra@redhat.com> Subject: Unexpected output in test qa/914 MIME-Version: 1.0 X-ASG-Orig-Subj: Unexpected output in test qa/914 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: Unexpected output in test qa/914 Thread-Index: EifvDCrl9UQ81r+0FY2q2UsmkEuR9w== X-Barracuda-Connect: mx6-phx2.redhat.com[209.132.183.39] X-Barracuda-Start-Time: 1413520281 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.03 X-Barracuda-Spam-Status: No, SCORE=0.03 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_SA_TO_FROM_DOMAIN_MATCH, THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10661 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... 0.01 BSF_SC0_SA_TO_FROM_DOMAIN_MATCH Sender Domain Matches Recipient Domain Hi Lukas, I'm seeing a few spurious messages from papi in test qa/914 - do these look worrying to you? This is an f20 machine, and has papi-5.2.0-5 installed... $ diff 914.out* 3a4,5 > PAPI Error: papi_preset: Error finding event MEM_LOAD_UOPS_RETIRED:L1_HIT:L1_MISS. > ibwarn: [9524] umad_init: can't read ABI version from /sys/class/infiniband_mad/abi_version (No such file or directory): is ib_umad module loaded? [snip ... same warning, repeated throughout test] I'm guessing its probably OK and the best we can do to achieve determinism is to filter these out? The infiniband message was a bit unexpected... *shrug*. cheers. -- Nathan From karthikeyan.ps@zohocorp.com Fri Oct 17 00:15:59 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id E33127F3F for ; Fri, 17 Oct 2014 00:15:59 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id D097E8F8040 for ; Thu, 16 Oct 2014 22:15:56 -0700 (PDT) X-ASG-Debug-ID: 1413522950-04cb6c2efc4d270001-S8gJnT Received: from sender1.zohomail.com (sender1.zohomail.com [74.201.84.154]) by cuda.sgi.com with ESMTP id MXDhnyXfake8lAAM (version=TLSv1 cipher=RC4-MD5 bits=128 verify=NO) for ; Thu, 16 Oct 2014 22:15:50 -0700 (PDT) X-Barracuda-Envelope-From: karthikeyan.ps@zohocorp.com X-Barracuda-Apparent-Source-IP: 74.201.84.154 Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1413522936105183.24537962612146; Thu, 16 Oct 2014 22:15:36 -0700 (PDT) Date: Fri, 17 Oct 2014 10:45:36 +0530 From: "karthikeyan.ps" To: Ken McDonell Cc: Message-ID: <1491c85ae35.11c1c395855827.2240099225956718949@zohocorp.com> In-Reply-To: <544043CF.8060309@internode.on.net> References: <149132de64a.fc32bc2f31382.7164760145933637595@zohocorp.com> <544043CF.8060309@internode.on.net> Subject: Re: Re: [pcp] reg install pcp in freebsd MIME-Version: 1.0 X-ASG-Orig-Subj: Re: Re: [pcp] reg install pcp in freebsd Content-Type: multipart/alternative; boundary="----=_Part_73750_357827917.1413522936046" X-Priority: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail X-Barracuda-Connect: sender1.zohomail.com[74.201.84.154] X-Barracuda-Start-Time: 1413522950 X-Barracuda-Encrypted: RC4-MD5 X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=HTML_MESSAGE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10662 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 HTML_MESSAGE BODY: HTML included in message ------=_Part_73750_357827917.1413522936046 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Thank you ken i installed pcp with normal source code install method. But the problem was make command not working in freebsd instead of i used gmake after that package installed without issue by the way thank you for your support Thanks & Regards Karthikeyan Saravanan ---- On Fri, 17 Oct 2014 03:46:18 +0530 Ken McDonell <kenj@internode.on.net> wrote ---- On 15/10/14 20:43, karthikeyan.ps wrote: > hi, > > I am karthikeyan i have planned to install pcp in freebsd 9.2. I tried > from source but unable to install pcp. If you don't mind can you suggest > procedure to install pcp in freebsd. I already tried all the given > procedure in your sides but no use. Karthikeyan, I (well, strictly speaking cron, not I) regularly build and install PCP on lots of platforms, including some FreeBSD ones. We don't have real *BSD packages (no one on the project has that skill set, volunteers would be most welcome), so we're using a tarball and some helper scripts. This is the recipe (extracted from qa/admin/pcp-daily) $ git pull $ ./Makepkgs This should produce a tarball in pcp-<buildversion>/build/tar $ . ./VERSION.pcp $ buildversion=$PACKAGE_MAJOR.$PACKAGE_MINOR.$PACKAGE_REVISION $ here=`pwd` $ tarball=$here/pcp-$buildversion/build/tar/pcp-[0-9]*[0-9].tar.gz $ cd pcp-$buildversion/build/tar $ sudo ./preinstall $ cd / $ sudo tar -zxpf $tarball $ cd $here $ cd pcp-$buildversion/build/tar $ sudo ./postinstall That's the end of the installation steps. Now start pmcd. $ . /etc/pcp.env $ sudo $PCP_RC_DIR/pcp start ------=_Part_73750_357827917.1413522936046 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =
Thank you ken i installed pcp with normal source code install= method. But the problem was make command  not working in freebsd inst= ead of i used gmake after that package installed without issue by the way t= hank you for your support

Thanks & Regards
Karthikeyan Saravanan
<= b>
---- = On Fri, 17 Oct 2014 03:46:18 +0530 Ken McDonell <kenj@internode.on.ne= t> wrote ----
On = 15/10/14 20:43, karthikeyan.ps wrote:
> hi,
>
> I am k= arthikeyan i have planned to install pcp in freebsd 9.2. I tried
> f= rom source but unable to install pcp. If you don't mind can you suggest > procedure to install pcp in freebsd. I already tried all the given > procedure in your sides but no use.

Karthikeyan,

I= (well, strictly speaking cron, not I) regularly build and install PCP on lots of platforms, including some FreeBSD ones.

We don't have = real *BSD packages (no one on the project has that skill
set, voluntee= rs would be most welcome), so we're using a tarball and
some helper sc= ripts.

This is the recipe (extracted from qa/admin/pcp-daily)
=
$ git pull
$ ./Makepkgs

This should produce a tarball in = pcp-<buildversion>/build/tar

$ . ./VERSION.pcp
$ buildve= rsion=3D$PACKAGE_MAJOR.$PACKAGE_MINOR.$PACKAGE_REVISION
$ here=3D`pwd` =
$ tarball=3D$here/pcp-$buildversion/build/tar/pcp-[0-9]*[0-9].tar.gz $ cd pcp-$buildversion/build/tar
$ sudo ./preinstall
$ cd /
$= sudo tar -zxpf $tarball
$ cd $here
$ cd pcp-$buildversion/build/ta= r
$ sudo ./postinstall

That's the end of the installation step= s. Now start pmcd.

$ . /etc/pcp.env
$ sudo $PCP_RC_DIR/pcp st= art


------=_Part_73750_357827917.1413522936046-- From fche@redhat.com Fri Oct 17 10:06:38 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 3480A7F3F for ; Fri, 17 Oct 2014 10:06:38 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 1461F30404E for ; Fri, 17 Oct 2014 08:06:34 -0700 (PDT) X-ASG-Debug-ID: 1413558390-04cb6c2ef95ce50001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id djAOTEAuq6oPHcmD (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Fri, 17 Oct 2014 08:06:30 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9HF6Tnl011028 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Fri, 17 Oct 2014 11:06:30 -0400 Received: from fche.csb (vpn-234-89.phx2.redhat.com [10.3.234.89]) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9HF6Tra014876; Fri, 17 Oct 2014 11:06:29 -0400 Received: by fche.csb (Postfix, from userid 2569) id 91B53582E4; Fri, 17 Oct 2014 11:06:23 -0400 (EDT) To: Nathan Scott Cc: Lukas Berk , pcp@oss.sgi.com Subject: Re: Unexpected output in test qa/914 References: <1953450796.70372138.1413520090797.JavaMail.zimbra@redhat.com> <227053328.70372701.1413520281062.JavaMail.zimbra@redhat.com> X-ASG-Orig-Subj: Re: Unexpected output in test qa/914 From: fche@redhat.com (Frank Ch. Eigler) Date: Fri, 17 Oct 2014 11:06:23 -0400 In-Reply-To: <227053328.70372701.1413520281062.JavaMail.zimbra@redhat.com> (Nathan Scott's message of "Fri, 17 Oct 2014 00:31:21 -0400 (EDT)") Message-ID: User-Agent: Gnus/5.1008 (Gnus v5.10.8) Emacs/21.4 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1413558390 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 nathans wrote: > [...] > I'm seeing a few spurious messages from papi in test qa/914 - > do these look worrying to you? [...] It's some goofy papi-internal thing, yeah ignore it. > I'm guessing its probably OK and the best we can do to achieve > determinism is to filter these out? [...] (Or with -h mode, such stuff would go only into pmcd/papi.log, not into the test case stderr/stdout.) - FChE From kenj@internode.on.net Fri Oct 17 15:08:47 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 64E427F3F for ; Fri, 17 Oct 2014 15:08:47 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id E3C36AC005 for ; Fri, 17 Oct 2014 13:08:46 -0700 (PDT) X-ASG-Debug-ID: 1413576523-04cbb070c86baf0001-S8gJnT Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by cuda.sgi.com with ESMTP id AT1OCpSdOF3uicmj for ; Fri, 17 Oct 2014 13:08:43 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.145 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqgKALN2QVR5LBtMPGdsb2JhbABbgw5TWIMGhDfEfIdQBAICgRMXAQYBAQEBODuEAgEBAQQIAhkFLiMMAQMCBgMVBQIjAwICGSAKAxECBBMLBYguuTKUZCyBLI8hB4J3gVQFj2SCHF2DaYhDPJA1iA8pL4JLAQEB Received: from ppp121-44-27-76.lns20.syd6.internode.on.net (HELO bozohorize) ([121.44.27.76]) by ipmail06.adl6.internode.on.net with ESMTP; 18 Oct 2014 06:38:41 +1030 From: "Ken McDonell" To: "'karthikeyan.ps'" Cc: References: <149132de64a.fc32bc2f31382.7164760145933637595@zohocorp.com> <544043CF.8060309@internode.on.net> <1491c85ae35.11c1c395855827.2240099225956718949@zohocorp.com> In-Reply-To: <1491c85ae35.11c1c395855827.2240099225956718949@zohocorp.com> Subject: RE: Re: [pcp] reg install pcp in freebsd Date: Sat, 18 Oct 2014 07:08:35 +1100 X-ASG-Orig-Subj: RE: Re: [pcp] reg install pcp in freebsd Message-ID: <002301cfea46$23b309a0$6b191ce0$@internode.on.net> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Outlook 15.0 Thread-Index: AQIYKpVeW9zYVq4U9mffPTdOFdGudwH+TbS4AnCW8V2bgL4n0A== Content-Language: en-au X-Barracuda-Connect: ipmail06.adl6.internode.on.net[150.101.137.145] X-Barracuda-Start-Time: 1413576523 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.01 X-Barracuda-Spam-Status: No, SCORE=0.01 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, THREAD_INDEX X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10687 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header > -----Original Message----- > From: karthikeyan.ps [mailto:karthikeyan.ps@zohocorp.com] > ... > Thank you ken i installed pcp with normal source code install method. = But the > problem was make command not working in freebsd instead of i used = gmake > after that package installed without issue by the way thank you for = your > support Glad it worked out. The qa/admin/pcp-daily script contains the snippet below to set MAKE to = gmake in your circumstances ... sorry I forgot to mention that. check=3D`ssh -t $host uname -sr 2>/dev/null` $debug && echo "check=3D$check" case "$check" in FreeBSD\ *) osname=3Dfreebsd ;; *) osname=3Dunknown ;; esac $debug && echo "osname=3D$osname" # per-host settings # MAKE=3Dgmake for *BSD hosts # MAKE=3Dmake case "$osname" in freebsd|netbsd) MAKE=3Dgmake ;; esac From michele@acksyn.org Tue Oct 21 17:24:03 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id C6C867F50 for ; Tue, 21 Oct 2014 17:24:03 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 472B1AC005 for ; Tue, 21 Oct 2014 15:24:00 -0700 (PDT) X-ASG-Debug-ID: 1413930237-04cbb070c6133a70001-S8gJnT Received: from palahniuk.acksyn.org (palahniuk.acksyn.org [5.9.7.26]) by cuda.sgi.com with ESMTP id LVWlmXsJYYiFW3Gg for ; Tue, 21 Oct 2014 15:23:58 -0700 (PDT) X-Barracuda-Envelope-From: michele@acksyn.org X-Barracuda-Apparent-Source-IP: 5.9.7.26 Received: from localhost (localhost [127.0.0.1]) by palahniuk.acksyn.org (Postfix) with ESMTP id 48B75289AD for ; Tue, 21 Oct 2014 18:23:57 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=acksyn.org; h= user-agent:content-disposition:content-type:content-type :mime-version:message-id:subject:subject:from:from:date:date :received:received; s=2010; t=1413930236; bh=JVyO/JNrNacy7O9Nlxd BqrAHMJiZO9/0AH15Yh+tB+0=; b=f12O/BO7MBpz5xiNaX3a0BE2KecwHKmptkU GfB5AwDD+rAXyLQRYc6JiJC103arU0WwiRyxD4wagntbRnraDWJqdB65m3QqXH9e NZdj95lYGgRLgazuKnT67YMomHd8j5i9GqG76fLEk++iVrlJTi2JlTIG+CvbPcIw w71dp8Nk= Received: from palahniuk.acksyn.org ([127.0.0.1]) by localhost (mail.acksyn.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id XkmJzJR-juYA for ; Tue, 21 Oct 2014 18:23:56 -0400 (EDT) Received: from localhost (host41-190-dynamic.24-79-r.retail.telecomitalia.it [79.24.190.41]) by palahniuk.acksyn.org (Postfix) with ESMTPSA id 1DFCC1FFB4 for ; Tue, 21 Oct 2014 18:23:56 -0400 (EDT) Date: Wed, 22 Oct 2014 00:23:55 +0200 From: Michele Baldessari To: pcp@oss.sgi.com Subject: hinv.* Message-ID: <20141021222355.GC6656@marquez.int.rhx> X-ASG-Orig-Subj: hinv.* MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2012-12-30) X-Barracuda-Connect: palahniuk.acksyn.org[5.9.7.26] X-Barracuda-Start-Time: 1413930237 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10811 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature Hi all, while for most hinv.* metrics I do not expect any changes over time (hinv.pagesize, hinv.ncpu, ...), I was a bit surprised that also hinv.cpu.clock seems to not change over time (F21 updated today, here). For example: $ pmval -h localhost -s 5 hinv.cpu.clock metric: hinv.cpu.clock host: marquez.int.rhx semantics: discrete instantaneous value units: count samples: 5 interval: 1.00 sec cpu0 cpu1 cpu2 cpu3 cpu4 cpu5 cpu6 cpu7 3060. 3060. 1862. 3060. 2926. 3060. 1596. 2926. 3060. 3060. 1862. 3060. 2926. 3060. 1596. 2926. 3060. 3060. 1862. 3060. 2926. 3060. 1596. 2926. 3060. 3060. 1862. 3060. 2926. 3060. 1596. 2926. 3060. 3060. 1862. 3060. 2926. 3060. 1596. 2926. Since the description implies it should be read from /proc: $ pminfo -t -h localhost hinv.cpu.clock hinv.cpu.clock [clock rate in Mhz for each CPU as reported by /proc/cpuinfo] A quick 'egrep "^processor|cpu\ MHz" /proc/cpuinfo' does show that they change quite often and all the time. Is this an issue in the pmda or in the description or have I missed something else? Thanks and regards, Michele -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D From nscott@redhat.com Wed Oct 22 12:41:13 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id AA2907F59 for ; Wed, 22 Oct 2014 12:41:13 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 883498F8033 for ; Wed, 22 Oct 2014 10:41:10 -0700 (PDT) X-ASG-Debug-ID: 1413999665-04cb6c2efa15e4b0001-S8gJnT Received: from mx6-phx2.redhat.com (mx6-phx2.redhat.com [209.132.183.39]) by cuda.sgi.com with ESMTP id HXnekBwy1BYdpA2P (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 22 Oct 2014 10:41:06 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.39 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx6-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9MHexoK011463; Wed, 22 Oct 2014 13:40:59 -0400 Date: Wed, 22 Oct 2014 13:40:59 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Michele Baldessari Cc: pcp@oss.sgi.com Message-ID: <573592010.74725901.1413999659304.JavaMail.zimbra@redhat.com> In-Reply-To: <20141021222355.GC6656@marquez.int.rhx> References: <20141021222355.GC6656@marquez.int.rhx> Subject: Re: [pcp] hinv.* MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] hinv.* Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF32 (Linux)/8.0.6_GA_5922) Thread-Topic: hinv.* Thread-Index: jIdc483XPhbEuNjcihgPnBlOP8aVcA== X-Barracuda-Connect: mx6-phx2.redhat.com[209.132.183.39] X-Barracuda-Start-Time: 1413999665 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10830 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Hi Michele, ----- Original Message ----- > Hi all, > > while for most hinv.* metrics I do not expect any changes over time > (hinv.pagesize, hinv.ncpu, ...), I was a bit surprised that also > hinv.cpu.clock seems to not change over time (F21 updated today, here). > For example: > $ pmval -h localhost -s 5 hinv.cpu.clock > metric: hinv.cpu.clock > host: marquez.int.rhx > semantics: discrete instantaneous value > units: count > samples: 5 > interval: 1.00 sec > > cpu0 cpu1 cpu2 cpu3 cpu4 > cpu5 cpu6 cpu7 > 3060. 3060. 1862. 3060. 2926. > 3060. 1596. 2926. > 3060. 3060. 1862. 3060. 2926. > 3060. 1596. 2926. > 3060. 3060. 1862. 3060. 2926. > 3060. 1596. 2926. > 3060. 3060. 1862. 3060. 2926. > 3060. 1596. 2926. > 3060. 3060. 1862. 3060. 2926. > 3060. 1596. 2926. > > Since the description implies it should be read from /proc: > $ pminfo -t -h localhost hinv.cpu.clock > hinv.cpu.clock [clock rate in Mhz for each CPU as reported by /proc/cpuinfo] > > A quick 'egrep "^processor|cpu\ MHz" /proc/cpuinfo' does show that they > change quite often and all the time. > > Is this an issue in the pmda or in the description or have I missed > something else? Yeah, I'm finding this a bit odd too - the code over in refresh_proc_cpuinfo (src/pmdas/linux/proc_cpuinfo.c) certainly *looks* like it should reevaluate the values on every sample. I can look a bit closer next week if its still an unsolved mystery then (travelling atm), will see if I can reproduce it. cheers. -- Nathan From mgoodwin@redhat.com Wed Oct 22 18:09:53 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 9B5D67F59 for ; Wed, 22 Oct 2014 18:09:53 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 88C6D30405F for ; Wed, 22 Oct 2014 16:09:50 -0700 (PDT) X-ASG-Debug-ID: 1414019386-04cb6c2efb16a590001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id wxR4R3KNF0rFugSp (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 22 Oct 2014 16:09:46 -0700 (PDT) X-Barracuda-Envelope-From: mgoodwin@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9MN9hu2020479 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 22 Oct 2014 19:09:43 -0400 Received: from [10.64.50.254] (vpn1-50-254.bne.redhat.com [10.64.50.254]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9MN9fkx011200; Wed, 22 Oct 2014 19:09:42 -0400 Message-ID: <54483934.4060802@redhat.com> Date: Thu, 23 Oct 2014 10:09:40 +1100 From: Mark Goodwin User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: Michele Baldessari CC: Nathan Scott , pcp@oss.sgi.com Subject: Re: [pcp] hinv.* References: <20141021222355.GC6656@marquez.int.rhx> <573592010.74725901.1413999659304.JavaMail.zimbra@redhat.com> X-ASG-Orig-Subj: Re: [pcp] hinv.* In-Reply-To: <573592010.74725901.1413999659304.JavaMail.zimbra@redhat.com> Content-Type: multipart/mixed; boundary="------------040501040401070305000401" X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1414019386 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 This is a multi-part message in MIME format. --------------040501040401070305000401 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 10/23/2014 04:40 AM, Nathan Scott wrote: > Hi Michele, > > ----- Original Message ----- >> Hi all, >> >> while for most hinv.* metrics I do not expect any changes over time >> (hinv.pagesize, hinv.ncpu, ...), I was a bit surprised that also >> hinv.cpu.clock seems to not change over time (F21 updated today, here). The original code was written before CPU clock throttling, cstates and so forth, and hence didn't expect the clock to change. It does nowdays! Attached patch should fix it (untested but should be fine). I'll also need to check if the default pmlogger config only logs hinv.cpu.clock once - that might need an update too, and of course some QA. Regards -- Mark --------------040501040401070305000401 Content-Type: text/x-patch; name="pcp_hinv_cpu_clock.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="pcp_hinv_cpu_clock.patch" hinv.cpu.clock can change due to speed throttling. diff --git a/src/pmdas/linux/proc_cpuinfo.c b/src/pmdas/linux/proc_cpuinfo.c index f76ac27..b944575 100644 --- a/src/pmdas/linux/proc_cpuinfo.c +++ b/src/pmdas/linux/proc_cpuinfo.c @@ -220,7 +220,7 @@ refresh_proc_cpuinfo(proc_cpuinfo_t *proc_cpuinfo) info->cache_align = atoi(val); else if (info->bogomips == 0.0 && strncasecmp(buf, "bogo", 4) == 0) info->bogomips = atof(val); - else if (info->clock == 0.0 && strncasecmp(buf, "cpu MHz", 7) == 0) + else if (strncasecmp(buf, "cpu MHz", 7) == 0) /* cpu MHz can change */ info->clock = atof(val); else if (info->clock == 0.0 && strncasecmp(buf, "cycle frequency", 15) == 0) { if ((p = strchr(val, ' ')) != NULL) --------------040501040401070305000401-- From michele@acksyn.org Thu Oct 23 02:26:46 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 5EDE17F51 for ; Thu, 23 Oct 2014 02:26:46 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 4CDE4304051 for ; Thu, 23 Oct 2014 00:26:43 -0700 (PDT) X-ASG-Debug-ID: 1414049198-04cb6c2efa1772b0001-S8gJnT Received: from palahniuk.acksyn.org (palahniuk.acksyn.org [5.9.7.26]) by cuda.sgi.com with ESMTP id Q5P8x4UCkmN83gPO for ; Thu, 23 Oct 2014 00:26:38 -0700 (PDT) X-Barracuda-Envelope-From: michele@acksyn.org X-Barracuda-Apparent-Source-IP: 5.9.7.26 Received: from localhost (localhost [127.0.0.1]) by palahniuk.acksyn.org (Postfix) with ESMTP id B8DA528E78; Thu, 23 Oct 2014 03:26:37 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=acksyn.org; h= user-agent:in-reply-to:content-disposition:content-type :content-type:mime-version:references:message-id:subject:subject :from:from:date:date:received:received; s=2010; t=1414049197; bh=wzi8uEsxcR8fTb1stATtH7/tPNUjPmwHBWcQepkzY9Q=; b=Fh3GhvX/vVw1 Up8AO6waCYocMrc8J985G6M1+Lmp4u5vfH7eUg9OAKAee6VDZI1Q2P5Me/vc/Sw9 Z+USHEdoDLVz8aOLWmZ3AfBC23aZ9YdtQXvwShK+hihTlYW1eGU3E9H0g6bKC7iA BE7XYx0bHHBYVFZ70YnDwji0T5K9FxU= Received: from palahniuk.acksyn.org ([127.0.0.1]) by localhost (mail.acksyn.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id oJUJ4p52byca; Thu, 23 Oct 2014 03:26:37 -0400 (EDT) Received: from localhost (host41-190-dynamic.24-79-r.retail.telecomitalia.it [79.24.190.41]) by palahniuk.acksyn.org (Postfix) with ESMTPSA id DC9282641B; Thu, 23 Oct 2014 03:26:36 -0400 (EDT) Date: Thu, 23 Oct 2014 09:26:36 +0200 From: Michele Baldessari To: Mark Goodwin Cc: Nathan Scott , pcp@oss.sgi.com Subject: Re: [pcp] hinv.* Message-ID: <20141023072636.GB6158@marquez.int.rhx> X-ASG-Orig-Subj: Re: [pcp] hinv.* References: <20141021222355.GC6656@marquez.int.rhx> <573592010.74725901.1413999659304.JavaMail.zimbra@redhat.com> <54483934.4060802@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <54483934.4060802@redhat.com> User-Agent: Mutt/1.5.21 (2012-12-30) X-Barracuda-Connect: palahniuk.acksyn.org[5.9.7.26] X-Barracuda-Start-Time: 1414049198 X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10845 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature Hi Mark, On Thu, Oct 23, 2014 at 10:09:40AM +1100, Mark Goodwin wrote: > On 10/23/2014 04:40 AM, Nathan Scott wrote: > >>while for most hinv.* metrics I do not expect any changes over time > >>(hinv.pagesize, hinv.ncpu, ...), I was a bit surprised that also > >>hinv.cpu.clock seems to not change over time (F21 updated today, here). > > The original code was written before CPU clock throttling, > cstates and so forth, and hence didn't expect the clock to > change. It does nowdays! Attached patch should fix it (untested > but should be fine). I'll also need to check if the default > pmlogger config only logs hinv.cpu.clock once - that might > need an update too, and of course some QA. Thanks Mark, the patch works fine, I see the different values now. Given how dynamic CPUs can be these days, it might be nice to add the online/offline status (/sys/devices/system/cpu/cpu1/online) as well. The default logging here is "once" for hinv, so that might need tweaking as well. Cheers, Michele -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D From kenj@internode.on.net Thu Oct 23 16:52:15 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id B4D977F60 for ; Thu, 23 Oct 2014 16:52:15 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id A3EFD304039 for ; Thu, 23 Oct 2014 14:52:12 -0700 (PDT) X-ASG-Debug-ID: 1414101126-04cb6c2efb263010001-S8gJnT Received: from ipmail06.adl2.internode.on.net (ipmail06.adl2.internode.on.net [150.101.137.129]) by cuda.sgi.com with ESMTP id GphNUNbJmssPJmUh for ; Thu, 23 Oct 2014 14:52:07 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.129 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AlQCAKJ3SVR20YErPGdsb2JhbAANT4Nig17JeYh6AQYBAQEBOIRnVTAGAgUNAQgLAgsDAgECATEOGQYCAQGISrJreJUHgSyPRQEDgmGBVAWGLZAehDadFViBCAeBPAEBAQ Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail06.adl2.internode.on.net with ESMTP; 24 Oct 2014 08:22:05 +1030 Message-ID: <544978C1.5010508@internode.on.net> Date: Fri, 24 Oct 2014 08:53:05 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: pcp updates Content-Type: text/plain; charset=utf-8 X-ASG-Orig-Subj: pcp updates Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl2.internode.on.net[150.101.137.129] X-Barracuda-Start-Time: 1414101127 X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10864 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Changes committed to git://git.performancecopilot.org/kenj/pcp.git dev Makepkgs | 18 +++++++----------- qa/069 | 6 +++--- qa/542 | 1 + qa/645 | 2 +- qa/645.out | 4 ++-- qa/common.avahi | 7 +++++-- qa/qa_hosts.master | 42 +----------------------------------------- 7 files changed, 20 insertions(+), 60 deletions(-) commit 698b501bc9fe738966c6e5fe1270a64c5045a574 Author: Ken McDonell Date: Fri Oct 24 08:51:39 2014 +1100 qa/common.avahi - domain name games Strip any domain part from hostname(1) output ... e.g. if hostname is vm08.localdomain, avahi reports this as vm08.local commit 43e6214ff23490a645bc05109db927bb27a6b34e Author: Ken McDonell Date: Fri Oct 24 08:49:12 2014 +1100 qa/645 - use fqdn hostnames and re-instate hyphen in hostname commit d8839910d8a5b0d31f1d82ae0fc335ff5217598c Author: Ken McDonell Date: Fri Oct 24 06:47:34 2014 +1100 qa/069 - add diags for failure cases For --- attempt N --- lines, show from which remote host the command that failed was run (the problem is sometimes there, not here). commit c82a1209805c723ebb309d388a7b34155b978e61 Author: Ken McDonell Date: Fri Oct 24 06:47:10 2014 +1100 qa/qa_hosts.master - simplify kenj sections commit fd917c3c0cb8fdb80fbffdf1ac62b603613c6a4e Author: Ken McDonell Date: Thu Oct 23 21:42:25 2014 +1100 qa/542 - timezone is sometimes 4 chars long, e.g. AEST commit 6b8d9d18ea9a772adbb6ceefbb3c4373a153a510 Author: Ken McDonell Date: Thu Oct 23 11:27:22 2014 +1100 Makepkgs - debian build, don't try to sign packages ... unless you're login is "nathans". Was causing non-zero exit for me on some platforms, which killed the pcp-daily QA runs. From kenj@internode.on.net Sun Oct 26 22:31:15 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id EB71F7F4E for ; Sun, 26 Oct 2014 22:31:14 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 7B6CAAC001 for ; Sun, 26 Oct 2014 20:31:11 -0700 (PDT) X-ASG-Debug-ID: 1414380668-04cbb070c83c95e0001-S8gJnT Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by cuda.sgi.com with ESMTP id iJ0sAz9v8CE0Fz6y for ; Sun, 26 Oct 2014 20:31:09 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.145 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AhoKAAu8TVR20SxTPGdsb2JhbABcgw6DYoUHzRAEgRAXAQYBAQEBODuECQgCHhIcMAUGYiAKFQEEHgWIMKRmpAiRJYQ1BZIHXYwug0mSewEBAQcBAQEBAQGCNSmCegEBAQ Received: from ppp118-209-44-83.lns20.mel4.internode.on.net (HELO bozohorize) ([118.209.44.83]) by ipmail06.adl6.internode.on.net with ESMTP; 27 Oct 2014 14:00:48 +1030 From: "Ken McDonell" To: Subject: user/group access control question Date: Mon, 27 Oct 2014 14:30:44 +1100 X-ASG-Orig-Subj: user/group access control question Message-ID: <001f01cff196$66cf3e00$346dba00$@internode.on.net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 15.0 Thread-Index: Ac/xlTlzXuD9W9FUTGezwUKon0oQ6w== Content-Language: en-au X-Barracuda-Connect: ipmail06.adl6.internode.on.net[150.101.137.145] X-Barracuda-Start-Time: 1414380668 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.01 X-Barracuda-Spam-Status: No, SCORE=0.01 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10950 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== I think this one for Dave Brolley, but I'd welcome insight from any quarter. I have a bunch of qa/944 failures, and they appear to be correlated with hosts for which pmconfig -L reports ... kenj@vm05:~/src/pcp/src/libpcp/src$ pmconfig -L pcp_version=3.10.0 ... secure_sockets=false ... authentication=false unix_domain_sockets=true ... qa/944 seems to be failing in the client connecting to pmcd which is returning an error code of -95. The problem seems to be in the logic around DoCreds() where we never get to CheckAccountAccess() because the earlier call to __pmSecureServerHandshake() fails, with, er, -95. I am not sure how things are supposed to work in this config setup, but the patch below makes qa/944 pass on at least one of these failing platforms. Note this is for the "not secure sockets" variant of the implementation of pmSecureServerHandshake() (there are two implementations in the code). Could someone who knows please take a look and let me know if this is even close to the "correct" way to fix this issue? Cheers and thanks, Ken. diff --git a/src/libpcp/src/auxserver.c b/src/libpcp/src/auxserver.c index 498bac4..c18e6a5 100644 --- a/src/libpcp/src/auxserver.c +++ b/src/libpcp/src/auxserver.c @@ -867,7 +867,17 @@ __pmSecureServerHandshake(int fd, int flags, __pmHashCtl *attrs) (void)fd; (void)flags; (void)attrs; - return -EOPNOTSUPP; + + /* for things that require a secure server, return -EOPNOTSUPP */ + if ((flags & (PDU_FLAG_SECURE | PDU_FLAG_SECURE_ACK | PDU_FLAG_COMPRESS + | PDU_FLAG_AUTH)) != 0) + return -EOPNOTSUPP; + + /* CREDS_REQD is a special case that does not need a secure server */ + if ((flags & PDU_FLAG_CREDS_REQD) != 0) + return 0; + /* otherwise the flags are not expected */ + return PM_ERR_IPC; } int From kenj@internode.on.net Mon Oct 27 19:08:03 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 3FEC47F99 for ; Mon, 27 Oct 2014 19:08:03 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 2E5468F8033 for ; Mon, 27 Oct 2014 17:07:59 -0700 (PDT) X-ASG-Debug-ID: 1414454873-04cbb070c53f30e0001-S8gJnT Received: from ipmail06.adl2.internode.on.net (ipmail06.adl2.internode.on.net [150.101.137.129]) by cuda.sgi.com with ESMTP id RK7LvgXdZjJNetXc for ; Mon, 27 Oct 2014 17:07:54 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.129 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ai4CALfdTlR20YErPGdsb2JhbAANT4NiWIMGhDfGGokBAQYBAQEBOIRnVTAGAgUWCwILAwIBAgExJwYCAQGISrU4eJUugSyPeYJhgVQFhi2QIqFMWIJLAQEB Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail06.adl2.internode.on.net with ESMTP; 28 Oct 2014 10:37:42 +1030 Message-ID: <544EDE90.6010808@internode.on.net> Date: Tue, 28 Oct 2014 11:08:48 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: pcp updates - sample pmda and qa/260 Content-Type: text/plain; charset=utf-8 X-ASG-Orig-Subj: pcp updates - sample pmda and qa/260 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl2.internode.on.net[150.101.137.129] X-Barracuda-Start-Time: 1414454873 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10976 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Changes committed to git://git.performancecopilot.org/kenj/pcp.git dev qa/260 | 73 ++++++-- qa/260.out | 352 +++++++++++++++++++++--------------------- src/pmdas/sample/src/sample.c | 4 3 files changed, 234 insertions(+), 195 deletions(-) commit b35afb641034df502b7fc17e7ed2342fee0e6985 Author: Ken McDonell Date: Tue Oct 28 10:44:44 2014 +1100 qa/260 - refactor choice of metrics We were previously using pmcd.pdu_in.* and pmcd.pdu_out.* for the derived expressions and the pmie expressions. This does not work in the presence of concurrent access to pmcd outside this QA test, e.g. a local system pmie or pmlogger or some bozo from a remote machine. So, change to sample.const_rate.* and sampledso.const_rate so we can control the rate of counter increase and the values of the counters in ways that make the test more likely to pass. commit 3832c538a64c6c3157a8e2c1cd3715da1ad9abaf Author: Ken McDonell Date: Tue Oct 28 10:42:53 2014 +1100 sample pmda - make sample.const_rate.value available to pmstore For QA, I'd like to be able to reset this counter, so the subsequent sequence of values is deterministic. From nscott@redhat.com Tue Oct 28 00:37:42 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 88E6B7F96 for ; Tue, 28 Oct 2014 00:37:42 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 6824E8F8049 for ; Mon, 27 Oct 2014 22:37:39 -0700 (PDT) X-ASG-Debug-ID: 1414474652-04cb6c2efb3c5fd0001-S8gJnT Received: from mx6-phx2.redhat.com (mx6-phx2.redhat.com [209.132.183.39]) by cuda.sgi.com with ESMTP id LSnq7uZTMsZM6bia (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Mon, 27 Oct 2014 22:37:33 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.39 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx6-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9S5bWu5020384 for ; Tue, 28 Oct 2014 01:37:32 -0400 Date: Tue, 28 Oct 2014 01:37:32 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: pcp developers Message-ID: <573773476.1355528.1414474652048.JavaMail.zimbra@redhat.com> In-Reply-To: <1119080428.1354537.1414474404348.JavaMail.zimbra@redhat.com> Subject: pcp updates: merges (mgoodwin, fche, kenj, self), qa MIME-Version: 1.0 X-ASG-Orig-Subj: pcp updates: merges (mgoodwin, fche, kenj, self), qa Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: pcp updates: merges (mgoodwin, fche, kenj, self), qa Thread-Index: CfGBq73QLyu8Mqp5O4nPN+gmNJZgeQ== X-Barracuda-Connect: mx6-phx2.redhat.com[209.132.183.39] X-Barracuda-Start-Time: 1414474652 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10981 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Changes committed to git://git.pcp.io/pcp.git dev Mark Goodwin (1): pmdalinux: fix hinv.cpu.clock refresh logic so value can change Ken McDonell (8): Makepkgs - debian build, don't try to sign packages qa/542 - timezone is sometimes 4 chars long, e.g. AEST qa/qa_hosts.master - simplify kenj sections qa/069 - add diags for failure cases qa/645 - use fqdn hostnames and re-instate hyphen in hostname qa/common.avahi - domain name games sample pmda - make sample.const_rate.value available to pmstore qa/260 - refactor choice of metrics Nathan Scott (9): build: extend Makepkgs rpm path trickery to non-RedHat platforms build: update mac gitignore for generated file coverity: a handful of trivial string-related improvements build: rework the web-manager split to just javascript files now build: switch fedora spec from web-manager to webjs model build: fix fallout from a makefile LSRCFILES cull pmwebd qa: fix qa/780 - not updated since -f option removed from pmwebd pmwebd qa: allow qa/660 to handle missing jsdemos directory pmwebd qa: add qa/782 to verify options file back-compatibility Frank Ch. Eigler (62): pmwebd: make foreground mode less magic whitespace: gnu-indent pmwebd to keep it looking spic & span pmwebd build: link in cairo graphics library pmwebd: TODO++ pmwebd: pmgraphite rough-in snapshot pmwebd: initial conversion to C++ pmwebd: eliminate mhdb_* miniapi in favour of ordinary c++ stringstreams pmwebapi: snapshot for web graphing pmwebapi: switched to astyle as indenter pmwebapi: signs of life snapshot: graphlot /rawdata data supplying pmwebapi: snapshot, grafana signs of life pmwebd: grafana snapshot, TZ=UTC & absolute timestamp parsing pmwebd: metric enumeration snapshot pmwebd: remind self of numerical methods, loss of significance pmwebd: respect /graphite/render maxDataPoints more vigorously pmwebd: merge /rawdata & /render&format=json modes pmwebd: graphlot-invoked-from-graphite-browser js improvements pmwebd: metric enumeration coup de grace: add instance domains too pmwebd: handle metric/query format=completer pmwebd: use isnanf() to make double-sure to detect float (vs. doubles) pmwebd: add Access-Control-Allow-Origin: * headers to all json responses pmwebd: respond to pending exits (^C) more rapidly pmwebd: add periodic client-stats notification pmwebd: dumpstats tweak pmwebd: tweak client dumpstats more pmwebd: reindent the lot pmwebd: beginnings of cairo server-side rendering of graphs pmwebd: graphite png rendering: fix rendering of null sequences pmwebd: graphite color name parsing pmwebd: draw legend, title pmwebd graphite: draw grid, axis labels too pmwebd graphite: cairo rendering: fix time axis pmwebd graphite: during pmns enumeration, filter out non-numerics pmwebd graphite: support graphite metric-regex-search pmwebd: correct result mime/type for file server pmwebd graphite: add graphlot metric-box live-search too pmwebd graphite, multithread prep pmwebd graphite: add -M multithreading option pmwebd: tweak verbosity; add png cache-suppression headers pmwebd: log http query params tighter for -vv pmwebd graphite png rendering: intelligent legend/curve sorting-by-visibility pmwebd: correct a memory leak pmwebd graphite archive processing: avoid out-of-range times pmwebapi classic pmapi: correct a c++ conversion problem, caught by old 660 testcase pmwebd: fix typo in new rc_pmwebd option loader pmwebd: whitespace update pmwebd graphite: tweak messages, y-axis rounding, time parsing pmwebd config sample: add commented-out $PCP_DERIVED_CONFIG clause pmwebd: add $LIB_FOR_PTHREADS pmwebd diagnostics: make -vvv Just Right for graphite non-empty curves pmwebd: add gcov (test-coverage) build option; enlarge 660 test case pmwebd qa++, now gcov-guided pmwebd qa: now with ipv6 output too pmwebd: switch to pkgconfig configury to extract cflags/libs paths pmwebd: extend precision for some floating point outputs pmwebd: eliminate compiler warnings about unused variables pmwebapi: ACAO and QA improvements pmwebd: support old $PMWEBDOPTS configuration file format pmwebd: correct graphite cairo status formatting in startup message pmmgr: handle case of pm{ie,logger} daemon that refuses SIGTERM Makepkgs | 45 build/mac/.gitignore | 1 build/rpm/GNUmakefile | 4 build/rpm/fedora.spec | 46 build/rpm/pcp.spec.in | 131 build/tar/postinstall.tail | 4 configure | 235 + configure.ac | 57 debian/GNUmakefile | 35 debian/control | 25 debian/control.master | 17 debian/control.webapi | 8 debian/pcp-manager.dirs | 2 debian/pcp-manager.install | 11 debian/pcp-manager.postinst | 14 debian/pcp-manager.postrm | 6 debian/pcp-manager.prerm | 8 debian/pcp-webapi.dirs | 2 debian/pcp-webapi.install | 5 debian/pcp-webapi.postinst | 14 debian/pcp-webapi.postrm | 6 debian/pcp-webapi.prerm | 8 debian/rules | 13 man/man1/pmmgr.1 | 340 + man/man1/pmwebd.1 | 285 + man/man3/pmwebapi.3 | 263 + qa/.gitignore | 1 qa/069 | 8 qa/260 | 73 qa/260.out | 352 - qa/518 | 2 qa/542 | 1 qa/645 | 2 qa/645.out | 4 qa/660 | 254 + qa/660.out.4 | 2176 +++++++++++ qa/660.out.46 | 2457 ++++++++++++ qa/666 | 5 qa/780 | 106 qa/780.out | 8 qa/782 | 56 qa/782.out | 13 qa/GNUmakefile | 2 qa/common.avahi | 7 qa/common.webapi | 39 qa/group | 7 qa/qa_hosts.master | 42 qa/src/test_webapi.python | 158 src/GNUmakefile | 2 src/include/builddefs.in | 8 src/include/pcp.conf.in | 8 src/libpcp/src/logmeta.c | 2 src/pmcd/rc-proc.sh | 4 src/pmchart/view.cpp | 8 src/pmdas/linux/proc_cpuinfo.c | 2 src/pmdas/sample/src/sample.c | 4 src/pmdumptext/pmdumptext.cpp | 2 src/pmmgr/.gitignore | 2 src/pmmgr/GNUmakefile | 59 src/pmmgr/TODO | 12 src/pmmgr/config/GNUmakefile | 40 src/pmmgr/config/README | 13 src/pmmgr/config/target-discovery.example-avahi | 1 src/pmmgr/pmmgr.cxx | 1305 ++++++ src/pmmgr/pmmgr.h | 133 src/pmmgr/pmmgr.options | 27 src/pmmgr/pmmgr.service.in | 14 src/pmmgr/rc_pmmgr | 296 + src/pmns/pmnsdel.c | 12 src/pmwebapi/.gitignore | 4 src/pmwebapi/.indent.pro | 4 src/pmwebapi/GNUmakefile | 109 src/pmwebapi/TODO | 38 src/pmwebapi/main.c | 1896 ++++----- src/pmwebapi/main.cxx | 1886 ++++++--- src/pmwebapi/pmgraphite.c | 122 src/pmwebapi/pmgraphite.cxx | 4676 +++++++++++++++++------- src/pmwebapi/pmresapi.c | 452 +- src/pmwebapi/pmresapi.cxx | 462 +- src/pmwebapi/pmwebapi.c | 4206 ++++++++++----------- src/pmwebapi/pmwebapi.cxx | 3369 +++++++++++------ src/pmwebapi/pmwebapi.h | 485 +- src/pmwebapi/pmwebd.options | 88 src/pmwebapi/pmwebd.service.in | 14 src/pmwebapi/rc_pmwebd | 360 + src/pmwebapi/util.c | 264 - src/pmwebapi/util.cxx | 296 + src/pmwebapi/util.h | 70 88 files changed, 20899 insertions(+), 7214 deletions(-) From nscott@redhat.com Tue Oct 28 00:40:33 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id CE6ED7F96 for ; Tue, 28 Oct 2014 00:40:33 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id AD8B2304032 for ; Mon, 27 Oct 2014 22:40:30 -0700 (PDT) X-ASG-Debug-ID: 1414474828-04cbb070c6407b40001-S8gJnT Received: from mx3-phx2.redhat.com (mx3-phx2.redhat.com [209.132.183.24]) by cuda.sgi.com with ESMTP id NI3HwEKP5E8HiEx4 (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Mon, 27 Oct 2014 22:40:29 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.24 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx3-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s9S5ePBT010240; Tue, 28 Oct 2014 01:40:25 -0400 Date: Tue, 28 Oct 2014 01:40:24 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Michele Baldessari , Mark Goodwin Cc: pcp@oss.sgi.com Message-ID: <1558759660.1356853.1414474824972.JavaMail.zimbra@redhat.com> In-Reply-To: <20141023072636.GB6158@marquez.int.rhx> References: <20141021222355.GC6656@marquez.int.rhx> <573592010.74725901.1413999659304.JavaMail.zimbra@redhat.com> <54483934.4060802@redhat.com> <20141023072636.GB6158@marquez.int.rhx> Subject: Re: [pcp] hinv.* MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] hinv.* Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: hinv.* Thread-Index: 899D2RSelvv5XIvzZrni95vW4ldwFg== X-Barracuda-Connect: mx3-phx2.redhat.com[209.132.183.24] X-Barracuda-Start-Time: 1414474828 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10981 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Hi guys, ----- Original Message ----- > [...] > Thanks Mark, the patch works fine, I see the different values now. > Given how dynamic CPUs can be these days, it might be nice to > add the online/offline status (/sys/devices/system/cpu/cpu1/online) > as well. Good idea - I'll hack that up tomorrow unless someone beats me to it. > The default logging here is "once" for hinv, so that might need tweaking as > well. Yeah, hmmm, thats a bit awkward - I don't really want us to have to explicitly list each hinv metric in the logging configs. Not sure what else we can do though. cheers. -- Nathan From kenj@internode.on.net Tue Oct 28 00:45:56 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 66EEC7F96 for ; Tue, 28 Oct 2014 00:45:56 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 53B948F8033 for ; Mon, 27 Oct 2014 22:45:56 -0700 (PDT) X-ASG-Debug-ID: 1414475154-04cbb070c7407d50001-S8gJnT Received: from ipmail06.adl2.internode.on.net (ipmail06.adl2.internode.on.net [150.101.137.129]) by cuda.sgi.com with ESMTP id xtsAycW7fEzLNKhk for ; Mon, 27 Oct 2014 22:45:54 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.129 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AsQBAL0sT1R20YErPGdsb2JhbAANT4t3ywODIAKBNAEGAQEBATiEPgEBBDhRCxgJJQ8CMhQTCAEBvUCWEAEBCAIBH5EPFoQ1AQS4G4MjAQEB Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail06.adl2.internode.on.net with ESMTP; 28 Oct 2014 16:15:53 +1030 Message-ID: <544F2DD5.40108@internode.on.net> Date: Tue, 28 Oct 2014 16:47:01 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: Re: [pcp] hinv.* References: <20141021222355.GC6656@marquez.int.rhx> <573592010.74725901.1413999659304.JavaMail.zimbra@redhat.com> <54483934.4060802@redhat.com> <20141023072636.GB6158@marquez.int.rhx> <1558759660.1356853.1414474824972.JavaMail.zimbra@redhat.com> X-ASG-Orig-Subj: Re: [pcp] hinv.* In-Reply-To: <1558759660.1356853.1414474824972.JavaMail.zimbra@redhat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl2.internode.on.net[150.101.137.129] X-Barracuda-Start-Time: 1414475154 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10981 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- On 28/10/14 16:40, Nathan Scott wrote: > > Yeah, hmmm, thats a bit awkward - I don't really want us to have to > explicitly list each hinv metric in the logging configs. Not sure > what else we can do though. You can log hinv once (as we do today), _and_ add another clause to log hinv.something and hinv.otherthing with the default logging frequency. I think that is manageable, unless I'm misunderstanding the issue. From mgoodwin@redhat.com Tue Oct 28 00:50:00 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 2BEE17F96 for ; Tue, 28 Oct 2014 00:50:00 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 1613A8F8033 for ; Mon, 27 Oct 2014 22:49:59 -0700 (PDT) X-ASG-Debug-ID: 1414475397-04cbb070c5407fb0001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id zqdXF1djjK64anf1 (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Mon, 27 Oct 2014 22:49:57 -0700 (PDT) X-Barracuda-Envelope-From: mgoodwin@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9S5nsZJ013867 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Tue, 28 Oct 2014 01:49:54 -0400 Received: from [10.64.51.65] (vpn1-51-65.bne.redhat.com [10.64.51.65]) by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9S5nqAi029280; Tue, 28 Oct 2014 01:49:53 -0400 Message-ID: <544F2E80.9080604@redhat.com> Date: Tue, 28 Oct 2014 16:49:52 +1100 From: Mark Goodwin User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: Nathan Scott , Michele Baldessari CC: pcp@oss.sgi.com Subject: Re: [pcp] hinv.* References: <20141021222355.GC6656@marquez.int.rhx> <573592010.74725901.1413999659304.JavaMail.zimbra@redhat.com> <54483934.4060802@redhat.com> <20141023072636.GB6158@marquez.int.rhx> <1558759660.1356853.1414474824972.JavaMail.zimbra@redhat.com> X-ASG-Orig-Subj: Re: [pcp] hinv.* In-Reply-To: <1558759660.1356853.1414474824972.JavaMail.zimbra@redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1414475397 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On 10/28/2014 04:40 PM, Nathan Scott wrote: > Hi guys, > > ----- Original Message ----- >> [...] >> Thanks Mark, the patch works fine, I see the different values now. >> Given how dynamic CPUs can be these days, it might be nice to >> add the online/offline status (/sys/devices/system/cpu/cpu1/online) >> as well. > > Good idea - I'll hack that up tomorrow unless someone beats me to it. that could be opening a nice big juicy can of worms - would we also want the cpu instance domain to become dynamic? Or keep it how it is with online and offline CPUs included? >> The default logging here is "once" for hinv, so that might need tweaking as >> well. > > Yeah, hmmm, thats a bit awkward - I don't really want us to have to > explicitly list each hinv metric in the logging configs. Not sure > what else we can do though. The 'atop-summary' logger config already includes it : #+ tools/atop-summary:y:default: ## metrics sampled once by the atop command log advisory on default { hinv.cpu.clock hinv.ncpu mem.physmem } Regards -- Mark From mgoodwin@redhat.com Tue Oct 28 00:53:21 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 1983C7F96 for ; Tue, 28 Oct 2014 00:53:21 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 9B10BAC007 for ; Mon, 27 Oct 2014 22:53:20 -0700 (PDT) X-ASG-Debug-ID: 1414475599-04cbb070c8408190001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id KrAnYqwPTYYUex2A (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Mon, 27 Oct 2014 22:53:19 -0700 (PDT) X-Barracuda-Envelope-From: mgoodwin@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9S5rGS6032738 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Tue, 28 Oct 2014 01:53:17 -0400 Received: from [10.64.51.65] (vpn1-51-65.bne.redhat.com [10.64.51.65]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9S5rDKe030745; Tue, 28 Oct 2014 01:53:14 -0400 Message-ID: <544F2F49.9030404@redhat.com> Date: Tue, 28 Oct 2014 16:53:13 +1100 From: Mark Goodwin User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: Nathan Scott , Michele Baldessari CC: pcp@oss.sgi.com Subject: Re: [pcp] hinv.* References: <20141021222355.GC6656@marquez.int.rhx> <573592010.74725901.1413999659304.JavaMail.zimbra@redhat.com> <54483934.4060802@redhat.com> <20141023072636.GB6158@marquez.int.rhx> <1558759660.1356853.1414474824972.JavaMail.zimbra@redhat.com> <544F2E80.9080604@redhat.com> X-ASG-Orig-Subj: Re: [pcp] hinv.* In-Reply-To: <544F2E80.9080604@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1414475599 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On 10/28/2014 04:49 PM, Mark Goodwin wrote: > The 'atop-summary' logger config already includes it : > > #+ tools/atop-summary:y:default: > ## metrics sampled once by the atop command > log advisory on default { > hinv.cpu.clock > hinv.ncpu > mem.physmem > } actually scratch that, it logs it once by default. My runtime config has been tweaked to log it every default seconds. Regards From nscott@redhat.com Tue Oct 28 00:53:36 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 9E0217F96 for ; Tue, 28 Oct 2014 00:53:36 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 391C0AC006 for ; Mon, 27 Oct 2014 22:53:36 -0700 (PDT) X-ASG-Debug-ID: 1414475614-04cbb070c84081c0001-S8gJnT Received: from mx6-phx2.redhat.com (mx6-phx2.redhat.com [209.132.183.39]) by cuda.sgi.com with ESMTP id KkfBn6H6eK9jFubW (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Mon, 27 Oct 2014 22:53:34 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.39 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx6-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9S5mX3R022479; Tue, 28 Oct 2014 01:48:33 -0400 Date: Tue, 28 Oct 2014 01:48:33 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Ken McDonell Cc: pcp@oss.sgi.com Message-ID: <1512922869.1359434.1414475313452.JavaMail.zimbra@redhat.com> In-Reply-To: <544F2DD5.40108@internode.on.net> References: <20141021222355.GC6656@marquez.int.rhx> <573592010.74725901.1413999659304.JavaMail.zimbra@redhat.com> <54483934.4060802@redhat.com> <20141023072636.GB6158@marquez.int.rhx> <1558759660.1356853.1414474824972.JavaMail.zimbra@redhat.com> <544F2DD5.40108@internode.on.net> Subject: Re: [pcp] hinv.* MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] hinv.* Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: hinv.* Thread-Index: 2CW8i9UdgqigTocQp7wlhjjRyQirWA== X-Barracuda-Connect: mx6-phx2.redhat.com[209.132.183.39] X-Barracuda-Start-Time: 1414475614 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10982 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... ----- Original Message ----- > On 28/10/14 16:40, Nathan Scott wrote: > > > > Yeah, hmmm, thats a bit awkward - I don't really want us to have to > > explicitly list each hinv metric in the logging configs. Not sure > > what else we can do though. > > You can log hinv once (as we do today), _and_ add another clause to log > hinv.something and hinv.otherthing with the default logging frequency. > > I think that is manageable, unless I'm misunderstanding the issue. > Ah, yes, that'll do the trick - thanks Ken. cheers. -- Nathan From kenj@internode.on.net Tue Oct 28 01:39:16 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 315CA7F6B for ; Tue, 28 Oct 2014 01:39:16 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 0E6E78F8035 for ; Mon, 27 Oct 2014 23:39:15 -0700 (PDT) X-ASG-Debug-ID: 1414478351-04cb6c2efc3c81e0001-S8gJnT Received: from ipmail06.adl2.internode.on.net (ipmail06.adl2.internode.on.net [150.101.137.129]) by cuda.sgi.com with ESMTP id GVOrISqv9eCD9AEG for ; Mon, 27 Oct 2014 23:39:11 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.129 Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail06.adl2.internode.on.net with ESMTP; 28 Oct 2014 17:08:43 +1030 Message-ID: <544F3A35.2020909@internode.on.net> Date: Tue, 28 Oct 2014 17:39:49 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: Re: [pcp] pcp updates: merges (mgoodwin, fche, kenj, self), qa References: <573773476.1355528.1414474652048.JavaMail.zimbra@redhat.com> X-ASG-Orig-Subj: Re: [pcp] pcp updates: merges (mgoodwin, fche, kenj, self), qa In-Reply-To: <573773476.1355528.1414474652048.JavaMail.zimbra@redhat.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl2.internode.on.net[150.101.137.129] X-Barracuda-Start-Time: 1414478351 X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10982 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- On 28/10/14 16:37, Nathan Scott wrote: > Changes committed to git://git.pcp.io/pcp.git dev > > Mark Goodwin (1): > pmdalinux: fix hinv.cpu.clock refresh logic so value can change > > Ken McDonell (8): > Makepkgs - debian build, don't try to sign packages > qa/542 - timezone is sometimes 4 chars long, e.g. AEST > qa/qa_hosts.master - simplify kenj sections > qa/069 - add diags for failure cases > qa/645 - use fqdn hostnames and re-instate hyphen in hostname > qa/common.avahi - domain name games > sample pmda - make sample.const_rate.value available to pmstore > qa/260 - refactor choice of metrics > > Nathan Scott (9): > build: extend Makepkgs rpm path trickery to non-RedHat platforms > build: update mac gitignore for generated file > coverity: a handful of trivial string-related improvements > build: rework the web-manager split to just javascript files now > build: switch fedora spec from web-manager to webjs model > build: fix fallout from a makefile LSRCFILES cull > pmwebd qa: fix qa/780 - not updated since -f option removed from pmwebd > pmwebd qa: allow qa/660 to handle missing jsdemos directory > pmwebd qa: add qa/782 to verify options file back-compatibility > > Frank Ch. Eigler (62): > pmwebd: make foreground mode less magic > whitespace: gnu-indent pmwebd to keep it looking spic & span > pmwebd build: link in cairo graphics library > pmwebd: TODO++ > pmwebd: pmgraphite rough-in snapshot > pmwebd: initial conversion to C++ > pmwebd: eliminate mhdb_* miniapi in favour of ordinary c++ stringstreams > pmwebapi: snapshot for web graphing > pmwebapi: switched to astyle as indenter > pmwebapi: signs of life snapshot: graphlot /rawdata data supplying > pmwebapi: snapshot, grafana signs of life > pmwebd: grafana snapshot, TZ=UTC & absolute timestamp parsing > pmwebd: metric enumeration snapshot > pmwebd: remind self of numerical methods, loss of significance > pmwebd: respect /graphite/render maxDataPoints more vigorously > pmwebd: merge /rawdata & /render&format=json modes > pmwebd: graphlot-invoked-from-graphite-browser js improvements > pmwebd: metric enumeration coup de grace: add instance domains too > pmwebd: handle metric/query format=completer > pmwebd: use isnanf() to make double-sure to detect float (vs. doubles) > pmwebd: add Access-Control-Allow-Origin: * headers to all json responses > pmwebd: respond to pending exits (^C) more rapidly > pmwebd: add periodic client-stats notification > pmwebd: dumpstats tweak > pmwebd: tweak client dumpstats more > pmwebd: reindent the lot > pmwebd: beginnings of cairo server-side rendering of graphs > pmwebd: graphite png rendering: fix rendering of null sequences > pmwebd: graphite color name parsing > pmwebd: draw legend, title > pmwebd graphite: draw grid, axis labels too > pmwebd graphite: cairo rendering: fix time axis > pmwebd graphite: during pmns enumeration, filter out non-numerics > pmwebd graphite: support graphite metric-regex-search > pmwebd: correct result mime/type for file server > pmwebd graphite: add graphlot metric-box live-search too > pmwebd graphite, multithread prep > pmwebd graphite: add -M multithreading option > pmwebd: tweak verbosity; add png cache-suppression headers > pmwebd: log http query params tighter for -vv > pmwebd graphite png rendering: intelligent legend/curve sorting-by-visibility > pmwebd: correct a memory leak > pmwebd graphite archive processing: avoid out-of-range times > pmwebapi classic pmapi: correct a c++ conversion problem, caught by old 660 testcase > pmwebd: fix typo in new rc_pmwebd option loader > pmwebd: whitespace update > pmwebd graphite: tweak messages, y-axis rounding, time parsing > pmwebd config sample: add commented-out $PCP_DERIVED_CONFIG clause > pmwebd: add $LIB_FOR_PTHREADS > pmwebd diagnostics: make -vvv Just Right for graphite non-empty curves > pmwebd: add gcov (test-coverage) build option; enlarge 660 test case > pmwebd qa++, now gcov-guided > pmwebd qa: now with ipv6 output too > pmwebd: switch to pkgconfig configury to extract cflags/libs paths > pmwebd: extend precision for some floating point outputs > pmwebd: eliminate compiler warnings about unused variables > pmwebapi: ACAO and QA improvements > pmwebd: support old $PMWEBDOPTS configuration file format > pmwebd: correct graphite cairo status formatting in startup message > pmmgr: handle case of pm{ie,logger} daemon that refuses SIGTERM > > Makepkgs | 45 > build/mac/.gitignore | 1 > build/rpm/GNUmakefile | 4 > build/rpm/fedora.spec | 46 > build/rpm/pcp.spec.in | 131 > build/tar/postinstall.tail | 4 > configure | 235 + > configure.ac | 57 > debian/GNUmakefile | 35 > debian/control | 25 > debian/control.master | 17 > debian/control.webapi | 8 > debian/pcp-manager.dirs | 2 > debian/pcp-manager.install | 11 > debian/pcp-manager.postinst | 14 > debian/pcp-manager.postrm | 6 > debian/pcp-manager.prerm | 8 > debian/pcp-webapi.dirs | 2 > debian/pcp-webapi.install | 5 > debian/pcp-webapi.postinst | 14 > debian/pcp-webapi.postrm | 6 > debian/pcp-webapi.prerm | 8 > debian/rules | 13 > man/man1/pmmgr.1 | 340 + > man/man1/pmwebd.1 | 285 + > man/man3/pmwebapi.3 | 263 + > qa/.gitignore | 1 > qa/069 | 8 > qa/260 | 73 > qa/260.out | 352 - > qa/518 | 2 > qa/542 | 1 > qa/645 | 2 > qa/645.out | 4 > qa/660 | 254 + > qa/660.out.4 | 2176 +++++++++++ > qa/660.out.46 | 2457 ++++++++++++ > qa/666 | 5 > qa/780 | 106 > qa/780.out | 8 > qa/782 | 56 > qa/782.out | 13 > qa/GNUmakefile | 2 > qa/common.avahi | 7 > qa/common.webapi | 39 > qa/group | 7 > qa/qa_hosts.master | 42 > qa/src/test_webapi.python | 158 > src/GNUmakefile | 2 > src/include/builddefs.in | 8 > src/include/pcp.conf.in | 8 > src/libpcp/src/logmeta.c | 2 > src/pmcd/rc-proc.sh | 4 > src/pmchart/view.cpp | 8 > src/pmdas/linux/proc_cpuinfo.c | 2 > src/pmdas/sample/src/sample.c | 4 > src/pmdumptext/pmdumptext.cpp | 2 > src/pmmgr/.gitignore | 2 > src/pmmgr/GNUmakefile | 59 > src/pmmgr/TODO | 12 > src/pmmgr/config/GNUmakefile | 40 > src/pmmgr/config/README | 13 > src/pmmgr/config/target-discovery.example-avahi | 1 > src/pmmgr/pmmgr.cxx | 1305 ++++++ > src/pmmgr/pmmgr.h | 133 > src/pmmgr/pmmgr.options | 27 > src/pmmgr/pmmgr.service.in | 14 > src/pmmgr/rc_pmmgr | 296 + > src/pmns/pmnsdel.c | 12 > src/pmwebapi/.gitignore | 4 > src/pmwebapi/.indent.pro | 4 > src/pmwebapi/GNUmakefile | 109 > src/pmwebapi/TODO | 38 > src/pmwebapi/main.c | 1896 ++++----- > src/pmwebapi/main.cxx | 1886 ++++++--- > src/pmwebapi/pmgraphite.c | 122 > src/pmwebapi/pmgraphite.cxx | 4676 +++++++++++++++++------- > src/pmwebapi/pmresapi.c | 452 +- > src/pmwebapi/pmresapi.cxx | 462 +- > src/pmwebapi/pmwebapi.c | 4206 ++++++++++----------- > src/pmwebapi/pmwebapi.cxx | 3369 +++++++++++------ > src/pmwebapi/pmwebapi.h | 485 +- > src/pmwebapi/pmwebd.options | 88 > src/pmwebapi/pmwebd.service.in | 14 > src/pmwebapi/rc_pmwebd | 360 + > src/pmwebapi/util.c | 264 - > src/pmwebapi/util.cxx | 296 + > src/pmwebapi/util.h | 70 > 88 files changed, 20899 insertions(+), 7214 deletions(-) > > _______________________________________________ > pcp mailing list > pcp@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/pcp > On ubuntu I'm seeing build errors ... Makepkgs dies with === pmwebapi === gcc -fPIC -fno-strict-aliasing -D_GNU_SOURCE -fstack-protector-all -D_FORTIFY_SOURCE=2 -Wextra -fPIE -Wall -O2 -g -DPCP_DEBUG -DPCP_VERSION=\"3.10.0\" -I../../src/include -I../../src/include/pcp -c -o main.o main.c In file included from main.c:17:0: pmwebapi.h:21:18: fatal error: string: No such file or directory #include ^ compilation terminated. Strangely, make by hand, as in, $ cd src; make dies differently in the python code ... which suggests another problem related to perhaps some change in the configure plumbing between Makepkgs and qa/admin/myconfiguire (yep, see below, looks like Makepkgs and qa/admin/myconfigure don't agree on python3 or not) kenj@bozo:~/src/pcp/src/python$ make python3 setup.py build_ext --include-dirs=../../src/include --library-dirs=../../src/libpcp/src:../../src/libpcp_pmda/src:../../src/libpcp_gui/src:../../src/libpcp_import/src:../../src/libpcp_mmv/src running build_ext building 'cpmapi' extension x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -I../../src/include -I/usr/include/python3.4m -c pmapi.c -o build/temp.linux-x86_64-3.4/pmapi.o pmapi.c:27:20: fatal error: Python.h: No such file or directory #include ^ compilation terminated. error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 configure differences ... kenj@bozo:~/src/pcp/build/deb/pcp-3.10.0/src/include$ diff -r . ~/src/pcp/src/include diff -r ./builddefs /home/kenj/src/pcp/src/include/builddefs 36c36 < PACKAGE_CONFIGURE ?= --with-systemd=no --with-python3=no --- > PACKAGE_CONFIGURE ?= 226,227c226,227 < ENABLE_PYTHON = true < ENABLE_PYTHON3 = false --- > ENABLE_PYTHON = false > ENABLE_PYTHON3 = true diff -r ./builddefs.install /home/kenj/src/pcp/src/include/builddefs.install 36c36 < PACKAGE_CONFIGURE ?= --with-systemd=no --with-python3=no --- > PACKAGE_CONFIGURE ?= 226,227c226,227 < ENABLE_PYTHON = true < ENABLE_PYTHON3 = false --- > ENABLE_PYTHON = false > ENABLE_PYTHON3 = true From nscott@redhat.com Tue Oct 28 01:48:06 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 9568E7F73 for ; Tue, 28 Oct 2014 01:48:06 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 2227EAC006 for ; Mon, 27 Oct 2014 23:48:02 -0700 (PDT) X-ASG-Debug-ID: 1414478880-04cb6c2efa3c89a0001-S8gJnT Received: from mx3-phx2.redhat.com (mx3-phx2.redhat.com [209.132.183.24]) by cuda.sgi.com with ESMTP id hcJlvrNMai93a81B (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Mon, 27 Oct 2014 23:48:01 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.24 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx3-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s9S6lu9B020185; Tue, 28 Oct 2014 02:47:56 -0400 Date: Tue, 28 Oct 2014 02:47:55 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Ken McDonell Cc: pcp@oss.sgi.com Message-ID: <1774911884.1381949.1414478875947.JavaMail.zimbra@redhat.com> In-Reply-To: <544F3A35.2020909@internode.on.net> References: <573773476.1355528.1414474652048.JavaMail.zimbra@redhat.com> <544F3A35.2020909@internode.on.net> Subject: Re: [pcp] pcp updates: merges (mgoodwin, fche, kenj, self), qa MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] pcp updates: merges (mgoodwin, fche, kenj, self), qa Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: pcp updates: merges (mgoodwin, fche, kenj, self), qa Thread-Index: ncxoRiTCkk4M+f5+1SJ/yEz6DqKs9w== X-Barracuda-Connect: mx3-phx2.redhat.com[209.132.183.24] X-Barracuda-Start-Time: 1414478880 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10983 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... ----- Original Message ----- > > On ubuntu I'm seeing build errors ... > > Makepkgs dies with > > === pmwebapi === > gcc -fPIC -fno-strict-aliasing -D_GNU_SOURCE -fstack-protector-all > -D_FORTIFY_SOURCE=2 -Wextra -fPIE -Wall -O2 -g -DPCP_DEBUG > -DPCP_VERSION=\"3.10.0\" -I../../src/include -I../../src/include/pcp -c -o > main.o main.c > In file included from main.c:17:0: > pmwebapi.h:21:18: fatal error: string: No such file or directory > #include > ^ This is a problem related to configure not being run on this build. On a clean build, you should see "g++" being used over gcc (above) and the C++ headers are then all found. I saw this here too, for a Debian build. > pmapi.c:27:20: fatal error: Python.h: No such file or directory > #include > ^ > compilation terminated. > error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 Yep, this is configure too - python support should have been switched off auto-magically in the build (IIRC) if no devel headers were found of appropriate version levels. > configure differences ... > > kenj@bozo:~/src/pcp/build/deb/pcp-3.10.0/src/include$ diff -r . Try "make distclean" and/or "rm -fr build/deb" and kick a fresh build off? cheers. -- Nathan From michele@acksyn.org Tue Oct 28 02:51:43 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 134527F90 for ; Tue, 28 Oct 2014 02:51:43 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 9394AAC006 for ; Tue, 28 Oct 2014 00:51:42 -0700 (PDT) X-ASG-Debug-ID: 1414482697-04cbb070c640d7c0001-S8gJnT Received: from palahniuk.acksyn.org (palahniuk.acksyn.org [5.9.7.26]) by cuda.sgi.com with ESMTP id LsHh5i8brn2RHa4F for ; Tue, 28 Oct 2014 00:51:37 -0700 (PDT) X-Barracuda-Envelope-From: michele@acksyn.org X-Barracuda-Apparent-Source-IP: 5.9.7.26 Received: from localhost (localhost [127.0.0.1]) by palahniuk.acksyn.org (Postfix) with ESMTP id 935C628043; Tue, 28 Oct 2014 03:51:36 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=acksyn.org; h= user-agent:in-reply-to:content-disposition:content-type :content-type:mime-version:references:message-id:subject:subject :from:from:date:date:received:received; s=2010; t=1414482696; bh=07dlGCrS7nyIezIspOOxqEqL/xv+A+9/q+1SI3Tt7fo=; b=XIl6DbOMUONT SYTgivNt/0iZlVmnWP/hG6zKy8ZLJkwFokXkEM4TFdp6vAwjckHp6V/kAI9NuuX9 m4ihm4J4IUDgDLivg1X9ZkSP8sXCqZU6vHXRuPelZthAAWKTZ5+nXdrM2DrpURij d/VuvVmAJEmlb9hWHz2TmA7EUWO4g6c= Received: from palahniuk.acksyn.org ([127.0.0.1]) by localhost (mail.acksyn.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id sLU52ikEIFgm; Tue, 28 Oct 2014 03:51:36 -0400 (EDT) Received: from localhost (host165-178-dynamic.6-79-r.retail.telecomitalia.it [79.6.178.165]) by palahniuk.acksyn.org (Postfix) with ESMTPSA id BC05A201F1; Tue, 28 Oct 2014 03:51:35 -0400 (EDT) Date: Tue, 28 Oct 2014 08:51:34 +0100 From: Michele Baldessari To: Nathan Scott Cc: Mark Goodwin , pcp@oss.sgi.com Subject: Re: [pcp] hinv.* Message-ID: <20141028075134.GB6825@marquez.int.rhx> X-ASG-Orig-Subj: Re: [pcp] hinv.* References: <20141021222355.GC6656@marquez.int.rhx> <573592010.74725901.1413999659304.JavaMail.zimbra@redhat.com> <54483934.4060802@redhat.com> <20141023072636.GB6158@marquez.int.rhx> <1558759660.1356853.1414474824972.JavaMail.zimbra@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1558759660.1356853.1414474824972.JavaMail.zimbra@redhat.com> User-Agent: Mutt/1.5.21 (2012-12-30) X-Barracuda-Connect: palahniuk.acksyn.org[5.9.7.26] X-Barracuda-Start-Time: 1414482697 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10984 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature Hi Nathan, On Tue, Oct 28, 2014 at 01:40:24AM -0400, Nathan Scott wrote: > > The default logging here is "once" for hinv, so that might need tweaking as > > well. > > Yeah, hmmm, thats a bit awkward - I don't really want us to have to > explicitly list each hinv metric in the logging configs. Not sure > what else we can do though. Good point. What would be disadvantage of simply logging hinv.* all the time? Too big of an increase in archive files? Or should we aim for a split of the hinv.* hierarchy into "extremely unlikely to ever change" and "likely to change" metrics? Or we just leave things as is and be it ;) Cheers, Michele -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D From jaehyuk.lee@boc.cn Tue Oct 28 03:28:18 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.9 required=5.0 tests=MISSING_HEADERS autolearn=no version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id B74FE7F6C for ; Tue, 28 Oct 2014 03:28:18 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 63DE8AC008 for ; Tue, 28 Oct 2014 01:28:18 -0700 (PDT) X-ASG-Debug-ID: 1414484893-04bdf038d14b38c0001-S8gJnT Received: from mail.mag-net.com (mail.mag-net.com [184.71.102.58]) by cuda.sgi.com with ESMTP id DyByC1JGbX6LTVKD (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 28 Oct 2014 01:28:14 -0700 (PDT) X-Barracuda-Envelope-From: jaehyuk.lee@boc.cn X-Barracuda-Apparent-Source-IP: 184.71.102.58 Received: (qmail 41865 invoked by uid 80); 28 Oct 2014 08:10:23 -0000 Cc: recipient list not shown: ; Received: from 67.221.255.94 (SquirrelMail authenticated user gilchris@mag-net.com) by webmail.mag-net.com with HTTP; Tue, 28 Oct 2014 01:10:23 -0700 Message-ID: <374555dff31da8232e33d701dc951541.squirrel@webmail.mag-net.com> Date: Tue, 28 Oct 2014 01:10:23 -0700 Subject: Hello From: "Lee Jaehyuk" X-ASG-Orig-Subj: Hello Reply-To: jaehyuk.lee1@qq.com User-Agent: SquirrelMail/1.4.22 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-Barracuda-Connect: mail.mag-net.com[184.71.102.58] X-Barracuda-Start-Time: 1414484894 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-Spam-Score: 1.41 X-Barracuda-Spam-Status: No, SCORE=1.41 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC7_SA298e, MISSING_HEADERS X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10984 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 1.21 MISSING_HEADERS Missing To: header 0.20 BSF_SC7_SA298e Custom Rule SA298e Hello, Did you receive my last email From bounces@newsletter.my Tue Oct 28 03:34:16 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=RCVD_NUMERIC_HELO autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 109C37F6D for ; Tue, 28 Oct 2014 03:34:16 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id E335D304048 for ; Tue, 28 Oct 2014 01:34:12 -0700 (PDT) X-ASG-Debug-ID: 1414485244-04bdf038cf4b3b00001-S8gJnT Received: from smtp-85.newsletter.my (smtp-85.newsletter.my [210.1.224.85]) by cuda.sgi.com with ESMTP id 0cAuUMfBy1ESUCd5 for ; Tue, 28 Oct 2014 01:34:05 -0700 (PDT) X-Barracuda-Envelope-From: bounces@newsletter.my X-Barracuda-Apparent-Source-IP: 210.1.224.85 Received: from e.newsletter.my/ (smtp-83.newsletter.my [210.1.224.83]) by smtp-85.newsletter.my (Postfix) with ESMTP id 56DF581DCC for ; Tue, 28 Oct 2014 16:34:03 +0800 (MYT) Received: from 202.43.101.30 [202.43.101.30] by e.newsletter.my with HTTP; Tue, 28 Oct 2014 16:34:02 +0800 Date: Tue, 28 Oct 2014 16:34:03 +0800 To: pcp@oss.sgi.com From: eVOX Newsletter Reply-To: eVOX Newsletter Subject: New Generation Of PABX System For Enterprise Message-ID: X-ASG-Orig-Subj: New Generation Of PABX System For Enterprise X-Priority: 3 X-Mailer: PHPMailer 5.2.5 (https://github.com/Synchro/PHPMailer/) X-phpList-version: 3.0.8 X-MessageID: 49 X-ListMember: pcp@oss.sgi.com Precedence: bulk Bounces-To: bounces@newsletter.my List-Help: List-Unsubscribe: List-Subscribe: List-Owner: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 X-Barracuda-Connect: smtp-85.newsletter.my[210.1.224.85] X-Barracuda-Start-Time: 1414485244 X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 1.25 X-Barracuda-Spam-Status: No, SCORE=1.25 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=RCVD_NUMERIC_HELO, RCVD_NUMERIC_HELO_2 X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.10984 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 RCVD_NUMERIC_HELO Received: contains an IP address used for HELO 1.25 RCVD_NUMERIC_HELO_2 Received: contains an IP address used for HELO This message was sent to pcp@oss.sgi.com by sales@evox.my To change your details and to choose which lists to be subscribed to, visit your personal preferences page From brolley@redhat.com Tue Oct 28 10:54:12 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id A0B047F6F for ; Tue, 28 Oct 2014 10:54:12 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 2DCE1AC00A for ; Tue, 28 Oct 2014 08:54:11 -0700 (PDT) X-ASG-Debug-ID: 1414511647-04cb6c2efb3e0810001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id KIxKQPJ4RBOUbRUl (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 28 Oct 2014 08:54:07 -0700 (PDT) X-Barracuda-Envelope-From: brolley@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9SFs6mo028884 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Tue, 28 Oct 2014 11:54:06 -0400 Received: from [10.15.16.139] (dhcp-10-15-16-139.yyz.redhat.com [10.15.16.139]) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9SFs5DO011897 for ; Tue, 28 Oct 2014 11:54:05 -0400 Message-ID: <544FBC96.2020709@redhat.com> Date: Tue, 28 Oct 2014 11:56:06 -0400 From: Dave Brolley User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: PCP Mailing List Subject: PCP Updates: Fix typo in __pmLogLoadLabel() Content-Type: text/plain; charset=ISO-8859-1; format=flowed X-ASG-Orig-Subj: PCP Updates: Fix typo in __pmLogLoadLabel() Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1414511647 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 I noticed this while stepping through the code as part of my efforts to learn about the logging infrastructure. Almost forgot about it, but here it is now. brolley/dev branch of pcpfans Dave -------------------------------------------------------------- commit f1e6c1742c658c140006d424d17e7695edaea858 Author: Dave Brolley Date: Tue Oct 28 10:53:09 2014 -0400 Typo in __pmLogLoadLabel(): '0' --> '\0' Caused no incorrect behaviour, but did cause the list of compressed file suffixes to be traversed unnecessarily. From brolley@redhat.com Tue Oct 28 10:55:17 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 301D27F6F for ; Tue, 28 Oct 2014 10:55:17 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 1DF998F8064 for ; Tue, 28 Oct 2014 08:55:14 -0700 (PDT) X-ASG-Debug-ID: 1414511712-04cb6c2efc3e0900001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id P1xPF1UwhkFZdEsv (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 28 Oct 2014 08:55:13 -0700 (PDT) X-Barracuda-Envelope-From: brolley@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9SFtCvL018536 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Tue, 28 Oct 2014 11:55:12 -0400 Received: from [10.15.16.139] (dhcp-10-15-16-139.yyz.redhat.com [10.15.16.139]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9SFtBsn032396 for ; Tue, 28 Oct 2014 11:55:12 -0400 Message-ID: <544FBCD8.3040902@redhat.com> Date: Tue, 28 Oct 2014 11:57:12 -0400 From: Dave Brolley User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: PCP Mailing List Subject: PCP Updates: Incorrect pmLogIndom Record Description in pcp-archive(5) Content-Type: text/plain; charset=ISO-8859-1; format=flowed X-ASG-Orig-Subj: PCP Updates: Incorrect pmLogIndom Record Description in pcp-archive(5) Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1414511713 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Another one found during my PCP logging education ... According to my reading of the code (logmeta.c:299-323), the instance numbers and string table offsets do not alternate. They are grouped together separately. brolley/dev/ in pcpfans. Dave ------------------------------------------------------------ Author: Dave Brolley Date: Tue Oct 28 11:45:15 2014 -0400 Fix pmLogIndom record description in pcp-archive(5). All instance numbers are grouped together first followed by all the offsets into the string table. From brolley@redhat.com Tue Oct 28 12:46:22 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 98F7B7F7C for ; Tue, 28 Oct 2014 12:46:22 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 86B688F8037 for ; Tue, 28 Oct 2014 10:46:19 -0700 (PDT) X-ASG-Debug-ID: 1414518377-04cb6c2efb3e89f0001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id L2YbEpUo3EiRCz8Q (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 28 Oct 2014 10:46:18 -0700 (PDT) X-Barracuda-Envelope-From: brolley@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9SHkExU009248 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Tue, 28 Oct 2014 13:46:14 -0400 Received: from [10.15.16.139] (dhcp-10-15-16-139.yyz.redhat.com [10.15.16.139]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9SHkDlr006392; Tue, 28 Oct 2014 13:46:14 -0400 Message-ID: <544FD6DF.8060206@redhat.com> Date: Tue, 28 Oct 2014 13:48:15 -0400 From: Dave Brolley User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: Ken McDonell , pcp@oss.sgi.com Subject: Re: [pcp] user/group access control question References: <001f01cff196$66cf3e00$346dba00$@internode.on.net> X-ASG-Orig-Subj: Re: [pcp] user/group access control question In-Reply-To: <001f01cff196$66cf3e00$346dba00$@internode.on.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1414518378 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On 10/26/2014 11:30 PM, Ken McDonell wrote: > I think this one for Dave Brolley, but I'd welcome insight from any quarter. Not completely my domain, but I have worked in this area and don't mind having a look. > qa/944 seems to be failing in the client connecting to pmcd which is > returning an error code of -95. The problem seems to be in the logic around > DoCreds() where we never get to CheckAccountAccess() because the earlier > call to __pmSecureServerHandshake() fails, with, er, -95. > > I am not sure how things are supposed to work in this config setup, but the > patch below makes qa/944 pass on at least one of these failing platforms. > Note this is for the "not secure sockets" variant of the implementation of > pmSecureServerHandshake() (there are two implementations in the code). > > Could someone who knows please take a look and let me know if this is even > close to the "correct" way to fix this issue? > Looks like you made a good catch and your are definitely on the right track. The non-secure server implementation of __pmSecureServerHandshake() definitely needs to handle PDU_FLAG_CREDS_REQD. However we need one additional bit from the secure server implementation. Here is an updated patch for which PDU_FLAG_CREDS_REQD is still not supported unless the connection is from a unix domain socket. I'll test it here, but would appreciate it if you could test it in your failing environment. Nathan, can you please also review the updated patch? Thanks, Dave ------------------------------------------- diff --git a/src/libpcp/src/auxserver.c b/src/libpcp/src/auxserver.c index 498bac4..f863b58 100644 --- a/src/libpcp/src/auxserver.c +++ b/src/libpcp/src/auxserver.c @@ -867,7 +867,23 @@ __pmSecureServerHandshake(int fd, int flags, __pmHashCtl *attrs) (void)fd; (void)flags; (void)attrs; - return -EOPNOTSUPP; + + /* for things that require a secure server, return -EOPNOTSUPP */ + if ((flags & (PDU_FLAG_SECURE | PDU_FLAG_SECURE_ACK | PDU_FLAG_COMPRESS + | PDU_FLAG_AUTH)) != 0) + return -EOPNOTSUPP; + + /* CREDS_REQD is a special case that does not need a secure server, but + does require a unix domain socket in the absence of secure server + support. */ + if ((flags & PDU_FLAG_CREDS_REQD) != 0) { + if (__pmHashSearch(PCP_ATTR_USERID, attrs) != NULL) + return 0; /* unix domain socket */ + return -EOPNOTSUPP; + } + + /* otherwise the flags are not expected */ + return PM_ERR_IPC; } int From brolley@redhat.com Tue Oct 28 13:19:09 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 4BAFD7F86 for ; Tue, 28 Oct 2014 13:19:09 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 39FAF304039 for ; Tue, 28 Oct 2014 11:19:06 -0700 (PDT) X-ASG-Debug-ID: 1414520341-04cb6c2efb3ea360001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id yHZRnkelGSzpjAcS (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 28 Oct 2014 11:19:02 -0700 (PDT) X-Barracuda-Envelope-From: brolley@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9SIJ1aE001603 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Tue, 28 Oct 2014 14:19:01 -0400 Received: from [10.15.16.139] (dhcp-10-15-16-139.yyz.redhat.com [10.15.16.139]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9SIJ0qi031628 for ; Tue, 28 Oct 2014 14:19:01 -0400 Message-ID: <544FDE8E.6060403@redhat.com> Date: Tue, 28 Oct 2014 14:21:02 -0400 From: Dave Brolley User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: PCP Mailing List Subject: QA Failures: 660 and 752 Content-Type: text/plain; charset=ISO-8859-1; format=flowed X-ASG-Orig-Subj: QA Failures: 660 and 752 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1414520342 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Looks like 660 needs to filter IPv4 and IPv6 addresses for local host into something more generic. Looks like 752 failed because today is Tuesday :-p Dave --------------------------------------------------------------------------------- 660 [failed, exit status 1] - output mismatch (see 660.out.bad) 511a512,513 > [Tue Oct 28 18:16:12] pmwebd(18387) Info: context (web2222=pm0) created, local, permanent > [Tue Oct 28 18:16:12] pmwebd(18387) Info: context (web2223=pm1) created, host localhost, permanent 515,516c517,518 < * Trying 127.0.0.1... < * Connected to LOCALHOST (127.0.0.1) port 44323 (####) --- > * Trying ::1... > * Connected to LOCALHOST (::1) port 44323 (####) 533,534c535,536 < * Trying 127.0.0.1... < * Connected to LOCALHOST (127.0.0.1) port 44323 (####) --- > * Trying ::1... > * Connected to LOCALHOST (::1) port 44323 (####) 550,551c552,553 < * Trying 127.0.0.1... < * Connected to LOCALHOST (127.0.0.1) port 44323 (####) --- > * Trying ::1... > * Connected to LOCALHOST (::1) port 44323 (####) 586,601c588 < < Performance Co-Pilot < < Web applications: < < < --- > PMWEBD error, code -22: Invalid argument Check local PMCD is still alive ... PMDA probe: pminfo -h brolley-t530 -f sample.milliseconds PMDA probe: pminfo -h brolley-t530 -f sampledso.milliseconds PMDA probe: pminfo -h brolley-t530 -f simple.numfetch --------------------------------------------------------------- 752 - output mismatch (see 752.out.bad) 116,117c116,117 < "next tuesday" NEXT TUESDAY < "next tuesday" NEXT TUESDAY --- > "next tuesday" TODAY > "next tuesday" TODAY Check local PMCD is still alive ... PMDA probe: pminfo -h brolley-t530 -f sample.milliseconds PMDA probe: pminfo -h brolley-t530 -f sampledso.milliseconds PMDA probe: pminfo -h brolley-t530 -f simple.numfetch Failures: 660 752 Failed 2 of 3 tests From kenj@internode.on.net Tue Oct 28 14:33:48 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 6460B7F37 for ; Tue, 28 Oct 2014 14:33:48 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 382FF304039 for ; Tue, 28 Oct 2014 12:33:44 -0700 (PDT) X-ASG-Debug-ID: 1414524822-04bdf038d24d9090001-S8gJnT Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by cuda.sgi.com with ESMTP id 3kzVCB5GMMXSpsIZ for ; Tue, 28 Oct 2014 12:33:43 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.145 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AsIBAP7uT1R20YErPGdsb2JhbAANT4t3yzODIAKBNQEBAQEBBgEBAQE4hD0BAQEDAThABgsLGAkWDwkDAgECATEUEwgBAYg0sgiWAQEBAQcCAR+REBaENQEEuCqDIwEBAQ Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail06.adl6.internode.on.net with ESMTP; 29 Oct 2014 06:03:42 +1030 Message-ID: <544FEFDC.9040005@internode.on.net> Date: Wed, 29 Oct 2014 06:34:52 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: Re: [pcp] QA Failures: 660 and 752 References: <544FDE8E.6060403@redhat.com> X-ASG-Orig-Subj: Re: [pcp] QA Failures: 660 and 752 In-Reply-To: <544FDE8E.6060403@redhat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl6.internode.on.net[150.101.137.145] X-Barracuda-Start-Time: 1414524823 X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11000 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Dave, I'm really encouraged to see others reporting QA issues, so thanks for that. On 29/10/14 05:21, Dave Brolley wrote: > Looks like 660 needs to filter IPv4 and IPv6 addresses for local host > into something more generic. Yep, but isn't the "PMWEBD error, code -22: Invalid argument" instead of html more significant? > Looks like 752 failed because today is Tuesday :-p I've been wrestling with 752 for sometime and not got to the bottom of the "fails on Tuesday" issue ... leave this one with me, unless someone else has a bright idea and choose to jump in. From nscott@redhat.com Tue Oct 28 16:31:06 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id C942A7F37 for ; Tue, 28 Oct 2014 16:31:06 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 6415AAC007 for ; Tue, 28 Oct 2014 14:31:03 -0700 (PDT) X-ASG-Debug-ID: 1414531858-04bdf038d14e25b0001-S8gJnT Received: from mx5-phx2.redhat.com (mx5-phx2.redhat.com [209.132.183.37]) by cuda.sgi.com with ESMTP id hvhpl3iMf30BvT5I (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 28 Oct 2014 14:30:58 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.37 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx5-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9SLUhGU015635; Tue, 28 Oct 2014 17:30:43 -0400 Date: Tue, 28 Oct 2014 17:30:43 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Ken McDonell , Dave Brolley Cc: pcp@oss.sgi.com Message-ID: <514153069.2165300.1414531843252.JavaMail.zimbra@redhat.com> In-Reply-To: <544FEFDC.9040005@internode.on.net> References: <544FDE8E.6060403@redhat.com> <544FEFDC.9040005@internode.on.net> Subject: Re: [pcp] QA Failures: 660 and 752 MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] QA Failures: 660 and 752 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: QA Failures: 660 and 752 Thread-Index: UKqpx5HCE7ZOaeCEiZfCdLmNs6oZsQ== X-Barracuda-Connect: mx5-phx2.redhat.com[209.132.183.37] X-Barracuda-Start-Time: 1414531858 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11003 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... ----- Original Message ----- > Dave, > > I'm really encouraged to see others reporting QA issues, so thanks for that. +1 > On 29/10/14 05:21, Dave Brolley wrote: > > Looks like 660 needs to filter IPv4 and IPv6 addresses for local host > > into something more generic. > > Yep, but isn't the "PMWEBD error, code -22: Invalid argument" instead of > html more significant? Yes. This test needs to be split into several tests now, as the presence of jsdemos below $PCP_SHARE_DIR is not guaranteed (it's possible you have an old copy of the demos there Dave? - there's a notrun guard at the head of the test, which must be succeeding for you). Either way, a bit of test refactoring is needed here - I'll push some changes in soon. I'm seeing also some output dependence on microhttpd versioning on some hosts, so we need a bit more filtering here too. > > Looks like 752 failed because today is Tuesday :-p > > I've been wrestling with 752 for sometime and not got to the bottom of > the "fails on Tuesday" issue ... leave this one with me, unless someone > else has a bright idea and choose to jump in. FWIW, I'm seeing this too - took a brief look yesterday but ran outta time before diagnosing it. cheers. -- Nathan From brolley@redhat.com Tue Oct 28 16:38:24 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 2AC077F37 for ; Tue, 28 Oct 2014 16:38:24 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id EF6B8304048 for ; Tue, 28 Oct 2014 14:38:20 -0700 (PDT) X-ASG-Debug-ID: 1414532299-04bdf038d24e2fd0001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 92mICt5Gwr9jAeRd (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 28 Oct 2014 14:38:19 -0700 (PDT) X-Barracuda-Envelope-From: brolley@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9SLcEUB001184 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Tue, 28 Oct 2014 17:38:15 -0400 Received: from [10.15.16.139] (dhcp-10-15-16-139.yyz.redhat.com [10.15.16.139]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9SL1aq3002930; Tue, 28 Oct 2014 17:01:37 -0400 Message-ID: <545004AA.1040702@redhat.com> Date: Tue, 28 Oct 2014 17:03:38 -0400 From: Dave Brolley User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: Ken McDonell , pcp@oss.sgi.com Subject: Re: [pcp] user/group access control question References: <001f01cff196$66cf3e00$346dba00$@internode.on.net> <544FD6DF.8060206@redhat.com> X-ASG-Orig-Subj: Re: [pcp] user/group access control question In-Reply-To: <544FD6DF.8060206@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1414532299 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On 10/28/2014 01:48 PM, Dave Brolley wrote: > Here is an updated patch for which PDU_FLAG_CREDS_REQD is still not > supported unless the connection is from a unix domain socket. > > I'll test it here, but would appreciate it if you could test it in > your failing environment. The updated patch passes QA for me with no regressions. Dave From kenj@internode.on.net Tue Oct 28 18:33:17 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 11F787F37 for ; Tue, 28 Oct 2014 18:33:17 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id E4D2D8F8035 for ; Tue, 28 Oct 2014 16:33:13 -0700 (PDT) X-ASG-Debug-ID: 1414539188-04cbb070c7442850001-S8gJnT Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by cuda.sgi.com with ESMTP id 5BRqXi3SxUrPYVi7 for ; Tue, 28 Oct 2014 16:33:08 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.145 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AsIBAIcmUFR20YErPGdsb2JhbAANT4t3yziDIAKBMwEBAQEBBgEBAQE4hD0BAQEDAScRQAYLCxgJFg8JAwIBAgExFAYBDAgBAYg0r2mVewEBAQEGAQEBAQEdkRCESwEEj22PLINJjTuIDYMjAQEB Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail06.adl6.internode.on.net with ESMTP; 29 Oct 2014 10:03:07 +1030 Message-ID: <545027F9.2020205@internode.on.net> Date: Wed, 29 Oct 2014 10:34:17 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: Dave Brolley , pcp@oss.sgi.com Subject: Re: [pcp] user/group access control question References: <001f01cff196$66cf3e00$346dba00$@internode.on.net> <544FD6DF.8060206@redhat.com> X-ASG-Orig-Subj: Re: [pcp] user/group access control question In-Reply-To: <544FD6DF.8060206@redhat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl6.internode.on.net[150.101.137.145] X-Barracuda-Start-Time: 1414539188 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.50 X-Barracuda-Spam-Status: No, SCORE=0.50 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=FB_YOUR_REFI X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11006 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.50 FB_YOUR_REFI BODY: Phrase: Your refi On 29/10/14 04:48, Dave Brolley wrote: > ... > Looks like you made a good catch and your are definitely on the right > track. The non-secure server implementation of > __pmSecureServerHandshake() definitely needs to handle > PDU_FLAG_CREDS_REQD. However we need one additional bit from the secure > server implementation. Here is an updated patch for which > PDU_FLAG_CREDS_REQD is still not supported unless the connection is from > a unix domain socket. Thanks Dave. First I modified qa/944 to add another test to explicitly use a tcp connection, expecting this to pass without your refinement to the patch ... it fails with PM_ERR_PERMISSION. Hmm ... several iterations and a bunch of additional diagnostics later I discover that from DoCreds() we sail through __pmSecureServerHandshake() and then CheckAccountAccess() is passing in the unix domain socket case and failing in the tcp socket case because cp->attrs is not null in the first case, but is null in the second case ... more digging and I find that much earlier in CheckNewClient() we have this guard if (sts >= 0 && family == AF_UNIX) in front of __pmServerSetLocalCreds(cp->fd, &cp->attrs) So it is working, but that is kinda fortuitous. I've added your check (to be sure, to be sure, and in case someone else ever calls __pmSecureServerHandshake() on a different code path, and the test is effectively the same test that was being done later in CheckAccountAccess()), but as expected (after the analysis) it does not change the behaviour ... 8^)> > I'll test it here, but would appreciate it if you could test it in your > failing environment. > > Nathan, can you please also review the updated patch? I have this change in the context of more diagnostics and the extensions to qa/944, so I'll push the commit if everyone is OK with that. From modesee@outlook.fr Wed Oct 29 05:41:56 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: * X-Spam-Status: No, score=1.2 required=5.0 tests=DRUGS_MUSCLE, FREEMAIL_FORGED_REPLYTO,HTML_MESSAGE,LOTS_OF_MONEY autolearn=no version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 06F1A7F3F for ; Wed, 29 Oct 2014 05:41:56 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 78719AC008 for ; Wed, 29 Oct 2014 03:41:55 -0700 (PDT) X-ASG-Debug-ID: 1414579310-04cb6c2efc413f00001-S8gJnT Received: from BLU004-OMC2S27.hotmail.com (blu004-omc2s27.hotmail.com [65.55.111.102]) by cuda.sgi.com with ESMTP id EhEn7XkNFRLTLTN3 (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 29 Oct 2014 03:41:50 -0700 (PDT) X-Barracuda-Envelope-From: modesee@outlook.fr X-Barracuda-Apparent-Source-IP: 65.55.111.102 Received: from BLU169-W112 ([65.55.111.71]) by BLU004-OMC2S27.hotmail.com over TLS secured channel with Microsoft SMTPSVC(7.5.7601.22751); Wed, 29 Oct 2014 03:41:50 -0700 X-TMN: [+OELquAKSUnUhjUei1t6WR/D32s8yL7w] X-Originating-Email: [modesee@outlook.fr] Message-ID: Content-Type: multipart/alternative; boundary="_21ef159c-260b-4004-98a2-4a705c7f09f1_" Reply-To: From: modese dossongui Subject: De Modese Dossongui, Date: Wed, 29 Oct 2014 11:41:50 +0100 X-ASG-Orig-Subj: De Modese Dossongui, Importance: Normal MIME-Version: 1.0 X-OriginalArrivalTime: 29 Oct 2014 10:41:50.0178 (UTC) FILETIME=[F2245420:01CFF364] X-Barracuda-Connect: blu004-omc2s27.hotmail.com[65.55.111.102] X-Barracuda-Start-Time: 1414579310 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-BRTS-Evidence: dd5adfa8efd5ee3fc0270ead950715bb-2001-htm X-Barracuda-Spam-Score: 1.21 X-Barracuda-Spam-Status: No, SCORE=1.21 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=DRUGS_MUSCLE, HTML_MESSAGE, MISSING_HEADERS, TO_CC_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11017 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 1.21 MISSING_HEADERS Missing To: header 0.00 HTML_MESSAGE BODY: HTML included in message 0.00 TO_CC_NONE No To: or Cc: header 0.00 DRUGS_MUSCLE Refers to a muscle relaxant To: undisclosed-recipients:; --_21ef159c-260b-4004-98a2-4a705c7f09f1_ Content-Type: text/plain; charset="iso-8859-15" Content-Transfer-Encoding: quoted-printable De Modese Dossongui=2C Eu sou deputada=2C Modese Dossongui a =FAnica filha do falecido Sr. e Sra. = Bazose Dossongui. Meu pai era um proeminente comerciante de ouro e cacau. A= ntes da morte de meu pai em um hospital particular aqui em Abidjan=2C ele m= e disse que ele tem a soma de tr=EAs milh=F5es e setecentos mil Euros =A4 3= .700.000. deixado em conta fixo / suspense em um dos banco principal aqui= =2C que ele usou o meu nome como sua =FAnica filha para o parente mais pr= =F3ximo em dep=F3sito do fundo=2C Ele tamb=E9m me explicou que era por causa dessa riqueza que ele foi envene= nado por seus colegas de trabalho que eu deveria procurar um parceiro estra= ngeiro em umaPa=EDs de minha escolha onde vou transferir esse dinheiro e us= =E1-lo para fins de investimento=2C tais como a gest=E3o de bens im=F3veis = ou de gest=E3o hoteleira. Querido=2C eu estou procurando honrosamente sua a= ssist=EAncia nas seguintes formas: (1) Para fornecer uma conta banc=E1ria e= m que o dinheiro seria transferido para. (2) Para servir como um guardi=E3o= deste fundo desde que eu sou apenas 20 anos. (3) Para fazer o arranjo para= me para vir para o seu pa=EDs para continuar a minha educa=E7=E3o e para g= arantir uma autoriza=E7=E3o de resid=EAncia em seu pa=EDs. Al=E9m disso=2C = querida=2C eu estou disposto a oferecer-lhe 15% do valor total como compens= a=E7=E3o pelo seu esfor=E7o / entrada ap=F3s o sucesso da transfer=EAncia d= o fundo de garantia para a sua conta indicada no exterior. Al=E9m disso=2C = voc=EA indica suas op=E7=F5es para me ajudar como eu acredito que esta tran= sa=E7=E3o ser=E1 conclu=EDda no prazo de quatro (4) dias voc=EA indica inte= resse para me ajudar.Antecipando para ouvir de voc=EA logo. Com os melhores cumprimentos=2CSra. Modese Dossongui. = --_21ef159c-260b-4004-98a2-4a705c7f09f1_ Content-Type: text/html; charset="iso-8859-15" Content-Transfer-Encoding: quoted-printable
De Modese Dossongui=2C

Eu sou deputada=2C Modese Dossongui a =FAnica filha do= falecido Sr. e Sra. Bazose Dossongui. Meu pai era um proeminente comercian= te de ouro e cacau. Antes da morte de meu pai em um hospital particular aqu= i em Abidjan=2C ele me disse que ele tem a soma de tr=EAs milh=F5es e setec= entos mil Euros =A4 3.700.000. deixado em conta fixo / suspense em um dos b= anco principal aqui=2C que ele usou o meu nome como sua =FAnica filha para = o parente mais pr=F3ximo em dep=F3sito do fundo=2C

Ele tamb=E9m me explicou que era por causa dessa riqueza que ele foi enven= enado por seus colegas de trabalho que eu deveria procurar um parceiro estr= angeiro em uma
Pa=EDs de minha escolha onde vou transferir esse d= inheiro e us=E1-lo para fins de investimento=2C tais como a gest=E3o de ben= s im=F3veis ou de gest=E3o hoteleira.
 =3B
Querido= =2C eu estou procurando honrosamente sua assist=EAncia nas seguintes formas= : (1) Para fornecer uma conta banc=E1ria em que o dinheiro seria transferid= o para. (2) Para servir como um guardi=E3o deste fundo desde que eu sou ape= nas 20 anos. (3) Para fazer o arranjo para me para vir para o seu pa=EDs pa= ra continuar a minha educa=E7=E3o e para garantir uma autoriza=E7=E3o de re= sid=EAncia em seu pa=EDs. Al=E9m disso=2C querida=2C eu estou disposto a of= erecer-lhe 15% do valor total como compensa=E7=E3o pelo seu esfor=E7o / ent= rada ap=F3s o sucesso da transfer=EAncia do fundo de garantia para a sua co= nta indicada no exterior. Al=E9m disso=2C voc=EA indica suas op=E7=F5es par= a me ajudar como eu acredito que esta transa=E7=E3o ser=E1 conclu=EDda no p= razo de quatro (4) dias voc=EA indica interesse para me ajudar.
A= ntecipando para ouvir de voc=EA logo.

Com os melho= res cumprimentos=2C
Sra. Modese Dossongui.
= --_21ef159c-260b-4004-98a2-4a705c7f09f1_-- From brolley@redhat.com Wed Oct 29 09:54:29 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 43F807F3F for ; Wed, 29 Oct 2014 09:54:29 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 31EE38F8037 for ; Wed, 29 Oct 2014 07:54:29 -0700 (PDT) X-ASG-Debug-ID: 1414594464-04cbb070c645f1b0001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id AAaWrOYJQe0pkeLT (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 29 Oct 2014 07:54:25 -0700 (PDT) X-Barracuda-Envelope-From: brolley@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9TEsJe3028416 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 29 Oct 2014 10:54:20 -0400 Received: from [10.15.16.139] (dhcp-10-15-16-139.yyz.redhat.com [10.15.16.139]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9TEsHD5012928; Wed, 29 Oct 2014 10:54:17 -0400 Message-ID: <54510015.1000405@redhat.com> Date: Wed, 29 Oct 2014 10:56:21 -0400 From: Dave Brolley User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: Ken McDonell , pcp@oss.sgi.com Subject: Re: [pcp] user/group access control question References: <001f01cff196$66cf3e00$346dba00$@internode.on.net> <544FD6DF.8060206@redhat.com> <545027F9.2020205@internode.on.net> X-ASG-Orig-Subj: Re: [pcp] user/group access control question In-Reply-To: <545027F9.2020205@internode.on.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1414594465 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On 10/28/2014 07:34 PM, Ken McDonell wrote: > > I have this change in the context of more diagnostics and the > extensions to qa/944, so I'll push the commit if everyone is OK with > that. > OK with me. Thanks. Dave From brolley@redhat.com Wed Oct 29 11:06:05 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 528AE7F3F for ; Wed, 29 Oct 2014 11:06:05 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id E3387AC002 for ; Wed, 29 Oct 2014 09:06:01 -0700 (PDT) X-ASG-Debug-ID: 1414598758-04bdf038cf50a6a0001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id NdbFiZaffxqzcf7P (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 29 Oct 2014 09:06:01 -0700 (PDT) X-Barracuda-Envelope-From: brolley@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9TG5rYc027567 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 29 Oct 2014 12:05:53 -0400 Received: from [10.15.16.139] (dhcp-10-15-16-139.yyz.redhat.com [10.15.16.139]) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9TG5r6e006243; Wed, 29 Oct 2014 12:05:53 -0400 Message-ID: <545110DC.2020104@redhat.com> Date: Wed, 29 Oct 2014 12:07:56 -0400 From: Dave Brolley User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: Ken McDonell , "'PCP Mailing List'" Subject: Re: [pcp] Multi-Volume Archive + Live Data Playback for PCP Client Tools References: <542C21AE.1010504@redhat.com> <007e01cfe010$7867f090$6937d1b0$@internode.on.net> X-ASG-Orig-Subj: Re: [pcp] Multi-Volume Archive + Live Data Playback for PCP Client Tools In-Reply-To: <007e01cfe010$7867f090$6937d1b0$@internode.on.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1414598761 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 I re-read this thread extracting the ideas that I need to focus on for the initial task of multi-archive support. When I got to the end, I found that Ken had already pretty much summed it up ... On 10/04/2014 04:19 PM, Ken McDonell wrote: > As others have pointed out ... each -a gets mapped to a context, so we need > some sort of syntax that can name more than one archive in a single command > line argument to be used with -a ... so this leads to the following options: > - dirname > - glob-like , probably not just * but the whole shooting match of ?, [...] > and {...,...} > - a list, e.g. -a 20141001,20140930 > > Now this could be handed off to pmNewContext, and the client could use a > single PMAPI context as a handle to access this _set_ of archives > > For this to work, we need some restrictions on the set of archives that can > be combined in this way: > - all for the same host > - non-overlapping time windows > > If these are not satisfied, pmNewContext needs to return a (new) error code. > > Then we need to consider the metadata: > - timezone could change - this will require some further investigation > before a cunning plan can be proposed > - PMNS - merge 'em all the while there are no conflicts ... in the case of a > conflict (different names map to the same PMID or the same name is assigned > more than one PMID) we probably need dynamic remapping ("first one found > wins" is probably the right strategy) > - metric descriptors - if these change it gets very messy, although is rare > in practice > - instance domains - should be close to OK, as these are already expected to > vary over time ... it would be bad if the semantics of the instance domain > members changed between archives, but this is more of a PMDA botch issue > than a problem for libpcp to solve > > One simple solution that might be acceptable for 95% of the cases would be > to rule all of the metadata data differences (except instance domains) to be > unsupported. So pmNewContext would fail. The user's option for resolving > this is to use pmlogrewrite to amend one or more of the archives and remove > the differences. I think this is definitely an OK plan. > I like Ken's thinking of this as a set of archives and, I think that the restrictions that he has suggested (non-overlapping time windows, all from the same host) are practical to begin with (perhaps it could someday be possible to deal with overlapping time windows). The idea of disallowing meta data differences also seem like a good starting point, but I imagine that the idea of remapping (also mentioned) is possible as an enhancement. I'll ask Ken to elaborate on when he meant by "first one found wins" if/when we decide to do that (or any time before then that he has time to do so). Thanks to everyone for getting me going in the right direction. Dave From dsmith@redhat.com Wed Oct 29 13:14:28 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 116247F3F for ; Wed, 29 Oct 2014 13:14:28 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id D4CC88F8037 for ; Wed, 29 Oct 2014 11:14:27 -0700 (PDT) X-ASG-Debug-ID: 1414606466-04bdf038d1510e40001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id pqKKINrrynxgXgCG (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 29 Oct 2014 11:14:26 -0700 (PDT) X-Barracuda-Envelope-From: dsmith@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9TIEQ7u026269 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Wed, 29 Oct 2014 14:14:26 -0400 Received: from t540p.usersys.redhat.com (vpn-62-253.rdu2.redhat.com [10.10.62.253]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9TIEOo4017358 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Wed, 29 Oct 2014 14:14:25 -0400 Message-ID: <54512E80.9090302@redhat.com> Date: Wed, 29 Oct 2014 13:14:24 -0500 From: David Smith User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: pcp Subject: [RFC] pcp python patch Content-Type: text/plain; charset=utf-8 X-ASG-Orig-Subj: [RFC] pcp python patch Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1414606466 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 First, some background. I'm working on integrating systemtap and pcp, and I wrote a python PMDA that will be similar to the mmv pmda in that it will need to monitor a directory for new data. After working on it, I found out that the pcp python support doesn't really allow adding new metrics (or clearing metrics) after PMDA.run() has been called. So, commit 38983a5 in the pcpfans.git dsmith/dev branch adds a 'refresh metrics' callback to the pcp python support. This allows a python pmda to add/remove metrics after PMDA.run() has been called. Nathan helped a bit with the idea (but blame all errors on me). Here's how you'd use it. In your PMDA.init() function, you'd set the callback, like this: self.set_refresh_metrics(self.__refresh_metrics) Then your __refresh_metrics (or whatever you want to call it) function will get called when needed. I based this code on the mmv pmda, so it could get called during several PDUs. Then your callback function could add metrics. New code in the PMDA class will update everything. Comments welcome on the patch. -- David Smith dsmith@redhat.com Red Hat http://www.redhat.com 256.217.0141 (direct) 256.837.0057 (fax) From michele@acksyn.org Wed Oct 29 15:06:55 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id C62A87F3F for ; Wed, 29 Oct 2014 15:06:55 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 428D3AC004 for ; Wed, 29 Oct 2014 13:06:52 -0700 (PDT) X-ASG-Debug-ID: 1414613206-04cbb070c746f0d0001-S8gJnT Received: from palahniuk.acksyn.org (palahniuk.acksyn.org [5.9.7.26]) by cuda.sgi.com with ESMTP id XDpNrhqXoto53g0I for ; Wed, 29 Oct 2014 13:06:47 -0700 (PDT) X-Barracuda-Envelope-From: michele@acksyn.org X-Barracuda-Apparent-Source-IP: 5.9.7.26 Received: from localhost (localhost [127.0.0.1]) by palahniuk.acksyn.org (Postfix) with ESMTP id 9F4A726BAA for ; Wed, 29 Oct 2014 16:06:46 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=acksyn.org; h= user-agent:content-disposition:content-type:content-type :mime-version:message-id:subject:subject:from:from:date:date :received:received; s=2010; t=1414613205; bh=uRHof0kWlsbe2WZCmBF 7J8CbzU7ObnyLOSc/m3aAWqw=; b=eBBLj0sOBzfNdiuSOYvM6xYbtYRJjJUP0rf FeJXe8XrUlownZ1Or1GP4iyBKT8Yj6UdfUAFoHLam1v6Qc1UOD3VmSgP5yAQtnt1 ANrouDgODFHLBsnpYi7kGv3fIXgpEw6Z8+hHAm1pwgc6jrwAP+LwD85BQ+MpLFai gqq09l+g= Received: from palahniuk.acksyn.org ([127.0.0.1]) by localhost (mail.acksyn.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 7ZSQnCvne12h for ; Wed, 29 Oct 2014 16:06:45 -0400 (EDT) Received: from localhost (host165-178-dynamic.6-79-r.retail.telecomitalia.it [79.6.178.165]) by palahniuk.acksyn.org (Postfix) with ESMTPSA id C701A26377 for ; Wed, 29 Oct 2014 16:06:44 -0400 (EDT) Date: Wed, 29 Oct 2014 21:06:43 +0100 From: Michele Baldessari To: pcp@oss.sgi.com Subject: Performance of parsing an archive in python Message-ID: <20141029200642.GA19804@marquez.int.rhx> X-ASG-Orig-Subj: Performance of parsing an archive in python MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2012-12-30) X-Barracuda-Connect: palahniuk.acksyn.org[5.9.7.26] X-Barracuda-Start-Time: 1414613207 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11030 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature Hi all, while tinkering with pcp2pdf one of my goals is to be able to render a fairly big archive which also includes per process metrics. On one of my servers such a daily archive file is around 800MB or so. Currently my archive parsing looks more or less like this ~250 liner: https://gist.github.com/mbaldessari/30dc7ae2fe46d9b804f2 I basically use a function that returns a big dictionary in the following form: { metric1: {'indom1': [(ts0, ts1, .., tsN), (v0, v1, .., vN)], .... 'indomN': [(ts0, ts1, .., tsN), (v0, v1, .., vN)]}, metric2: {'indom1': [(ts0, ts1, .., tsX), (v0, v1, .., vX)], .... 'indomN': [(ts0, ts1, .., tsX), (v0, v1, .., vX)]}...} Then I use this dictionary and create an image with matplotlib and I parallelize this on all the available CPUs. Now when profiling the above script against a fairly large archive (~800MB), I get the following: """ Parsing files: 20140908.0 - 764.300262451 MB Before parsing: usertime=0.066546 systime=0.011918 mem=12.34375 MB After parsing: usertime=1161.825736 systime=2.364544 mem=1792.53125 MB Profiling of parse() 725026682 function calls in 1169.003 seconds Ordered by: cumulative time List reduced from 72 to 15 due to restriction <15> ncalls tottime percall cumtime percall filename:lineno(function) 1 124.550 124.550 1169.003 1169.003 ./fetch.py:140(parse) 29028435 111.320 0.000 693.559 0.000 ./fetch.py:113(_extract_value) 57876970 134.777 0.000 539.856 0.000 /usr/lib64/python2.7/site-packages/pcp/pmapi.py:379(get_vlist) 146339015 367.384 0.000 367.384 0.000 /usr/lib64/python2.7/ctypes/__init__.py:496(cast) 28848535 34.114 0.000 312.698 0.000 /usr/lib64/python2.7/site-packages/pcp/pmapi.py:384(get_inst) 57876970 89.000 0.000 254.070 0.000 /usr/lib64/python2.7/site-packages/pcp/pmapi.py:374(get_vset) 29028435 168.361 0.000 179.190 0.000 /usr/lib64/python2.7/site-packages/pcp/pmapi.py:1724(pmExtractValue) 29028435 61.598 0.000 141.777 0.000 /usr/lib64/python2.7/site-packages/pcp/pmapi.py:364(get_valfmt) 233809633 33.780 0.000 33.780 0.000 {_ctypes.POINTER} 58013698 9.081 0.000 9.081 0.000 {method 'append' of 'list' objects} 519120 4.551 0.000 8.556 0.000 /usr/lib64/python2.7/site-packages/pcp/pmapi.py:1243(pmLookupDesc) 36257575 8.167 0.000 8.167 0.000 {_ctypes.byref} 1442 6.801 0.005 6.803 0.005 /usr/lib64/python2.7/site-packages/pcp/pmapi.py:1578(pmFetch) 518760 4.574 0.000 4.884 0.000 /usr/lib64/python2.7/site-packages/pcp/pmapi.py:1172(pmNameID) 518760 1.280 0.000 2.924 0.000 /usr/lib64/python2.7/site-packages/pcp/pmapi.py:369(get_numval) real 19m31.860s user 19m24.391s sys 0m2.566s """ While 20 minutes to parse such a big archive might be relatively ok, I was wondering what options I have to improve this. The ones I can currently think of are: 1) Split the time interval parsing over multiple CPUs. I can divide the archive in subintervals (one per cpu) and have each CPU do its own subinterval parsing and then stitch everything together at the end. This is the approach I currently use to create the graph images that go in the pdf (as matplotlib+reportlab aren't the fastest thing on the planet) 2) Implement a function in the C python bindings which returns a python dictionary as described above. This would save me all the ctypes/__init__ costs and probably I would shave some time off as there would be less python->C function calls. Maybe we can find a generic enough API for this to be usable by other clients? 3) See if I can use Cython tricks to speed up things 4) Anything else I have not thought of? Grab your cluebats and feel free to point me in the right direction ;) Thanks, Michele NB: I've tried using pmFetchArchive() but a) there was no substantial difference and b) pmFetchArchive() does not allow interpolation so users could not specify a custom time interval -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D From fche@redhat.com Wed Oct 29 15:23:48 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id C2DBE7F3F for ; Wed, 29 Oct 2014 15:23:48 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 5EACFAC004 for ; Wed, 29 Oct 2014 13:23:48 -0700 (PDT) X-ASG-Debug-ID: 1414614223-04bdf038d0517120001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id soM32FftCogF2TD2 (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 29 Oct 2014 13:23:44 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9TKNhii024654 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Wed, 29 Oct 2014 16:23:43 -0400 Received: from fche.csb (vpn-234-89.phx2.redhat.com [10.3.234.89]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9TKNhD3013676 for ; Wed, 29 Oct 2014 16:23:43 -0400 Received: by fche.csb (Postfix, from userid 2569) id 8E16B5816D; Wed, 29 Oct 2014 16:23:42 -0400 (EDT) Date: Wed, 29 Oct 2014 16:23:42 -0400 From: "Frank Ch. Eigler" To: pcp developers Subject: notes from buffalo pcp meetathon Message-ID: <20141029202342.GB8665@redhat.com> X-ASG-Orig-Subj: notes from buffalo pcp meetathon Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.2i X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1414614224 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Hi - The following is a set of notes from meetings held at the SUNY @ Buffalo CCR team's lovely facilities last Thursday. Nathan and I were hosted by Martins Innus, Joe White, Thomas Yearke, and a few other locals. Thanks for having us! Let's get to work on those todos. - ccr.buffalo.edu background info - cluster - scientific & computational research - 900-node, 10000-cpu hetero cluster - 1-3 PB of storage - 5000ish jobs per day - serves university & associates & general public too - managed by slurm resource manager - openxtmod data warehouse - ccs includes hpc support specialists, parallelization experts - our contacts mostly software dev for the management tools - contrast with very large clusters - here they have lots of small jobs - STAMPEDE from texas uses non-pcp collector 'taccstats' - cluster os: centos6, puppet managed - management system - uses fancy web front-end and stats data warehouse - python scripts ingest data from pcp and/or taccstats (<= sar), aggregate across jobs/notes/projects - see also http://xdmod.sourceforge.net/ - includes running background health-assessment jobs "application kernels" that provide canonical histories for trends/anomalies - pcp history - nathans shared overall timeline - pcp went through boom & bust cycles at sgi & aconex (a victim of its own success) - buffalo.edu a user since 1999, sgi-based source code, heavy on 3d viewage - rh plans to rebase frequently in RHEL - even faster personal rebases/builds possible via fedora COPR - broad problem area for pcp to help with - need to track resource consumption on a per-job per-node basis - this corresponds to short-lived cgroups - existing pcp pmda seems to reuse pmids during run -- bug! - has other problems too (cgroup name lack-of-quoting in pmns causing missing most cgroups) - full cgroup names from job manager are usefully unique though - pcp todo item 1: rework cgroups pmda - possibly wrapping cgroup name into nested indom with prefixing/quoting, then using pmdaCache to persist indom codes to avoid collisions - quoting cgroup names and making them part of dynamic pmns also a possibility - need holistic mesh of data about containers both from the host side (e.g., cgroup instantaneous resource usage & allocation), and from within (e.g., introspective network/filesystem states) - buffalo folks don't have recent code progress on this - pcp todo item 2: pmlogger logging proc.* - authentication - pmlogger needs to authenticate in order to get at systemwide proc.* stats - maybe as root; maybe enough to use AF_UNIX / local: / uid-pcp - if full root access as needed, then the current pmcd authentication machinery needs to be automated & simplified, so out-of-box it works (maybe equivalently to local pam?) - similarly, configuration of pmlogger to pass that authentication info needs to be smooth; possibly requiring tls if authenticating via plaintext - pcp todo item 3: pmlogger logging proc.* - data quantity - it is desirable to log some textual proc.* bits for analysis, like a hypothetical (?) proc.psinfo.environ metric, but this is very large - the hotproc pmda work is in a way a kludge to work around having too much input data - if the proc indom changes frequently (as it does), the .meta files get large; http://oss.sgi.com/bugzilla/show_bug.cgi?id=1046 - repeated data values bloat the log file - (a sign of this is how gzip-compressible archives are: >>90% common) - pmlogger should get a mode to skip same-as-last-value entries - (though this would not reduce query load on target pmcd) - semantics may permit this sort of change without archive file format revision - or use libz to compress on the fly & decompress on the fly too - ideally retaining compatible .index file-offset format - pcp todo item 4: need to log some metrics on demand, oneshot type - to match exact moment of job start/stop - something like % pmlc log oneshot _metriclist perhaps? - or deploy job-lifetime-matched temporary pmloggers - almost as in % pmlogger --lifetime PID - related to earlier idea re. pmval -c CMD in papi context - pcp todo item 5: more analysis capability - if pcp had more native analytics (well in excess of pmdiff), the data warehouse aggregations could become secondary and enable drilldown to raw but analyzed data - example types of calculations: cross-correlations between metrics - anomaly detection (big area) - fche's hobbyhorse: connecting this to pmlogger to govern archiving - pcp todo item 6: 3d view - used because anomaly detection still manual with mk. 1 eyeball - old sgi pmview still attractive, especially if considering topology - an openscenegraph-based partial reimplementation exists - so dies a unity 3d/gl-based one, with browser blob etc. - maybe have pmwebd / webgl reimplementation - pcp todo item 7: qa testsuite deployability - individual developer quick testing also well-established - need ability to reproduce something like kenj's elaborate qa farm - something like a public qa server tied to git - pcp todo item 8: docs on pmie best practices - would like more documentation/samples on local / centralized / archive-targeted ("has this happened before?") usage - pcp todo item 9: centralized logging with dynamic instances - need better support for monitoring instances that come & go asynchronously - need more awareness of pcp discovery / scanning / pmmgr - probably also need an instance-invoked mechanism to notify central logger of arrival of new node to log (apart from avahi?), or cloud/grid-infrastructure apis for instance enumeration - pcp todo item 10: archive access api across network - to let analytics / guis run remotely from archive files - already in plans of one flavour of the "grand unified context" - pmwebd graphite interface one possible near-term hack for this - pcp todo item 11: more data importers/exporters - taccstats, ganglia importers interesting - it could allow pcp archives to be canonical long-term storage - kenj's tutorial needs more awareness - export to numpy for analytics there - pcp todo item 12: better tolerance of broken archive files - pmlogextract reportedly gives up too easily if truncated archives encountered - recent pmlogger shouldn't normally generate these, but old files exist - maybe fix them with a one-time pm-archive-lint? - or just let the archive consuming tools put up with errors without aborting - pcp todo item 13: proc pmda timeouts - well known problem, hits ccr (and other larger servers) good and hard - http://oss.sgi.com/bugzilla/show_bug.cgi?id=1036 - kenj had drafted work for pmda timeout-detecting thread for fetches - still need this! - pmie-based auto-pmcd-restarting possible too, though one must run with root/sudo - and restarting doesn't guarantee persistency of pmids etc. - elevated-privileged split-pmcd a possible solution too (it could restart pmda processes) - pcp todo item 14: pmfetch timestamps - especially for slightly tardy pmdas, the pmResult timestamp is ambiguous: is it at beginning, end, or middle of process? - having too-short inter-fetch timestamp intervals can lead to crazy-big rate-converted values - maybe just confirm/fix pmcd to return beginning-of-operation timestamps - apps can also confirm that timestamp is reasonable, by comparing to their own clocks around pmFetch() - pmlogger could emit diagnostics for violations - pcp todo item 15: per-peer stats - sometimes need to be able to measure inter-node traffic stats - not just on network interface basis, but per-peer e.g., network.traffic.ipv4.perpeer.in.packets ["addr"] - also came up in comparisons to ntop - this would could generate large indoms - potentially generate from userspace (a la tcpdump) - or perhaps systemtap (via json / mmv interfaces) - pcp todo item 16: syscall stats - sometimes need to be able to measure per-process per-syscall stats e.g., proc.syscall.FOO ["pid"] or proc.syscall ["pid:FOO"] - naturally matrix-valued - probably gather via systemtap or similar - pcp todo item 17: gpfs pmda - need it - pcp todo item 18: simplified pmapi - fche already prototyping in C - python pmcc also useful for python clients - pcp todo item 19: todo lists - need to collect ideas of small things for students to do - need to remember to formally report bugs - fche proposes bugzilla as hammer for both nails - hm, maybe the above list should be there too From kenj@internode.on.net Wed Oct 29 17:32:45 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id DAA297F3F for ; Wed, 29 Oct 2014 17:32:45 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id C8ADA304039 for ; Wed, 29 Oct 2014 15:32:42 -0700 (PDT) X-ASG-Debug-ID: 1414621957-04cbb070c5475910001-S8gJnT Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by cuda.sgi.com with ESMTP id qcA60wAGKtPpOGau for ; Wed, 29 Oct 2014 15:32:37 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.145 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AloCADZqUVR20YErPGdsb2JhbAANT4t3yxmDIAKBNQEBAQEBBgEBAQE4hD4BAQQ4QBELGAkPBwUKCQMCAQIBMRQGAQwIAQG6Z4ZHjzUBAQEBBgEBAQEBHZEXGIQzAQSPbqhIgyMBAQE Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail06.adl6.internode.on.net with ESMTP; 30 Oct 2014 09:02:36 +1030 Message-ID: <54516B4C.1070806@internode.on.net> Date: Thu, 30 Oct 2014 09:33:48 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: Dave Brolley , pcp@oss.sgi.com Subject: Re: [pcp] user/group access control question References: <001f01cff196$66cf3e00$346dba00$@internode.on.net> <544FD6DF.8060206@redhat.com> <545027F9.2020205@internode.on.net> <54510015.1000405@redhat.com> X-ASG-Orig-Subj: Re: [pcp] user/group access control question In-Reply-To: <54510015.1000405@redhat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl6.internode.on.net[150.101.137.145] X-Barracuda-Start-Time: 1414621957 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11032 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- On 30/10/14 01:56, Dave Brolley wrote: > On 10/28/2014 07:34 PM, Ken McDonell wrote: >> >> I have this change in the context of more diagnostics and the >> extensions to qa/944, so I'll push the commit if everyone is OK with >> that. >> > OK with me. Thanks. > > Dave > This got delayed a little. It turns out that if you use user or group access controls, and you try to access pmcd via a tcp socket (rather than a unix domain socket) then (a) if you DO NOT have secure sockets, the access fails, but (b) if you DO have secure sockets the SASL callback leads us back into libpcp where _if_ you have a console, we get a Username: and Password: prompt. Case (b) is a non-starter for automated QA, so I've had to make this test conditional, which in turn leads to variant output files being needed. From nscott@redhat.com Wed Oct 29 17:53:00 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 7D6687F3F for ; Wed, 29 Oct 2014 17:53:00 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 0A0AEAC004 for ; Wed, 29 Oct 2014 15:52:56 -0700 (PDT) X-ASG-Debug-ID: 1414623174-04cb6c2efa4327f0001-S8gJnT Received: from mx5-phx2.redhat.com (mx5-phx2.redhat.com [209.132.183.37]) by cuda.sgi.com with ESMTP id 6Et8Ar6A9Wjnk7N5 (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 29 Oct 2014 15:52:55 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.37 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx5-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9TMqprR026101; Wed, 29 Oct 2014 18:52:52 -0400 Date: Wed, 29 Oct 2014 18:52:51 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Michele Baldessari Cc: pcp@oss.sgi.com Message-ID: <670106284.3108531.1414623171877.JavaMail.zimbra@redhat.com> In-Reply-To: <20141029200642.GA19804@marquez.int.rhx> References: <20141029200642.GA19804@marquez.int.rhx> Subject: Re: [pcp] Performance of parsing an archive in python MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] Performance of parsing an archive in python Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: Performance of parsing an archive in python Thread-Index: dj3FQ50RiVrbCVuezOPA9/XBDOTEWw== X-Barracuda-Connect: mx5-phx2.redhat.com[209.132.183.37] X-Barracuda-Start-Time: 1414623175 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11033 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Hi Michele, ----- Original Message ----- > [...] > real 19m31.860s > user 19m24.391s > sys 0m2.566s > """ OOC, can you time a pmlogsummary run on this archive? > While 20 minutes to parse such a big archive might be relatively ok, I > was wondering what options I have to improve this. The ones I can > currently think of are: > > 1) Split the time interval parsing over multiple CPUs. I can divide the > archive in subintervals (one per cpu) and have each CPU do its own > subinterval parsing and then stitch everything together at the end. > This is the approach I currently use to create the graph images that go > in the pdf (as matplotlib+reportlab aren't the fastest thing on the > planet) Should definitely help, since it appears to be CPU bound currently. > 2) Implement a function in the C python bindings which returns a > python dictionary as described above. This would save me all the > ctypes/__init__ costs and probably I would shave some time off as there > would be less python->C function calls. Maybe we can find a generic > enough API for this to be usable by other clients? Yep, sounds good. > 3) See if I can use Cython tricks to speed up things > > 4) Anything else I have not thought of? pmlogsummary uses that raw archive fetching interface we talked about awhile back, which isn't always ideal for your needs - I'm interested in seeing the time difference though, if you could run that locally? cheers. -- Nathan From kenj@internode.on.net Wed Oct 29 18:12:54 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 5EB8B7F3F for ; Wed, 29 Oct 2014 18:12:54 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id E237EAC001 for ; Wed, 29 Oct 2014 16:12:50 -0700 (PDT) X-ASG-Debug-ID: 1414624367-04cb6c2ef94330f0001-S8gJnT Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by cuda.sgi.com with ESMTP id N3YK8uIVsOHT4cEI for ; Wed, 29 Oct 2014 16:12:48 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.145 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AloCAJZzUVR20YErPGdsb2JhbAANT4NiWIc9xmmJAgEBAQEBBgEBAQE4hGdVMAYCBRYLAgsDAgECATEOGQYCAQGISrIZeJUtgSyQAYJhgVQFhi2QKqFfWIJLAQEB Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail06.adl6.internode.on.net with ESMTP; 30 Oct 2014 09:42:47 +1030 Message-ID: <545174B7.3060105@internode.on.net> Date: Thu, 30 Oct 2014 10:13:59 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: pcp updates Content-Type: text/plain; charset=utf-8 X-ASG-Orig-Subj: pcp updates Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl6.internode.on.net[150.101.137.145] X-Barracuda-Start-Time: 1414624367 X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11033 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Changes committed to git://git.performancecopilot.org/kenj/pcp.git dev qa/.gitignore | 1 qa/944 | 96 ++++++++++++++++++++++++++++++++---- qa/944.out | 119 --------------------------------------------- qa/944.out.1 | 115 +++++++++++++++++++++++++++++++++++++++++++ qa/944.out.2 | 119 +++++++++++++++++++++++++++++++++++++++++++++ qa/admin/myconfigure | 41 +++++++++------ qa/admin/show-me-all | 2 src/libpcp/src/access.c | 7 ++ src/libpcp/src/auxserver.c | 40 +++++++++++++-- src/libpcp/src/connect.c | 15 +++++ src/pmprobe/pmprobe.c | 2 11 files changed, 408 insertions(+), 149 deletions(-) commit 09a5f130b610add1e03f41b690710e88b3d47bf0 Author: Ken McDonell Date: Thu Oct 30 07:00:15 2014 +1100 qa/admin/show-me-all - fix small botch in hostname for echo commit 071715ec93fd5cb0f1165f3fd74183b650164c8a Author: Ken McDonell Date: Thu Oct 30 06:59:10 2014 +1100 qa/admin/myconfigure - adjust configure options to match logic in Makepkgs commit 8992fa5ce0e42546e32c73bbab4b716ed4c0a695 Author: Ken McDonell Date: Thu Oct 30 06:57:20 2014 +1100 qa/944 - extensions 1. run pmcd and clients with more diagnostic ... syphon this output into 944.full 2. add another test to be sure the implementation of user and authentication, fails when the connection to pmcd is via a unix domain socket ... but only try this for platforms that do NOT have secure socket support (see the comment in 944 for more details) 3. add variant output files for the have/do not have secure sockets cases commit 747ae8cd0e447f0e379ba585c0e3e6050668d8f4 Author: Ken McDonell Date: Wed Oct 29 16:32:21 2014 +1100 libpcp - non-secure handshake changes User and/or group authorisation requires the non-secure implementation of __pmSecureServerHandshake() to accept the PDU_FLAG_CREDS_REQD credential, provided the connection has been made with a unix domain socket. commit 3933ec89ec199f736c65f051033140ac391f62ce Author: Ken McDonell Date: Wed Oct 29 16:31:15 2014 +1100 libpcp - extra diagnostics In the areas of pmcd connection mode selection and user/group authorisation. commit df7d5b1551c0396d4f6acb832a090cd266f7b826 Author: Ken McDonell Date: Wed Oct 29 14:26:37 2014 +1100 pmprobe - improve guard in error case If the context is valid, but access controls prevent fetching information from pmcd, then the number of metrics found in a PMNS traversal may be 0. Avoid calling pmLookupName() in this case. From nscott@redhat.com Wed Oct 29 19:29:52 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 4C81B7F3F for ; Wed, 29 Oct 2014 19:29:52 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id DD702AC001 for ; Wed, 29 Oct 2014 17:29:48 -0700 (PDT) X-ASG-Debug-ID: 1414628986-04cbb070c5479010001-S8gJnT Received: from mx6-phx2.redhat.com (mx6-phx2.redhat.com [209.132.183.39]) by cuda.sgi.com with ESMTP id 9bMfwfS0AhDHLOwZ (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 29 Oct 2014 17:29:47 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.39 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx6-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9U0Tkkr022749; Wed, 29 Oct 2014 20:29:46 -0400 Date: Wed, 29 Oct 2014 20:29:46 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Dave Brolley Cc: PCP Mailing List Message-ID: <1180763876.3139907.1414628986470.JavaMail.zimbra@redhat.com> In-Reply-To: <545110DC.2020104@redhat.com> References: <542C21AE.1010504@redhat.com> <007e01cfe010$7867f090$6937d1b0$@internode.on.net> <545110DC.2020104@redhat.com> Subject: Re: [pcp] Multi-Volume Archive + Live Data Playback for PCP Client Tools MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] Multi-Volume Archive + Live Data Playback for PCP Client Tools Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: Multi-Volume Archive + Live Data Playback for PCP Client Tools Thread-Index: 7H2i6ekdVfBSdMPZHqOGr9/Gz/NkPw== X-Barracuda-Connect: mx6-phx2.redhat.com[209.132.183.39] X-Barracuda-Start-Time: 1414628987 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.03 X-Barracuda-Spam-Status: No, SCORE=0.03 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_SA_TO_FROM_DOMAIN_MATCH, THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11035 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... 0.01 BSF_SC0_SA_TO_FROM_DOMAIN_MATCH Sender Domain Matches Recipient Domain ----- Original Message ----- > I re-read this thread extracting the ideas that I need to focus on for > the initial task of multi-archive support. When I got to the end, I > found that Ken had already pretty much summed it up ... > :) > I like Ken's thinking of this as a set of archives and, I think that the > restrictions that he has suggested (non-overlapping time windows, all > from the same host) are practical to begin with (perhaps it could > someday be possible to deal with overlapping time windows). *nod* > The idea of disallowing meta data differences also seem like a good > starting point, but I imagine that the idea of remapping (also > mentioned) is possible as an enhancement. See the pmlogrewrite(1) man page for some more context there Dave. cheers. -- Nathan From nscott@redhat.com Wed Oct 29 20:11:46 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 6324C7F3F for ; Wed, 29 Oct 2014 20:11:46 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 51B208F8035 for ; Wed, 29 Oct 2014 18:11:43 -0700 (PDT) X-ASG-Debug-ID: 1414631497-04cb6c2efb436470001-S8gJnT Received: from mx6-phx2.redhat.com (mx6-phx2.redhat.com [209.132.183.39]) by cuda.sgi.com with ESMTP id Ng0QYnhyWaQpFpsM (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 29 Oct 2014 18:11:38 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.39 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx6-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9U1BYsG029840; Wed, 29 Oct 2014 21:11:34 -0400 Date: Wed, 29 Oct 2014 21:11:34 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Michele Baldessari Cc: Mark Goodwin , pcp@oss.sgi.com Message-ID: <99637966.3149570.1414631494891.JavaMail.zimbra@redhat.com> In-Reply-To: <20141028075134.GB6825@marquez.int.rhx> References: <20141021222355.GC6656@marquez.int.rhx> <573592010.74725901.1413999659304.JavaMail.zimbra@redhat.com> <54483934.4060802@redhat.com> <20141023072636.GB6158@marquez.int.rhx> <1558759660.1356853.1414474824972.JavaMail.zimbra@redhat.com> <20141028075134.GB6825@marquez.int.rhx> Subject: Re: [pcp] hinv.* MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] hinv.* Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: hinv.* Thread-Index: S6OIVFzvOdJw2NF+z95+K4/jXgA3uA== X-Barracuda-Connect: mx6-phx2.redhat.com[209.132.183.39] X-Barracuda-Start-Time: 1414631498 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11036 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Hi Michele, ----- Original Message ----- > Hi Nathan, > > On Tue, Oct 28, 2014 at 01:40:24AM -0400, Nathan Scott wrote: > > > The default logging here is "once" for hinv, so that might need tweaking > > > as > > > well. > > > > Yeah, hmmm, thats a bit awkward - I don't really want us to have to > > explicitly list each hinv metric in the logging configs. Not sure > > what else we can do though. > > Good point. What would be disadvantage of simply logging hinv.* all the > time? Too big of an increase in archive files? Yep, that'd be the downside. > Or should we aim for a split of the hinv.* hierarchy into "extremely > unlikely to ever change" and "likely to change" metrics? > Or we just leave things as is and be it ;) Ken's pointed out the correct path here - keep the log-once, as-is, and add in logging on an interval of just those hinv metrics that change. cheers. -- Nathan From kenj@internode.on.net Wed Oct 29 23:19:58 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 9DC0A7F3F for ; Wed, 29 Oct 2014 23:19:58 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 2C588AC001 for ; Wed, 29 Oct 2014 21:19:57 -0700 (PDT) X-ASG-Debug-ID: 1414642791-04cb6c2ef943c1a0001-S8gJnT Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id 8KmUdG2wuBb1MfAP for ; Wed, 29 Oct 2014 21:19:52 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AsgBAE+7UVR20YErPGdsb2JhbAANT4NiWIc9xyKHVQKBMwEBAQEBBgEBAQE4hD4BAQQ4QBELGAkWDwkDAgECATEUEwgBAbo2lXcBAQgCAR+QOF8WhDUBBJZXiESDSpVQWIEGgUUBAQE Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail04.adl6.internode.on.net with ESMTP; 30 Oct 2014 14:49:51 +1030 Message-ID: <5451BCAF.8090403@internode.on.net> Date: Thu, 30 Oct 2014 15:21:03 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: Re: [pcp] notes from buffalo pcp meetathon References: <20141029202342.GB8665@redhat.com> X-ASG-Orig-Subj: Re: [pcp] notes from buffalo pcp meetathon In-Reply-To: <20141029202342.GB8665@redhat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1414642791 X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11039 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- On 30/10/14 07:23, Frank Ch. Eigler wrote: > Hi - > > The following is a set of notes from meetings held at the SUNY @ Buffalo > CCR team's lovely facilities last Thursday. Nathan and I were hosted by > Martins Innus, Joe White, Thomas Yearke, and a few other locals. Thanks > for having us! Let's get to work on those todos. > ... Wow, that's some TODO list ... 8^)> It is many years since I last visited CCR @ SUNY, and it is a pity CCR isn't closer to Melbourne, I would have loved to have been there. Might I suggest that we migrate this todo list to the bugzilla list at oss.sgi.com and file one todo item per "enhancement" bug (aka RFE). This would keep the discussion scope limited to one todo item per comment, keep the related comments together, allow people to assign items to themselves to avoid duplicated effort, etc, etc. If there is consensus around this suggestion, I'm willing to do the initial bugzilla leg work. If not, I'm open for suggestions of how we might provide care and feeding for this list of todo items ... please don't suggest a wiki ... the last thing the world needs is another mouldy repository of information that is not updated nor correct. From nscott@redhat.com Wed Oct 29 23:40:56 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 97B977F3F for ; Wed, 29 Oct 2014 23:40:56 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 84875304032 for ; Wed, 29 Oct 2014 21:40:53 -0700 (PDT) X-ASG-Debug-ID: 1414644051-04cb6c2ef943cc90001-S8gJnT Received: from mx6-phx2.redhat.com (mx6-phx2.redhat.com [209.132.183.39]) by cuda.sgi.com with ESMTP id h3pcPC6e4wFpeXOn (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Wed, 29 Oct 2014 21:40:51 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.39 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx6-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9U4eknS002261; Thu, 30 Oct 2014 00:40:46 -0400 Date: Thu, 30 Oct 2014 00:40:46 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Ken McDonell Cc: pcp@oss.sgi.com Message-ID: <1561193033.3211246.1414644046122.JavaMail.zimbra@redhat.com> In-Reply-To: <5451BCAF.8090403@internode.on.net> References: <20141029202342.GB8665@redhat.com> <5451BCAF.8090403@internode.on.net> Subject: Re: [pcp] notes from buffalo pcp meetathon MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] notes from buffalo pcp meetathon Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: notes from buffalo pcp meetathon Thread-Index: qaysRg25df7unoS1+tVshOcbrQLP5Q== X-Barracuda-Connect: mx6-phx2.redhat.com[209.132.183.39] X-Barracuda-Start-Time: 1414644051 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11039 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... ----- Original Message ----- > [...] > This would keep the discussion scope limited to one todo item per > comment, keep the related comments together, allow people to assign > items to themselves to avoid duplicated effort, etc, etc. > > If there is consensus around this suggestion, I'm willing to do the > initial bugzilla leg work. Sounds good to me. Chatting further to Joe in the evening about the cgroups item, that's something Red Hat need fairly quickly and isn't at the top of his list, so I agreed to take that one on (since I was the one who caused his last series to implode by adding in the blkio stats there :) ... I'll probably do that one next week, so feel free to leave that one off the bugzilla list. cheers. -- Nathan From kenj@internode.on.net Thu Oct 30 01:20:58 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id E8B8F7F3F for ; Thu, 30 Oct 2014 01:20:58 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id D776F304032 for ; Wed, 29 Oct 2014 23:20:58 -0700 (PDT) X-ASG-Debug-ID: 1414650052-04cb6c2efa440a90001-S8gJnT Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id bb5PM2NTunQEZJVN for ; Wed, 29 Oct 2014 23:20:53 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AsUBAMDXUVR20YErPGdsb2JhbAANT4t3y22EVwEBAQEBBgEBAQE4hGdVNgIFFgsCCwMCAQIBMRoNCAEBuk54lSmBLJJigVQFuDWDIwEBAQ Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail04.adl6.internode.on.net with ESMTP; 30 Oct 2014 16:50:52 +1030 Message-ID: <5451D90C.2090802@internode.on.net> Date: Thu, 30 Oct 2014 17:22:04 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: PCP Subject: QA Summary - status Content-Type: text/plain; charset=utf-8 X-ASG-Orig-Subj: QA Summary - status Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1414650052 X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11041 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Here's the current status, FYI. Nice to see some 0's in the Fail column. 944 should improve after recent rework. 722 is the long-standing "pmatop output is truncated" problem 752 is the "Tuesday" bug ... gets better later in the week .. 8^)> and after that we're down to failures on 3 or fewer hosts kenj@bozo:~$ pcp-qa-summary -fr ==== QA Summary ==== Date Run Pass Fail Nrun Host 2014-10-28 693 689 4 59|bozo PCP 3.10.0 x86_64 Ubuntu 14.04 2014-10-24 684 683 1 65|bozo-laptop PCP 3.10.0 i686 LinuxMint 15 2014-10-28 683 681 2 66|bozo-vm PCP 3.10.0 x86_64 Debian 7.6 Daily runs, but no QA |fuji PCP 3.10.0 i386 Darwin 10.8.0 2014-10-28 617 614 3 87|grundy PCP 3.10.0 ia64 SUSE SLES11 SP1 2014-10-28 684 682 2 65|vm00 PCP 3.10.0 x86_64 Ubuntu 12.04 2014-10-28 683 682 1 65|vm01 PCP 3.10.0 i686 Ubuntu 12.10 2014-10-28 682 674 8 67|vm02 PCP 3.10.0 i686 openSUSE 12.1 2014-10-28 696 692 4 56|vm03 PCP 3.10.0 x86_64 Fedora 18 2014-10-25 640 633 7 109|vm04 PCP 3.10.0 i586 CentOS 5.10 2014-10-25 620 612 8 129|vm05 PCP 3.10.0 i486 Gentoo 2.0.3 2014-10-29 60 59 1 4|vm06 PCP 3.10.0 amd64 FreeBSD 8.2-RELEASE-p9 2014-10-29 684 681 3 68|vm07 PCP 3.10.0 x86_64 Debian 6.0.9 2014-10-27 698 692 6 51|vm08 PCP 3.10.0 x86_64 CentOS Linux7.0.1406 2014-10-27 60 59 1 4|vm10 PCP 3.10.0 i386 FreeBSD 8.2-RELEASE-p9 2014-10-29 684 681 3 68|vm11 PCP 3.10.0 i686 Debian 6.0.9 2014-10-29 658 658 0 94|vm12 PCP 3.10.0 i686 Fedora 17 2014-10-30 682 681 1 70|vm14 PCP 3.10.0 x86_64 CentOS6.5 No daily runs |vm15 PCP 3.9.1 x86_64 Slackware 13.37.0 2014-10-30 687 686 1 65|vm18 PCP 3.10.0 x86_64 LinuxMint 12 2014-10-30 687 684 3 65|vm19 PCP 3.10.0 x86_64 openSUSE 12.2 2014-10-30 688 687 1 64|vm20 PCP 3.10.0 x86_64 Ubuntu 13.04 2014-10-26 686 686 0 63|vm21 PCP 3.10.0 i686 Debian 7.4 2014-10-23 667 662 5 82|vm22 PCP 3.10.0 x86_64 Fedora 19 2014-10-24 694 688 6 55|vm23 PCP 3.10.0 i686 Fedora 20 2014-10-24 653 644 9 96|vm24 PCP 3.10.0 i686 openSUSE 13.1 Summary: 14970 run, 80 failed (0.53%) ==== QA Failure (X) Map ==== Host bo bl bv gr 00 01 02 03 04 05 06 07 08 10 11 14 18 19 20 22 23 24 Test %bad Test QA groups 944 32% X X X X X X X 944 pmcd secure 722 18% X X X X 722 python 752 18% X X X X 752 libpcp 115 14% X X X 115 pmie 198 14% X X X 198 pmda context 652 14% X X X 652 pmda.systemd event flakey 1044 14% X X X 1044 pmie pmieconf 1045 14% X X X 1045 pmie pmieconf 1046 14% X X X 1046 pmie pmieconf 001 9% X X 001 pdu 023 9% X X 023 pmcd pmprobe 088 9% X X 088 archive pmval 206 9% X X 206 archive pmval 260 9% X X 260 derive pmie 374 9% X X 374 pmlc pmlogger 721 9% X X 721 dbpmda 753 9% X X 753 derive pmie 828 9% X X 828 valgrind archive context Host bo bl bv gr 00 01 02 03 04 05 06 07 08 10 11 14 18 19 20 22 23 24 986 9% X X 986 pmda.dmcache pmda.install 000 5% X 000 other pmcd 069 5% X 069 pmcd pmval 083 5% X 083 pmlc pmlogger compat 119 5% X 119 logutil 188 5% X 188 libpcp 276 5% X 276 pmie pmval indom 323 5% X 323 pmda.shping pmda.proc pmda.install 359 5% X 359 pmcd pminfo 360 5% X 360 pmie 375 5% X 375 pmlc pmlogger 464 5% X 464 pmns 511 5% X 511 pmimport pmdumplog pmlogsummary perl 533 5% X 533 dbpmda pmda.sample 578 5% X 578 pmcd pmda.install pmval 583 5% X 583 pmie 635 5% X 635 pmda.linux libirixpmda 727 5% X 727 avahi 731 5% X 731 pmda.proc valgrind 740 5% X 740 pmda.sample pmstore secure 775 5% X 775 pmfind Host bo bl bv gr 00 01 02 03 04 05 06 07 08 10 11 14 18 19 20 22 23 24 776 5% X 776 pmfind 840 5% X 840 avahi 842 5% X 842 python 843 5% X 843 pmda python 946 5% X 946 pmfind avahi 950 5% X 950 avahi 999 5% X 999 pmns Host bo bl bv gr 00 01 02 03 04 05 06 07 08 10 11 14 18 19 20 22 23 24 From nscott@redhat.com Thu Oct 30 02:10:26 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 993B17F3F for ; Thu, 30 Oct 2014 02:10:26 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 87CC08F8035 for ; Thu, 30 Oct 2014 00:10:23 -0700 (PDT) X-ASG-Debug-ID: 1414653021-04cb6c2ef9442260001-S8gJnT Received: from mx6-phx2.redhat.com (mx6-phx2.redhat.com [209.132.183.39]) by cuda.sgi.com with ESMTP id 6nc3Hy8pUi2zTSbF (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 30 Oct 2014 00:10:21 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.39 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx6-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9U7AK3s028481 for ; Thu, 30 Oct 2014 03:10:21 -0400 Date: Thu, 30 Oct 2014 03:10:20 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: PCP Mailing List Message-ID: <1263736894.3256283.1414653020958.JavaMail.zimbra@redhat.com> In-Reply-To: <1111678629.3255872.1414652916682.JavaMail.zimbra@redhat.com> Subject: pcp updates: kenj+brolley merges, gluster+linux pmdas, qa MIME-Version: 1.0 X-ASG-Orig-Subj: pcp updates: kenj+brolley merges, gluster+linux pmdas, qa Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: pcp updates: kenj+brolley merges, gluster+linux pmdas, qa Thread-Index: 1AUor7DrAapRFFEry8X0npYUOtFiIg== X-Barracuda-Connect: mx6-phx2.redhat.com[209.132.183.39] X-Barracuda-Start-Time: 1414653021 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11042 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Changes committed to git://git.pcp.io/pcp.git dev Ken McDonell (7): qa/828 - pander to more valgrind non-determinism pmprobe - improve guard in error case libpcp - extra diagnostics libpcp - non-secure handshake changes qa/944 - extensions qa/admin/myconfigure - adjust configure options to match logic in Makepkgs qa/admin/show-me-all - fix small botch in hostname for echo Nathan Scott (3): pmdagluster: thread-based timeout for too-long gluster query build: update gitignore file for systemd pmda pmdalinux: add hinv.{cpu,node}.online metrics Dave Brolley (2): Typo in __pmLogLoadLabel(): '0' --> '\0' Fix pmLogIndom record description in pcp-archive(5). man/man5/pcp-archive.5 | 9 +- qa/.gitignore | 1 qa/747 | 43 ++++++++++++ qa/747.out | 14 ++++ qa/828 | 46 +++++++++++++ qa/944 | 96 +++++++++++++++++++++++++--- qa/944.out | 119 ----------------------------------- qa/944.out.1 | 115 +++++++++++++++++++++++++++++++++ qa/944.out.2 | 119 +++++++++++++++++++++++++++++++++++ qa/admin/myconfigure | 41 +++++++----- qa/admin/show-me-all | 2 qa/group | 1 qa/linux/sysdev-root-001.tgz |binary src/libpcp/src/access.c | 7 ++ src/libpcp/src/auxserver.c | 40 ++++++++++- src/libpcp/src/connect.c | 15 ++++ src/libpcp/src/logutil.c | 2 src/pmdas/gluster/pmdagluster.python | 68 ++++++++++++++++---- src/pmdas/linux/clusters.h | 3 src/pmdas/linux/help | 6 + src/pmdas/linux/pmda.c | 34 ++++++++++ src/pmdas/linux/proc_cpuinfo.c | 28 +++++++- src/pmdas/linux/proc_cpuinfo.h | 5 - src/pmdas/linux/root_linux | 6 + src/pmdas/systemd/.gitignore | 2 src/pmprobe/pmprobe.c | 2 26 files changed, 651 insertions(+), 173 deletions(-) From nscott@redhat.com Thu Oct 30 02:12:26 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id A0B487F3F for ; Thu, 30 Oct 2014 02:12:26 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 8E4FF8F8033 for ; Thu, 30 Oct 2014 00:12:26 -0700 (PDT) X-ASG-Debug-ID: 1414653144-04cb6c2ef94423b0001-S8gJnT Received: from mx5-phx2.redhat.com (mx5-phx2.redhat.com [209.132.183.37]) by cuda.sgi.com with ESMTP id I3XK5hYiHLpowDur (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 30 Oct 2014 00:12:25 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.37 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx5-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9U7CKnx017591; Thu, 30 Oct 2014 03:12:20 -0400 Date: Thu, 30 Oct 2014 03:12:20 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Ken McDonell Cc: PCP Message-ID: <219313982.3256921.1414653140831.JavaMail.zimbra@redhat.com> In-Reply-To: <5451D90C.2090802@internode.on.net> References: <5451D90C.2090802@internode.on.net> Subject: Re: [pcp] QA Summary - status MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] QA Summary - status Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.7] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: QA Summary - status Thread-Index: lOKDftlw2obiAP9IypwJbGfaW2sf5A== X-Barracuda-Connect: mx5-phx2.redhat.com[209.132.183.37] X-Barracuda-Start-Time: 1414653145 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11042 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... ----- Original Message ----- > Here's the current status, FYI. Thanks! > Nice to see some 0's in the Fail column. :) > 944 should improve after recent rework. (IOU a review there, but outta time for today) > 722 is the long-standing "pmatop output is truncated" problem Hmmm thought I'd got that one, sorry - will dig into it again. > 752 is the "Tuesday" bug ... gets better later in the week .. 8^)> Hehe, guess we should always release on Fridays. > and after that we're down to failures on 3 or fewer hosts Nice! cheers. -- Nathan From michele@acksyn.org Thu Oct 30 02:53:57 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 154CB7F3F for ; Thu, 30 Oct 2014 02:53:57 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 82AABAC001 for ; Thu, 30 Oct 2014 00:53:53 -0700 (PDT) X-ASG-Debug-ID: 1414655631-04cbb070c748e580001-S8gJnT Received: from palahniuk.acksyn.org (palahniuk.acksyn.org [5.9.7.26]) by cuda.sgi.com with ESMTP id DJ1W4BCHTY3maP9W for ; Thu, 30 Oct 2014 00:53:51 -0700 (PDT) X-Barracuda-Envelope-From: michele@acksyn.org X-Barracuda-Apparent-Source-IP: 5.9.7.26 Received: from localhost (localhost [127.0.0.1]) by palahniuk.acksyn.org (Postfix) with ESMTP id 99FF828EF8; Thu, 30 Oct 2014 03:53:50 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=acksyn.org; h= user-agent:in-reply-to:content-disposition:content-type :content-type:mime-version:references:message-id:subject:subject :from:from:date:date:received:received; s=2010; t=1414655630; bh=KPAhApN1h1mT4i67ewKUUmh0FsqpbsbO8RgSg2JuTi0=; b=d7gUTD7loePn J52FvEdkPNC5zWrkh7FI8St+XsX/qX/xcnJDVaFGY85gKHJ32avcDA1gGa6/Fa94 QWAcpPtMcv4E/c5apdtFeRzSzQ7ptow6G1j1Og0LVYE7D1z4CImBsOiv+U9n8Xih DzWVwM6F1UmXG0pYJpZs84ZJJWKjzEA= Received: from palahniuk.acksyn.org ([127.0.0.1]) by localhost (mail.acksyn.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id xCnI_58gjVWe; Thu, 30 Oct 2014 03:53:50 -0400 (EDT) Received: from localhost (host165-178-dynamic.6-79-r.retail.telecomitalia.it [79.6.178.165]) by palahniuk.acksyn.org (Postfix) with ESMTPSA id 4055826377; Thu, 30 Oct 2014 03:53:49 -0400 (EDT) Date: Thu, 30 Oct 2014 08:53:48 +0100 From: Michele Baldessari To: Nathan Scott Cc: pcp@oss.sgi.com Subject: Re: [pcp] Performance of parsing an archive in python Message-ID: <20141030075348.GA20686@marquez.int.rhx> X-ASG-Orig-Subj: Re: [pcp] Performance of parsing an archive in python References: <20141029200642.GA19804@marquez.int.rhx> <670106284.3108531.1414623171877.JavaMail.zimbra@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <670106284.3108531.1414623171877.JavaMail.zimbra@redhat.com> User-Agent: Mutt/1.5.21 (2012-12-30) X-Barracuda-Connect: palahniuk.acksyn.org[5.9.7.26] X-Barracuda-Start-Time: 1414655631 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11043 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature Hi Nathan, On Wed, Oct 29, 2014 at 06:52:51PM -0400, Nathan Scott wrote: > ----- Original Message ----- > > [...] > > real 19m31.860s > > user 19m24.391s > > sys 0m2.566s > > """ > > OOC, can you time a pmlogsummary run on this archive? Sure: time pmlogsummary 20141029.00.10 &> /dev/null real 0m8.651s user 0m2.913s sys 0m0.229s It's a python issue due to all the type conversions mostly. > > While 20 minutes to parse such a big archive might be relatively ok, I > > was wondering what options I have to improve this. The ones I can > > currently think of are: > > > > 1) Split the time interval parsing over multiple CPUs. I can divide the > > archive in subintervals (one per cpu) and have each CPU do its own > > subinterval parsing and then stitch everything together at the end. > > This is the approach I currently use to create the graph images that go > > in the pdf (as matplotlib+reportlab aren't the fastest thing on the > > planet) > > Should definitely help, since it appears to be CPU bound currently. > > > 2) Implement a function in the C python bindings which returns a > > python dictionary as described above. This would save me all the > > ctypes/__init__ costs and probably I would shave some time off as there > > would be less python->C function calls. Maybe we can find a generic > > enough API for this to be usable by other clients? > > Yep, sounds good. > > > 3) See if I can use Cython tricks to speed up things > > > > 4) Anything else I have not thought of? > > pmlogsummary uses that raw archive fetching interface we talked about > awhile back, which isn't always ideal for your needs - I'm interested > in seeing the time difference though, if you could run that locally? You mean pmFetchArchive() I assume? I did not notice any speed improvements when using that one from python (plus it does not support INTERP). I'll give 2) a try and then we can see if it is generic enough for other python users. Thanks, Michele -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D From kenj@internode.on.net Thu Oct 30 05:24:23 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 5C8CA7F3F for ; Thu, 30 Oct 2014 05:24:23 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 3BEE8304048 for ; Thu, 30 Oct 2014 03:24:20 -0700 (PDT) X-ASG-Debug-ID: 1414664657-04cb6c2ef9447a90001-S8gJnT Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id Cmk5Ig3PePh5xmlX for ; Thu, 30 Oct 2014 03:24:17 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AsYBAI0RUlR20YErPGdsb2JhbAANT4NiWIc9xhWJEgEBAQEBBgEBAQE4hGdVNgIFFgsCCwMCAQIBMRoNCAEBuw14lSiBLJJigVQFllehXliCSwEBAQ Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail04.adl6.internode.on.net with ESMTP; 30 Oct 2014 20:54:14 +1030 Message-ID: <54521217.5030709@internode.on.net> Date: Thu, 30 Oct 2014 21:25:27 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: PCP Subject: rpm-based build failures Content-Type: text/plain; charset=utf-8 X-ASG-Orig-Subj: rpm-based build failures Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1414664657 X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11045 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- I'm seeing failures like below on FC19, FC20 and openSUSE 13.1... == Configuring pcp, log is in /home/kenj/src/pcp/Logs/pcp (--prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib --libexecdir=/usr/lib --localstatedir=/var --sharedstatedir=/usr/com --mandir=/usr/share/man) == Building rpm packages, log is in /home/kenj/src/pcp/Logs/pcp Packaging RPMs via pack_pcp failed, see log in /home/kenj/src/pcp/Logs/pcp config.status: creating src/include/pcp/configsz.h == Building rpm packages, log is in /home/kenj/src/pcp/Logs/pcp make: Entering directory `/home/kenj/src/pcp/pcp-../build/rpm' Generating pcp.spec from pcp.spec.in DEFS=`grep '^--define' rpmmacros`; \ eval /usr/bin/rpmbuild -ba $DEFS \ --clean pcp.spec error: line 3: Illegal sequence ".." in: Version: .. make: *** [pack_pcp] Error 1 make: Leaving directory `/home/kenj/src/pcp/pcp-../build/rpm' From kenj@internode.on.net Thu Oct 30 14:42:06 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 79DEC7F3F for ; Thu, 30 Oct 2014 14:42:06 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 67CDE304032 for ; Thu, 30 Oct 2014 12:42:06 -0700 (PDT) X-ASG-Debug-ID: 1414698120-04cb6c2efb45ec50001-S8gJnT Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by cuda.sgi.com with ESMTP id WzshARQeMC8TeArZ for ; Thu, 30 Oct 2014 12:42:00 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.145 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgMCABqUUlR20YErPGdsb2JhbAANT4NiWIc9xVqHVQKBOgEBAQEBBgEBAQE4hD4BAQR4EQsYCRYPCQMCAQIBMRQTCAEBvUSVZgEBCAIBH5BcOhaENQWWWokDmGNYgksBAQE Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail06.adl6.internode.on.net with ESMTP; 31 Oct 2014 06:11:59 +1030 Message-ID: <545294D1.3050001@internode.on.net> Date: Fri, 31 Oct 2014 06:43:13 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: Re: [pcp] rpm-based build failures .. moved to python code References: <54521217.5030709@internode.on.net> X-ASG-Orig-Subj: Re: [pcp] rpm-based build failures .. moved to python code In-Reply-To: <54521217.5030709@internode.on.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail06.adl6.internode.on.net[150.101.137.145] X-Barracuda-Start-Time: 1414698120 X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11056 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- On 30/10/14 21:25, Ken McDonell wrote: > I'm seeing failures like below on FC19, FC20 and openSUSE 13.1... > > ... At lease for one VM I know what is happening here, but not why. My VERSIONS.pcp file is empty (and there is a block of nulls in the tail of the Logs/pcp file) ... these VMs have preallocated disk space, so I am not sure why I'm seeing something like a file system full symptom. Anyway, that seems to have passed now, but I'm seeing build failures in the python code now (on Fedora 19, I've not delved into the other cases yet) ... I've tried make distclean, but Makepkgs fails the same way. If it makes any difference, python and python3 are both installed on this system kenj@vm22:~/src/pcp$ python --version Python 2.7.5 kenj@vm22:~/src/pcp$ python3 --version Python 3.3.2 === python === python3 setup.py build_ext --include-dirs=../../src/include --library-dirs=../../src/libpcp/src:../../src/libpcp_pmda/src:../../src/libpcp_gui/src:../../src/libpcp_import/src:../../src/libpcp_mmv/src running build_ext building 'cpmapi' extension creating build creating build/temp.linux-x86_64-3.3 gcc -pthread -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC -fno-strict-aliasing -D_GNU_SOURCE -fstack-protector-all -D_FORTIFY_SOURCE=2 -Wall -O2 -g -DPCP_DEBUG -DPCP_VERSION="3.10.0" -I./src/include -I./src/include/pcp -fPIC -fno-strict-aliasing -D_GNU_SOURCE -fstack-protector-all -D_FORTIFY_SOURCE=2 -Wall -O2 -g -DPCP_DEBUG -DPCP_VERSION="3.10.0" -I../src/include -I../src/include/pcp -fPIC -fno-strict-aliasing -D_GNU_SOURCE -fstack-protector-all -D_FORTIFY_SOURCE=2 -Wall -O2 -g -DPCP_DEBUG -DPCP_VERSION="3.10.0" -I../../src/include -I../../src/include/pcp -fPIC -I../../src/include -I/usr/include/python3.3m -c pmapi.c -o build/temp.linux-x86_64-3.3/pmapi.o pmapi.c:27:20: fatal error: Python.h: No such file or directory #include ^ compilation terminated. From fche@redhat.com Thu Oct 30 15:33:01 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id B42B27F3F for ; Thu, 30 Oct 2014 15:33:01 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 343F1AC004 for ; Thu, 30 Oct 2014 13:32:58 -0700 (PDT) X-ASG-Debug-ID: 1414701176-04cb6c2efa461210001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id cQdqYgZGP7f7uCF3 (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 30 Oct 2014 13:32:57 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9UKWulK021967 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Thu, 30 Oct 2014 16:32:56 -0400 Received: from fche.csb (vpn-234-89.phx2.redhat.com [10.3.234.89]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9UKWtA6002304 for ; Thu, 30 Oct 2014 16:32:56 -0400 Received: by fche.csb (Postfix, from userid 2569) id 467315816E; Thu, 30 Oct 2014 16:32:55 -0400 (EDT) Date: Thu, 30 Oct 2014 16:32:55 -0400 From: "Frank Ch. Eigler" To: pcp developers Subject: pcp updates, pcpfans.git fche/dev Message-ID: <20141030203255.GA1913@redhat.com> X-ASG-Orig-Subj: pcp updates, pcpfans.git fche/dev Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.2i X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1414701176 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Hi - It's great to see pmmgr/pmwebd mostly back in the fold. The pcpfans.git fche/dev branch contains fixes that were lost in the code movement. Most of them are documentation bits, and a few bugs (as QA'd by existing test cases running in odd environemnts, or by QA test cases as yet unwelcome in the fold). commit 0aeeb740dfcc1c21460e62a648fd4299b57c41f1 (HEAD, origin/fche/dev, fche/dev) Author: Frank Ch. Eigler Date: Wed Apr 9 13:28:14 2014 -0400 pmmgr testing: quicken, avoid some granularity-edge races After concerns, the time taken by the pmmgr 666 test case are now reduced to about 6 minutes. [removed from merge] Changes to pmmgr proper involve active avoidance of granular-mode period boundaries. pmloggers are instructed to shut down one second before, and new pmloggers are precluded from launching within that transitional second. This seems to make the resulting archives' timespans match the intuitive expetations. Conflicts: qa/666 qa/666.out commit 78a8bfeb7b309615655fe63d7095bd7450439835 Author: Frank Ch. Eigler Date: Mon Sep 15 12:40:21 2014 -0400 pmmgr todo++ commit 5f068f5287a946989385967d852f69622b6665e5 Author: Frank Ch. Eigler Date: Thu Sep 11 15:11:06 2014 -0400 pmmgr todo++ commit 58df55824264f6502aac214f9a8e0849bc5c2a2c Author: Frank Ch. Eigler Date: Fri Jul 4 13:07:47 2014 -0400 pmmgr TODO++ commit 976765d6940303556cc7772fe537b3ba5f51d039 Author: Frank Ch. Eigler Date: Wed Jun 25 20:10:35 2014 -0400 pmmgr todo++ commit 8fba8a70ed7849b9bb6c24b24a23c7ac786d985a Author: Frank Ch. Eigler Date: Tue Jul 1 23:34:42 2014 -0400 pmwebd man page: point out rate-conversion commit 5c22cc929016a854ed5fa71c698837c60ca5ccfe Author: Frank Ch. Eigler Date: Mon Oct 13 21:57:24 2014 -0400 pmwebapi: adapt to some libmicrohttpd's multiple "completed" callbacks In some circumstances, we could receive two callbacks, and would try to delete the postprocessor/context objects each time. Now, clear the state variable after delete so we don't try it again. Tested by preexisting 660 test case, failing with libmicrohttpd 0.9.34-4.fc22.x86_64 commit 9bfedf7496eb865c78c04b36293d3909ef49758b Author: Frank Ch. Eigler Date: Mon Oct 13 16:39:18 2014 -0400 pmwebapi.3: continue elaborating on -h local: vs. -L commit 72a5a381f1b3ed8722205471603159bee2b52aa5 Author: Frank Ch. Eigler Date: Thu Jun 26 18:18:12 2014 -0400 pmwebd: update docs for graphite capabilities commit b40ea4c33e4d290f5f2eca51c191bd6f14cfc225 Author: Michele Baldessari Date: Mon Oct 13 22:08:43 2014 +0200 Clarify the limits of PM_CONTEXT_LOCAL when using pmwebd From nscott@redhat.com Thu Oct 30 15:54:40 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 93A697F3F for ; Thu, 30 Oct 2014 15:54:40 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 31212AC006 for ; Thu, 30 Oct 2014 13:54:40 -0700 (PDT) X-ASG-Debug-ID: 1414702470-04bdf038d0550b80001-S8gJnT Received: from mx4-phx2.redhat.com (mx4-phx2.redhat.com [209.132.183.25]) by cuda.sgi.com with ESMTP id o3aHXfDkwjK1HnYx (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 30 Oct 2014 13:54:30 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.25 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx4-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s9UKsTcb028760; Thu, 30 Oct 2014 16:54:29 -0400 Date: Thu, 30 Oct 2014 16:54:29 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: "Frank Ch. Eigler" Cc: pcp developers Message-ID: <494689218.4400197.1414702469822.JavaMail.zimbra@redhat.com> In-Reply-To: <20141030203255.GA1913@redhat.com> References: <20141030203255.GA1913@redhat.com> Subject: Re: [pcp] pcp updates, pcpfans.git fche/dev MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] pcp updates, pcpfans.git fche/dev Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.6] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: pcp updates, pcpfans.git fche/dev Thread-Index: LyJOVqJ5A8PSJKAqWVzWbZ+5Gb2sHQ== X-Barracuda-Connect: mx4-phx2.redhat.com[209.132.183.25] X-Barracuda-Start-Time: 1414702470 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.03 X-Barracuda-Spam-Status: No, SCORE=0.03 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_SA_TO_FROM_DOMAIN_MATCH, THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11058 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.01 BSF_SC0_SA_TO_FROM_DOMAIN_MATCH Sender Domain Matches Recipient Domain ----- Original Message ----- > [...] > commit 0aeeb740dfcc1c21460e62a648fd4299b57c41f1 (HEAD, origin/fche/dev, > fche/dev) > Author: Frank Ch. Eigler > Date: Wed Apr 9 13:28:14 2014 -0400 > > pmmgr testing: quicken, avoid some granularity-edge races > > After concerns, the time taken by the pmmgr 666 test case are now > reduced to about 6 minutes. Hmmm, sorry, I thought you said you were going to fix this? Couple releases back, most recently ... http://www.pcp.io/pipermail/pcp/2014-August/005374.html So, I've been waiting on that - 6 minutes is still far longer than any other test of course (more than double?) - even though it is down from the original 20 minutes, which is great. Those other long running tests (in the 2-3 minute range) are already a problem & need to be improved too if we can. thanks. -- Nathan From nscott@redhat.com Thu Oct 30 16:02:04 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 946547F3F for ; Thu, 30 Oct 2014 16:02:04 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 733258F8035 for ; Thu, 30 Oct 2014 14:02:01 -0700 (PDT) X-ASG-Debug-ID: 1414702919-04cb6c2efb4629c0001-S8gJnT Received: from mx4-phx2.redhat.com (mx4-phx2.redhat.com [209.132.183.25]) by cuda.sgi.com with ESMTP id 4YtAT9ELvviXSZPu (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 30 Oct 2014 14:01:59 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.25 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx4-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s9UL1twY030258; Thu, 30 Oct 2014 17:01:55 -0400 Date: Thu, 30 Oct 2014 17:01:55 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Ken McDonell Cc: pcp@oss.sgi.com Message-ID: <1970763283.4406287.1414702915209.JavaMail.zimbra@redhat.com> In-Reply-To: <545294D1.3050001@internode.on.net> References: <54521217.5030709@internode.on.net> <545294D1.3050001@internode.on.net> Subject: Re: [pcp] rpm-based build failures .. moved to python code MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] rpm-based build failures .. moved to python code Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.6] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: rpm-based build failures .. moved to python code Thread-Index: 2wZRvM+RdvAcF91L1zpczvb6QGZ7CQ== X-Barracuda-Connect: mx4-phx2.redhat.com[209.132.183.25] X-Barracuda-Start-Time: 1414702919 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11058 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... ----- Original Message ----- > [...] > Anyway, that seems to have passed now, but I'm seeing build failures in the > python code now (on Fedora 19, I've not delved into the other cases yet) ... > > I've tried make distclean, but Makepkgs fails the same way. > > If it makes any difference, python and python3 are both installed on this > system (yep, it does make a difference) > kenj@vm22:~/src/pcp$ python --version > Python 2.7.5 > kenj@vm22:~/src/pcp$ python3 --version > Python 3.3.2 Are both versions of python[3]-devel installed? > -I../../src/include -I/usr/include/python3.3m -c pmapi.c -o > build/temp.linux-x86_64-3.3/pmapi.o > pmapi.c:27:20: fatal error: Python.h: No such file or directory > #include > ^ > compilation terminated. This looks sorta like configure has decided to go ahead with python3 but later we find no python3 devel environment. There are checks in configure.ac for this though, its not clear how we are circumventing those in your case here? cheers. -- Nathan From nscott@redhat.com Thu Oct 30 16:14:05 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 8A6EB7F3F for ; Thu, 30 Oct 2014 16:14:05 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 360E4AC001 for ; Thu, 30 Oct 2014 14:14:02 -0700 (PDT) X-ASG-Debug-ID: 1414703640-04bdf038d2551710001-S8gJnT Received: from mx3-phx2.redhat.com (mx3-phx2.redhat.com [209.132.183.24]) by cuda.sgi.com with ESMTP id nDuZzxU4rTEqF628 (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 30 Oct 2014 14:14:00 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.24 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx3-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s9ULDjZV028669; Thu, 30 Oct 2014 17:13:45 -0400 Date: Thu, 30 Oct 2014 17:13:45 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Ken McDonell Cc: pcp@oss.sgi.com Message-ID: <495790646.4414999.1414703625215.JavaMail.zimbra@redhat.com> In-Reply-To: <1970763283.4406287.1414702915209.JavaMail.zimbra@redhat.com> References: <54521217.5030709@internode.on.net> <545294D1.3050001@internode.on.net> <1970763283.4406287.1414702915209.JavaMail.zimbra@redhat.com> Subject: Re: [pcp] rpm-based build failures .. moved to python code MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] rpm-based build failures .. moved to python code Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.6] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: rpm-based build failures .. moved to python code Thread-Index: 2wZRvM+RdvAcF91L1zpczvb6QGZ7CdDIrAxz X-Barracuda-Connect: mx3-phx2.redhat.com[209.132.183.24] X-Barracuda-Start-Time: 1414703640 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11058 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... ----- Original Message ----- > [...] > This looks sorta like configure has decided to go ahead with python3 > but later we find no python3 devel environment. There are checks in > configure.ac for this though, its not clear how we are circumventing > those in your case here? Ah, think I found the problem - could you try again with current dev? thanks. -- Nathan From kenj@internode.on.net Thu Oct 30 18:24:36 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 885167F3F for ; Thu, 30 Oct 2014 18:24:36 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 596518F8035 for ; Thu, 30 Oct 2014 16:24:33 -0700 (PDT) X-ASG-Debug-ID: 1414711470-04bdf038d0556320001-S8gJnT Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id q3UnAVqIXlalwj6D for ; Thu, 30 Oct 2014 16:24:31 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgcCAOvHUlR20YErPGdsb2JhbAANT4NiWIMGhDfFWodVAoE9AQEBAQEGAQEBATiEPQEBAQQjFUABDAQLGAICBRYLAgIJAwIBAgExFAYNAQcBAb0leJRvAQEBAQEBAQMBAQEBAQEBG4Esj2MHgneBVAEEj3GGaYhHPJBPiBRYgksBAQE Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail04.adl6.internode.on.net with ESMTP; 31 Oct 2014 09:51:33 +1030 Message-ID: <5452C847.7010906@internode.on.net> Date: Fri, 31 Oct 2014 10:22:47 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: Nathan Scott CC: pcp@oss.sgi.com Subject: Re: [pcp] rpm-based build failures .. moved to python code References: <54521217.5030709@internode.on.net> <545294D1.3050001@internode.on.net> <1970763283.4406287.1414702915209.JavaMail.zimbra@redhat.com> <495790646.4414999.1414703625215.JavaMail.zimbra@redhat.com> X-ASG-Orig-Subj: Re: [pcp] rpm-based build failures .. moved to python code In-Reply-To: <495790646.4414999.1414703625215.JavaMail.zimbra@redhat.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1414711470 X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11060 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- On 31/10/14 08:13, Nathan Scott wrote: > > > ----- Original Message ----- >> [...] >> This looks sorta like configure has decided to go ahead with python3 >> but later we find no python3 devel environment. There are checks in >> configure.ac for this though, its not clear how we are circumventing >> those in your case here? > > Ah, think I found the problem - could you try again with current dev? python3 devel was not installed on each of these failing platforms. And the t-o-t code has fixed the problem. I'll install python3 devel on a couple of these systems to increase that coverage, but leave one without it to include that case as well. Thanks. From kenj@internode.on.net Thu Oct 30 18:53:28 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 5D4FF7F3F for ; Thu, 30 Oct 2014 18:53:28 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 3CB8F8F8033 for ; Thu, 30 Oct 2014 16:53:28 -0700 (PDT) X-ASG-Debug-ID: 1414713202-04cbb070c84b3840001-S8gJnT Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id VoA9J4fbgX8wTJPr for ; Thu, 30 Oct 2014 16:53:22 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgMCAC3PUlR20YErPGdsb2JhbAANT4Q6hz3NLwKBPwEBAQEBBgEBAQE4hD0BAQEEOEANBAsUBAkWDwkDAgECATEUEwYCAQG9KpVlAQEIAgEfkRYWhDUBBLhAWIJLAQEB Received: from ppp118-209-129-43.lns20.mel8.internode.on.net (HELO [192.168.1.100]) ([118.209.129.43]) by ipmail04.adl6.internode.on.net with ESMTP; 31 Oct 2014 10:23:05 +1030 Message-ID: <5452CFAB.5050703@internode.on.net> Date: Fri, 31 Oct 2014 10:54:19 +1100 From: Ken McDonell User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: pcp@oss.sgi.com Subject: Re: [pcp] pcp updates, pcpfans.git fche/dev References: <20141030203255.GA1913@redhat.com> <494689218.4400197.1414702469822.JavaMail.zimbra@redhat.com> X-ASG-Orig-Subj: Re: [pcp] pcp updates, pcpfans.git fche/dev In-Reply-To: <494689218.4400197.1414702469822.JavaMail.zimbra@redhat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1414713202 X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11061 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- On 31/10/14 07:54, Nathan Scott wrote: > > > ----- Original Message ----- >> [...] >> commit 0aeeb740dfcc1c21460e62a648fd4299b57c41f1 (HEAD, origin/fche/dev, >> fche/dev) >> Author: Frank Ch. Eigler >> Date: Wed Apr 9 13:28:14 2014 -0400 >> >> pmmgr testing: quicken, avoid some granularity-edge races >> >> After concerns, the time taken by the pmmgr 666 test case are now >> reduced to about 6 minutes. > > ... Let me offer some background and guidance for the duration of QA tests. While a quick test is a good test, there are some cases where longer running tests are required ... for example - slow memory leaks that require lots of iterations to expose 'em - tests involving timeouts - performance tests (to run long enough to be statistically stable) - ... But when a test is long running because it includes a large number of test cases, this creates some operational problems. As one of the people who probably spends more time than most trying to triage failures of test cases written by others in a distant time or place, I would much rather have 10 tests each running for 30 seconds than one long test that runs for 5 minutes. Multiple small tests provide more focus on the area of failure, and even in systemic problems triage is much easier with a small test case. Hope this helps. From nscott@redhat.com Thu Oct 30 19:23:04 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 1E17F7F3F for ; Thu, 30 Oct 2014 19:23:04 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 0C49430404E for ; Thu, 30 Oct 2014 17:23:00 -0700 (PDT) X-ASG-Debug-ID: 1414714978-04cbb070c54b4720001-S8gJnT Received: from mx4-phx2.redhat.com (mx4-phx2.redhat.com [209.132.183.25]) by cuda.sgi.com with ESMTP id BeHUqFFIpr7Zbt9a (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 30 Oct 2014 17:22:59 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.25 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx4-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s9V0MtBC029119; Thu, 30 Oct 2014 20:22:55 -0400 Date: Thu, 30 Oct 2014 20:22:55 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Ken McDonell Cc: pcp developers Message-ID: <1627457629.4480548.1414714975111.JavaMail.zimbra@redhat.com> In-Reply-To: <5452CFAB.5050703@internode.on.net> References: <20141030203255.GA1913@redhat.com> <494689218.4400197.1414702469822.JavaMail.zimbra@redhat.com> <5452CFAB.5050703@internode.on.net> Subject: Re: [pcp] pcp updates, pcpfans.git fche/dev MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] pcp updates, pcpfans.git fche/dev Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.6] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: pcp updates, pcpfans.git fche/dev Thread-Index: /Tpy73klD/XoqcRXu9I3tDeL8UdnlA== X-Barracuda-Connect: mx4-phx2.redhat.com[209.132.183.25] X-Barracuda-Start-Time: 1414714979 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11061 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... ----- Original Message ----- > > ----- Original Message ----- > >> [...] > >> commit 0aeeb740dfcc1c21460e62a648fd4299b57c41f1 (HEAD, origin/fche/dev, > >> fche/dev) > >> Author: Frank Ch. Eigler > >> Date: Wed Apr 9 13:28:14 2014 -0400 > >> > >> pmmgr testing: quicken, avoid some granularity-edge races > >> > >> After concerns, the time taken by the pmmgr 666 test case are now > >> reduced to about 6 minutes. > > > > ... > > Let me offer some background and guidance for the duration of QA tests. > FWIW, this advice has been offered before. It turns out, the above commit message was incorrect & there was no regression testing in this series (666 was not added back in here); that confused me initially too, from just looking at the git-log. I've cherry-picked the change, and reworded the commit message a bit to reflect that its only about "avoid some granularity-edge races". cheers. -- Nathan From nscott@redhat.com Thu Oct 30 21:13:56 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id D05B47F3F for ; Thu, 30 Oct 2014 21:13:56 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 9EDFE8F8035 for ; Thu, 30 Oct 2014 19:13:56 -0700 (PDT) X-ASG-Debug-ID: 1414721631-04bdf038cf55b850001-S8gJnT Received: from mx4-phx2.redhat.com (mx4-phx2.redhat.com [209.132.183.25]) by cuda.sgi.com with ESMTP id muNI7YEtzfUE6vX5 (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 30 Oct 2014 19:13:51 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.25 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx4-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s9V2Dl1E012422; Thu, 30 Oct 2014 22:13:47 -0400 Date: Thu, 30 Oct 2014 22:13:47 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: Ken McDonell , Dave Brolley Cc: pcp@oss.sgi.com Message-ID: <1640459441.4506195.1414721627876.JavaMail.zimbra@redhat.com> In-Reply-To: <545027F9.2020205@internode.on.net> References: <001f01cff196$66cf3e00$346dba00$@internode.on.net> <544FD6DF.8060206@redhat.com> <545027F9.2020205@internode.on.net> Subject: Re: [pcp] user/group access control question MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [pcp] user/group access control question Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.6] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: user/group access control question Thread-Index: nfBlual6fptIpHWZqnJBXEpRtGn2Gw== X-Barracuda-Connect: mx4-phx2.redhat.com[209.132.183.25] X-Barracuda-Start-Time: 1414721631 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11063 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... ----- Original Message ----- > > I'll test it here, but would appreciate it if you could test it in your > > failing environment. > > > > Nathan, can you please also review the updated patch? > > I have this change in the context of more diagnostics and the extensions > to qa/944, so I'll push the commit if everyone is OK with that. Looks good to me - thanks Ken, nice catch & correct diagnosis. cheers. -- Nathan From nscott@redhat.com Thu Oct 30 22:08:51 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 313887F3F for ; Thu, 30 Oct 2014 22:08:51 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 11065304051 for ; Thu, 30 Oct 2014 20:08:47 -0700 (PDT) X-ASG-Debug-ID: 1414724924-04cb6c2efb46e080001-S8gJnT Received: from mx5-phx2.redhat.com (mx5-phx2.redhat.com [209.132.183.37]) by cuda.sgi.com with ESMTP id RvJEYrrEne09mMLo (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Thu, 30 Oct 2014 20:08:45 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.37 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx5-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9V38iFv031357 for ; Thu, 30 Oct 2014 23:08:44 -0400 Date: Thu, 30 Oct 2014 23:08:44 -0400 (EDT) From: Nathan Scott Reply-To: Nathan Scott To: pcp developers Message-ID: <235786320.4523709.1414724924661.JavaMail.zimbra@redhat.com> In-Reply-To: <2003819788.4523432.1414724874497.JavaMail.zimbra@redhat.com> Subject: pcp updates: fche-merge, pmlogextract, qa MIME-Version: 1.0 X-ASG-Orig-Subj: pcp updates: fche-merge, pmlogextract, qa Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.6] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: pcp updates: fche-merge, pmlogextract, qa Thread-Index: zvvF7jffgvFhUkVup/4o/TjfIy1TkQ== X-Barracuda-Connect: mx5-phx2.redhat.com[209.132.183.37] X-Barracuda-Start-Time: 1414724925 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11064 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Changes committed to git://git.pcp.io/pcp.git dev Nathan Scott (8): build: fix typo in python3-devel configure disablement man page: fix typos on dbpmda.1 that Lukas noticed pmlogextract: improve handling of corrupt archives build: add license info for pcp-doc rpm packages qa: fix qa/660 filtering with python 2.6 simplejson.tool qa: fix test qa/660 so it can pass when run from qa/check build: changelog, spec, deb pkg updates for release qa: split qa/660 in two (qa/661) for when jsdemos not installed Frank Ch. Eigler (7): pmwebd: update docs for graphite capabilities pmwebapi.3: continue elaborating on -h local: vs. -L pmwebapi: adapt to some libmicrohttpd's multiple "completed" callbacks pmwebd man page: point out rate-conversion pmmgr todo++ pmmgr: avoid some granularity-edge races pmwebd todo++ Michele Baldessari (1): Clarify the limits of PM_CONTEXT_LOCAL when using pmwebd Paul Cuzner (1): pmdagluster: thread-based timeout for too-long gluster query CHANGELOG | 43 build/rpm/fedora.spec | 2 build/rpm/pcp.spec.in | 1 configure | 2 configure.ac | 2 debian/changelog | 2 man/man1/dbpmda.1 | 4 man/man1/pmwebd.1 | 130 +- man/man3/pmwebapi.3 | 91 + qa/660 | 144 -- qa/660.out.4 | 1841 ----------------------------------- qa/660.out.46 | 1841 ----------------------------------- qa/661 | 88 + qa/661.out | 1828 ++++++++++++++++++++++++++++++++++ qa/997 | 37 qa/997.out | 18 qa/common.webapi | 41 qa/group | 4 src/pmdas/gluster/pmdagluster.python | 136 +- src/pmlogextract/pmlogextract.c | 4 src/pmmgr/TODO | 9 src/pmmgr/pmmgr.cxx | 21 src/pmwebapi/TODO | 1 src/pmwebapi/main.cxx | 4 24 files changed, 2342 insertions(+), 3952 deletions(-) From dak-unpriv@franck.debian.org Fri Oct 31 02:16:49 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id C505F7F3F for ; Fri, 31 Oct 2014 02:16:49 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id A40A18F8033 for ; Fri, 31 Oct 2014 00:16:49 -0700 (PDT) X-ASG-Debug-ID: 1414739802-04cbb070c64c2840001-S8gJnT Received: from mailly.debian.org (mailly.debian.org [82.195.75.114]) by cuda.sgi.com with ESMTP id hgsyuPPcZmshjSLz (version=TLSv1 cipher=AES128-SHA bits=128 verify=NO) for ; Fri, 31 Oct 2014 00:16:43 -0700 (PDT) X-Barracuda-Envelope-From: dak-unpriv@franck.debian.org X-Barracuda-Apparent-Source-IP: 82.195.75.114 Received: from franck.debian.org ([138.16.160.12]) from C=NA,ST=NA,L=Ankh Morpork,O=Debian SMTP,OU=Debian SMTP CA,CN=franck.debian.org,EMAIL=hostmaster@franck.debian.org (verified) by mailly.debian.org with esmtps (TLS1.2:DHE_RSA_AES_128_CBC_SHA1:128) (Exim 4.80) (envelope-from ) id 1Xk6Rt-0003oF-Gl for pcp@oss.sgi.com; Fri, 31 Oct 2014 07:16:41 +0000 Received: from dak-unpriv by franck.debian.org with local (Exim 4.80) (envelope-from ) id 1Xk6Rs-0005x1-6a for pcp@oss.sgi.com; Fri, 31 Oct 2014 07:16:40 +0000 To: pcp@oss.sgi.com From: Debian FTP Masters Subject: Processing of pcp_3.10.0_i386.changes Date: Fri, 31 Oct 2014 07:16:40 +0000 X-ASG-Orig-Subj: Processing of pcp_3.10.0_i386.changes X-Debian: DAK X-DAK: DAK Precedence: bulk Auto-Submitted: auto-generated X-Debian-Package: pcp Message-Id: X-Barracuda-Connect: mailly.debian.org[82.195.75.114] X-Barracuda-Start-Time: 1414739802 X-Barracuda-Encrypted: AES128-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11068 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- pcp_3.10.0_i386.changes uploaded successfully to localhost along with the files: pcp_3.10.0_i386.deb pcp-conf_3.10.0_i386.deb libpcp3-dev_3.10.0_i386.deb libpcp3_3.10.0_i386.deb libpcp-gui2-dev_3.10.0_i386.deb libpcp-gui2_3.10.0_i386.deb libpcp-mmv1-dev_3.10.0_i386.deb libpcp-mmv1_3.10.0_i386.deb libpcp-pmda3-dev_3.10.0_i386.deb libpcp-pmda3_3.10.0_i386.deb libpcp-trace2-dev_3.10.0_i386.deb libpcp-trace2_3.10.0_i386.deb libpcp-import1-dev_3.10.0_i386.deb libpcp-import1_3.10.0_i386.deb python-pcp_3.10.0_i386.deb libpcp-pmda-perl_3.10.0_i386.deb libpcp-import-perl_3.10.0_i386.deb libpcp-logsummary-perl_3.10.0_i386.deb libpcp-mmv-perl_3.10.0_i386.deb pcp-import-sar2pcp_3.10.0_all.deb pcp-import-mrtg2pcp_3.10.0_all.deb pcp-import-sheet2pcp_3.10.0_all.deb pcp-import-iostat2pcp_3.10.0_all.deb pcp-import-collectl2pcp_3.10.0_i386.deb pcp-doc_3.10.0_all.deb pcp-testsuite_3.10.0_i386.deb pcp-manager_3.10.0_i386.deb pcp-webapi_3.10.0_i386.deb pcp-gui_3.10.0_i386.deb pcp_3.10.0.dsc pcp_3.10.0.tar.xz Greetings, Your Debian queue daemon (running on host franck.debian.org) From envelope@ftp-master.debian.org Fri Oct 31 02:19:48 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 11C5A7F3F for ; Fri, 31 Oct 2014 02:19:48 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id B2DA5AC001 for ; Fri, 31 Oct 2014 00:19:44 -0700 (PDT) X-ASG-Debug-ID: 1414739981-04bdf038cf5648c0001-S8gJnT Received: from mailly.debian.org (mailly.debian.org [82.195.75.114]) by cuda.sgi.com with ESMTP id WXKgLwWQWblIPP5q (version=TLSv1 cipher=AES128-SHA bits=128 verify=NO) for ; Fri, 31 Oct 2014 00:19:42 -0700 (PDT) X-Barracuda-Envelope-From: envelope@ftp-master.debian.org X-Barracuda-Apparent-Source-IP: 82.195.75.114 Received: from franck.debian.org ([138.16.160.12]) from C=NA,ST=NA,L=Ankh Morpork,O=Debian SMTP,OU=Debian SMTP CA,CN=franck.debian.org,EMAIL=hostmaster@franck.debian.org (verified) by mailly.debian.org with esmtps (TLS1.2:DHE_RSA_AES_128_CBC_SHA1:128) (Exim 4.80) (envelope-from ) id 1Xk6Um-000426-IA; Fri, 31 Oct 2014 07:19:40 +0000 Received: from dak by franck.debian.org with local (Exim 4.80) (envelope-from ) id 1Xk6Ul-0007I3-1L; Fri, 31 Oct 2014 07:19:39 +0000 From: Debian FTP Masters To: PCP Development Team , Nathan Scott X-DAK: dak process-upload X-Debian: DAK X-Debian-Package: pcp Precedence: bulk Auto-Submitted: auto-generated MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Subject: pcp_3.10.0_i386.changes ACCEPTED into unstable Message-Id: X-ASG-Orig-Subj: pcp_3.10.0_i386.changes ACCEPTED into unstable Date: Fri, 31 Oct 2014 07:19:39 +0000 X-Barracuda-Connect: mailly.debian.org[82.195.75.114] X-Barracuda-Start-Time: 1414739982 X-Barracuda-Encrypted: AES128-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11068 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Accepted: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Format: 1.8 Date: Fri, 31 Oct 2014 13:16:51 +1100 Source: pcp Binary: pcp pcp-conf libpcp3-dev libpcp3 libpcp-gui2-dev libpcp-gui2 libpcp-mmv1-dev libpcp-mmv1 libpcp-pmda3-dev libpcp-pmda3 libpcp-trace2-dev libpcp-trace2 libpcp-import1-dev libpcp-import1 python-pcp libpcp-pmda-perl libpcp-import-perl libpcp-logsummary-perl libpcp-mmv-perl pcp-import-sar2pcp pcp-import-mrtg2pcp pcp-import-sheet2pcp pcp-import-iostat2pcp pcp-import-collectl2pcp pcp-doc pcp-testsuite pcp-manager pcp-webapi pcp-gui Architecture: source i386 all Version: 3.10.0 Distribution: unstable Urgency: low Maintainer: PCP Development Team Changed-By: Nathan Scott Description: libpcp-gui2 - Performance Co-Pilot graphical client tools library libpcp-gui2-dev - Performance Co-Pilot graphical client tools library and headers libpcp-import-perl - Performance Co-Pilot log import Perl module libpcp-import1 - Performance Co-Pilot data import library libpcp-import1-dev - Performance Co-Pilot data import library and headers libpcp-logsummary-perl - Performance Co-Pilot historical log summary module libpcp-mmv-perl - Performance Co-Pilot Memory Mapped Value Perl module libpcp-mmv1 - Performance Co-Pilot Memory Mapped Value client library libpcp-mmv1-dev - Performance Co-Pilot Memory Mapped Value library and headers libpcp-pmda-perl - Performance Co-Pilot Domain Agent Perl module libpcp-pmda3 - Performance Co-Pilot Domain Agent library libpcp-pmda3-dev - Performance Co-Pilot Domain Agent library and headers libpcp-trace2 - Performance Co-Pilot application tracing library libpcp-trace2-dev - Performance Co-Pilot application tracing library and headers libpcp3 - Performance Co-Pilot library libpcp3-dev - Performance Co-Pilot library and headers pcp - System level performance monitoring and performance management pcp-conf - Performance Co-Pilot runtime configuration pcp-doc - Documentation and tutorial for the Performance Co-Pilot pcp-gui - Visualisation tools for the Performance Co-Pilot toolkit pcp-import-collectl2pcp - Tool for importing data from collectl into PCP archive logs pcp-import-iostat2pcp - Tool for importing data from iostat into PCP archive logs pcp-import-mrtg2pcp - Tool for importing data from MRTG into PCP archive logs pcp-import-sar2pcp - Tool for importing data from sar into PCP archive logs pcp-import-sheet2pcp - Tool for importing data from a spreadsheet into PCP archive logs pcp-manager - Performance Co-Pilot (PCP) manager daemon pcp-testsuite - Performance Co-Pilot (PCP) Test Suite pcp-webapi - Performance Co-Pilot (PCP) web API service python-pcp - Performance Co-Pilot Python PMAPI module Changes: pcp (3.10.0) unstable; urgency=low . * New release (full details in CHANGELOG). Checksums-Sha1: 1413af57b4812afce108b086a41c95a7aec9a811 2817 pcp_3.10.0.dsc 8083a631499f310226c0c207ab991b24472050fe 9398032 pcp_3.10.0.tar.xz 49e4ba44a9dcab87fe632a5b44e68e0fdb0653ce 1249240 pcp_3.10.0_i386.deb a76cd3e2979ddf44095ea6bf9e3d7e687f4ed8f6 17188 pcp-conf_3.10.0_i386.deb 7b248a4b7b0e78480f8a1ef34338d19fcc7ab3dc 416756 libpcp3-dev_3.10.0_i386.deb 44106a14dbe01b172d8103dc2c247d46e77dabfb 189142 libpcp3_3.10.0_i386.deb e0ae79c8d1094dee784ebc77225286d1fd55e1ae 16670 libpcp-gui2-dev_3.10.0_i386.deb 6e34f44a1fdc7c69aa9c4fe412665b99bddb4cfb 15560 libpcp-gui2_3.10.0_i386.deb db33054ad4837006de850dbd0d4bdb759a1dcc69 19244 libpcp-mmv1-dev_3.10.0_i386.deb 71b7767422c8f89b11f9e20d9f02d607fd0eb900 12582 libpcp-mmv1_3.10.0_i386.deb 6af09a2b66cd456c67b450f25a02cf063eb96a0d 95390 libpcp-pmda3-dev_3.10.0_i386.deb 8693eeb5c065fd352babc8661cc766389569ac7f 36586 libpcp-pmda3_3.10.0_i386.deb c76eb1598d8766dd7c6b976abd839e54f0217811 27024 libpcp-trace2-dev_3.10.0_i386.deb 04c0bd4ca2ed9ffe0cb2ce4a997e80e1ac53c738 19756 libpcp-trace2_3.10.0_i386.deb 37d86e6942541b5a51f115076d23c2e53e2fa23b 16304 libpcp-import1-dev_3.10.0_i386.deb f63df8f696ced85d67a707f91a4bd2e0c4c533b6 15842 libpcp-import1_3.10.0_i386.deb 7c4135d3009f9f3bb6741ff4cb098b3fcf11c728 50618 python-pcp_3.10.0_i386.deb 61f9c9c410a993ea8b02366ee8d90a59f0d4a338 39546 libpcp-pmda-perl_3.10.0_i386.deb aa4731aa4ecb72f7dd7b74096b74322a012d8e2b 16980 libpcp-import-perl_3.10.0_i386.deb 09546eeba861178e0af94b3bc3d42fd987cd7fd3 11898 libpcp-logsummary-perl_3.10.0_i386.deb db309909c76e2e8c9c75d261a3d6e1385915e954 18234 libpcp-mmv-perl_3.10.0_i386.deb 81d5e76f0a6a443262bf3ce2a825ce3b6db5c03b 14908 pcp-import-sar2pcp_3.10.0_all.deb d24b694ba6cbda2a05952057a0efe28c027b48ec 9182 pcp-import-mrtg2pcp_3.10.0_all.deb 17fb5a09484dc57428a4985c8d6fb27bd64a2ea0 15662 pcp-import-sheet2pcp_3.10.0_all.deb d1b83af7f88b2b2113e57b64b47af86b4f9628b5 16112 pcp-import-iostat2pcp_3.10.0_all.deb 3195bd8f37520dfcf1783e9e7ce75c047e45aab5 23700 pcp-import-collectl2pcp_3.10.0_i386.deb dcd4baac77f81d1ca9753a5c70a12bb12453d8f0 3146464 pcp-doc_3.10.0_all.deb 9aaeb4eec84b6eb844127042133df4cf8a92df3a 2745362 pcp-testsuite_3.10.0_i386.deb 7dba7ebcc5e60ebb4f5389aec114c680d30d3954 47946 pcp-manager_3.10.0_i386.deb 3cb58cba9ccf022c9f770d6576f19f217781e675 72752 pcp-webapi_3.10.0_i386.deb 9232f529732c4165f4b16d1d6a821e0809359359 654042 pcp-gui_3.10.0_i386.deb Checksums-Sha256: 68e891dbe321153285af4d4c8759cb608edba8639017b1a5694991f41f920432 2817 pcp_3.10.0.dsc ec765dd342019a61387b7d59f0ee2008dac7786ec74c1c3e3a3b3f866e9d6d3e 9398032 pcp_3.10.0.tar.xz 1dd5b4d2cb8fab79999e23ce3e9b9bc1ead4812fdbfd4c24e12e4f6149473812 1249240 pcp_3.10.0_i386.deb 6b935cb964965645408a721c02764a9dee3b6dc558faebe4e892e2f99c4a5106 17188 pcp-conf_3.10.0_i386.deb c45e54563b7d4b1832dd81872e95c41fc5d16e2038341c054a465866b9984cea 416756 libpcp3-dev_3.10.0_i386.deb 6aea0baba666c9ad8b6ab9cb11e553989c07c75ee2c560f464408f124de60796 189142 libpcp3_3.10.0_i386.deb 34f7a0f81459181b44ad94afa18805bf2c9a88e50c969b6bb40af1b5f3cdb136 16670 libpcp-gui2-dev_3.10.0_i386.deb acb4ea3f515adbff4a5413030e07f3daf37cd9438264ebe632f4817129bbc7a9 15560 libpcp-gui2_3.10.0_i386.deb 9955313bbdbb09c4f9d211f83b830ac76c6f1f38bab250601147fcf2cdaa2e6b 19244 libpcp-mmv1-dev_3.10.0_i386.deb d63425179aef3a14d03b628d9e402a5a0467e622e2a1520a94122b5635bc3a81 12582 libpcp-mmv1_3.10.0_i386.deb ea34d7cb0f1b34b9c2bfe845141feed94d5bacd48f857af8b9eb10c260f204cd 95390 libpcp-pmda3-dev_3.10.0_i386.deb 8e3ff50061942fbe4dc89e39d30a2cc1fd9de6ea53be370d62a806239576c403 36586 libpcp-pmda3_3.10.0_i386.deb cad5f1c27f5f581b300136bbd2679071ffbd0c6d76850196303ca9cb1415eba3 27024 libpcp-trace2-dev_3.10.0_i386.deb d03d068431694b765f1e0f3e1188e3e71d7264c41260c42e304426e0a8fffb43 19756 libpcp-trace2_3.10.0_i386.deb 879f742a85b45c9bb4cbfe18459d17c741648cd3a06be1647c5ded29fe7de930 16304 libpcp-import1-dev_3.10.0_i386.deb 1c0d2ede75f510a1c6c2cd1bb1a102cef4e13688282c11f9e22cb6a57737d7e7 15842 libpcp-import1_3.10.0_i386.deb db79e345dd11b5f075a21ed26519ed1ce545cc08b029f881b56a75eaceead8b3 50618 python-pcp_3.10.0_i386.deb 8824e6f3e50d8b333f89af2c9d8d540fecdf83e760ecb62350aa3b128558a5e3 39546 libpcp-pmda-perl_3.10.0_i386.deb 60bf5964a14af86f91f5b68c08d00bb72f8b8b44621b85fa699a1db0a022460c 16980 libpcp-import-perl_3.10.0_i386.deb ad0111cb8c95190ee777f694b6a8d7250408200e33bfe0b0e63a3b24d8991fac 11898 libpcp-logsummary-perl_3.10.0_i386.deb fde0ef9ab079a93ec102fad9345f14dd6b372cf7cd369ebda91198227cdd23b7 18234 libpcp-mmv-perl_3.10.0_i386.deb 3e89ea58bcec1a9f2ff983a94d56060cba0c04f3605cdb16c4486384475ac469 14908 pcp-import-sar2pcp_3.10.0_all.deb 04d8b61c62966f69129e751ad255958d9473d2107e742ab18d603af3d85bacbd 9182 pcp-import-mrtg2pcp_3.10.0_all.deb 9d877ee64ff13170e775343871e350122d59dd87bdc5d0679c2ac1e56c5ae83d 15662 pcp-import-sheet2pcp_3.10.0_all.deb fd07c49bf30597c2ec7930d7938cc47a39de99d44012a5785b6e59d21ad3eda3 16112 pcp-import-iostat2pcp_3.10.0_all.deb 8a2b2ca27680d4eb815b9393419547d650d7236e3893a8f4439a2b5fee9e0229 23700 pcp-import-collectl2pcp_3.10.0_i386.deb 2e7549d147385c5d64d8a5b70a4cac702b863d5f649476548edcbd61e7d3840e 3146464 pcp-doc_3.10.0_all.deb b325a4f9e924cfabd1de900e3316901575b684157a763b444fbe8c17109cd9e1 2745362 pcp-testsuite_3.10.0_i386.deb 81c865033302d36e48823a1faf90685fb01f6d21f5d694f4b7ab38d48749be7b 47946 pcp-manager_3.10.0_i386.deb 84e9fea28829b1667916d11eee7944a32dac9150c67899cb0d6320551e52cc22 72752 pcp-webapi_3.10.0_i386.deb a97e29dfadc279a940061b95af45c8a1f2b8483ac11efaf986655177d4757eca 654042 pcp-gui_3.10.0_i386.deb Files: 2079aed4475b0ed2a18651969548be16 1249240 utils extra pcp_3.10.0_i386.deb dca4108f9d27440c09cb895f2b6acedc 17188 libs extra pcp-conf_3.10.0_i386.deb fb3126928efeed125814d746177bf166 416756 libdevel extra libpcp3-dev_3.10.0_i386.deb 2f88d003b5a57bb5eebe3d966afcdb9e 189142 libs extra libpcp3_3.10.0_i386.deb 2f4093f3f16c3a18b02a6484b46d7e4f 16670 libdevel extra libpcp-gui2-dev_3.10.0_i386.deb bebb16ccf3546d0de5bb801a2f1ec60f 15560 libs extra libpcp-gui2_3.10.0_i386.deb ffc05069d5dfd686104788bee3d732ef 19244 libdevel extra libpcp-mmv1-dev_3.10.0_i386.deb 28f64a9c9057749a7124f96e166486c5 12582 libs extra libpcp-mmv1_3.10.0_i386.deb 255eb9abda0e756a75d2dffc0a85f4fd 95390 libdevel extra libpcp-pmda3-dev_3.10.0_i386.deb e123bbe173e89d7c2a2c66cfb1269215 36586 libs extra libpcp-pmda3_3.10.0_i386.deb 610770a4a7546a849f4f85e32327c312 27024 libdevel extra libpcp-trace2-dev_3.10.0_i386.deb 4eec339a94a529a86f34c67ce215cb60 19756 libs extra libpcp-trace2_3.10.0_i386.deb db5669d7604ed8263b1f4491a0a0e088 16304 libdevel extra libpcp-import1-dev_3.10.0_i386.deb 4db9ffa29f17875a01ba6541fdf22515 15842 libs extra libpcp-import1_3.10.0_i386.deb 45a15d26c40b2127c675e0e97a4fec3b 50618 python extra python-pcp_3.10.0_i386.deb 478acb788fa976b7732516ca8c43e1ed 39546 perl extra libpcp-pmda-perl_3.10.0_i386.deb b20d820dbc5a82d4b245daaf26310a2a 16980 perl extra libpcp-import-perl_3.10.0_i386.deb 9e26f2313de558da5293de314bb04be6 11898 perl extra libpcp-logsummary-perl_3.10.0_i386.deb 8cc4849b0a0a8c4495a2a9d848cea261 18234 perl extra libpcp-mmv-perl_3.10.0_i386.deb cbfebd5a69c9c08559202fca77339499 14908 utils extra pcp-import-sar2pcp_3.10.0_all.deb 9b4518c1778bd0fae04ca7452635f3be 9182 utils extra pcp-import-mrtg2pcp_3.10.0_all.deb fc9f21c8e548489d70a9c23cb71fc331 15662 utils extra pcp-import-sheet2pcp_3.10.0_all.deb 4ed42378fabb43859611131c78aec18a 16112 utils extra pcp-import-iostat2pcp_3.10.0_all.deb 14523e37c50214caa22c72b0ab589357 23700 utils extra pcp-import-collectl2pcp_3.10.0_i386.deb e0509b15f7231f59f9b1707b915941ca 3146464 doc extra pcp-doc_3.10.0_all.deb 7c1f2e3acabfce8e6cfe97001327c569 2745362 utils extra pcp-testsuite_3.10.0_i386.deb 0efa4a4897890b7a56130178115e987c 47946 utils extra pcp-manager_3.10.0_i386.deb 9bf78ff7d0c1c3d3ad7ece7fa81531ef 72752 utils extra pcp-webapi_3.10.0_i386.deb 934f2b3a42f497abf8dab0c6e34640dc 654042 utils extra pcp-gui_3.10.0_i386.deb 03cc7c7f78c94bcc2de6819b8a32a6f8 2817 utils extra pcp_3.10.0.dsc 494aea8d58e644365b1077dd32af2360 9398032 utils extra pcp_3.10.0.tar.xz -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlRTDpMACgkQm8fl3HSIa2M0eQCeIv44jFvGJwA9+vNH9Wr/QDrj 7tIAoJa00p5zQh0NZdRQAw+s9+dDkwmI =qopX -----END PGP SIGNATURE----- Thank you for your contribution to Debian. From pcp-announce-bounces@oss.sgi.com Fri Oct 31 03:18:57 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from oss.sgi.com (localhost [IPv6:::1]) by oss.sgi.com (Postfix) with ESMTP id F32A87F4E; Fri, 31 Oct 2014 03:18:56 -0500 (CDT) X-Original-To: pcp-announce@oss.sgi.com Delivered-To: pcp-announce@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 981B57F3F for ; Fri, 31 Oct 2014 03:18:55 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 87078304051 for ; Fri, 31 Oct 2014 01:18:52 -0700 (PDT) X-ASG-Debug-ID: 1414743526-04cbb070c64c51b0001-87ZIJf Received: from mx4-phx2.redhat.com (mx4-phx2.redhat.com [209.132.183.25]) by cuda.sgi.com with ESMTP id JCmGXWEuQgnBUO1u (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Fri, 31 Oct 2014 01:18:46 -0700 (PDT) X-Barracuda-Envelope-From: nscott@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.25 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx4-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s9V8IkQl001294 for ; Fri, 31 Oct 2014 04:18:46 -0400 Date: Fri, 31 Oct 2014 04:18:45 -0400 (EDT) From: Nathan Scott To: pcp-announce Message-ID: <1563319379.4659883.1414743525995.JavaMail.zimbra@redhat.com> In-Reply-To: <2125014089.4655859.1414742862455.JavaMail.zimbra@redhat.com> MIME-Version: 1.0 X-ASG-Orig-Subj: Performance Co-Pilot 3.10.0 release X-Originating-IP: [10.5.82.6] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF17 (Linux)/8.0.6_GA_5922) Thread-Topic: Performance Co-Pilot 3.10.0 release Thread-Index: cLuulIqIM08ofOGvcAS/14Muj0Yd3w== X-Barracuda-Connect: mx4-phx2.redhat.com[209.132.183.25] X-Barracuda-Start-Time: 1414743526 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11069 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... Subject: [pcp-announce] Performance Co-Pilot 3.10.0 release X-BeenThere: pcp-announce@oss.sgi.com X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Nathan Scott List-Id: Performance Co-Pilot announcements List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pcp-announce-bounces@oss.sgi.com Sender: pcp-announce-bounces@oss.sgi.com Hi all, The pcp-3.10.0 release has been tagged and released into the wild. This release contains a veritable plethora of activity in the web front-end space - pmwebd now supports interaction with graphite, grafana and other javascript interfaces in the new pcp-webjs tree. Of course, there are fixes and features across the board - python3 pcp modules make their debut in this release, and there are new PMDAs, many new kernel metrics & improvements to several monitoring and importing tools tool. The hidden gem of this release is the huge effort that some folks have put into quality assurance testing - we continue to move forward in leaps and bounds thanks largely to Ken's efforts there. You'll find the latest code in the master branches here: git://git.pcp.io/pcp git://git.pcp.io/pcp-webjs And can download source/binaries from: http://www.pcp.io/ pcp-3.10.0 (31 October 2014) - pmlogextract: improve handling of corrupt archives - linux pmda: add hinv.{cpu,node}.online metrics - gluster pmda: thread-based timeout for long queries - linux pmda: fix hinv.cpu.clock refresh logic - dmcache pmda: add missing instance request handler - iostat2pcp: cater for iostat output format changes - packaging: fix debian suggests vs recommends usage - sample pmda: add pmStore support for some metrics - python: pmda module object refcounts improvements - pmiostat: support archives converted from collectl - FreeBSD pmda: changes for 32-bit platforms - docs: html validation fixes for the tutorial - pmie: rework control and config files - pmlogger: rework control and config files - pmstat: add pmlogger config as per man page - proc pmda: parser rework to improve robustness - proc pmda: per-proc context switch & other metrics - man pages: pmdiscoverservices(3) and pmfind(1) - ds389 pmda: 389 Directory Server PMDA - ds389log pmda: 389 Directory Server log processing PMDA - linux pmda: add rpc.server and nfs v4.1 ops metrics - telnet-probe: fix byte-by-byte copying - papi pmda: default enable when possible - docs: improve quick reference guide, use man7.org - pmproxy: fix new client init for secure connections - pmdiff: minor output formatting improvements - linux pmda: fix initialization for netstat metrics - pmlogger: fix small race on exit condition - timeval refactoring for improved double arithmetic - python: drop support for versions older than 2.6 - python3: add pcp module support for 3.3 and newer - build: workaround qmake handling of library paths - Mac OSX build/install improvements - pmwebd: support for more javascript demos (including graphite/grafana - via separate pcp-webjs package) - pmwebd: extend precision for floating point outputs - pmwebd: Access-Control-Allow-Origin header additions - pmwebd: experimental pthread support - pmwebd: new options file configuration format - pmmgr: avoid some granularity-edge races cheers. -- Nathan _______________________________________________ pcp-announce mailing list pcp-announce@oss.sgi.com http://oss.sgi.com/mailman/listinfo/pcp-announce From fche@redhat.com Fri Oct 31 15:13:10 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 514937F3F for ; Fri, 31 Oct 2014 15:13:10 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 30A188F8050 for ; Fri, 31 Oct 2014 13:13:10 -0700 (PDT) X-ASG-Debug-ID: 1414786385-04bdf038d25843d0001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 1sNfNDuj4VOyGeDf (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Fri, 31 Oct 2014 13:13:06 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s9VKD5pB000569 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Fri, 31 Oct 2014 16:13:05 -0400 Received: from fche.csb (vpn-236-171.phx2.redhat.com [10.3.236.171]) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s9VKD5jR018181 for ; Fri, 31 Oct 2014 16:13:05 -0400 Received: by fche.csb (Postfix, from userid 2569) id 8A6185816C; Fri, 31 Oct 2014 16:13:04 -0400 (EDT) Date: Fri, 31 Oct 2014 16:13:04 -0400 From: "Frank Ch. Eigler" To: pcp developers Subject: qa/518 tweaks on pcpfans.git fche/dev Message-ID: <20141031201304.GE1913@redhat.com> X-ASG-Orig-Subj: qa/518 tweaks on pcpfans.git fche/dev Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.2i X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1414786386 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Hi - Please consider these two ditties. They make the test case conclude successfully & reliably on my fedora-22 rawhide vm. Author: Frank Ch. Eigler Date: Fri Oct 31 16:03:37 2014 -0400 qa/518: stretch timing to run more reliably This test case involves a race between pmie running rules a certain number of times, and a separate task waiting (by time) to match the results via /usr/bin/pcp. It was observed as flakey, not always winning the race. This version stretches the pmie time and fine-tunes the waiting interval to give the race a larger (multi-second) window for success. commit 808aad5a20cd3b29a9368e60aedf6bb92acc7521 Author: Frank Ch. Eigler Date: Fri Oct 31 16:02:03 2014 -0400 qa/518: It was observed that the $sudo pmie invocation didn't always receive the closing signal at this test case, leading it to hang. Now we kill it harder, both by pid and later by name. From kenj@internode.on.net Fri Oct 31 19:41:15 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id F3B347F3F for ; Fri, 31 Oct 2014 19:41:14 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id D08BE8F8035 for ; Fri, 31 Oct 2014 17:41:14 -0700 (PDT) X-ASG-Debug-ID: 1414802471-04cb6c2efa499710001-S8gJnT Received: from ipmail05.adl6.internode.on.net (ipmail05.adl6.internode.on.net [150.101.137.143]) by cuda.sgi.com with ESMTP id cRneJFvjrONKHW4A for ; Fri, 31 Oct 2014 17:41:12 -0700 (PDT) X-Barracuda-Envelope-From: kenj@internode.on.net X-Barracuda-Apparent-Source-IP: 150.101.137.143 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AocJAOorVFR5LELyPGdsb2JhbABcgw5UzgUKh0cEAgKBFhcBAQEBAQYBAQEBODuEAgEBAQQBAQEFAh4SHBgXAQMCBgMRAwEBASgHGQ4SCgMJCAIEARILBYgwDspKAQEBAQYBAQEBAQEcjnOCJAaERQWSGGCDb4hHg02VWykvgQYFgUABAQE Received: from ppp121-44-66-242.lns20.syd6.internode.on.net (HELO bozohorize) ([121.44.66.242]) by ipmail05.adl6.internode.on.net with ESMTP; 01 Nov 2014 11:10:33 +1030 From: "Ken McDonell" To: "'Frank Ch. Eigler'" , "'pcp developers'" References: <20141031201304.GE1913@redhat.com> In-Reply-To: <20141031201304.GE1913@redhat.com> Subject: RE: [pcp] qa/518 tweaks on pcpfans.git fche/dev Date: Sat, 1 Nov 2014 11:40:27 +1100 X-ASG-Orig-Subj: RE: [pcp] qa/518 tweaks on pcpfans.git fche/dev Message-ID: <002c01cff56c$6febc830$4fc35890$@internode.on.net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 15.0 Thread-Index: AQIWIQMIOeKcOGylRqOgmw6wi16M5Zu+lStA Content-Language: en-au X-Barracuda-Connect: ipmail05.adl6.internode.on.net[150.101.137.143] X-Barracuda-Start-Time: 1414802471 X-Barracuda-URL: http://192.48.176.15:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.01 X-Barracuda-Spam-Status: No, SCORE=0.01 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=THREAD_INDEX X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.11090 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== I've reviewed these Frank, and cherry-picked 'em into my tree. qa/518 passes, but that does not prove much because I was not seeing any failures with this test on my QA machines. Reviewing the code, ... the second commit seems fine. I wonder about the first one, as it seems killing -a pmie may produce collateral damage and indeed on the one system I tried this a pmie instance that has nothing to do with qa/518 was nuked. This does not seem right. Do you have any additional information on the circumstances in which the first kill does not cause the pmie process launched by qa/518 to exit? Perhaps we need to try a bit harder to get the pid of the pmie process, rather than the pid of the sudo shell? > -----Original Message----- > From: pcp-bounces@oss.sgi.com [mailto:pcp-bounces@oss.sgi.com] On > Behalf Of Frank Ch. Eigler > Sent: Saturday, 1 November 2014 7:13 AM > To: pcp developers > Subject: [pcp] qa/518 tweaks on pcpfans.git fche/dev > > Hi - > > Please consider these two ditties. They make the test case conclude > successfully & reliably on my fedora-22 rawhide vm. > > > Author: Frank Ch. Eigler > Date: Fri Oct 31 16:03:37 2014 -0400 > > qa/518: stretch timing to run more reliably > > This test case involves a race between pmie running rules a certain > number of times, and a separate task waiting (by time) to match the > results via /usr/bin/pcp. It was observed as flakey, not always > winning the race. This version stretches the pmie time and fine-tunes > the waiting interval to give the race a larger (multi-second) window > for success. > > commit 808aad5a20cd3b29a9368e60aedf6bb92acc7521 > Author: Frank Ch. Eigler > Date: Fri Oct 31 16:02:03 2014 -0400 > > qa/518: > > It was observed that the $sudo pmie invocation didn't always > receive the closing signal at this test case, leading it to > hang. Now we kill it harder, both by pid and later by name. > > _______________________________________________ > pcp mailing list > pcp@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/pcp From fche@redhat.com Fri Oct 31 20:04:38 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 5514D7F3F for ; Fri, 31 Oct 2014 20:04:38 -0500 (CDT) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id D790EAC001 for ; Fri, 31 Oct 2014 18:04:34 -0700 (PDT) X-ASG-Debug-ID: 1414803873-04cbb070c84eca70001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 03lYBdb5EGDlFptl (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Fri, 31 Oct 2014 18:04:33 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id sA114SbG021314 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 31 Oct 2014 21:04:28 -0400 Received: from fche.csb (vpn-236-171.phx2.redhat.com [10.3.236.171]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id sA114RDB021595; Fri, 31 Oct 2014 21:04:28 -0400 Received: by fche.csb (Postfix, from userid 2569) id 7599A5816C; Fri, 31 Oct 2014 21:04:27 -0400 (EDT) Date: Fri, 31 Oct 2014 21:04:27 -0400 From: "Frank Ch. Eigler" To: Ken McDonell Cc: "'pcp developers'" Subject: Re: [pcp] qa/518 tweaks on pcpfans.git fche/dev Message-ID: <20141101010427.GF1913@redhat.com> X-ASG-Orig-Subj: Re: [pcp] qa/518 tweaks on pcpfans.git fche/dev References: <20141031201304.GE1913@redhat.com> <002c01cff56c$6febc830$4fc35890$@internode.on.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <002c01cff56c$6febc830$4fc35890$@internode.on.net> User-Agent: Mutt/1.4.2.2i X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1414803873 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.176.25:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Hi, Ken - > I've reviewed these Frank, and cherry-picked 'em into my tree. > qa/518 passes, but that does not prove much because I was not seeing any > failures with this test on my QA machines. Yeah, figured. > [...] I wonder about the first one, as it seems killing -a pmie may > produce collateral damage and indeed on the one system I tried this > a pmie instance that has nothing to do with qa/518 was nuked. This > does not seem right. It is a bit confusing. Note that near the top of the 518 test case, there was already: # real QA test starts here $sudo $signal -a pmie >/dev/null 2>&1 ... so pmie instances not associated with the test are already on the hit list. > Do you have any additional information on the circumstances in which the > first kill does not cause the pmie process launched by qa/518 to exit? I'll try to trace it with something like systemtap. (Even with pmmgr I encountered cases where a single SIGTERM sent to pmie was blocked/ignored, so sudo is probably not a necessary component of the problem.) - FChE From fche@redhat.com Fri Oct 31 21:40:20 2014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: pcp@oss.sgi.com Delivered-To: pcp@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 9F44B7F3F for ; Fri, 31 Oct 2014 21:40:20 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 7ED8F8F8033 for ; Fri, 31 Oct 2014 19:40:17 -0700 (PDT) X-ASG-Debug-ID: 1414809613-04bdf038d15906e0001-S8gJnT Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 5PpYOPUATNmcZayF (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Fri, 31 Oct 2014 19:40:13 -0700 (PDT) X-Barracuda-Envelope-From: fche@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id sA12e9pe020975 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 31 Oct 2014 22:40:09 -0400 Received: from fche.csb (vpn-236-171.phx2.redhat.com [10.3.236.171]) by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id sA12e9FL019053; Fri, 31 Oct 2014 22:40:09 -0400 Received: by fche.csb (Postfix, from userid 2569) id B2F4C5816C; Fri, 31 Oct 2014 22:40:08 -0400 (EDT) To: Ken McDonell Cc: pcp@oss.sgi.com Subject: Re: pcp updates, pcpfans.git fche/dev References: <20141030203255.GA1913@redhat.com> <494689218.4400197.1414702469822.JavaMail.zimbra@redhat.com> <5452CFAB.5050703@internode.on.net> X-ASG-Orig-Subj: Re: pcp updates, pcpfans.git fche/dev From: fche@redhat.com (Frank Ch. Eigler) Date: Fri, 31 Oct 2014 22:40:08 -0400 In-Reply-To: <5452CFAB.5050703@internode.on.net> (Ken McDonell's message of "Fri, 31 Oct 2014 10:54:19 +1100") Message-ID: User-Agent: Gnus/5.1008 (Gnus v5.10.8) Emacs/21.4 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1414809613 X-Barracuda-Encrypted: AES256-SHA X-Barracuda-URL: http://192.48.157.11:80/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 kenj wrote: > [...] While a quick test is a good test, there are some cases where > longer running tests are required [...] Thanks. The reason the pmmgr qa/666 test (not merged) takes relatively long is because it exercises pmmgr's time-based polling and governing of pmlogger/pmie daemons. Normally, polling / pm*conf generation / archive rotation takes a miniscule fraction of time compared to the overall daemon cycle time (default 24 hours for pmlogger). While the latter can be compressed for testsuite purposes, the former cannot be much (polling / pm*conf still take a number of seconds), so similar sorts of race conditions to the qa/518 case arise. So far, finding a sweet spot of cycle-times (to guarantee fixed-time ops conclude, but not waste time) has been difficult. Anyway, still thinking about it; all advice welcome. - FChE