Merged this lot - some of it got remerged to reorder the commits
(as git does sometimes). The containers work now has functioning
crontab in the pcp-pmlogger image so it's ready to push out to
a docker registry for wider exposure, testing and to figure out
how to qa it all.
Changes committed to git@xxxxxx/pcp master
Ken McDonell (14):
qa/common: need to descend a bit deeper for make setup
qa/888: promote building of src/qa_shmctl.so to GNUlocaldefs
qa/1024: rework to slice, sort and reassemble output
linux pmda: clean up disk stats
qa/1024: minor change to improve debugging diagnostics
qa/1024 and 1091: new .out files after disk.*.total_rawactive changes
qa/183: avoid leaving a pmlogger.log file in /tmp
qa/build: don't try to build qa_shmctl.so for non-Linux platforms
derived metrics: extend $PCP_DERIVED_CONFIG to be a $PATH-like list
configure.ac: force gcc check earlier, rewrite $cc_is_gcc tests
Makepkgs & configure: make, tar and zip setup
qa/admin/myconfigure: mirror logic elsewhere for rc dir
qa/183: avoid leaving a Latest pmafm file in /tmp
libpcp: add instant() method
Mark Goodwin (9):
New crontab entry for containers.
Build: changes to install container crontab entry.
build/containers - don't fix pcp user since pcp-base already does.
build: change Makepkgs --with-containers to --with-docker.
Containers: crontab entries should unconditionaly succeed (exit 0).
build: install python-pcp in pcp-monitor container.
build/packaging: add version and arch to container image filenames.
build/containers: minor PATH tweaks and python-pcp deps fixes
packaging: fix pcp-systemtools dependency on python-pcp.
Nathan Scott (5):
build: update deb packaging homepage url to shortform
build: gpg key passphrase use throughout upload process
build: rework configure test for ps(1) output formats
pcp-atop: resolve a handful of Coverity-reported code issues
pmdalinux: correct mnt/net namespace setns call issue
Makepkgs | 36 +-
build/containers/GNUlocalrules | 8
build/containers/pcp-base/Dockerfile | 1
build/containers/pcp-base/GNUmakefile | 4
build/containers/pcp-collector/Dockerfile | 8
build/containers/pcp-collector/GNUmakefile | 6
build/containers/pcp-monitor/Dockerfile | 8
build/containers/pcp-monitor/GNUmakefile | 6
build/containers/pcp-pmlogger/Dockerfile | 10
build/containers/pcp-pmlogger/GNUmakefile | 6
build/containers/pcp-testsuite/Dockerfile | 4
build/containers/pcp-testsuite/GNUmakefile | 6
build/rpm/fedora.spec | 3
build/rpm/pcp.spec.in | 3
config.mingw | 2
configure | 188 +++++--------
configure.ac | 152 +++-------
debian/control | 2
man/man1/pcpintro.1 | 27 +
man/man3/pmregisterderived.3 | 16 +
qa/1024 | 52 +++
qa/1024.out | 414 +++++++++++++++--------------
qa/1091.out | 2
qa/183 | 6
qa/225.out | 3
qa/773 | 83 +++++
qa/773.out | 160 +++++++++++
qa/777 | 84 +++++
qa/777.out | 150 ++++++++++
qa/888 | 3
qa/admin/myconfigure | 7
qa/common | 17 +
qa/group | 2
qa/src/GNUlocaldefs | 14
scripts/bintray-upload | 66 ++--
src/include/builddefs.in | 4
src/include/pcp.conf.in | 1
src/libpcp/src/derive.c | 173 ++++++++++--
src/libpcp/src/derive.h | 1
src/libpcp/src/derive_fetch.c | 12
src/pcp/atop/atop.c | 15 -
src/pcp/atop/photoproc.c | 10
src/pcp/atop/photoproc.h | 2
src/pcp/atop/showprocs.c | 3
src/pcp/atop/showsys.c | 3
src/pmdas/linux/clusters.h | 13
src/pmdas/linux/help | 231 +++++++++++-----
src/pmdas/linux/namespaces.c | 91 +++---
src/pmdas/linux/namespaces.h | 7
src/pmdas/linux/pmda.c | 211 ++++++++------
src/pmdas/linux/proc_net_dev.c | 226 +++++++++------
src/pmdas/linux/proc_net_dev.h | 5
src/pmdas/linux/proc_partitions.c | 21 +
src/pmdas/linux/root_linux | 4
src/pmlogger/GNUmakefile | 12
src/pmlogger/crontab.docker.in | 19 +
src/pmlogger/rc_pmlogger | 20 +
57 files changed, 1800 insertions(+), 843 deletions(-)
Details ...
commit 1952cf790470db9dcf566b22761d447441326479
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date: Mon Jul 6 09:18:55 2015 +1000
packaging: fix pcp-systemtools dependency on python-pcp.
pcp-systemtools depends on either or both python-pcp and python3-pcp
if they're available on the platform. There is no "one or the other"
restriction - either or both can be installed.
This fixes a reported issue where python tools in pcp-systemtools failed
to run because the platform default python2 interpreter was not installed
(even though python3 was installed).
modified: build/containers/pcp-collector/Dockerfile
modified: build/containers/pcp-monitor/Dockerfile
modified: build/rpm/fedora.spec
modified: build/rpm/pcp.spec.in
commit 8c59d3c7529e2f1571a6509c973aa7b74644e4fe
Author: Ken McDonell <kenj@xxxxxxxxxxxxxxxx>
Date: Sun Jul 5 18:31:55 2015 +1000
libpcp: add instant() method
- code change
- man page updates
- (new) qa/777
commit fb3be296f84b6e6c902131b8778e00d8c1550f8d
Author: Ken McDonell <kenj@xxxxxxxxxxxxxxxx>
Date: Sun Jul 5 18:28:04 2015 +1000
qa/183: avoid leaving a Latest pmafm file in /tmp
commit b9a478b48482364d3a78a96e4745e9a884b86423
Author: Ken McDonell <kenj@xxxxxxxxxxxxxxxx>
Date: Sat Jul 4 20:37:32 2015 +1000
qa/admin/myconfigure: mirror logic elsewhere for rc dir
commit 4265f47d7c1487d52567cc650fde87ccb2cfdddf
Author: Ken McDonell <kenj@xxxxxxxxxxxxxxxx>
Date: Sat Jul 4 20:36:57 2015 +1000
Makepkgs & configure: make, tar and zip setup
Makepkgs determines path to GNU-compatible make, tar and zip and
sets these in the environment. This works for the non-rpm builds,
but rpmbuild strips the environment, so in this case pass these
via the command line via new new --with-make, --with-tar and
--with-zip options to configure.
commit ea90e32c13e450cb272a38efbdb58f9b66397db8
Author: Ken McDonell <kenj@xxxxxxxxxxxxxxxx>
Date: Sat Jul 4 14:32:31 2015 +1000
configure.ac: force gcc check earlier, rewrite $cc_is_gcc tests
commit a33cd05f4ce84ef017802b2701aa5db24de0e1cb
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date: Thu Jul 2 18:37:01 2015 +1000
build/containers: minor PATH tweaks and python-pcp deps fixes
modified: build/containers/pcp-base/Dockerfile
modified: build/containers/pcp-collector/Dockerfile
commit fc04ee9398ebcd653e7287a302530b862169e24e
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date: Thu Jul 2 18:06:25 2015 +1000
build/packaging: add version and arch to container image filenames.
Container images are still tagged :latest but the image filenames now
have version.arch appended to the container name.
modified: build/containers/GNUlocalrules
modified: build/containers/pcp-base/GNUmakefile
modified: build/containers/pcp-collector/GNUmakefile
modified: build/containers/pcp-monitor/GNUmakefile
modified: build/containers/pcp-pmlogger/GNUmakefile
modified: build/containers/pcp-testsuite/GNUmakefile
modified: src/include/builddefs.in
commit 2d0f593c9ea96c5837d1a6f52ca8f854662dff3e
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date: Thu Jul 2 13:26:55 2015 +1000
build: install python-pcp in pcp-monitor container.
Work around an apparent dependency issue with python client
tools that require python-pcp even though python3-pcp is
installed.
modified: build/containers/pcp-monitor/Dockerfile
commit bdcea92921602bc5a0407b3e87c74a4bae357a36
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date: Thu Jul 2 11:03:46 2015 +1000
Containers: crontab entries should unconditionaly succeed (exit 0).
If the pcp-pmlogger container isn't running but the crontab entry is
still installed on the host, we don't want to spam the host logs. So
the crontab scriptlets need to redirect all output to dev/null and
unconditionally succeed.
modified: src/pmlogger/crontab.docker.in
commit 711d9b0b78f0f05aff31673bacf8310e2f781e86
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date: Thu Jul 2 10:52:09 2015 +1000
build: change Makepkgs --with-containers to --with-docker.
This is docker specific. Note there are quite a few more
build infrastructure changes needed, mostly to move the
docker checks out of Makepkgs into configure.ac as the
preferred way of checking we have a working docker daemon
that doesn't need sudo root. Deferring that work for now
since it doesn't actually affect the resulting container
images.
commit 82e7a9f344b8f280f5bb0cd7219703d9b0d94d88
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date: Thu Jul 2 10:43:59 2015 +1000
build/containers - don't fix pcp user since pcp-base already does.
The pcp-base container installs the pcp-config package and already edits
/etc/pcp.conf to switch to the root user in a container environment. So
the layered packages don't need to do this (they inherit the pcp-base
minimal installation).
commit 7cda6d7402170c347926c89284997bc262a87011
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date: Thu Jul 2 10:37:10 2015 +1000
Build: changes to install container crontab entry.
For details see the commit message for a50e3c28
modified: build/containers/pcp-pmlogger/Dockerfile
modified: src/pmlogger/GNUmakefile
modified: src/pmlogger/rc_pmlogger
commit a50e3c286e4ece43faaf14f45af90ca4aa35206a
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date: Thu Jul 2 10:13:56 2015 +1000
New crontab entry for containers.
This crontab entry is installed by the pmlogger rc script to /etc/cron.d
on the host, which is a bind mount in the pcp-pmloger container. The host
cron service runs this to docker exec the pmlogger_daily and pmlogger_check
scripts in the running pcp-pmlogger container. If that container isn't
running, the rc script redirects all output to dev/null since there is
no reliable way of removing it on the host if the container is killed.
The rc script will not re-install the crontab entry if it already exists,
so modifications are persistent. The pcp-pmlogger Dockerfile explicitly
removes the standard /etc/cron.d/pcp-pmlogger entry after the pcp packages
are installed in the container image, so when the rc script is run for the
first time the new contab entry will be installed.
new file: src/pmlogger/crontab.docker.in
commit 156e3dcd8cd73c07207b8af33f9b39b9d29dd1ca
Author: Ken McDonell <kenj@xxxxxxxxxxxxxxxx>
Date: Thu Jul 2 06:51:36 2015 +1000
derived metrics: extend $PCP_DERIVED_CONFIG to be a $PATH-like list
Allows more than one configuration file to be specified, and if a
directory is specified that means recursive descent to load all
configuration files in and below that directory.
Code change is confined to libpcp/derive.c.
This commit includes man page changes, a new qa test (773) and some
qa fallout fixes (cosmetic changes to -Dderive debug messages).
commit 384efb77dd37cc388680f04ca3ae990a0c347234
Author: Ken McDonell <kenj@xxxxxxxxxxxxxxxx>
Date: Thu Jul 2 06:48:15 2015 +1000
qa/build: don't try to build qa_shmctl.so for non-Linux platforms
This code relies on Linux-specific extensions to the Unix SystemV
ipc/shm/sem APIs. The associated qa/888 test is already guarded
to be notrun unless the platform is Linux.
commit d3e35457cc3422d114c44965d621e7d139792812
Author: Ken McDonell <kenj@xxxxxxxxxxxxxxxx>
Date: Thu Jul 2 06:45:46 2015 +1000
qa/183: avoid leaving a pmlogger.log file in /tmp
commit 7e721d96fd62874c7c8ce5e3d705fcab1fe1e9b5
Author: Nathan Scott <nathans@xxxxxxxxxx>
Date: Wed Jul 1 17:08:35 2015 +1000
pmdalinux: correct mnt/net namespace setns call issue
Resolves a problem in the networking metrics when used with
containers, reported by folks on #pcp IRC channel. Fixes up
an issue in one section of qa/042 in the process.
Also, since mass refactoring was needed to ensure we do not
attempt to setns both mnt and net namespaces simultaneously,
I went ahead with splitting the fetch/instance refresh logic
at the same time. Then switched over to defaulting to using
sysfs for resolution of network per-interface stats (iow, we
now prefer this over ioctl when available) - this required a
bit of additional logic to use sysfs for resolving all ioctl
metrics, which was missing previously.
commit 35ba4cfcda607ebc7ec5b1c16985342eae1c1543
Author: Nathan Scott <nathans@xxxxxxxxxx>
Date: Wed Jul 1 17:03:09 2015 +1000
pcp-atop: resolve a handful of Coverity-reported code issues
commit 79addd9f32ad6f0e241bbb3d78e0f49b352ee12f
Author: Nathan Scott <nathans@xxxxxxxxxx>
Date: Wed Jul 1 17:01:40 2015 +1000
build: rework configure test for ps(1) output formats
Drop all attempts to dynamically figure this out based on
running ps(1) on the host - instead, statically configure
PCP_PS_ALL_FLAGS for every supported platform. Dropped a
dated, unused variable while there - PCP_PS_HAVE_BSD. We
should now be immune to building inside Linux containers,
and in Solaris zones.
commit 1c9aa5100f975e0b9b99fe9dfa4e446aa312c0f3
Author: Nathan Scott <nathans@xxxxxxxxxx>
Date: Wed Jul 1 16:56:18 2015 +1000
build: gpg key passphrase use throughout upload process
commit d31f41ad3c9f93b9271a18bc06e0ea2a501b6f38
Author: Nathan Scott <nathans@xxxxxxxxxx>
Date: Wed Jul 1 16:55:27 2015 +1000
build: update deb packaging homepage url to shortform
commit 3f70b9c4f04661f4e6e01890e1671a8a3c7a6ad2
Author: Ken McDonell <kenj@xxxxxxxxxxxxxxxx>
Date: Tue Jun 30 06:28:34 2015 +1000
qa/1024 and 1091: new .out files after disk.*.total_rawactive changes
commit b644cf19b10c4192e3ce1305114c851a85a79d1f
Author: Ken McDonell <kenj@xxxxxxxxxxxxxxxx>
Date: Tue Jun 30 06:27:43 2015 +1000
qa/1024: minor change to improve debugging diagnostics
commit ce1df51ff0782cfde6363b764d6ba16d412f8062
Author: Ken McDonell <kenj@xxxxxxxxxxxxxxxx>
Date: Tue Jun 30 06:23:33 2015 +1000
linux pmda: clean up disk stats
Add total_rawactive to complement read_rawactive and write_rawactive
for per disk, all disk, per partition and per-device-mapper device
metrics.
Rework the help text to improve the description of the metrics
related to disk's being active and disk service times, namely
disk.*.avactive and disk.*.*_rawactive.
commit 07fec0e92ba3deb95c1226179bdf0edd52567cf2
Author: Ken McDonell <kenj@xxxxxxxxxxxxxxxx>
Date: Mon Jun 29 10:08:33 2015 +1000
qa/1024: rework to slice, sort and reassemble output
From the script ...
instance assignments (internal ids) are not deterministic
due to the local indom cache state that is outside the control
of QA, so map all internal instance ids to N and sort ...
need to alternate slicing between sort and don't sort blocks
commit b95a5d2e513e93702d6c31eaf0f38f405474c76f
Author: Ken McDonell <kenj@xxxxxxxxxxxxxxxx>
Date: Fri Jun 26 10:28:01 2015 +1000
qa/888: promote building of src/qa_shmctl.so to GNUlocaldefs
Better to take control of building here (and in particular avoid
any exotic CFLAGS) rather than in the qa script itself.
commit e19e0c5b2d8b8a518281c8aad93145d8608f61b2
Author: Ken McDonell <kenj@xxxxxxxxxxxxxxxx>
Date: Fri Jun 26 10:25:04 2015 +1000
qa/common: need to descend a bit deeper for make setup
In particular things below pmdas and pmdas/broken need to be built
running from a newly set up git workarea.
|