pcp
[Top] [All Lists]

pcp updates - global derived metrics, pmiostat, pmchart and QA

To: pcp <pcp@xxxxxxxxxxx>
Subject: pcp updates - global derived metrics, pmiostat, pmchart and QA
From: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date: Wed, 30 Mar 2016 16:53:40 +1100
Delivered-to: pcp@xxxxxxxxxxx
In-reply-to: <56E7480B.5080907@xxxxxxxxxx>
References: <56E01091.8070706@xxxxxxxxxx> <56E7480B.5080907@xxxxxxxxxx>
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0
This lot was held over from 3.11.1 (it was ready too late during QA week).

- enable global shared derived metrics defined in $PCP_VAR_DIR/config/derived/
  and ship iostat.conf containing derived metrics for the iostat metrics
  disk.{dev,dm,md}.{await,r_await,w_await,avg_qlen,avg_rqsz,util}
  The defs are included in the pcp-conf package, as suggested by Nathan.

- as previously discussed, this can be turned off with PCP_DERIVED_CONFIG=""
  or can be redirected to one or more alternative derived metrics defs using
  a ':' separated list of files or directories. If the environment variable
  is NOT set, the entries in the default config directory are loaded, as above.

- QA updates for the above - mostly to set or unset $PCP_DERIVED_CONFIG,
  most notably in qa/common where it is set to "". I will be working on
  additional QA updates (mostly to filter the additional 18 derived
  metrics definitions we have so far, and any additional configs that are
  added in the future). QA is passing for now with the qa/common change.

- new pmchart views for Iostat, Iostat.DM and Iostat.MD. These views basically
  display all the fields of pmiostat in graphical charts using two tabs - the
  first for disk traffic, and the second for await, avgqlen, avgrqsz and %util.
  Obviously they use the Iostat.conf derived metrics definitions, and thus work
  with existing archives and in live mode with downrev remote pmcd - i.e. it's
  all client side using both existing and derived metrics.

- merged Nikhil's pmiostat -P patch (see previous mail), along with a
  couple of additional minor pmiostat tweaks, and associated QA updates.

- pmchart: fix an autoscale bug when loading a view in archive mode with the
  origin set to the end of the archive (-O-0), and a few pmchart doc/man 
updates.

- have run several full QA runs on f23/x86_64

- a remaining QA task is to extend 842 to also check the new derived iostat
  metrics reported by pmval with the values reported by pmiostat when replaying
  the same archive.


Changes committed to git://git.pcp.io/mgoodwin/pcp master


commit fc1f228b1062cd8ebf4db8fa06443fb9784f7e48
Merge: b536b11 9d962cb
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Wed Mar 30 09:42:04 2016 +1100

    Merge branch 'master' of git://pcp.io/pcp

commit b536b113c16a61a5d0dfc260d73c7842ac1be13f
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Tue Mar 29 17:31:44 2016 +1100

    qa - extend 842 to check reporting for pmiostat -P precision
modified: qa/842
    modified:   qa/842.out

commit f12bced41dbe934ca14d790d79c927cda9f26675
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Tue Mar 29 17:13:20 2016 +1100

    qa - updates for new pmiostat -P precision flag.
The pmiostat report format changed slightly with the -P flag defaulting
    to 2 decimals, so remake tests in the 'pmiostat' group. i.e. qa/1099.out,
    qa/536.out, qa/810.out and a minor tweak for qa/842 and qa/842.out

commit 0325f568b4a8ddfebbc80f84d0d0894088434f81
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Tue Mar 29 17:08:49 2016 +1100

    docs - pmiostat man page, mention extra default precision for avgrq-sz and 
avgqu-sz
avgrq-sz and avgqu-sz fields are reported with N+1 decimals because they're
    usually small values.

commit 8ef4588986189f0f08420b0d8faa14919c614bae
Merge: 50a47b1 1d171f9
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Tue Mar 29 15:31:13 2016 +1100

    Merge branch 'master' of git://pcp.io/pcp

commit 50a47b1be7d3323824a24bac1b01e733310f6408
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Fri Mar 25 11:22:32 2016 +1100

    derived metrics: add derived iostat metrics for disk.md
This adds iostat derived metrics for disk.md devices, complementing
    similar derived metrics based on disk.dev and disk.dm. These are
    used by the new pmchart view 'Iostat.MD'.

commit fdf0d364956a227c3bc3c20447a201e07a2633cd
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Fri Mar 25 10:46:36 2016 +1100

    pmchart - split out Iostat view to SD, DM and MD
This splits out the Iostat view to Iostat (for SD devices using disk.dev),
    Iostat.DM (for device mapper logical devices using disk.dm) and Iostat.MD
    (for logical multiple devices, using disk.md metrics).
modified: src/pmchart/views/GNUmakefile
        modified:   src/pmchart/views/Iostat
        new file:   src/pmchart/views/Iostat.DM
        new file:   src/pmchart/views/Iostat.MD

commit 9c42ba123a40c3ebe4671ea404d7cad4ce5c0615
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Fri Mar 25 10:34:22 2016 +1100

    qa/common - set PCP_DERIVED_CONFIG to an empty string
Avoid QA fallout whilst I add new filtering for global derived
    metrics configs.

commit f1d6ad6791e976e1486e7b3777692287b45708d0
Merge: 5232ea2 dd79509
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Fri Mar 25 10:26:18 2016 +1100

    Merge branch 'master' of git://pcp.io/pcp

commit 5232ea23730ad0bfe3e3b4865e00d29ac27af32f
Merge: e319b0b 6a44882
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Thu Mar 24 15:28:26 2016 +1100

    Merge branch 'master' of git://pcp.io/pcp

commit e319b0b49420b3ff0e73169fd53d3f5f07c980e5
Merge: c96253c 0ee38da
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Mon Mar 21 18:33:45 2016 +1100

    Merge branch 'master' of git://pcp.io/pcp

commit c96253cf403e75b27965f53ea8c5c8f5b08a4bbd
Merge: c9d335b 237f9b7
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Fri Mar 18 10:32:14 2016 +1100

    Merge https://github.com/nkshirsagar/nkshirsa_pcp

commit c9d335bda5f85e42d2f0b3ad7264c44976c07fd6
Merge: c200425 b23487a
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Fri Mar 18 08:37:31 2016 +1100

    Merge branch 'master' of git://pcp.io/pcp

commit 237f9b729d18f4ee320c4fd948108fc1932b3882
Author: Nikhil <nkshirsa@xxxxxxxxxx>
Date:   Thu Mar 17 10:32:54 2016 +0530

    Commit for bz 1297498.
pcp-iostat now takes a -P parameter that specifies the number of
    decimal places to display in the output.

commit c20042526218cb66950b2a1226b3653d3d2cf2de
Merge: 77231e5 bfa07b4
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Thu Mar 17 13:44:25 2016 +1100

    Merge branch 'master' of git://git.pcp.io/kenj/pcp

commit 77231e5796034eb8482cb2d13a8abb37db90e719
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Thu Mar 17 13:43:16 2016 +1100

    pmiostat: improve time delta calculation
reduces the possibility of arithmetic overflow.
    No QA updates needed.

commit 688f7107a12c82f3bdeb898b2661c34324c72173
Merge: f68c604 c39fff3
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Wed Mar 16 16:39:02 2016 +1100

    Merge branch 'master' of git://pcp.io/pcp

commit f68c60427318969196efc006428d8055417c442c
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Tue Mar 15 12:23:00 2016 +1100

    qa: updates to 'derive' group tests for global derived metrics
updates qa/225 qa/246 qa/247 qa/249 qa/256 qa/370 qa/509 qa/969
    but no new qualified output.

commit 1362ff11b9c96574efe392c2f63747615d4cac18
Merge: d00d400 054a5bb
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Tue Mar 15 10:24:10 2016 +1100

    Merge branch 'master' of git://pcp.io/pcp

commit d00d400dd591adba8403b97d2705c3bda052ee72
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Tue Mar 15 09:53:51 2016 +1100

    libpcp: PCP_DERIVED_CONFIG=="" disables loading derived metrics
Also add a comment explaining the three use-cases: If PCP_DERIVED_CONFIG is NOT set, then by default we load global
    derived configs from the directory $PCP_VAR_DIR/config/derived.
If PCP_DERIVED_CONFIG is set to a zero length string, then don't
    load any derived metrics definitions.
Else if PCP_DERIVED_CONFIG is set then load user-defined derived
    metrics from one or more files or directories separated by ':'.

commit b83ef983a98979528a3d70736c0dc3c6677075ce
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Wed Mar 9 22:05:16 2016 +1100

    packaging: packaging for global derived metrics configs
Add new dir PCP_VAR_DIR/config/derived and config file iostat.config
    to the pcp-config package.
modified: build/rpm/fedora.spec
        modified:   debian/pcp-conf.dirs
        modified:   debian/pcp-conf.install

commit 7a4d099ce75c035c701e153bd3bfedbdcd61f3e0
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Wed Mar 9 21:57:32 2016 +1100

    pmchart: add Iostat view using disk metrics and derived iostat metrics
This is a new multi-tabbed view showing disk and device-mapper traffic
    and most of the traditional iostat metrics. Some of these metrics are
    derived, see /var/lib/pcp/config/derived/iostat.conf.
modified: src/pmchart/views/GNUmakefile
        new file:   src/pmchart/views/Iostat

commit 5d5620645415100a4734d379f111b179c25267d6
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Wed Mar 9 17:41:05 2016 +1100

    libpcp: load global derived metrics config by default
If PCP_DERIVED_CONFIG is NOT defined in the environment, then __dminit()
    now loads derived configs from $PCP_VAR_DIR/config/derived by default.
    This follows on from previous commit 689c031ca6f5, where we now ship
    an iostat.config.
A follow-up change will move the global derived config directory and
    iostat.config into the pcp-conf package (currently in pcp base package).
Also, some QA changes to follow.

commit 0b91a221b8187f11d7c1dd05a222ab3f94f63d35
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Wed Mar 9 17:26:33 2016 +1100

    pmchart: fix an autoscale bug when loading a view in archive mode
Fix an issue in pmchart when loading a view in archive mode, autoscale
    isn't properly performed until a pmtime event is received (was initially
    fixed scale at 0 .. 1000). Also improves console diagnostics in related
    areas of code. Fix is in GroupControl::adjustArchiveWorldViewForward().

commit df5ec5921a8dab639530d939fb43382d18921305
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Wed Mar 9 17:17:46 2016 +1100

    docs: clarify pmchart -geometry and -g flags (former is for display 
geometry)

commit 5c6ed952f33490aedd0a0ebb898e32001cc0146e
Merge: 7a2d529 689c031
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Wed Mar 9 17:15:43 2016 +1100

    Merge branch 'master' of git://pcp.io/mgoodwin/pcp

commit 689c031ca6f5db455cce6aab8f145eb5b8029223
Author: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date:   Mon Mar 7 15:23:56 2016 +1100

    config: install a global derived metrics config directory with iostat.conf
Installs a global derived metrics config dir $PCP_VAR_DIR/config/derived
    along with iostat.conf (containing iostat derived metrics definitions).
modified: src/GNUmakefile
        new file:   src/derived/GNUmakefile
        new file:   src/derived/iostat.conf

<Prev in Thread] Current Thread [Next in Thread>