Hi,
On 2016-07-20 23:43, Nathan Scott wrote:
> ----- Original Message -----
>>> [...]
>>> What kind of "highly problematic" scenarios there has been in the past?
>>
>> So, one example is pmie - there is no equivalent of some_inst, etc for
>> metric names, so rules end up having to be expanded for every metric.
>> pmchart configs have similar issues, regex matching on instance names
>> is available but metric names are expected to be more static (and need
>> individual expansion in the configuration files).
If we think metrics with PM_INDOM_NULL as one-dimensional metrics,
metrics with instance domains as two-dimensional metrics, and these
metrics which have two dynamic dimensions as three-dimensional metrics,
are there other such "3D" metrics in standard / often used PMDAs
present already, how are they set up?
>> Another class of problems is around naming - metric names are defined
>> to be less flexible than instance names (as per that pmns(5) extract,
>> from earlier).
I don't see this any kind of issue here. The case that a component must
begin with an alphabetic is documented in pmns(5) but for some reason
it is not checked/forced and in fact everything seemed to work with
components beginning with a digit.
(Btw, how well pmns(5) is up-to-date, it lists several domain IDs for
Oracle, Sybase, Informix, and e.g. OpenBSD and libvirt are in the
"End-user and demo PMDA" range? Perhaps a component is allowed to begin
with a digit these days?)
> I forgot another biggie yesterday - related to the persistence of PMIDs.
> Its important to a number of the client tools (pmlogger, pmie, pmchart,
> hmmm pretty much all actually) that if pmcd is restarted, metrics return
> with the same PMID as before (this is pretty much part of the protocol
> over-the-wire, and its also necessary on-disk between archives of the
> same host).
>
> This becomes very difficult to ensure in the case of dynamic metrics -
> it was the main reason we switch cgroups to using this model IIRC. And
> the pmdaCache interfaces support persisting instance identifiers well of
> course.
This worked already earlier ok, PMIDs were never different for added
metrics (IOW, the names and PMIDs for, e.g., VCPU4 related metrics were
always the same). I've now double-checked it by starting/stopping VMs,
libvirt PMDA, PMCD, and libvirtd in random order and as should, PMIDs
and names never change.
> Also, along the vein of PMIDs, and especially depending on how the PMID
> "cluster" identifier is used, the size limits of 10 bits (1024 metrics)
> and/or 22 bits (if full cluster space available also) are less appealing
> than the convenience of the full 32 bit instance identifier space.
Hmm, ok, so in practice this means that we're limited to 338 vCPUs, 71
block devices, and 111 net devices with this approach, do we consider
this a show-stopper? Because otherwise the "convenience" of some
internal limit being 22 or 32 bit is not that apparent.
Martins, you were already trying out this libvirt PMDA after its
initial posting and you have lots of valuable experience and insight
with PCP also in general, how do you see these two approaches, which
one would be more suitable for you and your use cases?
Below is a rebased patch (on top of the backing patch) and which has
the one extra line needed to allow using e.g. ..net.net0.. instead of
..net.0.. etc.
PS. I'll be offline for a few weeks, I'll catch up with this then.
---
src/pmdas/libvirt/pmdalibvirt.python | 80 ++++++++++++++++++++++++++++++++++++
1 file changed, 80 insertions(+)
diff --git a/src/pmdas/libvirt/pmdalibvirt.python
b/src/pmdas/libvirt/pmdalibvirt.python
index bd3c23b..5646c96 100755
--- a/src/pmdas/libvirt/pmdalibvirt.python
+++ b/src/pmdas/libvirt/pmdalibvirt.python
@@ -417,6 +417,30 @@ class LibvirtPMDA(PMDA):
elif i == 2:
res['vcpu.' + nrstr + '.time'] =
stats[nr][i]
self.vm_vcpustats_res.append([dom, res])
+
+ if self.vm_vcpustats_res:
+ high = 0
+ for r in self.vm_vcpustats_res:
+ if r[1]['vcpu.current'] > high:
+ high = r[1]['vcpu.current']
+ if not high:
+ return
+ base = self.read_name() + '.domstats.vcpu.vcpu'
+ if base + str(high-1) + '.time' not in
self._metric_names.values():
+ # New high for vCPUs, add needed per-vCPU metrics
+ metrics = ['state', 'time', 'wait']
+ for nr in range(high):
+ nrstr = str(nr)
+ if base + nrstr + '.time' not in
self._metric_names.values():
+ for j, m in enumerate(metrics):
+ metric = base + nrstr + '.' + m
+ help = 'VM vCPU' + nrstr + ', ' + m
+ # 5 - nr of static items, 2 - nr of items
before total metrics
+ self.add_metric(metric,
pmdaMetric(self.pmid(self.vm_vcpustats_cluster, 5+nr*len(metrics)+j),
+ self.vm_vcpustats[2+j][2],
self.vm_indom, self.vm_vcpustats[2+j][3],
+ self.vm_vcpustats[2+j][4]), help, help)
+
self.vm_vcpustats.append([metric.replace(self.read_name() + '.', ''), None,
self.vm_vcpustats[2+j][2]])
+ self.pmns_refresh()
except libvirt.libvirtError as error:
self.log("Failed to get domain vcpu stats: %s" % error)
return
@@ -480,6 +504,34 @@ class LibvirtPMDA(PMDA):
elif i == 3:
res['block.' + nrstr + '.wr.bytes'] =
stats[i]
self.vm_blockstats_res.append([dom, res])
+
+ if self.vm_blockstats_res:
+ high = 0
+ for r in self.vm_blockstats_res:
+ if r[1]['block.count'] > high:
+ high = r[1]['block.count']
+ if not high:
+ return
+ base = self.read_name() + '.domstats.block.block'
+ if base + str(high-1) + '.rd.reqs' not in
self._metric_names.values():
+ # New high for block devices, add needed per-block
device metrics
+ metrics = ['name', 'backingIndex', 'path', 'rd.reqs',
'rd.bytes', 'rd.times', 'wr.reqs', 'wr.bytes', 'wr.times', 'fl.reqs',
'fl.times', 'allocation', 'capacity', 'physical']
+ backing = ['backingIndex', 'allocation', 'capacity',
'physical']
+ for nr in range(high):
+ nrstr = str(nr)
+ if base + nrstr + '.rd.reqs' not in
self._metric_names.values():
+ for j, m in enumerate(metrics):
+ metric = base + nrstr + '.' + m
+ if m not in backing:
+ help = 'VM block dev ' + nrstr + ', '
+ m.replace('.', ' ')
+ else:
+ help = 'VM backing img ' + nrstr + ',
' + m
+ # 15 - nr of static items, 1 - nr of items
before total metrics
+ self.add_metric(metric,
pmdaMetric(self.pmid(self.vm_blockstats_cluster, 15+nr*len(metrics)+j),
+ self.vm_blockstats[1+j][2],
self.vm_indom, self.vm_blockstats[1+j][3],
+ self.vm_blockstats[1+j][4]), help,
help)
+
self.vm_blockstats.append([metric.replace(self.read_name() + '.', ''), None,
self.vm_blockstats[1+j][2]])
+ self.pmns_refresh()
except libvirt.libvirtError as error:
self.log("Failed to get domain block stats: %s" % error)
return
@@ -521,6 +573,33 @@ class LibvirtPMDA(PMDA):
elif i == 7:
res['net.' + nrstr + '.tx.drop'] = stats[i]
self.vm_netstats_res.append([dom, res])
+
+ if self.vm_netstats_res:
+ high = 0
+ for r in self.vm_netstats_res:
+ if r[1]['net.count'] > high:
+ high = r[1]['net.count']
+ if not high:
+ return
+ base = self.read_name() + '.domstats.net.net'
+ if base + str(high-1) + '.rx.bytes' not in
self._metric_names.values():
+ # New high for NICs, add needed per-NIC metrics
+ metrics = ['name', 'rx.bytes', 'rx.pkts', 'rx.errs',
'rx.drop', 'tx.bytes', 'tx.pkts', 'tx.errs', 'tx.drop']
+ for nr in range(high):
+ nrstr = str(nr)
+ if base + nrstr + '.rx.bytes' not in
self._metric_names.values():
+ for j, m in enumerate(metrics):
+ metric = base + nrstr + '.' + m
+ if m == 'name':
+ help = 'VM NIC ' + nrstr + ', name'
+ else:
+ help = 'VM NIC ' + nrstr + ', ' +
m.replace('.', ' ')
+ # 10 - nr of static items, 1 - nr of items
before total metrics
+ self.add_metric(metric,
pmdaMetric(self.pmid(self.vm_netstats_cluster, 10+nr*len(metrics)+j),
+ self.vm_netstats[1+j][2],
self.vm_indom, self.vm_netstats[1+j][3],
+ self.vm_netstats[1+j][4]), help, help)
+
self.vm_netstats.append([metric.replace(self.read_name() + '.', ''), None,
self.vm_netstats[1+j][2]])
+ self.pmns_refresh()
except libvirt.libvirtError as error:
self.log("Failed to get domain net stats: %s" % error)
return
@@ -639,6 +718,7 @@ class LibvirtPMDA(PMDA):
return [PM_ERR_INST, 0]
key = mtx[item][0].partition('.')[2]
+ key = key.replace('.' + key.partition('.')[0], '.')
# All done for non-dynamic clusters
if cluster != self.vm_vcpustats_cluster and \
Thanks,
--
Marko Myllynen
|