Hi,
below is a 389 Directory Server Log PMDA, ds389log. It provides client/server
metrics from the 389 DS access log using logconv.pl(1). Additional metrics
provided by logconv.pl would be easy to add if deemed useful, defining them in
the %data hash should be enough for most cases.
Running logconv.pl on an access log on a busy server without defining the
start/end time can be a long, CPU consuming task. Thus, we always ask metrics
for a certain time slice only and calculate the actual metrics since the PMDA
was started to minimize resource usage and to avoid pmcd(1) dropping the PMDA
(see http://oss.sgi.com/bugzilla/show_bug.cgi?id=1036). In few cases the code
could perhaps a bit more Perlish but OTOH TIMTOWTDI.
There is, however, one minor issue currently with the PMDA which I think I
should mention - it doesn't work :) This is because of logconv.pl(1) fails to
honor the start/end time definitions but this will hopefully get addressed
soon, see https://bugzilla.redhat.com/show_bug.cgi?id=1145948. I've tested
using logconv.pl without those switches and then the metrics are as expected
(but as said, this wouldn't work in practice in production).
To test this locally one would ideally have 389 DS running but testing against
a previously captured access log (e.g. from the above RHBZ) should help to do
basic sanity checking.
>From c9ff631c930ff9569eadc62b324aac495025011f Mon Sep 17 00:00:00 2001
From: Marko Myllynen <myllynen@xxxxxxxxxx>
Date: Thu, 25 Sep 2014 17:12:13 +0300
Subject: [PATCH] 389 DS Log PCP PMDA
---
src/pmdas/ds389log/Install | 32 +++++
src/pmdas/ds389log/Remove | 23 +++
src/pmdas/ds389log/pmdads389log.pl | 263 ++++++++++++++++++++++++++++++++++++
3 files changed, 318 insertions(+), 0 deletions(-)
create mode 100755 src/pmdas/ds389log/Install
create mode 100755 src/pmdas/ds389log/Remove
create mode 100644 src/pmdas/ds389log/pmdads389log.pl
diff --git a/src/pmdas/ds389log/Install b/src/pmdas/ds389log/Install
new file mode 100755
index 0000000..226afa8
--- /dev/null
+++ b/src/pmdas/ds389log/Install
@@ -0,0 +1,32 @@
+#!/bin/sh
+#
+# Copyright (C) 2014 Marko Myllynen <myllynen@xxxxxxxxxx>
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+# or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+# for more details.
+#
+
+. $PCP_DIR/etc/pcp.env
+. $PCP_SHARE_DIR/lib/pmdaproc.sh
+
+iam=ds389log
+perl_opt=true
+daemon_opt=false
+forced_restart=true
+
+perl -e "use Date::Manip" 2>/dev/null
+if test $? -ne 0; then
+ echo "Date::Manip Perl module is not installed"
+ exit 1
+fi
+
+pmdaSetup
+pmdaInstall
+exit 0
diff --git a/src/pmdas/ds389log/Remove b/src/pmdas/ds389log/Remove
new file mode 100755
index 0000000..6af8902
--- /dev/null
+++ b/src/pmdas/ds389log/Remove
@@ -0,0 +1,23 @@
+#!/bin/sh
+#
+# Copyright (C) 2014 Marko Myllynen <myllynen@xxxxxxxxxx>
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+# or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+# for more details.
+#
+
+. $PCP_DIR/etc/pcp.env
+. $PCP_SHARE_DIR/lib/pmdaproc.sh
+
+iam=ds389log
+
+pmdaSetup
+pmdaRemove
+exit 0
diff --git a/src/pmdas/ds389log/pmdads389log.pl
b/src/pmdas/ds389log/pmdads389log.pl
new file mode 100644
index 0000000..855ed37
--- /dev/null
+++ b/src/pmdas/ds389log/pmdads389log.pl
@@ -0,0 +1,263 @@
+#
+# Copyright (C) 2014 Marko Myllynen <myllynen@xxxxxxxxxx>
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+# or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+# for more details.
+#
+
+use strict;
+use warnings;
+use PCP::PMDA;
+use Date::Manip;
+use POSIX;
+
+my $lc_opts = '-D /dev/shm -s all';
+my $lc_ival = 30; # minimal query interval in seconds, must be >= 30
+my $ds_alog = ''; # empty - guess; ok if only one DS instance in use
+my $ds_logd = '/var/log/dirsrv';
+my $ds_user = 'nobody'; # empty - use root
+
+my %data = (
+ # logconv.pl string - name - subtree - cluster - id - type
+ # type : 0 - cumulative, 1 - peak
+ 'Total Connections:' => [ 'totalconns', 'conns', 0, 0, 0 ],
+ 'Peak Concurrent Connections:' => [ 'peakconns', 'conns', 0, 1, 1 ],
+ 'U1' => [ 'cleanclose', 'conns', 0, 2, 0 ],
+ 'B1' => [ 'badclose', 'conns', 0, 3, 0 ],
+ 'Total Operations:' => [ 'totalops', 'ops', 1, 0, 0 ],
+ 'Total Results:' => [ 'totalres', 'ops', 1, 1, 0 ],
+ 'Searches:' => [ 'searches', 'ops', 1, 2, 0 ],
+ 'Modifications:' => [ 'mods', 'ops', 1, 3, 0 ],
+ 'Adds:' => [ 'adds', 'ops', 1, 4, 0 ],
+ 'Deletes:' => [ 'dels', 'ops', 1, 5, 0 ],
+ 'Mod RDNs:' => [ 'modrdns', 'ops', 1, 6, 0 ],
+ 'Compares:' => [ 'comps', 'ops', 1, 7, 0 ],
+ 'Binds:' => [ 'binds', 'ops', 1, 8, 0 ],
+ 'Paged Searches:' => [ 'pagedsearches', 'searches', 2, 0,
0 ],
+ 'Unindexed Searches:' => [ 'unindexedsearches', 'searches',
2, 1, 0 ],
+ 'err=0' => [ 'noerror', 'errors', 3, 0, 0 ],
+ 'err=X' => [ 'error', 'errors', 3, 1, 0 ], #
custom
+ 'Highest FD Taken:' => [ 'fdhigh', 'fd', 4, 0, 1 ],
+);
+
+use vars qw( $pmda %metrics );
+
+# Timestamps
+my @lc_prev = localtime();
+my @lc_curr;
+
+# Configuration files for overriding the above settings
+for my $file (pmda_config('PCP_PMDAS_DIR') . '/ds389log/ds389log.conf',
'./ds389log.conf') {
+ eval `cat $file` unless ! -f $file;
+}
+
+sub ds389log_set_ds_access_log {
+ $ds_alog = `ls -1 $ds_logd/slapd-*/access 2>/dev/null | tail -n 1`;
+ my $un = `id -un`;
+ chomp($ds_alog); chomp($un);
+ die "$un can't read access log file \"$ds_alog\"" unless -f $ds_alog;
+ $pmda->log("Using access log file $ds_alog");
+}
+
+sub ds389log_fetch {
+ ds389log_set_ds_access_log() if $ds_alog eq '';
+ return if $ds_alog eq '';
+
+ # Server might not have written entries for operations during
+ # the past few seconds yet so we will collect them next round.
+ @lc_curr = localtime();
+ $lc_curr[0] -= 30; # secs
+
+ if ((strftime("%s", @lc_curr) - strftime("%s", @lc_prev)) < $lc_ival) {
+ return;
+ }
+
+ # Don't include anything twice
+ $lc_prev[0] += 1; # secs
+
+ # Include the previous rotated log only if needed
+ my $prev_log = `ls -1rt $ds_alog.2* 2>/dev/null | tail -n 1`;
+ if ($prev_log ne '') {
+ my $lastline = `tail -n 1 $prev_log`;
+ $lastline =~ tr/\[//d; $lastline =~ s/\].*//;
+ my $log_ts = UnixDate($lastline, "%s");
+ if (strftime("%s", @lc_prev) > $log_ts) {
+ $prev_log = '';
+ }
+ }
+
+ my $lc_start = strftime("[%d/%m/%Y:%H:%M:%S %z]", @lc_prev);
+ my $lc_end = strftime("[%d/%m/%Y:%H:%M:%S %z]", @lc_curr);
+ @lc_prev = @lc_curr;
+
+ my $ds_stats = "logconv.pl -cpe $lc_opts -S $lc_start -E $lc_end
$ds_alog $prev_log 2>/dev/null";
+ open(STATS, "$ds_stats |") or
+ die $pmda->err("pmda389log failed to open $ds_stats pipe: $!");
+ my @stats = <STATS>;
+ close(STATS);
+
+ my $errors = 0; # combined
+ foreach my $line (@stats) {
+ my $key;
+ my @metric;
+
+ if ($line =~ /^.*:/ || $line =~ /^U1/ || $line =~ /^B1/) {
+ $key = $&;
+ }
+ if ($line =~ /^err=.?/) {
+ $key = 'err=X';
+ }
+ if ($line =~ /^err=0/) {
+ $key = 'err=0';
+ }
+
+ if (defined($key) && defined $data{$key}) {
+ $key = 'err=' if $key eq 'err=X';
+ if ($line =~ /($key)\s+(\d+)/ || $line =~
/($key\d+)\s+(\d+)/) {
+ my $value = $2;
+
+ if ($key eq 'err=') {
+ $key = 'err=X';
+ $errors += $value;
+ $value = $errors;
+ }
+
+ my $id = 'ds389log.' . $data{$key}[1] . '.' .
$data{$key}[0];
+
+ if ($data{$key}[4] eq 1) {
+ my $prev = $metrics{$id}[1];
+ $value = $prev if $prev > $value;
+ } else {
+ $value = $metrics{$id}[1] + $value;
+ }
+
+ @metric = ($id , $value);
+ $metrics{$id} = \@metric;
+ }
+ }
+ }
+}
+
+sub ds389log_fetch_callback {
+ my ($cluster, $item, $inst) = @_;
+
+ if ($inst != PM_INDOM_NULL) { return (PM_ERR_INST, 0); }
+
+ my $pmnm = pmda_pmid_name($cluster, $item);
+ my $value = $metrics{$pmnm};
+
+ if (!defined($value)) { return (PM_ERR_APPVERSION, 0); }
+
+ return ($value->[1], 1);
+}
+
+$pmda = PCP::PMDA->new('ds389log', 137);
+
+# Add and zero metrics
+foreach my $key (keys %data) {
+ my $name = 'ds389log.' . $data{$key}->[1] . '.' . $data{$key}->[0];
+ $pmda->add_metric(pmda_pmid($data{$key}->[2], $data{$key}->[3]),
+ PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_COUNTER,
+ pmda_units(0,0,1,0,0,PM_COUNT_ONE),
+ $name, '', '');
+ my @value = ($name, 0);
+ $metrics{$name} = \@value;
+}
+
+$pmda->set_refresh(\&ds389log_fetch);
+$pmda->set_fetch_callback(\&ds389log_fetch_callback);
+# NB: needs to run as root or as a user having read access to the logs
+$pmda->set_user($ds_user) if $ds_user ne '';
+$pmda->run;
+
+=pod
+
+=head1 NAME
+
+pmdads389log - 389 Directory Server Access Log PMDA
+
+=head1 DESCRIPTION
+
+B<pmdads389log> is a Performance Metrics Domain Agent (PMDA) which
+extracts statistics from 389 Directory Server access log by using
+the B<logconv.pl>(1) utility.
+
+=head1 INSTALLATION
+
+B<pmdads389log> uses configuration file:
+
+=over
+
+=item * $PCP_PMDAS_DIR/ds389log/ds389log.conf
+
+=back
+
+This file can contain overridden values (Perl code) for the settings
+listed at the start of pmdads389log.pl, namely:
+
+=over
+
+=item * logconv.pl parameters
+
+=item * minimal query interval
+
+=item * 389 DS access log to use
+
+=item * 389 DS log directory to use (used if no file specified)
+
+=item * non-root user having read access to the access log file
+
+=back
+
+Once this is setup, you can access the names and values for the
+ds389log access log metrics by doing the following as root:
+
+ # cd $PCP_PMDAS_DIR/ds389log
+ # ./Install
+
+If you want to undo the installation, do the following as root:
+
+ # cd $PCP_PMDAS_DIR/ds389log
+ # ./Remove
+
+B<pmdads389log> is launched by pmcd(1) and should never be executed
+directly. The Install and Remove scripts notify pmcd(1) when the agent
+is installed or removed.
+
+=head1 FILES
+
+=over
+
+=item $PCP_PMDAS_DIR/ds389log/ds389log.conf
+
+configuration file for B<pmdads389log>
+
+=item $PCP_PMDAS_DIR/ds389log/Install
+
+installation script for the B<pmdads389log> agent
+
+=item $PCP_PMDAS_DIR/ds389log/Remove
+
+undo installation script for the B<pmdads389log> agent
+
+=item $PCP_LOG_DIR/pmcd/ds389log.log
+
+default log file for messages from B<pmdads389log>
+
+=item /var/log/dirsrv/slapd-.../access
+
+389 Directory Server access log parsed by B<logconv.pl>
+
+=back
+
+=head1 SEE ALSO
+
+logconv.pl(1), pmcd(1), pmdads389(1).
+
--
1.7.1
--
Marko Myllynen
|