Hi,
It's been a while since my last update to pmdagfs2, I've been on PTO for
the last couple of weeks around Western Europe camping, but in-between
been continuing testing on our side with the sas-calibration workloads.
I plan on following up with a quick succession additional metric updates
with the rest of the trace_pipe based gfs2 tracepoints as well as
starting work on a client side tool to bring information for each node
in the cluster to one central place.
Please let me know of any issues and feedback is welcome :).
Changes committed to git://github.com/pauljevans/pcp.git dev
qa/654.out | 2 +-
qa/655.out | 2 +-
src/pmdas/gfs2/GNUmakefile | 5 +-
src/pmdas/gfs2/control.c | 4 +-
src/pmdas/gfs2/control.h | 1 +
src/pmdas/gfs2/gfs2_ftrace.c | 156
++++++++++++++++++++++++++++++++++++++++++
src/pmdas/gfs2/gfs2_ftrace.h | 36 ++++++++++
src/pmdas/gfs2/help | 5 ++
src/pmdas/gfs2/lock_time.c | 158
+++++++++++++++----------------------------
src/pmdas/gfs2/lock_time.h | 8 ++-
src/pmdas/gfs2/pmda.c | 16 ++++-
src/pmdas/gfs2/pmdagfs2.h | 2 +-
src/pmdas/gfs2/pmns | 1 +
13 files changed, 278 insertions(+), 118 deletions(-)
commit 3163ff6a3c55f83da9b40bf83d089c5d80466468
Author: Paul Evans <pevans@xxxxxxxxxx>
Date: Wed Oct 9 18:01:06 2013 +0100
pmdagfs2: Updates from continued testing and changes for future
updates.
From continued testing with the gfs2 sas-calibration workload there
have been
a couple of changes made to increase the stability of the PMDA
along with
performance improvement with regards to glock throughput when
reading glock
data on trace_pipe based metrics. These improvements are the result
of two main
changes, the first involving changing how the glocks are temporary
stored after
being read from the trace_pipe and the second from controlling the
maximum
number of glocks that are read through each refresh of the metrics.
During testing it because increasingly obviously that due to the
number of
updates that occur for each glock in the event of inode contention
on a node
(resulting in locks being passed around and updated) that hash
collisions when
reading data into the pmdaCache structure was a major bottleneck in
throughput
before time-outs under load. To combat this the lock data taken
from the
trace_pipe is now stored in an array. The duplication issues is
rectified by
checking that the latest update for a given glock is placed towards
the front
of the array during comparison. This has resulted in a total
throughput for
glocks from the trace_pipe to 1 million glocks from 120 thousand using
pmdaCache as the temp storage method.
It has also become obvious that because we need to compute the
worst glock on
each refresh and in order to minimise the chances of time-out of
the PMDA by
PMCD during calculation a new control metric has been introduced.
This metric
gfs2.control.glock_threshold controls the max number of accepted
glock entries
that are taken from the trace_pipe each refresh call. This can be
either
increased or decreased by the user in the event that a machine is
experiencing
time-outs. (Future improvement ideas involve having this value
computed and
changed either reducing or increase with regards to the load of the
system/time
it is taking to process the lock data).
Limiting the number of locks that can be read out on in refresh
call comes with
the potential issue of us not reading all of the locks from the
trace_pipe and
a queue of old lock data collecting. To combat this
gfs2_trace_clear_buffer()
has been introduced to allow the trace_pipe buffer to be completely
cleared at
the end of a metric refresh clearing any unread glocks from trace_pipe.
The final change involves the introduction of gfs2_ftrace.c this
file is a
split of the trace_pipe reading functions from lock_time.c I felt
that with
future improvements being the introduction of the rest of the
trace_pipe based
metrics it was best to split the trace_pipe reading functions from
lock_time.c
and have it contain the specific information regarding the
glock_lock_time
based metrics thus allowing the smoother introduction of the
additional metrics
in the following weeks.
- control.c, control.h
Contains the code to handle the new gfs2.glock_threshold metric
allowing the
user to set a new max threshold value with pmstore.
- gfs2_ftrace.c, gfs2_ftrace.h
Newly created files to separate the trace_pipe handling code. Also
includes new
code to flush the trace_pipe after each refresh. Handling of the glock
threshold to control the maximum number of glocks processed in each
refresh
cycle.
- lock_time.c, lock_time.h
Updated to remove the trace_pipe open/close and reading code which
has be
relocated into gfs2_trace.c. Updates to the method used to
temporary store the
glock data before processing to a dynamic array instead of a
pmdaCache use to
increased performance (not having to worry about hash collision).
Any issues
with duplication are handled during the comparison of the locks before
assigning a worst glock for the filesystem.
- pmda.c, pmdagfs2.h
Contains changes made reflecting the additional control metric and the
splitting of the trace_pipe reading code to it's own files.
- qa/654, qa/655, help, pmns
Updated to reflect the additional metric for controlling the processing
threshold regarding trace_pipe based metrics.
- GNUmakefile
Updates to reflect the additional trace files which need to be
compiled.
Cheers,
Paul
|