----- "Mark Nelson" <mark@xxxxxxxxxxx> wrote:
>
> Moab is a policy based job scheduler for clusters. It schedules jobs
> and provides logs with details about how resources were allocated. I
> collect and graph this information and am working on comparing and
> contrasting what was allocated with what was used (as reported by
> collectl). One of the things I'm especially interested in doing is
> looking at resource allocation vs resource usage on a per job basis.
> Right now I can use the moab logs and collectl logs to trace process
> statistics and tie it back to specific users but it is very difficult
> to tie processes back to their associated moab jobs.
In an ideal PCP deployment for your situation, a moab PCP agent would
export the data you extract from those logs as moab is scheduling and
you would be able to monitor moab and kernel stats at the same time.
A starting point would be a moab log parser PMDA (some examples exist
in PCP already of such approaches - e.g. pmdazimbra - straightfwd perl
code to extract data from logfiles, live). Longer term you might be
better off directly instrumenting moab itself.
Some of the current discussion on the PCP list in the last couple of
weeks could be of interest to you too (its funny how when you have a
hammer in your hand, everything starts to look like a nail!) - the
tracing infrastructure being designed now could help you track jobs,
job completion and users who submitted them, processes involved, etc.
> A lot of the things you mention here sound like they would be
> extremely
> useful for monitoring a data center with various servers fulfilling
> different roles (database servers, web servers, mailservers, dns,
> etc).
> Is this kind of the direction pcp is going? Our infrastructure
Its a direction PCP went around 10 - 15 years ago. One of the more
recent new directions is investigating incorporating performance data
with tracing semantics alongside the existing data.
> group
> would be very interested in this but I'm not sure PCP is quite the
> right fit for what I'm trying to do.
If collectl is useful to you, then PCP would be - you could use PCP as
a drop in replacement for collectl, initially, then extend it with moab
stats and I expect you would have a good win there & start to see a bit
more of what PCP might be able to do for you.
> I doubt it will really help you, but feel free to take a look at how I
> am doing heatmaps in msica (the software I've been writing). Right
> now I only really deal with time-series data but I'm working on changing
> that. I'm still putting documentation together so if you have any
> questions feel free to ask.
>
> http://code.google.com/p/msica
Almost everything PCP is doing centers around time series data. Thanks
for sending that link, will take a look.
cheers.
--
Nathan
|