pmmgr normally generates relatively large merged-archive-* files. It seems
that somewhere in the vicinity of 300MB, performance starts drastically
dropping for operations like time-scrolling in pmchart, when a single Overview
chart-set is active.
For a peer server's 230MB merged-archive, time-panning (720 points, 60-second
interval) takes a delightful second or so to redraw. For the troubled 300MB
one, each takes more like 30-90 seconds (!!). Their pmloglabel -l output is
almost identical: they contain two weeks' worth of data.
A systemtap pf4.stp profiling run during the slow-refresh interval indicates
many hits to backtraces such as these, implicating the __pmLogFetchInterp()
function in libpcp/src/interp.c as a common root:
112 hits on cpu 0
__pmHashSearch+0x21 [libpcp.so.3]
update_bounds.isra.1+0xcc [libpcp.so.3]
__pmLogFetchInterp+0x1a9a [libpcp.so.3]
__pmLogFetch+0x68f [libpcp.so.3]
pmFetch+0xb6 [libpcp.so.3]
_ZN10QmcContext5fetchEb+0x368 [pmchart]
_ZN8QmcGroup5fetchEb+0xbd [pmchart]
_ZN12GroupControl29adjustArchiveWorldViewForwardEPN6PmTime6PacketEb+0x1c5
[pmchart]
_ZN11TimeControl15protocolMessageEbPN6PmTime6PacketEP10QTcpSocketPNS_13ProtocolStateE+0x308
[pmchart]
_ZN11TimeControl19protocolMessageLoopEbPN6PmTime6PacketEP10QTcpSocketPNS_13
77 hits on cpu 0
__ntohpmValueBlock+0x2b [libpcp.so.3]
__pmDecodeResult+0x1ba [libpcp.so.3]
__pmLogRead+0x565 [libpcp.so.3]
cache_read.isra.0+0x141 [libpcp.so.3]
__pmLogFetchInterp+0x1a16 [libpcp.so.3]
__pmLogFetch+0x68f [libpcp.so.3]
pmFetch+0xb6 [libpcp.so.3]
_ZN10QmcContext5fetchEb+0x368 [pmchart]
_ZN8QmcGroup5fetchEb+0xbd [pmchart]
_ZN12GroupControl29adjustArchiveWorldViewForwardEPN6PmTime6PacketEb+0x1c5
[pmchart]
_ZN11TimeControl15protocolMessageEbPN6PmTime6PacketEP10QTcpSocketPNS_13ProtocolStateE+0x308
[pmchart]
_ZN1
41 hits on cpu 0
__GI_memcpy+0x15b [libc-2.17.so]
__GI__IO_file_xsgetn+0x113 [libc-2.17.so]
fread+0x88 [libc-2.17.so]
__pmLogRead+0x347 [libpcp.so.3]
cache_read.isra.0+0x141 [libpcp.so.3]
__pmLogFetchInterp+0x1a16 [libpcp.so.3]
__pmLogFetch+0x68f [libpcp.so.3]
pmFetch+0xb6 [libpcp.so.3]
_ZN10QmcContext5fetchEb+0x368 [pmchart]
_ZN8QmcGroup5fetchEb+0xbd [pmchart]
_ZN12GroupControl29adjustArchiveWorldViewForwardEPN6PmTime6PacketEb+0x1c5
[pmchart]
_ZN11TimeControl15protocolMessageEbPN6PmTime6PacketEP10QTcpSocketPNS_13ProtocolS