Re: [performancecopilot/pcp] pmwebd impossibly slow when using grafana with 300 archives (#117)

[Marko Kevac <notifications@xxxxxxxxxx>]

So about 2 GB of data per server per day, times seven days, times 300 servers, so 4200 GB of data on disk. Wow. Even the current day's data won't fit into your machine's RAM, so any scanning would have to rely on libpcp/archive optimally using .index files to seek to just the the parts being requested by the client (pmwebd/grafana). I don't know if pcp developers have much experience with such RAM-starved configurations.

Why query for host list has to read whole and each archive? It's very strange. Shouldn't host be somewhere in the beginning of the file?

After initial "get me host list" query, when I chose one host, pmwebd would have to read only 7 archives for one host (14GB).

And if only one day or one hour is selected in Grafana, then only one file (2GB).

But right now problem is on the first stage. Getting host list. And IMHO (without knowing about pcp internals), this should not need 4Tb read.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.