[performancecopilot/pcp] pmwebd impossibly slow when using grafana with 300 archives (#117)

[Marko Kevac <notifications@xxxxxxxxxx>]

Hello.

We are collecting metrics from ~300 servers with 1s granularity using pmlogger. It works fine. Data is going to appropriate PCP databases on disk.

But visually browsing these archives is impossible using pmwebd and grafana.

After clicking on a host list in grafana

Nothing happens.

I can see in pmwebd log that it got request for /graphite/render:

[Thu Sep 29 14:06:55] pmwebd(22926): [192.168.3.129:33220] HTTP/1.1 GET /grafana/config.js
[Thu Sep 29 14:06:55] pmwebd(22926): [192.168.3.129:33220] HTTP/1.1 GET /grafana/app/dashboards/hostselect.js
[Thu Sep 29 14:06:55] pmwebd(22926): [192.168.3.129:33220] HTTP/1.1 GET /graphite/render

pmwebd process is stuck in D reading files. I've waited for 10 minutes, but nothing changed.

strace shows something like this:

[pid 22926] lseek(60, 666161152, SEEK_SET) = 666161152
[pid 22926] read(60, "\3\0\0\f\0\0\0\0\310\326\375\262\3\0\0\f\0\0\0\0\316\7\242Z\3\0\0\f\0\0\0\0"..., 4096) = 4096
[pid 22926] lseek(60, 666165248, SEEK_SET) = 666165248
[pid 22926] lseek(60, 666165248, SEEK_SET) = 666165248
[pid 22926] lseek(60, 666136576, SEEK_SET) = 666136576
[pid 22926] read(60, "\3\0\0\f\0\0\0\0\0\0\0\0\3\0\0\f\0\0\0\0\0\0\0\0\3\0\0\f\0\0\0\0"..., 4096) = 4096
[pid 22926] read(60, "\0\0\0\20\0\0\v\265\0\0\0\21\0\0\v\270\0\0\0\22\0\0\v\273\0\0\0\23\0\0\v\276"..., 20480) = 20480
[pid 22926] read(60, "\3\0\0\f\0\0\0\0\310\326\375\262\3\0\0\f\0\0\0\0\316\7\242Z\3\0\0\f\0\0\0\0"..., 4096) = 4096
[pid 22926] lseek(60, 666136576, SEEK_SET) = 666136576
[pid 22926] read(60, "\3\0\0\f\0\0\0\0\0\0\0\0\3\0\0\f\0\0\0\0\0\0\0\0\3\0\0\f\0\0\0\0"..., 4096) = 4096
[pid 22926] lseek(60, 666140672, SEEK_SET) = 666140672
[pid 22926] lseek(60, 666140672, SEEK_SET) = 666140672
[pid 22926] lseek(60, 666116096, SEEK_SET) = 666116096
[pid 22926] read(60, "\0\0\0\0\22\240\357\267\3\0\0\f\0\0\0\0\0\23\222G\3\0\0\f\0\0\0\33\204\245\3714"..., 4096) = 4096
[pid 22926] read(60, "\0\0\0\1\0\0\0\1\377\377\377\377\0\0\r\324\17\0pD\0\0\0\1\0\0\0\1\377\377\377\377"..., 16384) = 16384
[pid 22926] read(60, "\3\0\0\f\0\0\0\0\0\0\0\0\3\0\0\f\0\0\0\0\0\0\0\0\3\0\0\f\0\0\0\0"..., 4096) = 4096
[pid 22926] lseek(60, 666116096, SEEK_SET) = 666116096
[pid 22926] read(60, "\0\0\0\0\22\240\357\267\3\0\0\f\0\0\0\0\0\23\222G\3\0\0\f\0\0\0\33\204\245\3714"..., 4096) = 4096
[pid 22926] lseek(60, 666120192, SEEK_SET) = 666120192
[pid 22926] lseek(60, 666120192, SEEK_SET) = 666120192
[pid 22926] lseek(60, 666091520, SEEK_SET) = 666091520
[pid 22926] read(60, "\0\0\0\0\0\0\0\0\3\0\0\f\0\0\0\0\0\0\0\0\3\0\0\f\0\0\0\0\0\0\0\0"..., 4096) = 4096
[pid 22926] read(60, "\0\0\v[\0\0\0\23\0\0\v^\0\0\0\24\0\0\va\0\0\0\25\0\0\vd\0\0\0\26"..., 20480) = 20480
[pid 22926] read(60, "\0\0\0\0\22\240\357\267\3\0\0\f\0\0\0\0\0\23\222G\3\0\0\f\0\0\0\33\204\245\3714"..., 4096) = 4096
[pid 22926] lseek(60, 666091520, SEEK_SET) = 666091520
[pid 22926] read(60, "\0\0\0\0\0\0\0\0\3\0\0\f\0\0\0\0\0\0\0\0\3\0\0\f\0\0\0\0\0\0\0\0"..., 4096) = 4096
[pid 22926] lseek(60, 666095616, SEEK_SET) = 666095616
[pid 22926] lseek(60, 666095616, SEEK_SET) = 666095616
[pid 22926] lseek(60, 666071040, SEEK_SET) = 666071040
[pid 22926] read(60, "\0\21\302T\3\0\0\f\0\0\0\0\0\7\233D\3\0\0\f\0\0\0\0\0\31O\210\3\0\0\f"..., 4096) = 4096

I will try and provide some more info, but maybe you, pcp developers, already know what is going on...

This is a IO Wait time for server after requesting /graphite/render:

Server is barely working :-)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.