[issue] pmwebd graphite api performance issue

[Aurelien Gonnay <aurelien.gonnay@xxxxxxxxx>]

Hi,

 

When using grafana via pmwebd graphite api, I’m experiencing fairly long response times:

 

time wget 'http://xyz:44323/graphite/metrics/find/?query=*'

--2015-07-27 14:56:04--  http://xyz:45323/graphite/metrics/find/?query=*

…

2015-07-27 14:58:14 (217 MB/s) - “index.html?query=*” saved [4210/4210]

 

real    2m10.057s

user    0m0.003s

sys     0m0.003s

 

This makes the system completely impossible to use in any non-trivial context.

You can find thereafter the directory containing the archives:

du -sh */*

286M    xyz-901/archive-20150717.000145.0

52K     xyz-901/archive-20150717.000145.index

44K     xyz-901/archive-20150717.000145.meta

490M    xyz-901/archive-20150718.000241.0

88K     xyz-901/archive-20150718.000241.index

56K     xyz-901/archive-20150718.000241.meta

490M    xyz-901/archive-20150718.000255.0

88K     xyz-901/archive-20150718.000255.index

28K     xyz-901/archive-20150718.000255.meta

490M    xyz-901/archive-20150719.000301.0

88K     xyz-901/archive-20150719.000301.index

28K     xyz-901/archive-20150719.000301.meta

489M    xyz-901/archive-20150721.000315.0

88K     xyz-901/archive-20150721.000315.index

104K    xyz-901/archive-20150721.000315.meta

499M    xyz-901/archive-20150721.000334.0

92K     xyz-901/archive-20150721.000334.index

32K     xyz-901/archive-20150721.000334.meta

565M    xyz-901/archive-20150723.000136.0

104K    xyz-901/archive-20150723.000136.index

60K     xyz-901/archive-20150723.000136.meta

616M    xyz-901/archive-20150723.000157.0

120K    xyz-901/archive-20150723.000157.index

32K     xyz-901/archive-20150723.000157.meta

1.8G    xyz-901/archive-20150727.000941.0

336K    xyz-901/archive-20150727.000941.index

48K     xyz-901/archive-20150727.000941.meta

290M    xyz-901/archive-20150727.001049.0

60K     xyz-901/archive-20150727.001049.index

32K     xyz-901/archive-20150727.001049.meta

47M     xyz-901/archive-20150727.112924.0

12K     xyz-901/archive-20150727.112924.index

32K     xyz-901/archive-20150727.112924.meta

4.0K    xyz-901/config.pmie

24K     xyz-901/config.pmlogger

4.0K    xyz-901/pmie.log

12K     xyz-901/pmlogger.log

861M    xyz-902/archive-20150718.000022.0

160K    xyz-902/archive-20150718.000022.index

13M     xyz-902/archive-20150718.000022.meta

479M    xyz-902/archive-20150718.000048.0

88K     xyz-902/archive-20150718.000048.index

28K     xyz-902/archive-20150718.000048.meta

479M    xyz-902/archive-20150719.000051.0

88K     xyz-902/archive-20150719.000051.index

28K     xyz-902/archive-20150719.000051.meta

511M    xyz-902/archive-20150721.000048.0

100K    xyz-902/archive-20150721.000048.index

11M     xyz-902/archive-20150721.000048.meta

1.4G    xyz-902/archive-20150722.000325.0

256K    xyz-902/archive-20150722.000325.index

40K     xyz-902/archive-20150722.000325.meta

1.8G    xyz-902/archive-20150723.000437.0

336K    xyz-902/archive-20150723.000437.index

5.1M    xyz-902/archive-20150723.000437.meta

1.9G    xyz-902/archive-20150723.000629.0

344K    xyz-902/archive-20150723.000629.index

3.2M    xyz-902/archive-20150723.000629.meta

2.0G    xyz-902/archive-20150725.000552.0

168M    xyz-902/archive-20150725.000552.1

340K    xyz-902/archive-20150725.000552.index

21M     xyz-902/archive-20150725.000552.meta

2.0G    xyz-902/archive-20150725.001407.0

198M    xyz-902/archive-20150725.001407.1

340K    xyz-902/archive-20150725.001407.index

29M     xyz-902/archive-20150725.001407.meta

2.0G    xyz-902/archive-20150726.000657.0

209M    xyz-902/archive-20150726.000657.1

344K    xyz-902/archive-20150726.000657.index

29M     xyz-902/archive-20150726.000657.meta

1.1G    xyz-902/archive-20150727.000654.0

164K    xyz-902/archive-20150727.000654.index

12M     xyz-902/archive-20150727.000654.meta

142M    xyz-902/archive-20150727.113442.0

32K     xyz-902/archive-20150727.113442.index

32K     xyz-902/archive-20150727.113442.meta

4.0K    xyz-902/config.pmie

24K     xyz-902/config.pmlogger

4.0K    xyz-902/pmie.log

12K     xyz-902/pmlogger.log

 

Finally, out of curiosity, I ran perf while performing the very same query, and you can find attached the results of the associated FlameGraph (http://www.brendangregg.com/flamegraphs.html).

The part about pmGetInDomArchive is kind of bothering me, since it looks like it’s spending most of its time in that method.

 

Any thoughts on how to improve my experience ?

 

Aurelien Gonnay

 ___________________________________

The integrity of this message cannot be guaranteed on the internet. Therefore EXANE cannot be considered responsible for the contents.
If you are not the intended recipient of this message, please delete it and notify the sender.

This message is provided for information purposes only and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments.
Although it may contain some elements from publications produced by Exane's research department, this message is not research.
Please consult our web site for important disclaimers and disclosures concerning Exane's research. (http://www.exane.com)
 ___________________________________

pmwebapi_g.perf.svg
Description: pmwebapi_g.perf.svg