nathans wrote:
I wish you had waited for a more thorough analysis of the issue before
committing this patch. (No, the pcpqa tests are not adequate for
performance testing.)
> commit f1fdb217a59506ee1b16ed113727bee3fa170d68
> Author: Nathan Scott <nathans@xxxxxxxxxx>
> Date: Mon May 9 09:33:12 2016 +1000
>
> Revert "PR1099 (compressed archive) mitigation in pmwebd: skip them in
> graphite mode"
>
> This reverts commit 738ccc07e942d295ec516814e7b52f9c910555d8.
>
> ( Thanks Marko for the reminder about this forgotten issue. )
>
> Commit 12da325fa449 properly fixed the archive de-compression
> issues, in libpcp, so the pmwebd workarounds for the observed
> slowdown can now be removed.
No. That commit helps one problem: filtering archives by their
start/end times. That commit does not help the other problem:
actually reading data from a compressed archive. That still requires
completely decompressing the data volume into a /tmp file, each time
the archive is opened.
> [...]
> Finally, its worth noting that the reduction of synchronous read
> I/O means certain situations will see improved performance using
> compressed archives. Modern filesystem optimisations even allow
> for avoiding the temporary file async write I/O, as well, under
> favourable conditions ... so nothing is as simple as it might at
> first seem.
That's all very well, but ignores the CPU & RAM cost of repeatedly
decompressing hundreds of megabytes. (Beyond that, if there is enough
RAM to cache the temporarily-decompressed-to-/tmp content, then there
is enough RAM to cache the never-compressed content from the
filesystem.) Some real performance numbers:
- 2 weeks of routine workstation pmlogger data: ~45 MB/day: 688MB total
- pmwebd -M8
- pmwebd fetch of standard grafana dashboard: 1700 ms (elapsed)
- pmwebd -M1:
- same dashboard: 2640 ms (elapsed)
- xz-compressed: by 97% down to 20MB
- on pmwebd -M8 8cpu server:
- pmwebd fetch of same dashboard: 7500 ms (elapsed)
- xz decompression cpu time: >10000 ms
- on pmwebd -M1 server:
- pmwebd fetch of same dashboard: 12000 ms (elapsed)
While unxz is relatively quick, it is far from instant, and is paid at
every single graphite timeline curve request. It's a user-perceptible
4x slowdown - and this is on a machine many fast CPUs and plenty of
RAM. On a resource-constrained server, this will break (e.g., filling
up /tmp or having to flush it to physical disk).
Until PR1072 on-the-fly decompression is implemented, or some other
countermeasure is implemented, this is not an appropriate change.
- FChE
|