Performance degradation over time

[Peter Grandi]

>>> [ ... ] open() syscalls (open for writing) were taking
>>> significantly more time than they should eg. 15-20ms vs
>>> 100-150us. [ ... ] That means that we create lots of files in
>>> /mountpoint/some/path/.tmp directory, but directory is empty
>>> as they are moved (rename() syscall) shortly after file
>>> creation to a different directory on the same filesystem.

>>> The workaround which I found so far is to remove that
>>> directory (/mountpoint/some/path/.tmp in our case) with its
>>> content and re-create it. After this operation open() syscall
>>> goes down to 100-150us again.

>>> Is this a known problem ?

Indeed, two known (for several decades) problems: using
filesystems as DBMSes and directories as spool queues.

[ ... ]

> After mounting XFS with inode64 I see performance improvement
> (open() now takes ~3ms vs ~15ms previous) although it's still
> not something I would expect (~150us.)

It would be amusing to know why ever to expect a random metadata
access operation to take 150µs on *average* on a storage system
that seems to have rotating disk with 10-15ms *average* access
time.

The metadata operations may have locality, but unsurprisingly
that decreases with time...