XFS fragmentation on file append

[Stefan Ring]

On Thu, Apr 24, 2014 at 11:49 PM, Keyur Govande <keyurgovande at gmail.com> wrote:
> On Thu, Apr 24, 2014 at 2:54 AM, Stefan Ring <stefanrin at gmail.com> wrote:
>> I've become interested in this topic, as I'm also running MySQL with
>> O_DIRECT and innodb_file_per_table. Out of curiosity, I immediately
>> ran xfs_bmap on a moderately sized table space (34GB). It listed
>> around 30000 fragments, on average one for every MB.
>>
>> I want to report what happened then: A flurry of activity started on
>> both disks (root/swap lives on one of them, the data volume containing
>> the MySQL files on another) and lasted for about two minutes.
>> Afterwards, all memory previously allocated to the file cache has
>> become free, and also everything XFS seems to keep buffered internally
>> (I think it's called SReclaimable) was released. Swap usage increased
>> only slightly. dmesg was silent during that time.
>>
>> This is a 2.6.32-358.2.1.el6.x86_64 kernel with xfsprogs 3.1.1 (CentOS
>> 6.4). The machine has 64GB of RAM (2 NUMA nodes) and 24 (virtual)
>> cores. Is this known behavior of xfs_bmap?
>
> Interesting...it looks like your box flushed all of the OS buffer
> cache. I am unable to reproduce this behavior on my test box with the
> 3.10.37 kernel. I also tried with 2.6.32-358.18.1.el6.x86_64 and
> didn't hit the issue, but obviously our access patterns differ wildly.

I tried it again, logging a few files in /proc periodically:
https://dl.dropboxusercontent.com/u/5338701/dev/xfs/memdump.tar.xz

Inside the archive, "memdump" is the simplistic script used to create
the other files. A few seconds in, I invoked xfs_bmap on the same file
again (this time weighing in at 81GB), and it spit out 36000
fragments. It took only a few seconds to completely drain 40 GB of
buffer memory.

cciss/c0d0 is the device where the XFS filesystem lives, while sda
contains root and swap.

If somebody could gain some insight from this, I'd be happy.