Thanks for the reply Dave!
> Oh, right, it's that workqueue we removed in late 2012 (in the 3.7
> cycle) because it was redundant. The only remaining fragment of it
> is the xfslogd. What kernel are you running?
I am running 3.13.0-39-generic on Ubuntu 14.04.
# uname -a
Linux tf-hippo-1 3.13.0-39-generic #66-Ubuntu SMP Tue Oct 28 13:30:27
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
>> I am seeing a behavior where the system pretty much stalls for ~5
>> seconds after every 30 seconds. I see that the # of ios goes up but
>> the actual write bandwidth during this 5 second period is very low
>> (see attached images). After a fair bit of investigation, we've
>> narrowed down the problem to XFS's syncd (fs.xfs.xfssyncd_centisecs).
>> This runs at a default interval of 30 seconds.
> It's doing background inode reclaim which, under some circumstances,
> involves truncating specualtive allocation beyond EOF before reclaim
> occurs, which results in transactions and inode writeback. It was
> highly inefficient, which is why we replaced it.
Oh.. I see. So, this isn't even actual filesystem metadata. And there
is no option to turn the speculative allocation on/off?
What's the downside of not doing the truncation of the speculative
allocation? Does that result in wasted disk space? If so, how much?
>> I have a couple of questions:
>> 1. If all file writes are done with an fsync() at the end, what is
>> xfssyncd doing for several seconds?
>> 2. How does xfssyncd actually work across several disks? Currently, it
>> seems that when it runs, it pretty much stalls the entire system.
> xfssyncd was actually a workqueue, so it services multiple
> filesystems at once. Before that, there was a kernel thread per
> filesystem for it. Anyway, it's doing lots of random write IO and
> saturating your disks, which will stall any system that is dependent
> on IO throughput to function.
>> 3. I see that fs.xfs.xfssyncd_centisecs is the parameter to tune the
>> interval. But that doesn't give us much. Increasing the interval
>> simply postpones the work. When xfssyncd runs, it takes more time. Are
>> there any other options I can try to make xfssyncd not stall the
>> system when it runs?
> Upgrade your kernel to something more recent, and the problem should
> go away.
We have several other dependencies on the OS. Not sure if upgrading
above Ubuntu 14.04 and kernel 3.13.0-39-generic is an option. Any
other options to try out?