High Fragmentation with XFS and NFS Sync

[Nick Fisk]

On 2 July 2016 at 21:12, Darrick J. Wong <darrick.wong at oracle.com> wrote:
> On Sat, Jul 02, 2016 at 09:52:40AM +0100, Nick Fisk wrote:
>> Hi, hope someone can help me here.
>>
>> I'm exporting some XFS fs's to ESX via NFS with the sync option enabled. I'm
>> seeing really heavy fragmentation when multiple VM's are copied onto the
>> share at the same time. I'm also seeing kmem_alloc failures, which is
>> probably the biggest problem as this effectively takes everything down.
>
> (Probably a result of loading the millions of bmbt extents into memory?)

Yes I thought that was the case.

>
>> Underlying storage is a Ceph RBD, the server the FS is running on, is
>> running kernel 4.5.7. Mount options are currently default. I'm seeing
>> Millions of extents, where the ideal is listed as a couple of thousand when
>> running xfs_db, there is only a couple of 100 files on the FS. It looks
>> like roughly the extent sizes roughly match the IO size that the VM's were
>> written to XFS with. So it looks like each parallel IO thread is being
>> allocated next to each other rather than at spaced out regions of the disk.
>>
>> From what I understand, this is because each NFS write opens and closes the
>> file which throws off any chance that XFS will be able to use its allocation
>> features to stop parallel write streams from interleaving with each other.
>>
>> Is there anything I can tune to try and give each write to each file a
>> little bit of space, so that it at least gives readahead a chance when
>> reading, that it might hit at least a few MB of sequential data?
>
> /me wonders if setting an extent size hint on the rootdir before copying
> the files over would help here...

I've set a 16M hint and will copy a new VM over, interested to see
what happens. Thanks for the suggestion.

>
> --D
>
>>
>> I have read that inode32 allocates more randomly compared to inode64, so I'm
>> not sure if it's worth trying this as there will likely be less than a 1000
>> files per FS.
>>
>> Or am I best just to run fsr after everything has been copied on?
>>
>> Thanks for any advice
>> Nick
>
>> _______________________________________________
>> xfs mailing list
>> xfs at oss.sgi.com
>> http://oss.sgi.com/mailman/listinfo/xfs
>