<div dir="ltr"><span style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);display:inline!important;float:none">Hi, hope someone can help me here.</span><br style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);display:inline!important;float:none">I'm exporting some XFS fs's to ESX via NFS with the sync option enabled. I'm</span><br style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);display:inline!important;float:none">seeing really heavy fragmentation when multiple VM's are copied onto the</span><br style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);display:inline!important;float:none">share at the same time. I'm also seeing kmem_alloc failures, which is probably the biggest problem as this effectively takes everything down. <br><br>Underlying storage is a Ceph RBD, the server the FS is running on, is running kernel 4.5.7. Mount options are currently default. I'm seeing Millions of extents, where the ideal is listed as a couple of thousand when running xfs_db, there is only a couple of 100 files on the FS. It looks like roughly the extent sizes roughly match the IO size that the VM's were written to XFS with. So it looks like each parallel IO thread is being allocated next to each other rather than at spaced out regions of the disk.</span><br style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);display:inline!important;float:none">From what I understand, this is because each NFS write opens and closes the</span><br style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);display:inline!important;float:none">file which throws off any chance that XFS will be able to use its allocation</span><br style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);display:inline!important;float:none">features to stop parallel write streams from interleaving with each other.</span><br style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);display:inline!important;float:none">Is there anything I can tune to try and give each write to each file a</span><br style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);display:inline!important;float:none">little bit of space, so that it at least gives readahead a chance when</span><br style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);display:inline!important;float:none">reading, that it might hit at least a few MB of sequential data?</span><br style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);display:inline!important;float:none">I have read that inode32 allocates more randomly compared to inode64, so I'm</span><br style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);display:inline!important;float:none">not sure if it's worth trying this as there will likely be less than a 1000</span><br style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);display:inline!important;float:none">files per FS.</span><br style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);display:inline!important;float:none">Or am I best just to run fsr after everything has been copied on?</span><br style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);display:inline!important;float:none">Thanks for any advice</span><br style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="color:rgb(0,0,0);font-family:monospace;font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);display:inline!important;float:none">Nick</span></div>