<div dir="ltr">ok, I have a simple reproducer. try out the following, noting you'll obviously have to change the directory pointed to by dname:<div><br></div><div><div>libc=ctypes.CDLL(ctypes.util.find_library('c'), use_errno=True)<br>
</div><div>falloc=getattr(libc, 'fallocate')</div><div><br></div><div>fb<span style="font-family:'courier new',monospace">#!/usr/bin/python -u</span></div><div><font face="courier new, monospace"><br></font></div>
<div><font face="courier new, monospace">import os</font></div><div><font face="courier new, monospace">import sys</font></div><div><font face="courier new, monospace">import time</font></div><div><font face="courier new, monospace">import ctypes</font></div>
<div><font face="courier new, monospace">import ctypes.util</font></div><div><font face="courier new, monospace">from tempfile import mkstemp</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">dname = '/srv/node/disk0/mjs'</font></div>
<div><font face="courier new, monospace">fname = 'foo'</font></div><div><font face="courier new, monospace">fsize = 1024</font></div><div><font face="courier new, monospace">nfiles = 1000</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace">body = ' ' * fsize</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">time0 = time.time()</font></div>
<div><font face="courier new, monospace">for i in range(nfiles):</font></div><div><font face="courier new, monospace"> file_name = '%s/%s-%s' % (dname, fname, i)</font></div><div><font face="courier new, monospace"> fd, tmppath = mkstemp(dir=dname)</font></div>
<div><font face="courier new, monospace"> falloc(fd, 1, 0, fsize)</font></div><div><font face="courier new, monospace"> os.rename(tmppath, file_name)</font></div><div><font face="courier new, monospace"><br></font></div>
<div><font face="courier new, monospace">elapsed = time.time() - time0</font></div><div><font face="courier new, monospace">tbytes = fsize * nfiles</font></div><div><font face="courier new, monospace">rate = tbytes/elapsed/1024/1024</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">print "DName: %s" % dname</font></div><div><font face="courier new, monospace">print "Bytes: %d" % (tbytes/1024/1024)</font></div>
<div><font face="courier new, monospace">print "Time: %.2f secs" % elapsed</font></div><div><font face="courier new, monospace">print "Rate: %.2f/sec" % rate</font></div></div><div><br></div><div style>
when I run it I see this:</div><div style><br></div><div style><div><font face="courier new, monospace">segerm@az1-sw-object-0006:~$ sudo ./falloc.py</font></div><div><font face="courier new, monospace">DName: /srv/node/disk0/mjs</font></div>
<div><font face="courier new, monospace">Bytes: 9</font></div><div><font face="courier new, monospace">Time: 5.84 secs</font></div><div><font face="courier new, monospace">Rate: 1.67/sec</font></div><div><br></div><div style>
and while running collectl I see this:</div><div style><br></div><div style><div><font face="courier new, monospace">#<----CPU[HYPER]-----><----------Disks-----------><----------Network----------></font></div>
<div><font face="courier new, monospace">#cpu sys inter ctxsw KBRead Reads KBWrit Writes KBIn PktIn KBOut PktOut</font></div><div><font face="courier new, monospace"> 0 0 110 113 0 0 0 0 0 5 0 3<br>
</font></div><div><font face="courier new, monospace"> 1 0 1576 2874 0 0 170240 665 0 3 0 2</font></div><div><font face="courier new, monospace"> 4 3 2248 6623 0 0 406585 1596 0 1 0 1</font></div>
<div><font face="courier new, monospace"> 4 3 2145 7680 0 0 473600 1850 0 1 0 1</font></div><div><font face="courier new, monospace"> 2 1 2200 7406 0 0 456633 1875 0 2 0 1</font></div>
<div><font face="courier new, monospace"> 4 3 3696 7401 0 0 454606 1823 0 1 0 1</font></div><div><font face="courier new, monospace"> 3 2 3877 7354 0 0 453537 1806 0 1 0 1</font></div>
<div><font face="courier new, monospace"> 1 0 1610 2764 0 0 163793 684 0 3 0 3<br></font></div><div><br></div><div style>This is the same behavior I'm seeing on swift. 10K 1KB files X 4kb minimal block size still comes out to a lot less than the multiple GB of writes being reported. Actually since these whole thing only takes a few seconds and I know a single disk can't write that fast maybe it's just a bug in the way the kernel is reported allocated preallocated blocks and nothing to do with XFS? Or iis xfs responsible for the stats?</div>
<div style><br></div><div style>If I remove the fallocate call I see the expected amount of disk traffic.</div><div style><br></div><div style>-mark</div><div><br></div></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Sat, Jun 15, 2013 at 8:11 PM, Dave Chinner <span dir="ltr"><<a href="mailto:david@fromorbit.com" target="_blank">david@fromorbit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On Sat, Jun 15, 2013 at 12:22:35PM -0400, Mark Seger wrote:<br>
> I was thinking a little color commentary might be helpful from a<br>
> perspective of what the functionally is that's driving the need for<br>
> fallocate. I think I mentioned somewhere in this thread that the<br>
> application is OpenStack Swift, which is a highly scalable cloud object<br>
> store.<br>
<br>
</div>I'm familiar with it and the problems it causes filesystems. What<br>
application am I talking about here, for example?<br>
<br>
<a href="http://oss.sgi.com/pipermail/xfs/2013-June/027159.html" target="_blank">http://oss.sgi.com/pipermail/xfs/2013-June/027159.html</a><br>
<br>
Basically, Swift is trying to emulate Direct IO because python<br>
does't support Direct IO. Hence Swift is hacking around that problem<br>
and causing secondary issues that would never have occurred if<br>
Direct IO was used in the first place.<br>
<div class="HOEnZb"><div class="h5"><br>
Cheers,<br>
<br>
Dave.<br>
--<br>
Dave Chinner<br>
<a href="mailto:david@fromorbit.com">david@fromorbit.com</a><br>
</div></div></blockquote></div><br></div>