Re: understanding speculative preallocation

[Jason Rosenberg <jbr@xxxxxxxxxxxx>]

Is it safe to say that speculative preallocation will not be used if a file is opened read-only?

It turns out that the kafka server does indeed write lots of log files, and rotate them after they reach a max size, but never closes the files until the app exits, or until it deletes the files.  This is because it needs to make them available for reading, etc.   So, an obvious change for kafka might be to close each log file after rotating, and then re-open it read-only for consumers of the data.  Does that sound like a solution that would pro-actively release pre-allocated storage?

Thanks,

Jason

On Fri, Jul 26, 2013 at 5:04 PM, Jason Rosenberg <jbr@xxxxxxxxxxxx> wrote:

Thanks Ben,

This is helpful.

On Fri, Jul 26, 2013 at 4:50 PM, Ben Myers <bpm@xxxxxxx> wrote:

Hi Jason,

On Fri, Jul 26, 2013 at 04:38:21PM -0400, Jason Rosenberg wrote:
> Thanks for the info (most of it was, in fact, news to me).  I'm an
> application developer trying to debug a disk space problem, that's all.  So
> far, I've tracked it down to being an XFS issue.

The speculative block reservations have been an issue for awhile.  You are not
the first person to take issue with it.

> Regardless, is there a version history for XFS vis-a-via mainline Linux?

You can find a full version history for XFS back to 2.6.12 or so here:
http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=summary

If you're interested in going older than that look here:
http://oss.sgi.com/cgi-bin/gitweb.cgi?p=archive/xfs-import.git;a=summary

The function you'll most likely want to track is xfs_iomap_write_delay, which
calls xfs_iomap_eof_want_preallocate, both of which are in fs/xfs/xfs_iomap.c.

Recently Brian Foster added a scanner to remove the speculative block
reservations on a timer which may give you some relief.  See
xfs_queue_eofblocks in fs/xfs/xfs_icache.c

Regards,
        Ben