sleeps and waits during io_submit

Tue Dec 1 10:01:34 CST 2015

On Tue, Dec 01, 2015 at 05:22:38PM +0200, Avi Kivity wrote:
> 
> 
> On 12/01/2015 04:56 PM, Brian Foster wrote:
> >On Tue, Dec 01, 2015 at 03:58:28PM +0200, Avi Kivity wrote:
> >>
> >>On 12/01/2015 03:11 PM, Brian Foster wrote:
> >>>On Tue, Dec 01, 2015 at 11:08:47AM +0200, Avi Kivity wrote:
> >>>>On 11/30/2015 06:14 PM, Brian Foster wrote:
> >>>>>On Mon, Nov 30, 2015 at 04:29:13PM +0200, Avi Kivity wrote:
> >>>>>>On 11/30/2015 04:10 PM, Brian Foster wrote:
> >>>...
> >>>>>The agsize/agcount mkfs-time heuristics change depending on the type of
> >>>>>storage. A single AG can be up to 1TB and if the fs is not considered
> >>>>>"multidisk" (e.g., no stripe unit/width is defined), 4 AGs is the
> >>>>>default up to 4TB. If a stripe unit is set, the agsize/agcount is
> >>>>>adjusted depending on the size of the overall volume (see
> >>>>>xfsprogs-dev/mkfs/xfs_mkfs.c:calc_default_ag_geometry() for details).
> >>>>We'll experiment with this.  Surely it depends on more than the amount of
> >>>>storage?  If you have a high op rate you'll be more likely to excite
> >>>>contention, no?
> >>>>
> >>>Sure. The absolute optimal configuration for your workload probably
> >>>depends on more than storage size, but mkfs doesn't have that
> >>>information. In general, it tries to use the most reasonable
> >>>configuration based on the storage and expected workload. If you want to
> >>>tweak it beyond that, indeed, the best bet is to experiment with what
> >>>works.
> >>We will do that.
> >>
> >>>>>>Are those locks held around I/O, or just CPU operations, or a mix?
> >>>>>I believe it's a mix of modifications and I/O, though it looks like some
> >>>>>of the I/O cases don't necessarily wait on the lock. E.g., the AIL
> >>>>>pushing case will trylock and defer to the next list iteration if the
> >>>>>buffer is busy.
> >>>>>
> >>>>Ok.  For us sleeping in io_submit() is death because we have no other thread
> >>>>on that core to take its place.
> >>>>
> >>>The above is with regard to metadata I/O, whereas io_submit() is
> >>>obviously for user I/O.
> >>Won't io_submit() also trigger metadata I/O?  Or is that all deferred to
> >>async tasks?  I don't mind them blocking each other as long as they let my
> >>io_submit alone.
> >>
> >Yeah, it can trigger metadata reads, force the log (the stale buffer
> >example) or push the AIL (wait on log space). Metadata changes made
> >directly via your I/O request are logged/committed via transactions,
> >which are generally processed asynchronously from that point on.
> >
> >>>  io_submit() can probably block in a variety of
> >>>places afaict... it might have to read in the inode extent map, allocate
> >>>blocks, take inode/ag locks, reserve log space for transactions, etc.
> >>Any chance of changing all that to be asynchronous?  Doesn't sound too hard,
> >>if somebody else has to do it.
> >>
> >I'm not following... if the fs needs to read in the inode extent map to
> >prepare for an allocation, what else can the thread do but wait? Are you
> >suggesting the request kick off whatever the blocking action happens to
> >be asynchronously and return with an error such that the request can be
> >retried later?
> 
> Not quite, it should be invisible to the caller.
> 
> That is, the code called by io_submit() (file_operations::write_iter, it
> seems to be called today) can kick off this operation and have it continue
> from where it left off.
> 

Isn't that generally what happens today? We submit an I/O which is
asynchronous in nature and wait on a completion, which causes the cpu to
schedule and execute another task until the completion is set by I/O
completion (via an async callback). At that point, the issuing thread
continues where it left off. I suspect I'm missing something... can you
elaborate on what you'd do differently here (and how it helps)?

> Seastar (the async user framework which we use to drive xfs) makes writing
> code like this easy, using continuations; but of course from ordinary
> threaded code it can be quite hard.
> 
> btw, there was an attempt to make ext[34] async using this method, but I
> think it was ripped out.  Yes, the mortal remains can still be seen with
> 'git grep EIOCBQUEUED'.
> 
> >
> >>>It sounds to me that first and foremost you want to make sure you don't
> >>>have however many parallel operations you typically have running
> >>>contending on the same inodes or AGs. Hint: creating files under
> >>>separate subdirectories is a quick and easy way to allocate inodes under
> >>>separate AGs (the agno is encoded into the upper bits of the inode
> >>>number).
> >>Unfortunately our directory layout cannot be changed.  And doesn't this
> >>require having agcount == O(number of active files)?  That is easily in the
> >>thousands.
> >>
> >I think Glauber's O(nr_cpus) comment is probably the more likely
> >ballpark, but really it's something you'll probably just need to test to
> >see how far you need to go to avoid AG contention.
> >
> >I'm primarily throwing the subdir thing out there for testing purposes.
> >It's just an easy way to create inodes in a bunch of separate AGs so you
> >can determine whether/how much it really helps with modified AG counts.
> >I don't know enough about your application design to really comment on
> >that...
> 
> We have O(cpus) shards that operate independently.  Each shard writes 32MB
> commitlog files (that are pre-truncated to 32MB to allow concurrent writes
> without blocking); the files are then flushed and closed, and later removed.
> In parallel there are sequential writes and reads of large files using 128kB
> buffers), as well as random reads.  Files are immutable (append-only), and
> if a file is being written, it is not concurrently read.  In general files
> are not shared across shards.  All I/O is async and O_DIRECT.  open(),
> truncate(), fdatasync(), and friends are called from a helper thread.
> 
> As far as I can tell it should a very friendly load for XFS and SSDs.
> 
> >
> >>>  Reducing the frequency of block allocation/frees might also be
> >>>another help (e.g., preallocate and reuse files,
> >>Isn't that discouraged for SSDs?
> >>
> >Perhaps, if you're referring to the fact that the blocks are never freed
> >and thus never discarded..? Are you running fstrim?
> 
> mount -o discard.  And yes, overwrites are supposedly more expensive than
> trim old data + allocate new data, but maybe if you compare it with the work
> XFS has to do, perhaps the tradeoff is bad.
> 

Ok, my understanding is that '-o discard' is not recommended in favor of
periodic fstrim for performance reasons, but that may or may not still
be the case.

Brian

> 
> >
> >If so, it would certainly impact that by holding blocks as allocated to
> >inodes as opposed to putting them in free space trees where they can be
> >discarded. If not, I don't see how it would make a difference, but
> >perhaps I misunderstand the point. That said, there's probably others on
> >the list who can more definitively discuss SSD characteristics than I...
> 
> 
> 
> >
> >>We can do that for a subset of our files.
> >>
> >>We do use XFS_IOC_FSSETXATTR though.
> >>
> >>>'mount -o ikeep,'
> >>Interesting.  Our files are large so we could try this.
> >>
> >Just to be clear... this behavior change is more directly associated
> >with file count than file size (though indirectly larger files might
> >mean you have less of them, if that's your point).
> 
> Yes, that's what I meant, and especially that if a lot of files are removed
> we'd be losing the inode space allocated to them.
> 
> >
> >To generalize a bit, I'd be more weary of using this option if your
> >filesystem can be used in an unstructured manner in any way. For
> >example, if the file count can balloon up and back down temporarily,
> >that's going to allocate a bunch of metadata space for inodes that won't
> >ever be reclaimed or reused for anything other than inodes.
> 
> Exactly.  File count can balloon, but files will be large, so even the worst
> case waste is very limited.
> 
> >
> >>>etc.). Beyond that, you probably want to make sure the log is large
> >>>enough to support all concurrent operations. See the xfs_log_grant_*
> >>>tracepoints for a window into if/how long transaction reservations might
> >>>be waiting on the log.
> >>I see that on an 400G fs, the log is 180MB.  Seems plenty large for write
> >>operations that are mostly large sequential, though I've no real feel for
> >>the numbers.  Will keep an eye on this.
> >>
> >FWIW, XFS on recent kernels has grown some sysfs entries that might help
> >give an idea of log reservation state at runtime. See the entries under
> >/sys/fs/xfs/<dev>/log for details.
> 
> Great.  We will study those with great interest.
> 
> >
> >Brian
> >
> >>Thanks for all the info.
> >>
> >>>Brian
> >>>
> >>>>_______________________________________________
> >>>>xfs mailing list
> >>>>xfs at oss.sgi.com
> >>>>http://oss.sgi.com/mailman/listinfo/xfs
> 
> 