On Tue, Dec 1, 2015 at 8:58 AM, Avi Kivity <avi@xxxxxxxxxxxx> wrote:
>
>
> On 12/01/2015 03:11 PM, Brian Foster wrote:
>>
>> On Tue, Dec 01, 2015 at 11:08:47AM +0200, Avi Kivity wrote:
>>>
>>> On 11/30/2015 06:14 PM, Brian Foster wrote:
>>>>
>>>> On Mon, Nov 30, 2015 at 04:29:13PM +0200, Avi Kivity wrote:
>>>>>
>>>>> On 11/30/2015 04:10 PM, Brian Foster wrote:
>>
>> ...
>>>>
>>>> The agsize/agcount mkfs-time heuristics change depending on the type of
>>>> storage. A single AG can be up to 1TB and if the fs is not considered
>>>> "multidisk" (e.g., no stripe unit/width is defined), 4 AGs is the
>>>> default up to 4TB. If a stripe unit is set, the agsize/agcount is
>>>> adjusted depending on the size of the overall volume (see
>>>> xfsprogs-dev/mkfs/xfs_mkfs.c:calc_default_ag_geometry() for details).
>>>
>>> We'll experiment with this. Surely it depends on more than the amount of
>>> storage? If you have a high op rate you'll be more likely to excite
>>> contention, no?
>>>
>> Sure. The absolute optimal configuration for your workload probably
>> depends on more than storage size, but mkfs doesn't have that
>> information. In general, it tries to use the most reasonable
>> configuration based on the storage and expected workload. If you want to
>> tweak it beyond that, indeed, the best bet is to experiment with what
>> works.
>
>
> We will do that.
>
>>>>> Are those locks held around I/O, or just CPU operations, or a mix?
>>>>
>>>> I believe it's a mix of modifications and I/O, though it looks like some
>>>> of the I/O cases don't necessarily wait on the lock. E.g., the AIL
>>>> pushing case will trylock and defer to the next list iteration if the
>>>> buffer is busy.
>>>>
>>> Ok. For us sleeping in io_submit() is death because we have no other
>>> thread
>>> on that core to take its place.
>>>
>> The above is with regard to metadata I/O, whereas io_submit() is
>> obviously for user I/O.
>
>
> Won't io_submit() also trigger metadata I/O? Or is that all deferred to
> async tasks? I don't mind them blocking each other as long as they let my
> io_submit alone.
>
>> io_submit() can probably block in a variety of
>> places afaict... it might have to read in the inode extent map, allocate
>> blocks, take inode/ag locks, reserve log space for transactions, etc.
>
>
> Any chance of changing all that to be asynchronous? Doesn't sound too hard,
> if somebody else has to do it.
>
>>
>> It sounds to me that first and foremost you want to make sure you don't
>> have however many parallel operations you typically have running
>> contending on the same inodes or AGs. Hint: creating files under
>> separate subdirectories is a quick and easy way to allocate inodes under
>> separate AGs (the agno is encoded into the upper bits of the inode
>> number).
>
>
> Unfortunately our directory layout cannot be changed. And doesn't this
> require having agcount == O(number of active files)? That is easily in the
> thousands.
Actually, wouldn't agcount == O(nr_cpus) be good enough?
>
>> Reducing the frequency of block allocation/frees might also be
>> another help (e.g., preallocate and reuse files,
>
>
> Isn't that discouraged for SSDs?
>
> We can do that for a subset of our files.
>
> We do use XFS_IOC_FSSETXATTR though.
>
>> 'mount -o ikeep,'
>
>
> Interesting. Our files are large so we could try this.
>
>> etc.). Beyond that, you probably want to make sure the log is large
>> enough to support all concurrent operations. See the xfs_log_grant_*
>> tracepoints for a window into if/how long transaction reservations might
>> be waiting on the log.
>
>
> I see that on an 400G fs, the log is 180MB. Seems plenty large for write
> operations that are mostly large sequential, though I've no real feel for
> the numbers. Will keep an eye on this.
>
> Thanks for all the info.
>
>
>> Brian
>>
>>> _______________________________________________
>>> xfs mailing list
>>> xfs@xxxxxxxxxxx
>>> http://oss.sgi.com/mailman/listinfo/xfs
>
>
|