On Mon, Aug 18, 2014 at 11:16:12AM -0500, Stan Hoeppner wrote:
> On Mon, 18 Aug 2014 17:01:53 +1000, Dave Chinner <david@xxxxxxxxxxxxx>
> wrote:
> > On Sun, Aug 17, 2014 at 10:29:21PM -0500, Stan Hoeppner wrote:
> >> Say I have a single 4TB disk in an md linear device. The md device has
> a
> >> filesystem on it formatted with defaults. It has 4 AGs, 0-3. I have
> >> created 4 directories. Each should reside in a different AG, the first
> >> in
> >> AG0. Now I expand the linear device with an identical 4TB disk and
> >> execute
> >> xfs_growfs. I now have 4 more AGs, 4-7. I create 4 more directories.
> >>
> >> Will these 4 new dirs be created sequentially in AGs 4-7, or in the
> first
> >> 4 AGs? Is this deterministic, or is there any chance involved? On the
> >
> > Deterministic, assuming single threaded *file-system-wide* directory
> > creation. Completely unpredictable under concurrent directory
> > creations. See xfs_ialloc_ag_select/xfs_ialloc_next_ag.
> >
> > Note that the rotor used to select the next AG is set to
> > zero at mount.
> >
> > i.e. single threaded behaviour at agcount = 4:
> >
> > dir number rotor value destination AG
> > 1 0 0
> > 2 1 1
> > 3 2 2
> > 4 3 3
> > 5 0 0
> > 6 1 1
> > ....
> >
> > So, if you do what you suggest, and grow *after* the first 4 dirs
> > are created, the above is what you'll get because the rotor goes
> > back to zero on the fourth directory create. Now, with changing from
> > 4 to 8 AGs after the first 4:
> >
> > dir number rotor value new inode location (AG)
> > 1 0 0
> > 2 1 1
> > 3 2 2
> > 4 3 3
> > <grow to 8 AGs>
> > 5 0 0
> > 6 1 1
> > 7 2 2
> > 8 3 3
> > 9 4 4
> > 10 5 5
> > 11 6 6
> > 13 7 7
> > 14 0 0
> >
> >> real system these 4TB drives are actually 48TB LUNs. I'm after
> >> deterministic parallel bandwidth to subsequently added RAIDs after each
> >> grow operation by simply writing to the proper directory.
> >
> > Just create new directories and use the inode number to
> > determine their location. If the directory is not in the correct AG,
> > remove it and create a new one, until you have directories located
> > in the AGs you want.
> >
> > Cheers,
> >
> > Dave.
>
>
> Thanks for the info Dave. Was hoping it would be more straightforward.
> Modifying the app for this is out of the question. They've spent 3+ years
> developing with EXT4 and decided to try XFS at the last minute. Product is
> to ship in October, so optimizations I can suggest are limited.
Perhaps you could actually tell us what the requirement for
layout/separation is, and how they are acheiving it with ext4. We
really need a more "directed" allocation ability, but it's not clear
exactly what requirements need to drive that.
Cheers,
Dave.
--
Dave Chinner
david@xxxxxxxxxxxxx
|