|To:||Brian Foster <bfoster@xxxxxxxxxx>, Sean Caron <scaron@xxxxxxxxx>|
|Subject:||Re: Is XFS suitable for 350 million files on 20TB storage?|
|From:||Sean Caron <scaron@xxxxxxxxx>|
|Date:||Fri, 5 Sep 2014 18:39:42 -0400|
|Cc:||Stefan Priebe <s.priebe@xxxxxxxxxxxx>, "xfs@xxxxxxxxxxx" <xfs@xxxxxxxxxxx>|
|References:||<540986B1.4080306@xxxxxxxxxxxx> <20140905123058.GA29710@xxxxxxxxxxxxxxx> <5409AF40.10801@xxxxxxxxxxxx> <20140905134810.GA3965@xxxxxxxxxxxxxx> <5409FBEA.9050708@xxxxxxxxxxxx> <20140905191815.GB8400@xxxxxxxxxxxxxx> <540A19BB.8040404@xxxxxxxxxxxx> <20140905212400.GA8904@xxxxxxxxxxxxxx>|
Generally speaking, this is a situation that you want to avoid. At 350 million files and 20 TB, you're looking at around 17-18 MB per file at minimum? That's pretty small. And with 350M files, a fair number of those 350M must be on the smaller side of things.
Memory is cheap these days... people can make 50 GB, 100 GB, ... files, go ahead and read those things directly into memory, 100%. And CPU cycles are pretty cheap, too. Certainly you get more bang per buck there, than in IOPS on your storage system!!
Empirically, I have found from experience (currently running Linux 3.4.61; many historical revs previous) in a reasonably large-scale (up to ~0.5 PB in single file system, up to 270 JBOD spindles on one machine) high-I/O (jobs running on a few-hundred-node compute cluster, or a few hundred threads running locally on the server) environment, that XFS (and things running on top of it, ESPECIALLY rsync) will perform MUCH better on smaller numbers of very large files, then they will on very large numbers of small files (I'm always trying to reinforce this to our end users).
I'm not really even saying XFS is to really blame here... in fact in 3.4.61 it has been very well-behaved; but Linux has many warts: poor implementation of I/O and CPU scheduling algorithms; kernel does not degrade gracefully in resource-constrained settings; if you are ultimately using this data store as a file share, the protocol implementations have their own issues... NFS, CIFS, etc... Not trying to dog all the hardworking free software devs out there but clearly much work remains to be done in many areas, to make Linux really ready to play in the "big big leagues" of computing (unless you have a local staff of good systems programmers with some free time on their hands...). XFS is just one piece of the puzzle we have to work with in trying to integrate a Linux system as a good high-throughput storage machine.
If there is any way that you can use simple catenation or some kind of archiver... even things as simple as shar, tar, zip... to get the file sizes up and the absolute number of files down, you should notice some big performance gains when trying to process your 20 TB worth of stuff.
If you can't dramatically increase individual file size while dramatically reducing the absolute number of files for whatever reason in your environment, I think you can still win by trying to reduce the number of files in any one directory. You want to look out for directories that have five or six figures worth of files in them, those can be real performance killers. If your claim of no more than 5,000 files per any directory is accurate, that shouldn't be a big deal for XFS at all, I don't think you're in bad shape there.
Rsync can be just the worst in this kind of scenario. It runs so slow, you feel sometimes like you might as well be on 10 Meg Ethernet (or worse).
I'm not sure exactly what your application is here... It sounds backup related. If you're doing rsync, you can win a little bit by dropping down a level or two in your directory hierarchy from the top of the tree where XFS is mounted, and running a number of rsync threads in parallel, per-directory, instead of just one top-level rsync thread for an entire filesystem. Experiment to find the best number of threads; run too many and they can deadlock, or just step all over one another.
Also, I have a suspicion (sorry can't back this up quantitatively) that if you are just trying to a straight copy from here to there, a 'cp -Rp' will be faster than an rsync. You might be better off doing an initial copy with 'cp -Rp' and then just synchronizing diffs at the end with an rsync pass, rather than trying to do the whole thing with rsync.
Hope some of this might help... just casual thoughts from a daily XFS-wrangler ;)
On Fri, Sep 5, 2014 at 5:24 PM, Brian Foster <bfoster@xxxxxxxxxx> wrote:
|<Prev in Thread]||Current Thread||[Next in Thread>|
|Previous by Date:||Re: Is XFS suitable for 350 million files on 20TB storage?, Brian Foster|
|Next by Date:||Re: Is XFS suitable for 350 million files on 20TB storage?, Dave Chinner|
|Previous by Thread:||Re: Is XFS suitable for 350 million files on 20TB storage?, Brian Foster|
|Next by Thread:||Re: Is XFS suitable for 350 million files on 20TB storage?, Dave Chinner|
|Indexes:||[Date] [Thread] [Top] [All Lists]|