allocsize mount option
Gim Leong Chin
chingimleong at yahoo.com.sg
Wed Jan 13 03:42:16 CST 2010
Hi,
The application is ANSYS, which writes 128 GB files. The existing computer with SUSE Linux Enterprise Desktop 11 which is used for running ANSYS, has two software RAID 0 devices made up of five 1 TB drives. The /home partition is 4.5 T, and it is now 4 TB full. I see a fragmentation > 19%.
I have just set up a new computer with 16 WD Cavair Black 1 TB drives connected to an Areca 1680ix-16 RAID with 4 GB cache. 14 of these drives are in RAID 6 with 128 kB stripes. The OS is also SLED 11. The system has 16 GB memory, and AMD Phenom II X4 965 CPU.
I have done tests writing 100 30 MB files and 1 GB, 10 GB and 20 GB files, with single instance and multiple instances.
There is a big difference in writing speed when writing 20 GB files when using allocsize=1g and not using the option. That is without the inode64 option, which gives further speed gains.
I use dd for writing the 1 GB, 10 GB and 20 GB files.
mkfs.xfs -f -b size=4k -d agcount=32,su=128k,sw=12 -i size=256,align=1,attr=2 -l version=2,su=128k,lazy-count=1 -n version=2 -s size=512 -L /data /dev/sdb1
defaults,nobarrier,usrquota,grpquota,noatime,nodiratime,allocsize=1g,logbufs=8,logbsize=256k,largeio,swalloc
The start of the partition has been set to LBA 3072 using GPT Fdisk to align the stripes.
The dd command is:
chingl at tsunami:/data/test/t2> dd if=/dev/zero of=bigfile20GB bs=1073741824 count=20
Single instance of 20 GB dd repeats were 214, 221, 123 MB/s with allocsize=1g, compared to 94, 126 MB/s without.
Two instances of 20 GB dd repeats were aggregate 331, 372 MB/s with allocsize=1g, compared to 336, 296 MB/s without.
Three instances of 20 GB dd was aggregate 400 MB/s with, 326 MB/s without.
Six instances of 20 GB dd was 606 MB/s with, 473 MB/s without.
My production configuration is
defaults,nobarrier,usrquota,grpquota,noatime,nodiratime,allocsize=1g,logbufs=8,logbsize=256k,largeio,swalloc,inode64
for which I got up to 297 MB/s for single instance 20 GB dd.
Chin Gim Leong
--- On Tue, 12/1/10, Eric Sandeen <sandeen at sandeen.net> wrote:
> From: Eric Sandeen <sandeen at sandeen.net>
> Subject: Re: allocsize mount option
> To: "Gim Leong Chin" <chingimleong at yahoo.com.sg>
> Cc: xfs at oss.sgi.com
> Date: Tuesday, 12 January, 2010, 2:16 AM
> Gim Leong Chin wrote:
> > Hi,
> >
> > Mount options for xfs allocsize=size Sets the
> buffered I/O
> > end-of-file preallocation size when doing delayed
> allocation writeout
> > (default size is 64KiB).
> >
> >
> > I read that setting allocsize to a big value can be
> used to combat
> > filesystem fragmentation when writing big files.
>
> That's not universally necessary though, depending on how
> you are
> writing them. I've only used it in the very specific
> case of mythtv
> calling "sync" every couple seconds, and defeating
> delalloc.
>
> > I do not understand how allocsize works. Say I
> set allocsize=1g, but
> > my file size is only 1 MB or even smaller. Will
> the rest of the 1 GB
> > file extent be allocated, resulting in wasted space
> and even file
> > fragmentation problem?
>
> possibly :) It's only speculatively allocated,
> though, so you won't
> have 1g for every file; when it's closed the preallocation
> goes
> away, IIRC.
>
> > Does setting allocsize to a big value result in
> performance gain when
> > writing big files? Is performance hurt by a big
> value setting when
> > writing files smaller than the allocsize value?
> >
> > I am setting up a system for HPC, where two different
> applications
> > have different file size characteristics, one writes
> files of GBs and
> > even 128 GB, the other is in MBs to tens of MBs.
>
> We should probably back up and say: are you seeing
> fragmentation
> problems -without- the mount option, and if so, what is
> your write pattern?
>
> -Eric
>
> > I am not able to find documentation on the behaviour
> of allocsize
> > mount option.
> >
> > Thank you.
> >
> >
> > Chin Gim Leong
> >
> >
> > New Email names for you! Get the Email name you've
> always wanted
> > on the new @ymail and @rocketmail. Hurry before
> someone else does!
> > http://mail.promotions.yahoo.com/newdomains/sg/
> >
> > _______________________________________________ xfs
> mailing list
> > xfs at oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs
> >
>
> _______________________________________________
> xfs mailing list
> xfs at oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
>
New Email addresses available on Yahoo!
Get the Email name you've always wanted on the new @ymail and @rocketmail.
Hurry before someone else does!
http://mail.promotions.yahoo.com/newdomains/sg/
More information about the xfs
mailing list