Hi,
The application is ANSYS, which writes 128 GB files. The existing computer
with SUSE Linux Enterprise Desktop 11 which is used for running ANSYS, has two
software RAID 0 devices made up of five 1 TB drives. The /home partition is
4.5 T, and it is now 4 TB full. I see a fragmentation > 19%.
I have just set up a new computer with 16 WD Cavair Black 1 TB drives connected
to an Areca 1680ix-16 RAID with 4 GB cache. 14 of these drives are in RAID 6
with 128 kB stripes. The OS is also SLED 11. The system has 16 GB memory, and
AMD Phenom II X4 965 CPU.
I have done tests writing 100 30 MB files and 1 GB, 10 GB and 20 GB files, with
single instance and multiple instances.
There is a big difference in writing speed when writing 20 GB files when using
allocsize=1g and not using the option. That is without the inode64 option,
which gives further speed gains.
I use dd for writing the 1 GB, 10 GB and 20 GB files.
mkfs.xfs -f -b size=4k -d agcount=32,su=128k,sw=12 -i size=256,align=1,attr=2
-l version=2,su=128k,lazy-count=1 -n version=2 -s size=512 -L /data /dev/sdb1
defaults,nobarrier,usrquota,grpquota,noatime,nodiratime,allocsize=1g,logbufs=8,logbsize=256k,largeio,swalloc
The start of the partition has been set to LBA 3072 using GPT Fdisk to align
the stripes.
The dd command is:
chingl@tsunami:/data/test/t2> dd if=/dev/zero of=bigfile20GB bs=1073741824
count=20
Single instance of 20 GB dd repeats were 214, 221, 123 MB/s with allocsize=1g,
compared to 94, 126 MB/s without.
Two instances of 20 GB dd repeats were aggregate 331, 372 MB/s with
allocsize=1g, compared to 336, 296 MB/s without.
Three instances of 20 GB dd was aggregate 400 MB/s with, 326 MB/s without.
Six instances of 20 GB dd was 606 MB/s with, 473 MB/s without.
My production configuration is
defaults,nobarrier,usrquota,grpquota,noatime,nodiratime,allocsize=1g,logbufs=8,logbsize=256k,largeio,swalloc,inode64
for which I got up to 297 MB/s for single instance 20 GB dd.
Chin Gim Leong
--- On Tue, 12/1/10, Eric Sandeen <sandeen@xxxxxxxxxxx> wrote:
> From: Eric Sandeen <sandeen@xxxxxxxxxxx>
> Subject: Re: allocsize mount option
> To: "Gim Leong Chin" <chingimleong@xxxxxxxxxxxx>
> Cc: xfs@xxxxxxxxxxx
> Date: Tuesday, 12 January, 2010, 2:16 AM
> Gim Leong Chin wrote:
> > Hi,
> >
> > Mount options for xfs allocsize=size Sets the
> buffered I/O
> > end-of-file preallocation size when doing delayed
> allocation writeout
> > (default size is 64KiB).
> >
> >
> > I read that setting allocsize to a big value can be
> used to combat
> > filesystem fragmentation when writing big files.
>
> That's not universally necessary though, depending on how
> you are
> writing them. I've only used it in the very specific
> case of mythtv
> calling "sync" every couple seconds, and defeating
> delalloc.
>
> > I do not understand how allocsize works. Say I
> set allocsize=1g, but
> > my file size is only 1 MB or even smaller. Will
> the rest of the 1 GB
> > file extent be allocated, resulting in wasted space
> and even file
> > fragmentation problem?
>
> possibly :) It's only speculatively allocated,
> though, so you won't
> have 1g for every file; when it's closed the preallocation
> goes
> away, IIRC.
>
> > Does setting allocsize to a big value result in
> performance gain when
> > writing big files? Is performance hurt by a big
> value setting when
> > writing files smaller than the allocsize value?
> >
> > I am setting up a system for HPC, where two different
> applications
> > have different file size characteristics, one writes
> files of GBs and
> > even 128 GB, the other is in MBs to tens of MBs.
>
> We should probably back up and say: are you seeing
> fragmentation
> problems -without- the mount option, and if so, what is
> your write pattern?
>
> -Eric
>
> > I am not able to find documentation on the behaviour
> of allocsize
> > mount option.
> >
> > Thank you.
> >
> >
> > Chin Gim Leong
> >
> >
> > New Email names for you! Get the Email name you've
> always wanted
> > on the new @ymail and @rocketmail. Hurry before
> someone else does!
> > http://mail.promotions.yahoo.com/newdomains/sg/
> >
> > _______________________________________________ xfs
> mailing list
> > xfs@xxxxxxxxxxx http://oss.sgi.com/mailman/listinfo/xfs
> >
>
> _______________________________________________
> xfs mailing list
> xfs@xxxxxxxxxxx
> http://oss.sgi.com/mailman/listinfo/xfs
>
New Email addresses available on Yahoo!
Get the Email name you've always wanted on the new @ymail and @rocketmail.
Hurry before someone else does!
http://mail.promotions.yahoo.com/newdomains/sg/
|