Ok, I had some time over the weekend to put some spare disks in the
machine and see what the effect of different mkfs options would be with
bonnie. This is all on a 700MHz Athlon, 512Mb RAM, a Mylex DAC960PD with
32Mb Cache, and the 3 18Gb IBM DPSS are on a separate channel each.
First, hardware set to 64k stripe, 8k segment, write-through caching;
mkfs.xfs with sunit=2,swidth=16:
-------Sequential Output-------- ---Sequential Input--
--Random--
-Per Char- --Block--- -Rewrite-- -Per Char- --Block---
--Seeks---
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU
/sec %CPU
sunit=2, 4096 4419 7.9 4420 2.3 2808 2.3 8292 19.9 9915 5.7
177.8 1.5
mkfs.xfs with sunit=16,swidth=128:
-------Sequential Output-------- ---Sequential Input--
--Random--
-Per Char- --Block--- -Rewrite-- -Per Char- --Block---
--Seeks---
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU
/sec %CPU
sunit=16 4096 4431 7.6 4439 2.2 2837 2.2 8593 20.2 10280 5.8
190.2 1.4
So there's really no difference to speak of in real terms... Next, set
the hardware to 64k stripe, 64k segment, write-through caching;
mkfs.xfs with sunit=16,swidth=128:
-------Sequential Output-------- ---Sequential Input--
--Random--
-Per Char- --Block--- -Rewrite-- -Per Char- --Block---
--Seeks---
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU
/sec %CPU
sunit=16 4096 5014 8.7 5013 2.5 3408 2.6 11876 28.3 14653 8.2
190.6 1.3
And now that last setup (64k stripe,64k segment, sunit=16,swidth=128)
with write-back caching enabled:
-------Sequential Output-------- ---Sequential Input--
--Random--
-Per Char- --Block--- -Rewrite-- -Per Char- --Block---
--Seeks---
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU
/sec %CPU
u16w48s6 4096 6153 10.8 6165 3.1 3968 3.0 12149 29.0 14533 8.6
165.5 1.2
So the 64k segment seems to really help (in Bonnie at least). To a
lesser degree switching on write-back also helps, so in case there's a
UPS sitting in the same rack (or a battery backup on the DAC960) it's
probably a good idea to use it. However, I was surprised to see how
relatively little the performance impact of different mkfs options were
assuming the same hardware parameters.
Of course, this is nowhere near a scientific study. It's also
interesting to see how badly the DAC does in comparison to an ata md
raid 5, even though there's only one disk per scsi channel in active
use. Oh and _do_ remember this is mostly focussing on sequential i/o!
Hope this is interesting to anyone,
Kai.
> -----Original Message-----
> From: Murthy Kambhampaty [mailto:murthy.kambhampaty@xxxxxxxxx]
> Sent: 03 February 2003 21:45
> To: Murthy Kambhampaty; 'Kai Leibrandt'; 'linux-xfs@xxxxxxxxxxx'
> Subject: RE: Another mkfs.xfs RAID Question
>
>
> Try sunit=8k, swidth=64k, for the Mylex raid controller with
> default settings
> (http://oss.sgi.com/projects/xfs/mail_archive/200205/msg00152.html)
>
> Note: With recent firmware, both segment size (8k, 64k) and
> stripe size (up to 1024k in several steps) are user
> selectable. However, we had problems with a 64k SEGMENT size,
> so we now just use 8k segment size and a stripe size of 64k.
> Also, for the postgresql data volume, we use a 64m external
> v2 log on md1 on a separate controller, and mount with
> logbufs=8,logbsize=256k (the server -- Supermicro Super8050
> -- has 2+1 hot pluggable power supplies, runs on a ups with
> apcusd to shut the machine down on power failure, and is
> backed up every night; if that motherboard fails we will
> likely be doing bare-metal recovery given that it is a busy
> server and the aggressive log settings). It was difficult to
> test the effects of tweaking the segment size and stripe size
> (if you are able to set up Mylex GAM server, you might find
> it easy enough); the log settings gave the lowest `time dd
> if=/dev/zero of=/mnt/testdisk/testfile bs=4096 count=1310720`
> in repeated testing with unmount/mount between reps; the
> mount options are at their maximums, increasing the logdev's
> size to 128m (which is the max, IIRC) gave a small decline
> rather than an improvement. (I'll admit the testing wasn't
> too formal, but we didn't think the differences would be so
> dramatic as to formalize the "protocol" and record the results.)
>
> PS: sorry if you see multiple messages; I seem to have been
> having problems posting
>
>
> -----Original Message-----
> From: Kai Leibrandt [mailto:k_leibrandt@xxxxxxxxxxx]
> Sent: Saturday, February 01, 2003 07:53
> To: linux-xfs@xxxxxxxxxxx
> Subject: Another mkfs.xfs RAID Question
>
>
> Hi all,
>
> Dragos' question about the su,sw options led me to have
> another look at my Mylex RAID setup, and I now am confused
> too... My RAID controller tells me that the primary RAID
> volume (a 3-disk RAID
> 5) has a Stripe Size of 64k and a Segment Size of 8k. So for
> this setup should the sunit be set to 64k or to 8k (the
> segment size)??? The manual fo the DAC says that the segment
> size is the preferred io size for caching, but xfs has a
> preferred io size equal to the stripe size - which to choose?
>
> Many thanks,
>
> Kai.
>
|