Re: XFS and RAID

[Sidik Isani <lksi@cfht.hawaii.edu>]

On Fri, May 24, 2002 at 10:23:22AM +0200, Seth Mos wrote:
> [...]
> Software raid5 will give you a much better performance. And if you make it 
> with an external log onto the raid1(hardware) device it will perform a few 
> hundred percent better then the hardware raid5 of the controller.

  I don't have a comparison of hardware vs. software raid, but I did
  experiment with various software raid5+XFS configurations.  All tests
  were on a single 400 MHz CPU with 6 13GB IDE (old, slowwww 7 MB/sec
  drives).  This is our oldest RAID... no one would let me do "experiments"
  on our newer >1TB configurations :-(  We have been running the 2.4.16
  kernel with XFS.

  o 2.4.18+XFS-1.1 shows significant performance improvements,
    especially in copying small files (30% faster) and removing
    lots of files (300% faster.)  This is with an internal log.

  o Switching to an external log (as raid1 on partitions of two of the
    same six disks that have the raid5) boosted small-file performance
    by another 80% or so, even compared to an internal log with the
    same 2.4.18 kernel.  So, even with XFS-1.1, an external log really
    helped a lot here.

  o In all tests, sustained writes of very large files (which is
    mostly what we do) stayed around 19 MBytes/second.

  Some charts are below . . .

Be seeing you,

- Sidik

LEGEND for left column:
kversion[raid5chunksize,raid5algorithm,internal/external log]

	UNTAR TREE, INCLUDING `sync' AFTER
		(Megabytes per second)
		0	0.5	1.0	1.5	2.0
2.4.18[32,0,e]	*******************************>2.0
2.4.18[32,2,e]	*******************************>2.0
2.4.16[32,2,e]	******************************>1.9
2.4.18[64,0,e]	******************************>1.9
2.4.18[128,0e]	*******************************>2.0
2.4.16[128,0e]	******************************>1.9
2.4.18[32,0,i]	********************>1.2
2.4.18[32,2,i]	********************>1.2
2.4.16[32,2,i]	********************>1.2
2.4.16[128,0i]	**********************>1.4

	COPY CACHED TREE, INCLUDING `sync' AFTER
		(Megabytes per second)
		0	0.5	1.0	1.5	2.0	2.5
2.4.18[32,0,e]	**************************************>2.4
2.4.18[32,2,e]	***************************************>2.5
2.4.16[32,2,e]	*************************************>2.3
2.4.18[64,0,e]	******************************************>2.8
2.4.18[128,0e]	*****************************************>2.7
2.4.16[128,0e]	****************************************>2.6
2.4.18[32,0,i]	**********************>1.4
2.4.18[32,2,i]	*************************>1.6
2.4.16[32,2,i]	****************>1.1
2.4.16[128,0i]	***************>1.0

	REMOVE TREE
		(Files removed per second)
		0	250	500	750	1000
2.4.18[32,0,e]	*********************************>1050
2.4.18[32,2,e]	**********************************>1150
2.4.16[32,2,e]	********************>600
2.4.18[64,0,e]	**********************************>1150
2.4.18[128,0e]	**********************************>1150
2.4.16[128,0e]	******************>550
2.4.18[32,0,i]	***************>400
2.4.18[32,2,i]	***********>300
2.4.16[32,2,i]	****>100
2.4.16[128,0i]	***>90

	SUSTAINED LARGE FILE WRITE
		(Megabytes per second)
		0	5	10	15	20
2.4.18[32,0,e]	*********************************>20
2.4.18[32,2,e]	********************************>19
2.4.16[32,2,e]	********************************>19
2.4.18[64,0,e]	*******************************>18
2.4.18[128,0e]	******************************>17
2.4.16[128,0e]	******************************>17
2.4.18[32,0,i]	*******************************>18
2.4.18[32,2,i]	********************************>19
2.4.16[32,2,i]	*******************************>18
2.4.16[128,0i]	******************************>17

  (I apologize for not using bonnie or something standard ... I
   untarred linux-2.4.18.tgz, sync'd, divided the time into 150MB,
   cp -a linux linux2 + sync, divided into 150MB, and time rm -r
   linux divided into the number of files in the source tree.)