On Sun, Apr 17, 2005 at 04:05:21PM -0700, delusion@xxxxxxxxxxxx wrote:
> The card is a PCI-X 133 although there is also a PCI-E 8x card. The
> PCI-E card uses a PCI-X 133 bus on the card so there shouldn't be
> much of a difference. The card supposedly can do the speeds but I
> haven't been able to get
I'm dubious that you can get 600MB/s on a single inexpensive card
right now. I would try it on a raw partition and see how it performs
for you to get some idea of the upper limit.
This is hardware RAID right? What level? RAID-5 is going to be quit
a bit slower.
> 14 x WD raptors (73GB), 3.2GHz dual EMT64, 2GB, Supermicro
> motherboard.
FWIW on lesser hardware I can write 400MB/s across two controllers
(each doing RAID-5, I assuming RAID-0 would be faster).
> This is essentially a box that needs to backup data coming in over
> infiniband. The data will be written in 16mb files.
Can you actually sink 600MB/s over IB and also push that out over the
bus to the controller?
> Thanks. Is opening files on the fly till a bit slow as it used to be
> on IRIX?
It's not been overly slow or problematic for me. What problems are
you seeing?
> Sounds like preallocating and opening is the best way to go then? Is
> there a fast way to preallocate in XFS (my xfs experience dates back
> a few years on irix and I need to catch up on what's been done since
> then).
'man xfsctl' for details on preallocation. AFAIK it's the same basic
interface as what IRIX has used for many years.
> I did try sunit=256 and swidth=256*14 (I'm not testing raid 5/6 yet)
> but it didn't seem to help much. I also tried setting both to zero
> but it still didn't help.
I assume this is RAID-0 then? Are you sure 128K RAID chunks are
optimal? When I was testing I was suprised to find that values over
32K actually were slower than smaller values.
RAID-5 & 6 are going to be a *lot* slower for you I suspect (I doubt
the card you are using has enough bandwidth or CPU to deal with the
speed you are after with RAID 5 or 6 --- what doe the specs claim?)
> It's a fairly straightforward case... 16MB files all written
> sequentially.
16MB files are pretty small for this sort of IO rate. In fact, if
they are all 16MBis in size --- why use a filesystem at all?
> In this case, would it be better to use one thread for writing or
> multiple threads?
Some experimentation would probably be needed. I've found a small
number of writing threads is faster than one but a larger number has
no gain or is slower.
It would be nice to try AIO + DIO but presently that doesn't work and
I've not really had a chance to revisit fixing that (since apparently
it will ge done eventually anyhow).
|