> I'm dubious that you can get 600MB/s on a single inexpensive card
> right now. I would try it on a raw partition and see how it performs
> for you to get some idea of the upper limit.
The 8port card seems to do fairly well but itremains to be seen
if the 16port card can perform. The claim it can but not many people
have used the card yet.
> This is hardware RAID right? What level? RAID-5 is going to be quit
> a bit slower.
Raid 0 at first and then hopefully raid 5 or 6 (dual parity)
at some point with more drives/cards.
> FWIW on lesser hardware I can write 400MB/s across two controllers
> (each doing RAID-5, I assuming RAID-0 would be faster).
Which cards are you using?
> Can you actually sink 600MB/s over IB and also push that out over the
> bus to the controller?
Mellanox claims they can get over 500MB/s with their SRP driver.
500MB/s if fine for me but I'd like to have the additional disk
bandwidth.
> It's not been overly slow or problematic for me. What problems are
> you seeing?
I hven't tested that fuly to know for sure yet.
> 'man xfsctl' for details on preallocation. AFAIK it's the same basic
> interface as what IRIX has used for many years.
Thanks.
> I assume this is RAID-0 then? Are you sure 128K RAID chunks are
> optimal? When I was testing I was suprised to find that values over
> 32K actually were slower than smaller values.
I assumed 128 would be better but I'll try 32 and 64k next.
> RAID-5 & 6 are going to be a *lot* slower for you I suspect (I doubt
> the card you are using has enough bandwidth or CPU to deal with the
> speed you are after with RAID 5 or 6 --- what doe the specs claim?)
The specs are vague but the bandwidth tests on areca.us for the
8port card seems to do ok. I might actually be better off with
two of the 8 port cards instead.
> 16MB files are pretty small for this sort of IO rate. In fact, if
> they are all 16MBis in size --- why use a filesystem at all?
Agreed, although I need a filesystem for a number of other applications
to access the data easily.
>
> Some experimentation would probably be needed. I've found a small
> number of writing threads is faster than one but a larger number has
> no gain or is slower.
>
> It would be nice to try AIO + DIO but presently that doesn't work and
> I've not really had a chance to revisit fixing that (since apparently
> it will ge done eventually anyhow).
Thanks for the advice!
|