Re: Playing around with NFS+XFS

[Seth Mos <knuffie@xxxxxxxxx>]

At 17:00 31-8-2001 +0800, you wrote:
> On Thu, 30 Aug 2001 at 22:37, Seth Mos wrote:
> > I use 16384 for 100Mbit at work which seems to be a decent size vs
> > reponse ratio.
>
> I'm too lazy to do yet more tests, so I'm taking your (Seth's) word for
> it. << changing /etc/fstab >> ;>

:)

> > I have a few standard bonnie results of linux -> linux tests on my
> > homepage http://iserv.nl/
>
> I just checked it out and it's _significantly_ better than my results, and
> my setup is supposed to pack more punch than yours in all aspects except
> tuning (where I'm slowly learning thanks to you).
>
> Hmm ...

It's just a matter of time.

> > Server was a pIII 450 with 256MB of ram and a 2c905B NIC and a 40GB
> > IDE disk in UDMA33 mode.
>
> Mine is a Pentium III 733MHz with 512MB RAM, a TLAN 10/100MBps NIC running
> 100Mbps full-duplex, and 4 x 36GB UDMA/66 hard drives in hardware RAID5.

Your write speeds will be lower but not more then a single disk if the card 
you are using is beefy enough. Raid5 involves a lot more overhead during 
writes, we once noted a windows NT sytem going faster when one of the 
harddisks in the raid5 failed. After that we converted it to raid10.

> This is starting to get a little off-topic in that it's not XFS-specific
> anymore. I hope everyone else will pardon it:
>
> > I used 8 nfsd processes
>
> Just a note, Debian's nfs-kernel-server package has this as the default
> via the following in /etc/init.d/nfs-kernel-server:
>
> RPCNFSDCOUNT=8

This is the redhat default as well as specified in /etc/rc.d/init.d/nfs

> > and enlarged the buffer size from 64KB to 256KB.
> > echo 262144 > /proc/sys/net/core/rmem_default
> > echo 262144 > /proc/sys/net/core/rmem_max

Note: The amount of buffers are shared between the nfsd processes, this 
will results in 8KB buffer per deamon. If you run 16 deamons this will mean 
you get 4KB per process.

Change acordingly.

> I read somewhere that this can lead to some not-so-nice situations when
> left like this instead of the default. Would you be able to qualify this?

This is noted from the NFS HowTo but they do not state what adverse results 
and under what version of the linux kernel. 2.4.9 might have this fixed but 
I just don't know. I have not run into funny behaviour yet and I am 
thinking of changing this option on the internet server gateway as well 
since that one has 3 networkcards and simultaneous activity on all.

> I do remember reading somewhere that the recommended action will be to set
> buffer sizes to 256KB, then start the NFS server, then revert to 64KB. I
> was wondering if you'd have any information on that.

That is what they say yes, I am just taking that risk. I have not 
experienced anomelies in the other sytems yet. I can imagine that if you 
are routing or packetfiltering this might lead to problems.

> Now for something that I _think_ is a little more on-topic (but still not
> quite XFS specific):
>
> I tweaked my /proc/sys/vm/bdflush a little, bringing down age_buffer to
> 100 jiffies (1 second) from 3000 jiffies (30 seconds). I did this in the
> hopes that idle systems would flush dirty buffers sooner. I left
> everything at their defaults, which are:
>
> nfract = 30
> ndirty = 64
> nrefill = 64
> nref_dirt = 256
> age_super = 60
>
>  o Did my bringing down age_buffer to 1 second make things too synchronous
>    that writes were significantly affected?
>
>  o Will bringing up nfract allow the system to perform better? Or will
>    this simply chunk up disk writes and make the system less responsive
>    during those buffer flushes?
>
>  o How will increasing or decreasing ndirty affect disk-intensive
>    operations like database updates or massive file transfers?
>
> Hopefully those (like Seth Mos, hehehe) with much more administrative
> experience than I can help out and recommend tuning parameters? I thank
> Seth for the NFS tuning tips. Maybe if he has some hidden bdflush (or even
> buffermem) secrets, he can share them with the list? ;>

Nope, to deep for me. Never changed any of those defaults.

Cheers

--
Seth
Every program has two purposes one for which
it was written and another for which it wasn't
I use the last kind.