On Wednesday 05 January 2005 10:30 am, Robert Olsson wrote:
> Jeremy M. Guthrie writes:
> > Yeah, the load will be high. I'm expecting this to be watching ~ 750
> > mbps by next December. The app profiles all traffic going in and out of
> > our data centers.
> BW itself or pps is that not much of challange as handling of concurrent
> flows.
Roger that.
> > I'm not showing the /proc/net/rt_cache_stat file. Was there a kernel
> > option I need to recompile with for rt_cache_stat to show up in proc?
>
> No it's there without any options. Would be nice to the output from rtstat
Output from rtstat:
rtstat
fopen: No such file or directory
cd /proc/net/
ls -la
total 0
dr-xr-xr-x 5 root root 0 2005-01-04 08:53 .
dr-xr-xr-x 72 root root 0 2005-01-04 08:53 ..
-r--r--r-- 1 root root 0 2005-01-05 10:32 anycast6
-r--r--r-- 1 root root 0 2005-01-05 10:32 arp
dr-xr-xr-x 2 root root 0 2005-01-05 10:32 atm
-r--r--r-- 1 root root 0 2005-01-05 10:32 dev
-r--r--r-- 1 root root 0 2005-01-05 10:32 dev_mcast
dr-xr-xr-x 2 root root 0 2005-01-05 10:32 dev_snmp6
-r--r--r-- 1 root root 0 2005-01-04 08:53 if_inet6
-r--r--r-- 1 root root 0 2005-01-05 10:32 igmp
-r--r--r-- 1 root root 0 2005-01-05 10:32 igmp6
-r--r--r-- 1 root root 0 2005-01-05 10:32 ip6_flowlabel
-r--r--r-- 1 root root 0 2005-01-05 10:32 ip_mr_cache
-r--r--r-- 1 root root 0 2005-01-05 10:32 ip_mr_vif
-r--r--r-- 1 root root 0 2005-01-05 10:32 ip_tables_matches
-r--r--r-- 1 root root 0 2005-01-05 10:32 ip_tables_names
-r--r--r-- 1 root root 0 2005-01-05 10:32 ip_tables_targets
-r--r--r-- 1 root root 0 2005-01-05 10:32 ipv6_route
-r--r--r-- 1 root root 0 2005-01-05 10:32 mcfilter
-r--r--r-- 1 root root 0 2005-01-05 10:32 mcfilter6
-r--r--r-- 1 root root 0 2005-01-05 10:32 netlink
-r--r--r-- 1 root root 0 2005-01-05 10:32 netstat
-r--r--r-- 1 root root 0 2005-01-05 10:32 psched
-r--r--r-- 1 root root 0 2005-01-05 10:32 raw
-r--r--r-- 1 root root 0 2005-01-05 10:32 raw6
-r--r--r-- 1 root root 0 2005-01-05 10:32 route
dr-xr-xr-x 6 root root 0 2005-01-05 10:32 rpc
-r--r--r-- 1 root root 0 2005-01-05 10:32 rt6_stats
-r--r--r-- 1 root root 0 2005-01-05 10:32 rt_acct
-r--r--r-- 1 root root 0 2005-01-05 10:32 rt_cache
-r--r--r-- 1 root root 0 2005-01-05 10:32 snmp
-r--r--r-- 1 root root 0 2005-01-05 10:32 snmp6
-r--r--r-- 1 root root 0 2005-01-05 10:32 sockstat
-r--r--r-- 1 root root 0 2005-01-05 10:32 sockstat6
-r--r--r-- 1 root root 0 2005-01-05 10:32 softnet_stat
dr-xr-xr-x 2 root root 0 2005-01-05 10:32 stat
-r--r--r-- 1 root root 0 2005-01-04 08:53 tcp
-r--r--r-- 1 root root 0 2005-01-05 10:32 tcp6
-r--r--r-- 1 root root 0 2005-01-05 10:32 tr_rif
-r--r--r-- 1 root root 0 2005-01-04 08:53 udp
-r--r--r-- 1 root root 0 2005-01-05 10:32 udp6
-r--r--r-- 1 root root 0 2005-01-05 10:32 unix
-r--r--r-- 1 root root 0 2005-01-05 10:32 wireless
> > > Also check that the CPU shares the RX packet load. CPU0 affinty to
> > > eth0 and CPU1 to eth1 seems to be best. It gives cache bouncing at
> > > "TX" and slab jobs but we have accept that for now.
> >
> > How would I go about doing this?
>
> Assume you route packets between eth0 <-> eth1
Yup
> Set eth0 irq to CPU0 and eth1 to CPU1 with /proc/irq/XX/smp_affinity
Done
> Disable irqbalancer etc.
Done
I'll let you know what I see for stats once I get some collected.
> > cat /proc/net/softnet_stat
>
> total droppped tsquz Throttl FR_hit FR_succe FR_defer FR_def_o
> cpu_coll
>
> > 5592c972 00000000 00001fc8 00000000 00000000 00000000 00000000 00000000
> > 00391c3f
> > 000f1991 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> > 001292ba
>
> See! One line per CPU. So CPU0 is handing almost all packets.
cat /proc/interrupts
CPU0 CPU1
0: 674862 93484967 IO-APIC-edge timer
1: 564 9 IO-APIC-edge i8042
7: 0 0 IO-APIC-level ohci_hcd
8: 1 1 IO-APIC-edge rtc
12: 268 62 IO-APIC-edge i8042
14: 2 0 IO-APIC-edge ide0
18: 2105131410 9140835 IO-APIC-level eth3
20: 1077 248075156 IO-APIC-level eth2
27: 118224 1 IO-APIC-level eth0
28: 36298 49 IO-APIC-level aic7xxx
30: 0 0 IO-APIC-level acpi
NMI: 0 0
LOC: 94168097 94168094
ERR: 0
MIS: 0
> > "%soft"
> > Show the percentage of time spent by the CPU or CPUs to service
> > softirqs. A softirq (software interrupt) is one of up to 32
> > enumerated software interrupts which can run on multiple CPUs
>
> Well yes. I had a more specific question. I'll look into mpstat where do
> find it? Kernel pacthes?
Sorry about that. New to the list. I'm not suggesting anything. I
appreciate the help!
Suse listed mpstat as part of sysstat 5.1.2. I'm running stock 2.6.10.
> Be also aware that packet forwarding with SMP/NUMA is very much research
> today it is not that easy or not even possible to get aggregated
> performance from several CPU's. in any setup. Well anyway we are beginning
> to see some benefits now as we better understand the problems.
Understood. As long as I know this, I can articulate this to my uppers for
bigger hardware. My current system is a dual P-III 700mhz. May be time for
an upgrade. However, I figure this may also offer a good environment to help
provide you guys with a taxed system running an load of flows. Nothing like
finding fun stuff while a system is ready to fall over.
Would a single hyper threaded CPU help this or should I default to a normal
dual-cpu system?
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@xxxxxxxxxx
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
pgpsh1CqAlnPh.pgp
Description: PGP signature
|