From owner-lockmeter@oss.sgi.com Tue Nov 2 12:23:56 1999 Date: Tue, 2 Nov 1999 21:27:53 +0100 (MET) From: Anonymous To: lockmeter@oss.sgi.com Subject: Please publish more details! MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-lockmeter@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;lockmeter-outgoing Dear Madam or Sir, lockmetering seems to be a useful indicator for lock contention, which is one of several interesting indicators for the overall performance (and the _growing_ performance) of Linux. I'm very interested in more detailed information on Linux speed but don't (yet) have an SMP box on which I could let the lockmetering patches run. So I'd like to read some more detailed results on your web pages (or somewhere else, but I couldn't find anything), especially for the current development kernels. - The latest kernel mentioned on "http://oss.sgi.com/projects/lockmeter/faq.html" is 2.3.11, which is quite old... ;-) Meanwhile lots of improvements have been made in the kernel. Best regards (and thanks for your lockmetering patches!) Anonymous From owner-lockmeter@oss.sgi.com Wed Nov 3 09:46:38 1999 Received: by oss.sgi.com id ; Wed, 3 Nov 1999 09:46:29 -0800 Received: from deliverator.sgi.com ([204.94.214.10]:28768 "EHLO deliverator.sgi.com") by oss.sgi.com with ESMTP id ; Wed, 3 Nov 1999 09:46:09 -0800 Received: from cthulhu.engr.sgi.com (cthulhu.engr.sgi.com [192.26.80.2]) by deliverator.sgi.com (980309.SGI.8.8.8-aspam-6.2/980310.SGI-aspam) via ESMTP id JAA07169 for ; Wed, 3 Nov 1999 09:46:40 -0800 (PST) mail_from (hawkes@cthulhu.engr.sgi.com) Received: from ppphawkeswa ([169.238.112.29]) by cthulhu.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via SMTP id JAA07096 for ; Wed, 3 Nov 1999 09:50:27 -0800 (PST) mail_from (hawkes@engr.sgi.com) Message-ID: <013501bf2624$0cae7060$1d70eea9@seattle.sgi.com> From: "John Hawkes" To: References: Subject: Re: Please publish more details! Date: Wed, 3 Nov 1999 09:51:20 -0800 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2615.200 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 Sender: owner-lockmeter@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;lockmeter-outgoing From: Anonymous ...[snip]... > I'm very interested in more detailed > information on Linux speed but don't (yet) have an SMP box on which I > could let the lockmetering patches run. You can, of course, build an SMP kernel and run it on a uniprocessor box, but the spinlock contention ought to be quite small for such a system. > So I'd like to read some more detailed results on your web pages (or > somewhere else, but I couldn't find anything), especially for the current > development kernels. - The latest kernel mentioned on > "http://oss.sgi.com/projects/lockmeter/faq.html" is 2.3.11, which is quite > old... ;-) Meanwhile lots of improvements have been made in the kernel. Yes, true. The latest available lockmetering patch is against 2.3.22. For various reasons I've had limited access lately to a 4xCPU SMP system running a more modern 2.3.x kernel, so I haven't been exploring spinlock performance issues. I expect more work will be forthcoming. I have my own 2xCPU system, but a 4xCPU system exhibits more interesting spinlock behavior. > Best regards (and thanks for your lockmetering patches!) Thank you. I appreciate your interest. I wish Linus had seen fit to include lockmetering in 2.3.x, but he chose otherwise, so lockmetering continues to have limited exposure in the Linux community. I intend to continue to provide patches for 2.3.x and 2.4. John Hawkes From owner-lockmeter@oss.sgi.com Wed Nov 10 09:12:31 1999 Received: by oss.sgi.com id ; Wed, 10 Nov 1999 09:12:21 -0800 Received: from mercury.Sun.COM ([192.9.25.1]:32915 "EHLO mercury.Sun.COM") by oss.sgi.com with ESMTP id ; Wed, 10 Nov 1999 09:12:07 -0800 Received: from emuc05-home.Germany.Sun.COM ([129.157.51.10]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id JAA01276 for ; Wed, 10 Nov 1999 09:17:14 -0800 (PST) Received: from sunhsc (sunhsc [129.157.167.92]) by emuc05-home.Germany.Sun.COM (8.8.8+Sun/8.8.8/ENSMAIL,v1.7) with SMTP id SAA13801 for ; Wed, 10 Nov 1999 18:17:13 +0100 (MET) Message-Id: <199911101717.SAA13801@emuc05-home.Germany.Sun.COM> Date: Wed, 10 Nov 1999 18:18:07 +0100 (MET) From: Frank Hofmann - Sun Germany Technical Solution Center - Munich Reply-To: Frank Hofmann - Sun Germany Technical Solution Center - Munich Subject: Some notes on portability of lockmetering code to non-IA32 architectures To: lockmeter@oss.sgi.com MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: q2MS54R6J0usKHOY+NlAPg== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.3.2 SunOS 5.7 sun4u sparc Sender: owner-lockmeter@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;lockmeter-outgoing Hi, have downloaded the lockmeter 1.1.1 code - this is great stuff ! Unfortunately I don't have an Intel-based SMP system to test this on, but I've found that with very few modifications it also compiles on sparc (only tried v8, i.e. 32bit). To accomplish this, apart from adding to the config.in, Makefile, ... stuff I also had to change the following code parts: In lockmeter.c, there are several points which prevent that it compiles on non-IA32: 1. the use of "cpucount". This variable is not an exported symbol resp. a global on sparc. Replacing this by the arch-independent global "smp_num_cpus" works. 2. Same for "cpu_hz". This is only used currently to fill in a field in the lockstat info struct. I don't know an arch-independent way to get this but it's not required for working lockstat and so I've just removed it for the moment. 3. The following assembler code: #define local_irq_save(x) \ __asm__ __volatile__("pushfl ; popl %0 ; cli":"=g" (x): /* no input */ :"memory") #define local_irq_restore(x) \ __asm__ __volatile__("pushl %0 ; popfl": /* no output */ :"g" (x):"memory") looking at the #define's in asm-i386/system.h, one finds that this is identical to: #define local_irq_save(x) __save_and_cli((x)) #define local_irq_restore(x) __restore_flags((x)) This way it also should be more portable. __save_and_cli() is not available on all architectures but it is on IA32 and sparc. 4. The following assembler code: static __inline__ int atomic_incr_and_test_nonneg(volatile atomic_t *v) { unsigned char c; __asm__ __volatile__( LOCK "incl %0; setns %1" :"=m" (__atomic_fool_gcc(v)), "=qm" (c) :"m" (__atomic_fool_gcc(v))); return c != 0; } There's tons of similar code in the includes asm-i386/atomic.h and the corresponding asm-includes for other platforms. I've not yet found a portable way of coding this using the macros from atomic.h but for sparc that reads: #define atomic_incr_and_test_nonneg(x) (atomic_add_return(1,(x)) >= 0) Modifications to other files: - smplock.h: minor, easy, but I don't see a way of getting this completely platform-independent. - spinlock.h: dito. - some mods to the $ARCH_ksyms.c file. This makes sense ;-) I've not yet been able to test lockmetering on my SS10 because my kernel codebase is a bit corrupted and therefore I did not get a working kernel in the few hours I coded last night ;-) This is not a fault of the lockmetering code - there were some code problems with other patches I was using that prevented the kernel from compiling. I will be able to test my sparc-adapted code tonight. If it works, I'm going to create diff's against a stock-2.2.13 source. It should be pretty straightforward to port this to other non-IA32 architectures then. Let's see - where could other arch-dependent pitfalls be located: - lockmeter.h: I'm not sure about the LSTAT_RA macro. - endianness issues: Have not yet been able to check for this. Thanks for the code - it's fun ! Bye Frank Hofmann From owner-lockmeter@oss.sgi.com Fri Nov 12 10:40:38 1999 Received: by oss.sgi.com id ; Fri, 12 Nov 1999 10:40:29 -0800 Received: from pneumatic-tube.sgi.com ([204.94.214.22]:7203 "EHLO pneumatic-tube.sgi.com") by oss.sgi.com with ESMTP id ; Fri, 12 Nov 1999 10:40:16 -0800 Received: from cthulhu.engr.sgi.com (gate3-relay.engr.sgi.com [130.62.1.234]) by pneumatic-tube.sgi.com (980327.SGI.8.8.8-aspam/980310.SGI-aspam) via ESMTP id KAA01003 for ; Fri, 12 Nov 1999 10:46:44 -0800 (PST) mail_from (hawkes@cthulhu.engr.sgi.com) Received: from pchawkes (dhcp-b8-42-217.engr.sgi.com [150.166.42.217]) by cthulhu.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via SMTP id KAA92481; Fri, 12 Nov 1999 10:45:15 -0800 (PST) mail_from (hawkes@engr.sgi.com) Message-ID: <00f201bf2d3e$5815a8e0$d92aa696@engr.sgi.com> From: "John Hawkes" To: "Frank Hofmann - Sun Germany Technical Solution Center - Munich" , References: <199911101717.SAA13801@emuc05-home.Germany.Sun.COM> Subject: Re: Some notes on portability of lockmetering code to non-IA32 architectures Date: Fri, 12 Nov 1999 10:47:15 -0800 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2615.200 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 Sender: owner-lockmeter@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;lockmeter-outgoing From: Frank Hofmann - Sun Germany Technical Solution Center - Munich To: <...snip...> > In lockmeter.c, there are several points which prevent that it > compiles on non-IA32: > > 1. the use of "cpucount". <...snip...> > Replacing this by the arch-independent global "smp_num_cpus" works. Yes. Thank you. > 2. Same for "cpu_hz". This is only used currently to fill in a field > in the lockstat info struct. I don't know an arch-independent way > to get this but it's not required for working lockstat and so I've > just removed it for the moment. cpu_hz is used by the user program "lockstat" to translate the raw "ticks" count into user-meaningful time. For x86 the "ticks" are cpu cycles, so cpu_hz allows "lockstat" to convert to microseconds or milliseconds for display purposes. I suppose to make this more multi-platform, there should be abstracted macros for obtaining some low-level raw time "ticks" (for x86 that macro expands to a call to get_cycles()) and for defining some divisor that converts these "ticks" into microseconds. Of course, a real multi-platform solution would have a include/linux/lockmeter.h containing the bulk of the architecture-independent code, and a handful of architecture-specific include/XYZ/lockmeter.h files which contain the various macro definitions that lockmeter.c and include/linux/lockmeter.h use. Anyway, you can remove the reference to cpu_hz in order to get a Sparc lockmeter.c to build, but "lockstat" isn't going to display correct numbers (and might even divide-by-zero if you don't initialize the kernel-to-user lstat_user_request_t "cycleval" field to a nonzero value). > 3. The following assembler code: and > 4. The following assembler code: I'll work on abstracting these. Thank you. > - lockmeter.h: I'm not sure about the LSTAT_RA macro. > - endianness issues: Have not yet been able to check for this. Oh, true. I'm afraid I was being rather x86-centric. Thanks for your suggestions! John Hawkes From owner-lockmeter@oss.sgi.com Mon Nov 15 01:57:28 1999 Received: by oss.sgi.com id ; Mon, 15 Nov 1999 01:57:19 -0800 Received: from mercury.Sun.COM ([192.9.25.1]:24029 "EHLO mercury.Sun.COM") by oss.sgi.com with ESMTP id ; Mon, 15 Nov 1999 01:57:07 -0800 Received: from emuc05-home.Germany.Sun.COM ([129.157.51.10]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id CAA02706 for ; Mon, 15 Nov 1999 02:02:30 -0800 (PST) Received: from sunhsc (sunhsc [129.157.167.92]) by emuc05-home.Germany.Sun.COM (8.8.8+Sun/8.8.8/ENSMAIL,v1.7) with SMTP id LAA11411 for ; Mon, 15 Nov 1999 11:02:23 +0100 (MET) Message-Id: <199911151002.LAA11411@emuc05-home.Germany.Sun.COM> Date: Mon, 15 Nov 1999 11:03:19 +0100 (MET) From: Frank Hofmann - Sun Germany Technical Solution Center - Munich Reply-To: Frank Hofmann - Sun Germany Technical Solution Center - Munich Subject: Re: Some notes on portability of lockmetering code to non-IA32 architectures To: lockmeter@oss.sgi.com MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: mU9tb1ys3OwQh1rPV0+c5Q== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.3.2 SunOS 5.7 sun4u sparc Sender: owner-lockmeter@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;lockmeter-outgoing > > From: Frank Hofmann - Sun Germany Technical Solution Center - Munich > > To: > <...snip...> > > In lockmeter.c, there are several points which prevent that it > > compiles on non-IA32: > > > > 1. the use of "cpucount". > <...snip...> > > Replacing this by the arch-independent global "smp_num_cpus" works. > > Yes. Thank you. > > > 2. Same for "cpu_hz". This is only used currently to fill in a field > > in the lockstat info struct. I don't know an arch-independent way > > to get this but it's not required for working lockstat and so I've > > just removed it for the moment. > > cpu_hz is used by the user program "lockstat" to translate the raw > "ticks" count into user-meaningful time. For x86 the "ticks" are cpu > cycles, so cpu_hz allows "lockstat" to convert to microseconds or > milliseconds for display purposes. I suppose to make this more > multi-platform, there should be abstracted macros for obtaining some > low-level raw time "ticks" (for x86 that macro expands to a call to > get_cycles()) and for defining some divisor that converts these "ticks" > into microseconds. > > Of course, a real multi-platform solution would have a > include/linux/lockmeter.h containing the bulk of the > architecture-independent code, and a handful of architecture-specific > include/XYZ/lockmeter.h files which contain the various macro > definitions that lockmeter.c and include/linux/lockmeter.h use. > > Anyway, you can remove the reference to cpu_hz in order to get a Sparc > lockmeter.c to build, but "lockstat" isn't going to display correct > numbers (and might even divide-by-zero if you don't initialize the > kernel-to-user lstat_user_request_t "cycleval" field to a nonzero > value). I'll check for a way to get the CPU clock frequency on SPARC. On my SS10, the reported BogoMIPS value corresponds to the CPU freq in MHz - this may be accidental, I'm going to look deeper into this code. I believe for any kind of statistics interface a way to get the CPU freq would be very useful. Still, to my knowledge the kernel does not have a generic interface for this. > > > 3. The following assembler code: > and > > 4. The following assembler code: > > I'll work on abstracting these. Thank you. > > > - lockmeter.h: I'm not sure about the LSTAT_RA macro. > > - endianness issues: Have not yet been able to check for this. > > Oh, true. > > I'm afraid I was being rather x86-centric. Thanks for your suggestions! Thank you for providing this code in the first place ;-) Still, I must have made some errors (or left out something) because my adaptions prevent the kernel from booting. I've made sure that it is the lockmetering stuff because if I build a kernel with nothing changed but CONFIG_LOCKMETER unset it works well. With lockmeter, initial loading stops after the message "Now booting the kernel", so problems must start in a very early phase. I've not yet tried to add printk()'s to narrow down where it blocks. A question aside: What's EXPORT_SYMBOL actually doing, and do I need to export the lockmetering _spin_lock_ ... symbols for it to work ? I had to leave this out because the kernel did not want to compile - I believe there's a problem with #define's. As soon as I have either a working lockmeter on sparc-linux or the reason why the kernel blocks so early in the initialization phase, I will update the list. Bye, Frank From owner-lockmeter@oss.sgi.com Mon Nov 15 09:52:47 1999 Received: by oss.sgi.com id ; Mon, 15 Nov 1999 09:52:28 -0800 Received: from sgi.SGI.COM ([192.48.153.1]:11890 "EHLO sgi.com") by oss.sgi.com with ESMTP id ; Mon, 15 Nov 1999 09:52:12 -0800 Received: from cthulhu.engr.sgi.com (cthulhu.engr.sgi.com [192.26.80.2]) by sgi.com (980305.SGI.8.8.8-aspam-6.2/980304.SGI-aspam: SGI does not authorize the use of its proprietary systems or networks for unsolicited or bulk email from the Internet.) via ESMTP id JAA271482 for ; Mon, 15 Nov 1999 09:57:40 -0800 (PST) mail_from (hawkes@cthulhu.engr.sgi.com) Received: from ppphawkeswa ([169.238.112.29]) by cthulhu.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via SMTP id JAA94795 for ; Mon, 15 Nov 1999 09:57:23 -0800 (PST) mail_from (hawkes@engr.sgi.com) Message-ID: <013401bf2f93$34ee6580$1d70eea9@seattle.sgi.com> From: "John Hawkes" To: Subject: porting lockmeter to Sparc Date: Mon, 15 Nov 1999 09:59:43 -0800 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2615.200 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 Sender: owner-lockmeter@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;lockmeter-outgoing > From: Frank Hofmann - Sun Germany Technical Solution Center - > Munich ... > With lockmeter, initial loading stops after the message "Now booting the > kernel", so problems must start in a very early phase. I've not yet tried > to add printk()'s to narrow down where it blocks. My guess is that the expanded CONFIG_LOCKMETER locks or unlocks are in error -- that is, the failure lies either in some change you have made in spinlock.h or smp.h, or more likely in the verbose locking or unlocking in lockmeter.c. Be careful when inserting printk()'s, since that code uses spinlocks and you'll encounter the same error. One way to debug this is to narrow down the error to either the locking or to the unlocking, or to the spinlocks vs. the MR locks. You can, for example, go into spinlock.h and turn one of those: #ifdef CONFIG_LOCKMETER around the redefinition of the locks, or the redefinition of the unlocks, into: #ifdef CONFIG_LOCKMETER_not and recompile, thus reverting that lock or unlock back to the traditional known-to-work instruction sequence. Once you've narrowed it down to "spinlock lock" or "MRlock unlock", then you can stare at the smaller hunk of lockmeter.c code. Yes, it is cumbersome to debug this when just enabling the CONFIG_LOCKMETER causes the system boot to hang. > A question aside: What's EXPORT_SYMBOL actually doing I am informed that the EXPORT_SYMBOL changes are necessary in order for a loadable module to find the correct lockmeter.c lock/unlock routines. I confess that the particular kernels I was building for x86 used no modules, and I don't have firsthand knowledge of the problem. John Hawkes From owner-lockmeter@oss.sgi.com Tue Nov 23 12:07:54 1999 Received: by oss.sgi.com id ; Tue, 23 Nov 1999 12:07:44 -0800 Received: from [192.48.153.1] ([192.48.153.1]:56896 "EHLO sgi.com") by oss.sgi.com with ESMTP id ; Tue, 23 Nov 1999 12:07:26 -0800 Received: from nodin.corp.sgi.com ([192.26.51.193]) by sgi.com (980327.SGI.8.8.8-aspam/980304.SGI-aspam: SGI does not authorize the use of its proprietary systems or networks for unsolicited or bulk email from the Internet.) via ESMTP id MAA04319 for ; Tue, 23 Nov 1999 12:13:24 -0800 (PST) mail_from (hawkes@babylon.engr.sgi.com) Received: from babylon.engr.sgi.com (babylon.engr.sgi.com [192.26.80.26]) by nodin.corp.sgi.com (980427.SGI.8.8.8/980728.SGI.AUTOCF) via ESMTP id MAA59181 for ; Tue, 23 Nov 1999 12:13:24 -0800 (PST) Received: (from hawkes@localhost) by babylon.engr.sgi.com (SGI-8.9.3/8.9.3) id MAA00371 for lockmeter@oss.sgi.com; Tue, 23 Nov 1999 12:11:57 -0800 (PST) Date: Tue, 23 Nov 1999 12:11:57 -0800 (PST) From: John Hawkes Message-Id: <199911232011.MAA00371@babylon.engr.sgi.com> To: lockmeter@oss.sgi.com Subject: Lockmeter patches now available for 2.2.13 and 2.3.28 Sender: owner-lockmeter@oss.sgi.com Precedence: bulk Return-Path: X-Orcpt: rfc822;lockmeter-outgoing The Lockmeter patches are now available against 2.2.13 and 2.3.28. I made a few small changes that move some of the i386-dependent stuff out of arch/i386/kernel/lockmeter.c and into various header files. This doesn't make Lockmeter architecture-independent, but it does go a small way toward helping others to port the code to non-Intel platforms. John Hawkes