File: [Development] / ltp-website / faq.html (download) (as text)
Revision 1.8, Thu Aug 31 21:18:16 2000 UTC (17 years, 1 month ago) by alaffin
Branch: MAIN
Changes since 1.7: +1 -1
lines
Update the ltp website to have a link to a page describing the anonymous
CVS access.
|
<HTML>
<HEAD>
<TITLE>LTP FAQ</TITLE>
<META NAME="description" content="LTP Frequently Asked Questions">
<META NAME="keywords" content="LTP, FAQ, Open Source, Linux, code, programming">
<META HTTP-EQUIV="Expires" CONTENT="Fri, 31 Dec 1990 23:59:59 GMT">
<SCRIPT LANGUAGE="JavaScript" SRC="/js/nav-top_oss.js"></SCRIPT>
</HEAD>
<SCRIPT LANGUAGE="JavaScript" SRC="/js/l3_images.js"></SCRIPT>
<BODY MARGINHEIGHT="0" MARGINWIDTH="0" LEFTMARGIN="0" TOPMARGIN="0" BGCOLOR="#FFFFFF" TEXT="#000000" ALINK="#FF0000" VLINK="#666666" LINK="#420077">
<SCRIPT LANGUAGE="Javascript">
<!---
function change(Name,Image,No,Msg) {
if (!document.images) {}
else {
document [Name].src = eval(Image + No + ".src");
}
if (Msg) self.status = eval(Image + "2");
return true
}
//-->
</SCRIPT>
<TABLE BORDER="0" CELLPADDING="0" CELLSPACING="0" WIDTH="620">
<!-- Start sgi.com Top Nav -->
<TR>
<TD COLSPAN="3"><A HREF="http://www.sgi.com/" onMouseover="change('hom','nav_hom',1)" onMouseout="change('hom','nav_hom',0)"><IMG
SRC="/images/common/nav_home0.gif" ALT="[Home] " WIDTH="48" HEIGHT="25" BORDER="0" NAME="hom"></A><A
HREF="http://www.sgi.com/search/" onMouseover="change('src','nav_src',1)" onMouseout="change('src','nav_src',0)"><IMG
SRC="/images/common/nav_search0.gif" ALT="[Search] " WIDTH="54" HEIGHT="25" BORDER="0" NAME="src"></A><A
HREF="http://www.sgi.com/sales/" onMouseover="change('sal','nav_sal',1)" onMouseout="change('sal','nav_sal',0)"><IMG
SRC="/images/common/nav_sales0.gif" ALT="[How to Buy] " WIDTH="79" HEIGHT="25" BORDER="0" NAME="sal"></A><A
HREF="http://www.sgi.com/products/" onMouseover="change('prd','nav_prd',1)" onMouseout="change('prd','nav_prd',0)"><IMG
SRC="/images/common/nav_prod0.gif" ALT="[Products] " WIDTH="70" HEIGHT="25" BORDER="0" NAME="prd"></A><A
HREF="http://www.sgi.com/solutions/" onMouseover="change('sol','nav_sol',1)" onMouseout="change('sol','nav_sol',0)"><IMG
SRC="/images/common/nav_solutions0.gif" ALT="[Solutions] " WIDTH="73" HEIGHT="25" BORDER="0" NAME="sol"></A><A
HREF="http://www.sgi.com/developers/"><IMG
SRC="/images/common/nav_dev1.gif" ALT="[Developers] " WIDTH="81" HEIGHT="25" BORDER="0" NAME="dev"></A><A
HREF="http://www.sgi.com/support/" onMouseover="change('srv','nav_srv',1)" onMouseout="change('srv','nav_srv',0)"><IMG
SRC="/images/common/nav_support0.gif" ALT="[Services and Support] " WIDTH="128" HEIGHT="25" BORDER="0" NAME="srv"></A><A
HREF="http://www.sgi.com/fun/" onMouseover="change('fun','nav_fun',1)" onMouseout="change('fun','nav_fun',0)"><IMG
SRC="/images/common/nav_fun0.gif" ALT="[Serious Fun] " WIDTH="87" HEIGHT="25" BORDER="0" NAME="fun"></A></TD>
</TR>
<TR>
<TD COLSPAN="3"><IMG SRC="/images/common/nav_dev_line.gif" ALT="" WIDTH="620" HEIGHT="7"></TD>
</TR>
<!-- End sgi.com Top Nav -->
<TR>
<TD VALIGN="TOP"><A HREF="http://www.sgi.com/"><IMG SRC="/images/common/sgilogo.gif" ALT="sgi" WIDTH="130" HEIGHT="108" BORDER="0"></A></TD>
<TD COLSPAN="2" ALIGN="RIGHT"> <BR><A HREF="http://www.sgi.com/developers/"><IMG src="/images/nav-devcentral.gif" WIDTH="203" HEIGHT="55" ALT="Developer Central" BORDER="0"></A>
<BR>
<A HREF="http://www.sgi.com/developers/devtools/index.html" onMouseOver="if (good == 1) turnOn('image1')" onMouseOut="if (good == 1) turnOff('image1')"><IMG
NAME="image1" src="/images/nav-dev_off.gif" ALT="Development Products" WIDTH="80" HEIGHT="55" BORDER="0"></A><A HREF="http://www.sgi.com/developers/program/index.html" onMouseOver="if (good == 1) turnOn('image2')" onMouseOut="if (good == 1) turnOff('image2')"><IMG
NAME="image2" src="/images/nav-prog_off.gif" ALT="Program & Benefits" WIDTH="75" HEIGHT="55" BORDER="0"></A><A HREF="http://www.sgi.com/developers/library/index.html" onMouseOver="if (good == 1) turnOn('image3')" onMouseOut="if (good == 1) turnOff('image3')"><IMG
NAME="image3" src="/images/nav-lib_off.gif" ALT="Library" WIDTH="72" HEIGHT="55" BORDER="0"></A><A HREF="http://www.sgi.com/developers/marketing/index.html" onMouseOver="if (good == 1) turnOn('image4')" onMouseOut="if (good == 1) turnOff('image4')"><IMG
NAME="image4" src="/images/nav-mkt_off.gif" ALT="Marketing Resources" WIDTH="72" HEIGHT="55" BORDER="0"></A><A HREF="http://www.sgi.com/developers/technology/index.html" onMouseOver="if (good == 1) turnOn('image5')" onMouseOut="if (good == 1) turnOff('image5')"><IMG
NAME="image5" src="/images/nav-tech_off.gif" ALT="Technology" WIDTH="73" HEIGHT="55" BORDER="0"></A><A HREF="http://www.sgi.com/developers/oss/index.html" onMouseOver="if (good == 1) turnOn('image6')" onMouseOut="if (good == 1) turnSel('image6')"><IMG
NAME="image6" src="/images/nav-oss_sel.gif" ALT="Open Source" WIDTH="75" HEIGHT="55" BORDER="0"></A><BR><A HREF="http://www.sgi.com/developers/oss/index.html" onMouseOver="if (good == 1) turnOn('image6')" onMouseOut="if (good == 1) turnSel('image6')"><IMG
src="/images/nav-title_oss.gif" WIDTH="118" HEIGHT="25" ALT="Open Source" BORDER="0"></TD>
</TR>
<TR><TD WIDTH="138" VALIGN="TOP">
<!-- Start oss Project Sub Nav -->
<TABLE BORDER="0" WIDTH="138" CELLSPACING="0" CELLPADDING="0">
<TR>
<TD><IMG src="/images/dot_clear.gif" WIDTH="5" HEIGHT="2" ALT="image"></TD>
<TD><IMG src="/images/dot_clear.gif" WIDTH="20" HEIGHT="2" ALT="image"></TD>
<TD><IMG src="/images/dot_clear.gif" WIDTH="116" HEIGHT="2" ALT="image"></TD>
</TR>
<TR>
<TD></TD>
<TD COLSPAN="2"><IMG src="/images/bullets_dev2.gif" ALT=""
WIDTH="18" HEIGHT="10"><FONT FACE="Geneva, Arial Narrow, Helvetica"
SIZE="2"><a href="../../about/system.html"><B>About this site</B></a></FONT><br>
<br></TD>
</TR>
<TR>
<TD></TD>
<TD COLSPAN="2"><IMG src="/images/bullets_dev2.gif" ALT=""
WIDTH="18" HEIGHT="10"><FONT FACE="Geneva, Arial Narrow, Helvetica"
SIZE="2"><B>Linux Test Project</B></FONT></TD>
</TR>
<TR>
<TD></TD>
<TD></TD>
<TD><FONT FACE="ARIAL, HELVETICA" SIZE="2">
<A href="">Overview</a><BR>
<A href="news.html">News</a><BR>
<font color="#207575"><b>FAQ</b></font><BR>
<A HREF="mail.html">Mailing List</A><BR>
<A HREF="contribute.html">How to Contribute</A><BR>
Source [<A HREF="cvs_download.html">CVS</A> | <A HREF="http://oss.sgi.com/cgi-bin/cvsweb.cgi/ltp/">Web</A>]<BR>
<A HREF="ftp://oss.sgi.com/www/projects/ltp/download/">Download</A><BR>
<A HREF="license.html">License</A>
</FONT><BR> </TD>
</TR>
<TR>
<TD></TD>
<TD COLSPAN="2"><IMG src="/images/bullets_dev2.gif" ALT=""
WIDTH="18" HEIGHT="10"><FONT FACE="Geneva, Arial Narrow, Helvetica" SIZE="2"><B>SGI Open Source</B></FONT></TD>
</TR>
<TR>
<TD></TD>
<TD></TD>
<TD><FONT FACE="ARIAL, HELVETICA" SIZE="2">
<A HREF="../">Project List</A>
</FONT><BR> </TD>
</TR>
</TD>
</TR>
</TABLE>
<!-- End oss Project Sub Nav -->
</TD>
<TD><IMG src="/images/dot_clear.gif" WIDTH="20" HEIGHT="1"></TD>
<TD VALIGN="TOP">
<P>
<!-- Start Project Content -->
<FONT FACE="ARIAL NARROW, HELVETICA" SIZE="5"><B>LTP FAQ</B></FONT>
<p>
<i>Our FAQ is a bit unorganized at this point. We're adding questions as they pop up. When topic areas
become clear, we'll reorder the questions into topic areas.</i>
<P>
<FONT FACE="ARIAL NARROW, HELVETICA">
<!-- questions section -->
<p>
<a href="#Q1"><b>What is LTP ?</b></a><br>
<a href="#Q2"><b>Linux already has a successful testing model. How will this project benefit Linux further?</b></a><br>
<a href="#Q3"><b>What is in the LTP, so far. </b></a><br>
<a href="#Q4"><b>How do I build this?</b></a><br>
<a href="#Q5"><b>How do I run this? How does it work?</b></a><br>
<a href="#Q6"><b>How do I analyze the results?</b></a><br>
<a href="#Q7"><b>What if a test program reports a failure?</b></a><br>
<a href="#Q8"><b>What is a test?</b></a><br>
<a href="#Q8"><b>Are you doing benchmarking?</b></a><br>
<a href="#Q8"><b>Are you doing standards testing?</b></a><br>
<hr noshade size=1>
<!-- answers section -->
<a name="Q1"></a>
<h3>What is LTP ?</h3>
LTP is the Linux Test Project. LTP is a project that aims to develop
a set of tools and tests to verify the functionality and
stability of the Linux kernel (including regression tests). We hope this will
support Linux development by making unit testing more complete and minimizing user impact
by building a barrier to keep bugs from making it to the user.
<p>
<a name="Q2"></a>
<h3>Linux already has a successful testing model. How will this project benefit Linux further?</h3>
The Linux development community utilizes two important (some would argue most important)
testing techniques in it's normal operations: Design and Code Inspections. The intent of LTP is
to support this by giving developers an ever growing set of tools to help identify any
operational problems in their code that may be missed by human review.
One of the toughest categories of problems to catch with inspection is that of interaction
of features. With a continuously improving set of tests and tools, developers can get
an indication of whether their changes may have broken some other functionality.
<p>
There is no such thing as a perfect test base. It's only useful if it keeps up with new and
changing functionality, and if it actually gets used.
<p>
<a name="Q3"></a>
<h3>What is in the LTP, so far? </h3>
The first release of code from SGI for LTP was a set of tools for testing file systems.
Since that first release, a group of tests we call <i>Quickhit</i> tests have been released.
These are simple tests aimed at running through a few execution paths in a number of system calls.
Watch the news link for updates on the content of the LTP releases.
<p>
<a name="Q4"></a>
<h3>How do I build this?</h3>
We've made a decision to not put effort into a grand build process in favor of a
minimal makefile approach. We did this because the project is just starting and we don't
want to impose an "SGI approach" in favor of a more open source approach decided more by
open source contributors. For now, from the root of the tree, a simple 'make' should do
it. Send your build errata to ltp@oss.sgi.com.
<p>
<a name="Q5"></a>
<h3>How do I run this? How does it work?</h3>
Currently, you can't really "run" this -- at least not in terms of a single starting command line.
Instead, LTP is made up of a number of individual test programs. These test programs usually test some
some portion of a feature's functionality. Most tests can be run without any command line options.
Others offer variable functionality specified from the command line.
<p>A problem that needs to be
addressed is how to store different command lines and determining when to run them. A couple examples are, should
a bigmem test be run on a 128Mb system? Should a file system test that requires 2Gb of scratch space be
run on a system without such a test space? These two example highlight a couple of issues that need to be addressed
in regard to the runtime framework for tests.
<p>
<a name="Q6"></a>
<h3>How do I analyze the results?</h3>
Our philosophy on writing functional tests is that each test program is responsible for setting up an
environment for testing the target functionality, performing an operation using the target functionality, and
then analyzing whether the functionality performed according to expectations and reporting the results.
Reporting the results can take a couple of different forms.
<p>The simplest form is the exit status of the program. If a test hits a scenario where it finds unexpected
or improper results, calling <font face="courier">exit()</font> with a non-zero exit status would flag the
caller that the test failed for some reason. Conversely, if the test completes with the functionality
performing as expected, a zero exit would flag the caller that the test passed. Summarizing: <font face="courier">
exit(0)==PASS, exit(!0)==FAIL</font>. Additionally, the test program could report some sort of information
on standard output indicating why the test passed or failed.
<p>
Another form of results reporting used within SGI is a hybrid of the <font face="courier">exit()</font> style
reporting. It still uses the exit status for reporting, but incorporates a standard format for output from
standard output of the test program. This accomplishes two things: allows the reporting of the functional
testing results to be more understandable for both humans and a computer analysis tool and allows more than one test
case result to be reported from within one test program. This is in contrast to a pure exit status analysis where
a test program may contain N test cases and one test case fails, the test program returns a non-zero value. It
appears that the whole test fails. Compare this to a scenario where a test reports for each test case a PASS or
FAIL token on standard output.
<p>
For quick clarity, take this example:
<font face="courier"><pre>
$ ./dup02
dup02 1 PASS : dup(-1) Failed, errno=9 : Bad file descriptor
dup02 2 PASS : dup(1500) Failed, errno=9 : Bad file descriptor
</pre></font>
dup02 is a test program that tests a couple of scenarios for the dup() call. Each of the test cases call dup()
in an effort to be sure dup() responds with a correct errno value after deliberately calling dup() with bad
arguments. In both cases above, dup() is called with non-existent (1500) or undefined (-1) file descriptor values.
In both cases, dup() should respond with an errno indicating a 'Bad File Descriptor'. In both cases, the test reports
that dup() responded as expected by printing a PASS token.
<p>
Additionally, the exit status should be zero, indicating all the test cases passed. Accessing the exit status of the
test program can be reported immediately after running the test program by printing the $? variable.
<font face="courier"><pre>
$ echo $?
0
</pre></font>
<p>
Finally, we can analyze the results of the dup02 test program on two fronts. We can see from the two PASS lines, that both
test cases in the dup02 program passed. We can also see from the exit status that the dup02 program as a whole passed.
<p>
<a name="Q7"></a>
<h3>What if a test program reports a failure?</h3>
After a failure report, analysis consists of three parts: determining what test case reported the failure, determining
if the reported failure was really a failure, and comparing results with a passing baseline.
<p>
Determining what test case is reporting a failure is sometimes a challenge. In some cases, in a multiple test case
test program, previous test cases can cause failure in subsequent test cases. If a test is well written, the output
on standard output can be useful.
<p>
Once the test case the "failure" is occuring under is osolated, determining whether the "failure" is really a failure
needs to be determined. If a test is poorly written and certain environmental conditions are not handled properly, a
false failure may occur.
<p>
Comparing the failure on the system under test against a baseline system is important. If the test fails on a system
but it is unknown if it ever passed on any other system, its not possible to determine if the problem is a new failure.
Comparing how the functionality under test performs on the baseline against how it acts on the test system is the most
important method for determining how the "failure" occured.
<p>
<a name="Q8"></a>
<h3>What is a test?</h3>
Take a look at the LTP Glossary at <a href="glossary.html">http://oss.sgi.com/projects/ltp/glossary.html</a>. It covers
some of the basic testing definitions, including 'test'.
<p>
In software testing, the word 'test' has become overloaded and increasingly tough to define. In operating system
testing, we might be able to categorize tests into a few different areas: functional (regression),
duration, stress (load), and performance.
<p>
<a name="Q9"></a>
<h3>Are you doing benchmarking?</h3>
Not at this time. We're more interested in functional, regression, and stress testing the Linux kernel.
Benchmarking may be workable to compare the performace among Linux versions.
<p>
<a name="Q10"></a>
<h3>Are you doing standards testing?</h3>
No, we're leaving that to Linux Standard Base (LSB). Check out
<a href="http://www.freestandards.org/">http://www.freestandards.org/</a>.
</font>
<IMG src="/images/dot_clear.gif" WIDTH="400" HEIGHT="1">
</TD>
</TR>
</TABLE>
<P>
<CENTER>
<!---- Virtual Footer ---->
<TABLE WIDTH="400" CELLPADDING="0" CELLPADDING="0" BORDER="0">
<TR>
<TD ALIGN="RIGHT">
<FONT FACE="Helvetica, Arial" SIZE="-1"><a
href="../../about/system.html">about this site</a> | <A
href="http://www.sgi.com/company_info/privacy.html">privacy policy</A></FONT>
</TD>
<TD ALIGN="CENTER">
<FONT FACE="Helvetica, Arial">
|
</FONT>
</TD>
<TD ALIGN="LEFT">
<FONT FACE="Helvetica, Arial" SIZE="-1"><A HREF="mailto:owner-ltp@oss.sgi.com">owner(s) of project ltp</A></FONT>
</TD>
</TR>
<TR>
<TD ALIGN="RIGHT">
<FONT FACE="Helvetica, Arial" SIZE="-2"><A HREF="http://www.sgi.com/company_info/copyright.html">Copyright © 1999 Silicon Graphics, Inc.</A> All rights reserved.</FONT>
</TD>
<TD ALIGN="CENTER">
<FONT FACE="Helvetica, Arial">
|
</FONT>
</TD>
<TD ALIGN="LEFT">
<FONT FACE="Helvetica, Arial" SIZE="-2"><A HREF="http://www.sgi.com/company_info/trademarks/">Trademark Information</A></FONT>
</TD>
</TR>
</TABLE>
</BODY>
</HTML>
![[BACK]](/icons/back.gif){kind=icon}
Return to faq.html CVS log ![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/dir.gif){kind=icon}
Up to [Development] / ltp-website