Michael T. Jones (mtj++at++babar.asd.sgi.com)
Thu, 2 May 1996 08:29:50 -0700
It's been like a rollercoaster ride where my expectations for a really
interesting result started low, went way up, then down really fast, up
up slightly, and now are back where they started. (As an aside for the
simile and metaphor impared, I don't mean that I've actually been ON a
rollercoaster lately, just that I feel like it ;-)
No one who's purchased an IG or who has extensive experience is likely
to have much hope in a generic benchmark. Image generators by definition
tend to push hardware right to the limits, and this tends to expose the
little rough edges to application developers and database modelers. My
original feeling about a generic benchmark started out low for just this
reason. For example, the fine Intergraph high-end PC graphics systems
are supposedly able to texture or fog, but not both at once. Likewise,
the CompuScene-IV could texture or Gouraud-shade, but not both at once.
If a user could live with one mode at a time (and perhaps many could)
then these are not serious problems, but what should a generic benchmark
do about it? Be "fair" and do no fog, no texture, no shading? Now you
have your system running in "Amiga" mode, so what's the point?
On the other hand, there are certain broad classes of visual simulation
applications (such as high-level flight, helicoper, armored vehicles,
commericial aviation, driving simulators, marine simulation, etc.) that
most IG purchases are targeted at. Somehow my general personal optimism
caused me to read John Archdeacon's original Gemini-benchmark-suite email
to info-performer as the announcement of some reference or benchmark
environments for these common application areas. This could be good. It
would not say everything about a machine, but you could have vendors put
each environment on their machine and tune it for whatever they feel is
the optimal feature/price/performance point. For example, an Intergraph
might leave fog off and say you did not need it, while SGI would have
fog, texture, and smooth-shading and argue it all three are critical.
This elevated my hopes.
Now I see that it's just the results of running one toolkit on several
machines. Machines where the vendors in some cases might have doubts
about the ability of that toolkit to optimally present the capability
of the hardware. This was the big drop, almost like free-fall. How
could a user interpret the results of such a benchmark?
The only way I see to validate the single-software benchmark is to
augment it with testing of other tools on the same systems and using
the same databases for comparative performance. There's a lot of work
in this, but it could be to good end for toolkit-users, since any
vendor who was significantly lower in performance than the "best" one
on a given machine would tend to focus on their area of weakness, which
would beneift their customers. At least it provides insight to
benchmark analysis tasks in the case where machine A is 9x the cost of
machine B, but only runs 8x the speed using toolkit #1. Then you look
at the full range of available software and see that you can get rates
of 3x and 12x using toolkits #2 and #3. This lets you see that machine
A costs 9x and runs at 12x the performance, an important point. (Some
benchmarking--or experience--will identify the names of toolkits A, B,
and C)
Now, sadly, I realize that we're back where we started with the single
toolkit benchmark approach where the benchmark is of the toolkit and
not of the hardware. I have had email with Gemini about this and there
is some hope for a change in the future, but not much. Still, we'll do
what we can to get the databases and eyepoint/model data from Gemini in
time for IMAGE. Perhaps John Archdeacon and I will cross paths on one
of the Magic Bus tours and we can make a tape then.
Michael "Through-with-benchmarking" Jones
Be seeing you, Phone:415.933.1455 Fax:415.965.2658 M/S:8U-590
Michael T. Jones Silicon Graphics, Advanced Systems Division
mtj++at++sgi.com 2011 N. Shoreline Blvd., Mtn. View, CA 94039-7311
"Du musst Amboss oder Hammer sein" -- Goethe
This archive was generated by hypermail 2.0b2 on Mon Aug 10 1998 - 17:52:51 PDT