The TPC-R Benchmark
The other TPC benchmark category that is of interest is the
TPC-R. This benchmark is analogous to knowing what the test questions are ahead
of time. In database terms then, the vendor can front load whatever to help
make the known query run as fast as possible. This benchmark is controversial
in that what is it really being measured and how can you compare it to another
system? Even the TPC has a disclaimer on the use of this benchmark ("The TPC believes
that comparisons of TPC-R results measured against different database sizes are
misleading and discourages such comparisons."). Only Oracle is shown on the
However, on Oracle's Web site, there is an entire page
dedicated to highlighting these results.
You can interpret the TPC-R results with a grain of salt.
The TPC discourages this particular benchmark test, and Oracle shows how well
it did by being the only contestant in the race. Is that a credibility problem,
or just market/sales spin?
The Benchmark Tests
You can download the TPC-C specifications from the TPC Web
site. Additionally, there are links to FAQ's and a PowerPoint presentation.
Aside from the dated results, the PowerPoint presentation may be quite useful
to you if you are the one tasked to explain what benchmarking is. Additionally,
you can use the information on the slides if you need some basis to form a
recommendation on which database product to purchase (given a particular
system) with respect to what metrics to use.
For the TPC-R test, you can actually download the C files
used to create ASCII files with delimited data, and this is something you can
try at home. Watch your disk space because the scale factor setting can use up
the available space on your system. Your Dell Dimension 8200 running XP
Professional with its 200MB database, unfortunately, was not what the TPC had
in mind for testing on high-end systems.
Whenever you see a claim by a database vendor about how its
RDBMS can do great and wondrous things on a fill-in-the-blank system, you now
know where to go to verify those claims for yourself, but more importantly, you
can see how that vendor's results compare to other database system/platform
combinations. Oracle, for example, gives great press to its TPC results, but
when viewed in comparison with how other combinations faired, you can develop a
more informed opinion about what the results mean.
See All Articles by Columnist Steve Callan