Database Benchmarking - Page 3

The TPC-R Benchmark

The other TPC benchmark category that is of interest is the TPC-R. This benchmark is analogous to knowing what the test questions are ahead of time. In database terms then, the vendor can front load whatever to help make the known query run as fast as possible. This benchmark is controversial in that what is it really being measured and how can you compare it to another system? Even the TPC has a disclaimer on the use of this benchmark ("The TPC believes that comparisons of TPC-R results measured against different database sizes are misleading and discourages such comparisons."). Only Oracle is shown on the results page.

However, on Oracle's Web site, there is an entire page dedicated to highlighting these results.

You can interpret the TPC-R results with a grain of salt. The TPC discourages this particular benchmark test, and Oracle shows how well it did by being the only contestant in the race. Is that a credibility problem, or just market/sales spin?

The Benchmark Tests

You can download the TPC-C specifications from the TPC Web site. Additionally, there are links to FAQ's and a PowerPoint presentation. Aside from the dated results, the PowerPoint presentation may be quite useful to you if you are the one tasked to explain what benchmarking is. Additionally, you can use the information on the slides if you need some basis to form a recommendation on which database product to purchase (given a particular system) with respect to what metrics to use.

For the TPC-R test, you can actually download the C files used to create ASCII files with delimited data, and this is something you can try at home. Watch your disk space because the scale factor setting can use up the available space on your system. Your Dell Dimension 8200 running XP Professional with its 200MB database, unfortunately, was not what the TPC had in mind for testing on high-end systems.

In Closing

Whenever you see a claim by a database vendor about how its RDBMS can do great and wondrous things on a fill-in-the-blank system, you now know where to go to verify those claims for yourself, but more importantly, you can see how that vendor's results compare to other database system/platform combinations. Oracle, for example, gives great press to its TPC results, but when viewed in comparison with how other combinations faired, you can develop a more informed opinion about what the results mean.

» See All Articles by Columnist Steve Callan