Database Benchmarking - Page 2

January 26, 2005

The TPC-C Benchmark

From the TPC Web site, the TPC-C benchmark is defined as:

TPC-C simulates a complete computing environment where a population of users executes transactions against a database. The benchmark is centered around the principal activities (transactions) of an order-entry environment. These transactions include entering and delivering orders, recording payments, checking the status of orders, and monitoring the level of stock at the warehouses. While the benchmark portrays the activity of a wholesale supplier, TPC-C is not limited to the activity of any particular business segment, but, rather represents any industry that must manage, sell, or distribute a product or service.

TPC-C involves a mix of five concurrent transactions of different types and complexity either executed on-line or queued for deferred execution. It does so by exercising a breadth of system components associated with such environments, which are characterized by:

• The simultaneous execution of multiple transaction types that span a breadth of complexity
• On-line and deferred transaction execution modes
• Multiple on-line terminal sessions
• Moderate system and application execution time
• Significant disk input/output
• Transaction integrity (ACID properties)
• Non-uniform distribution of data access through primary and secondary keys
• Databases consisting of many tables with a wide variety of sizes, attributes, and relationships
• Contention on data access and update

TPC-C performance is measured in new-order transactions per minute.  The primary metrics are the transaction rate (tpmC), the associated price per transaction ($/tpmC), and the availability date of the priced configuration.

TPC-C sounds like a reasonable test for a transactional database. However, if you look further into the TPC-C benchmark, you will see that there are versions of the test, so the results of how a system did using one version may not be comparable to the same system using the latest version (5.3). TPC-C is the most popular benchmark test, so it is probably the results you have seen Oracle advertise. As a test of this, perform a search on benchmark at Oracle's home page, and go to the Benchmark Results - Transaction Processing link.

Let's drill down into the first bulleted item (TPC-C: Oracle and HP ... on Linux).

Reading the fine print at the bottom, Oracle cites the Transaction Processing Council results, but look at the specifics of what it took to get those results. On that particular system/platform combination, using the specific test established by the TPC, Oracle exceeded one million transactions per minute. Again, this result is based on fairly exact conditions whose results may - and do - differ when run on other system/platform combinations. Using only the tpmC metric, did anyone else exceed the one million transactions per minute mark?

Now that you know about the TPC and the results it lists, two of the top three tpmC results were obtained by none other than IBM - and one of those spots is first place. See the page at http://www.tpc.org/tpcc/results/tpcc_results.asp?print=false&orderby=tpm&sortby=desc and order by descending tpmC. Oracle's results are impressive, but not nearly so as the results obtained with the IBM eServer/DB2 database on AIX combination and it is nearly three times as much tpmC as Oracle. Further, the IBM competitor has a lower price per tpmC. However, don't try this test at home, boys and girls. If you click on the "IBM eServer p5 595 64p" link in the System column, you may be surprised to see how much this system cost (over 16 million dollars). The results shown at Oracle's Web site only tell part of the story - when using either tpmC or Price/tpmC, Oracle is not the best overall. Nevertheless, technically speaking, the results (using tpmC) shown by Oracle are the best on the market today as the IBM "winner" isn't available until May 2005.