If you have seen advertising literature from major database
system vendors, invariably you have seen statements about how that database
performed against other systems (benchmark results). All of the major vendors
support and participate in benchmarking tests because, to some degree, they
have to. The mindset must be along the lines of "Everyone else is doing it, so
we'd look odd not participating in the tests ourselves - even though most
analysts agree the tests are somewhat artificial and flawed."
A good introduction into database performance benchmarking
can be found in an Oracle Magazine (March-April 2001) by Fred Sandsmark (http://www.oracle.com/oramag/oracle/01-mar/o21bench.html).
And despite what you may think before reading the article, Mr. Sandsmark does
not give Oracle preferential treatment just because he wrote this article for
Oracle. Who is the referee or judge of database benchmarking? No one is going
to believe Oracle, Microsoft, or anyone else is going to be impartial, and why
should they be? They are businesses with the same end goal in mind: increased
profitability for owners and shareholders. When Oracle, for example, claims it
has the best-performing database on System X using Processor Y, that is
probably true, and the results can be verified, but when you see just how many
system/platform combinations there are in the benchmarking results and what the
results were, the value of the claim may be diminished.
One important concept to take away from this discussion is
that there is no singular, all encompassing, definitive test that allows a
vendor to claim their system is the best one out there, no ands, ifs or buts.
For Oracle, Microsoft, IBM, or Sybase to claim they are the best overall, well,
it's simply not true. A particular system can be the best on a particular
platform under certain conditions, but to say a particular system is the best
overall is going to make that vendor suspect with respect to credibility.
The purpose of this article - from an Oracle DBA standpoint
- is to introduce you to what the benchmark tests are, who controls or
regulates them, and show you how Oracle compares to other database systems.
Three letters will over and over again come into play when vendors compare
themselves to one another, and those letters are TCO (total cost of ownership).
Defining what factors into and how exactly you measure TCO is an issue in of
When someone (your management at budget time?) says, "Sybase
rocks" or "SQL Server rules, just look at these benchmark results," it is
useful to know what those results mean, where they came from, and how they were
obtained (under what conditions).
The Transaction Processing Performance Council
Yes, it is another one of those somewhere-on-the-Internet
entities that wields power and control over our lives. The TPC "defines
transaction processing and database benchmarks and delivers trusted results to
the industry." The home page is at www.tpc.org,
and you can see who the major members are in the picture below.
As you can readily see, all of the big names - platform,
operating system and database system - are represented.
From the "About the TPC" page, the TPC "produces benchmarks
that measure transaction processing (TP) and database (DB) performance in terms
of how many transactions a given system and database can perform per unit of
time, e.g., transactions per second or transactions per minute." Two commonly
used metrics are variations of transactions per unit of time and cost per
The TPC benchmarks are divided into four categories: TPC-C,
TPC-H, TPC-R, and TPC-W, and each category has it own set of metrics as just mentioned.
In the interest of space and time, let's focus on two of them: TPC-C and TPC-R.