Monitoring Oracle 10g RAC with Quest Spotlight on RAC – Part I

You have heard the hype around a particular technology. It’s
cool and everyone wants a piece of it. Then it’s the post-hype era for a
technology where it is getting serious attention from the Big Players
and eventually its commoditization. Breakthrough technologies, like Oracle RAC
or Server Partitioning (fondly known as Virtualization), will eventually be
commonplace. Therefore, it is time for your DBA to monitor several oracle RACs.
The scenario could range from a RAC farm, as massive as 36-Node Oracle RAC, to
a 2-Node Oracle RAC. Being confronted with questions like these is very normal
to a curious DBA:

• How is my cluster performing?

• How is my Database performance as a whole?

• Contention issues disk, io, cluster, interconnect?

• Latency problems within HSI (High Speed Interconnects)? Are there
  poor settings? Are my NIC’s are set properly? Or is it a poorly configured RAC?

• Inter-node communication issues? Are too many "Hot
  Blocks" being transferred across nodes due to poor database design?

• Poor load balancing?

As you can see, the list is endless. You have migrated to a
heavy duty Oracle RAC and you see performance drop. Unfortunately, you cannot see
exactly what might be going wrong.

Introducing Spotlight on RAC (SoRAC)

It is time to say hello to SoRAC (Spotlight on RAC by Quest
Software) which is well out of beta phase
and very soon to be GA. I had the pleasure of participating in the beta program
and was rather pleased with the results (even though it was still in beta
phase).

SoRAC provides detailed diagnostic information by drilling
down individual instances and merging the appropriate calculations into one
interface called "SoRAC Homepage." And it’s all live!

Overview of Spotlight on Oracle RAC

What is this tool capable of?

It provides three classes of functionality to a RAC DBA.…

• You get to see the physical architecture. As you see below…

  Click for larger image
  

  You can clearly see all of the essential
  components of a cluster: the individual instances, the interconnect and the I/O
  subsystem are displayed, including the essential data flows between them. The
  great thing about this tool is that you can paint a picture of your RAC in your
  head, enhance your understanding of RAC, and localize problems. The above
  example runs against my 2-node RHEL 4.2 Oracle 10g R2 RAC on VMware GSX Server.

• Calculation of diagnostics and presenting them back to the
  interface as a whole. For instance, you might have latch contention on one node
  on a 6-node RAC and the aggregate of all the latches might just be fine for
  your RAC to function adequately.

• Several RAC specific alarms such as:
  1. Uneven
     Load Distribution to the instances.

  2. Overhead
     due to cluster maintenance.

  3. High
     latency problems within the HSI (High Speed Interconnects).

  4. Excessive
     HSI communication (might point to partitioning data across nodes, for instance
     a typical hybrid , meaning OLTP (Online Transaction Processing) /DSS (Decision
     Support System) might require a data partitioning on the DSS part of that RAC
     database).

  5. Block
     Corruption / Orphaned blocks across instances.

  6. OCFS
     problems / Shared Disk (Whether on SAN/NAS) contention.

What Platform does it support?

The initial release will still provide support to a RACs
running on Unix/Linux based systems such as RHEL/Solaris. There is theoretical
support for a typical 36-node cluster but formally, you monitor a 16-node
cluster with ease.

You will, however, need an Oracle Client software
installation and you can get it here
although you will need to adjust your TNSNAMES.ora file, as this will be used
to make a connection to your RAC Service.

Getting Spotlight on RAC

Spotlight on RAC should be available very soon. (So I heard
from the Marketing Manager) and then you should be able to download an
evaluation from here .