Monitoring Oracle 10g RAC with Quest Spotlight on RAC - Part I

January 24, 2006

You have heard the hype around a particular technology. It's cool and everyone wants a piece of it. Then it's the post-hype era for a technology where it is getting serious attention from the Big Players and eventually its commoditization. Breakthrough technologies, like Oracle RAC or Server Partitioning (fondly known as Virtualization), will eventually be commonplace. Therefore, it is time for your DBA to monitor several oracle RACs. The scenario could range from a RAC farm, as massive as 36-Node Oracle RAC, to a 2-Node Oracle RAC. Being confronted with questions like these is very normal to a curious DBA:

• How is my cluster performing?
• How is my Database performance as a whole?
• Contention issues disk, io, cluster, interconnect?
• Latency problems within HSI (High Speed Interconnects)? Are there poor settings? Are my NIC's are set properly? Or is it a poorly configured RAC?
• Inter-node communication issues? Are too many "Hot Blocks" being transferred across nodes due to poor database design?
• Poor load balancing?

As you can see, the list is endless. You have migrated to a heavy duty Oracle RAC and you see performance drop. Unfortunately, you cannot see exactly what might be going wrong.

Introducing Spotlight on RAC (SoRAC)

It is time to say hello to SoRAC (Spotlight on RAC by Quest Software) which is well out of beta phase and very soon to be GA. I had the pleasure of participating in the beta program and was rather pleased with the results (even though it was still in beta phase).

SoRAC provides detailed diagnostic information by drilling down individual instances and merging the appropriate calculations into one interface called "SoRAC Homepage." And it's all live!

Overview of Spotlight on Oracle RAC

What is this tool capable of?

It provides three classes of functionality to a RAC DBA.…

• You get to see the physical architecture. As you see below...

  Click for larger image
  

  You can clearly see all of the essential components of a cluster: the individual instances, the interconnect and the I/O subsystem are displayed, including the essential data flows between them. The great thing about this tool is that you can paint a picture of your RAC in your head, enhance your understanding of RAC, and localize problems. The above example runs against my 2-node RHEL 4.2 Oracle 10g R2 RAC on VMware GSX Server.

• Calculation of diagnostics and presenting them back to the interface as a whole. For instance, you might have latch contention on one node on a 6-node RAC and the aggregate of all the latches might just be fine for your RAC to function adequately.
• Several RAC specific alarms such as:
  1. Uneven Load Distribution to the instances.
  2. Overhead due to cluster maintenance.
  3. High latency problems within the HSI (High Speed Interconnects).
  4. Excessive HSI communication (might point to partitioning data across nodes, for instance a typical hybrid , meaning OLTP (Online Transaction Processing) /DSS (Decision Support System) might require a data partitioning on the DSS part of that RAC database).
  5. Block Corruption / Orphaned blocks across instances.
  6. OCFS problems / Shared Disk (Whether on SAN/NAS) contention.

What Platform does it support?

The initial release will still provide support to a RACs running on Unix/Linux based systems such as RHEL/Solaris. There is theoretical support for a typical 36-node cluster but formally, you monitor a 16-node cluster with ease.

You will, however, need an Oracle Client software installation and you can get it here although you will need to adjust your TNSNAMES.ora file, as this will be used to make a connection to your RAC Service.

Getting Spotlight on RAC

Spotlight on RAC should be available very soon. (So I heard from the Marketing Manager) and then you should be able to download an evaluation from here .