Monitoring Oracle 10g RAC with Quest Spotlight on RAC – Part I

You have heard the hype around a particular technology. It’s
cool and everyone wants a piece of it. Then it’s the post-hype era for a
technology where it is getting serious attention from the Big Players
and eventually its commoditization. Breakthrough technologies, like Oracle RAC
or Server Partitioning (fondly known as Virtualization), will eventually be
commonplace. Therefore, it is time for your DBA to monitor several oracle RACs.
The scenario could range from a RAC farm, as massive as 36-Node Oracle RAC, to
a 2-Node Oracle RAC. Being confronted with questions like these is very normal
to a curious DBA:

  • How is my cluster performing?

  • How is my Database performance as a whole?

  • Contention issues disk, io, cluster, interconnect?

  • Latency problems within HSI (High Speed Interconnects)? Are there
    poor settings? Are my NIC’s are set properly? Or is it a poorly configured RAC?

  • Inter-node communication issues? Are too many "Hot
    Blocks" being transferred across nodes due to poor database design?

  • Poor load balancing?

As you can see, the list is endless. You have migrated to a
heavy duty Oracle RAC and you see performance drop. Unfortunately, you cannot see
exactly what might be going wrong.

Introducing Spotlight on RAC (SoRAC)

It is time to say hello to SoRAC (Spotlight on RAC by Quest
Software) which is well out of beta phase
and very soon to be GA. I had the pleasure of participating in the beta program
and was rather pleased with the results (even though it was still in beta

SoRAC provides detailed diagnostic information by drilling
down individual instances and merging the appropriate calculations into one
interface called "SoRAC Homepage." And it’s all live!

Overview of Spotlight on Oracle RAC

What is this tool capable of?

It provides three classes of functionality to a RAC DBA.…

  • You get to see the physical architecture. As you see below…

    You can clearly see all of the essential
    components of a cluster: the individual instances, the interconnect and the I/O
    subsystem are displayed, including the essential data flows between them. The
    great thing about this tool is that you can paint a picture of your RAC in your
    head, enhance your understanding of RAC, and localize problems. The above
    example runs against my 2-node RHEL 4.2 Oracle 10g R2 RAC on VMware GSX Server.

  • Calculation of diagnostics and presenting them back to the
    interface as a whole. For instance, you might have latch contention on one node
    on a 6-node RAC and the aggregate of all the latches might just be fine for
    your RAC to function adequately.

  • Several RAC specific alarms such as:

    1. Uneven
      Load Distribution to the instances.

    2. Overhead
      due to cluster maintenance.

    3. High
      latency problems within the HSI (High Speed Interconnects).

    4. Excessive
      HSI communication (might point to partitioning data across nodes, for instance
      a typical hybrid , meaning OLTP (Online Transaction Processing) /DSS (Decision
      Support System) might require a data partitioning on the DSS part of that RAC

    5. Block
      Corruption / Orphaned blocks across instances.

    6. OCFS
      problems / Shared Disk (Whether on SAN/NAS) contention.

What Platform does it support?

The initial release will still provide support to a RACs
running on Unix/Linux based systems such as RHEL/Solaris. There is theoretical
support for a typical 36-node cluster but formally, you monitor a 16-node
cluster with ease.

You will, however, need an Oracle Client software
installation and you can get it here
although you will need to adjust your TNSNAMES.ora file, as this will be used
to make a connection to your RAC Service.

Getting Spotlight on RAC

Spotlight on RAC should be available very soon. (So I heard
from the Marketing Manager) and then you should be able to download an
evaluation from here .

Tarry Singh
Tarry Singh
I have been active in several industries since 1991. While working in the maritime industry I have worked for several Fortune 500 firms such as NYK, A.P. Møller-Mærsk Group. I made a career switch, emigrated, learned a new language and moved into the IT industry starting 2000. Since then I have been a Sr. DBA, (Technical) Project Manager, Sr. Consultant, Infrastructure Specialist (Clustering, Load Balancing, Networks, Databases) and (currently) Virtualization/Cloud Computing Expert and Global Sourcing in the IT industry. My deep understanding of multi-cultural issues (having worked across the globe) and international exposure has not only helped me successfully relaunch my career in a new industry but also helped me stay successful in what I do. I believe in "worknets" and "collective or swarm intelligence". As a trainer (technical as well as non-technical) I have trained staff both on national and international level. I am very devoted, perspicacious and hard working.

Latest Articles