Oracle RAC Administration - Part 3: Administering the Clusterware Components

August 17, 2006

Brief intro

Database administration can be a very daunting task. Will that change with all the cool technologies like Virtualization or market developments like Offshoring? NO WAY! You will still need a DBA or a typical "IT Versatilist," if you will, who will do the necessary tasks. Sure, things might get easier as the Oracle Enterprise Manager will be more robust. Administration tools like Spotlight on RAC will get even more advanced. However, the fact remains that you must still be able to troubleshoot your RAC. The need for these skills will not go away. Ok so in the future, (as I see it happening) maybe your RAC will float in a "Secure Virtual Vault" somewhere on the web, but someone will need to watch it. On the other hand, it could be the Extended RAC (RAC nodes geographically separated) thus providing real High Availability. Administration is administration--you will need it no matter what.

So here, we will take good look at the administration aspects of "Oracle Clusterware Components"

What is Clusterware composed of?

The Oracle Clusterware is comprised primarily of two components: the voting disk and the OCR (Oracle Cluster Registry). The voting disk is nothing but a file that contains and manages information of all the node memberships and the OCR is a file that manages the cluster and RAC configuration. Let's take a quick look at administering the voting disks and the OCR.

Administering voting disks

Backup and Recovery:

First, let's look at backing up the voting disks by running the following command:

dd if=voting_disk_name of=backup_file_name

This operation needs to be performed on all voting disks. Here, clearly you see that the if (input file) is the source file (replace the voting_disk_name with your voting disk) and the of (output file) is the destination backup file containing all information of the voting disk contents. You can do a lot of stuff with dd, such as splitting a file (using bs = block size, I just did it yesterday, to restore my Linux hacked iPAQ with WinCE), converting case sensitivity. Type dd –help for more information. Running the command with the names of the files reversed will help you recover your voting disk file(s).

dd if=backup_file_name of=voting_disk_name

You can use the ocopy command in Windows environments or use the crsctl commands to copy and administer the files. Also, note that if you have multiple voting disks, which are not abnormal to have, you can use the crsctl command to add and delete the voting disks. For instance:

crsctl delete css votedisk path

Here you delete the disk and the path, which is the complete path of the location of the file, and below you add your new or backup files by doing the following:

crsctl add css votedisk path

This way you can either statically or dynamically add or remove your voting disks in your RAC.

You must, however, note that if your cluster is down, then you can use the -force option

crsctl add css votedisk path -force

to modify the voting disk configuration. This way you don’t end up interfering with other Clusterware daemons. Using it in your active configuration may corrupt your configuration.

Administering Oracle Cluster Registry

OCR contains information pertaining to instance-to-node mapping, node list and resource profiles for customized applications in your Clusterware. Let's take a look at some of the following administrative tasks:

Adding, Replacing, Managing & Removing OCR

You can’t have more than two OCRs.You can add an OCR either after an upgrade or after installing the RAC installation. If you already mirror the OCR, then you do not need to add an OCR location; Oracle does that automatically. If your OCR is on the network, do create a target file before performing any tasks! In addition, you must be logged in as the root user to run the ocrconfig tool.

So if you created a single OCR, then add by doing the following. :

ocrconfig -replace ocr destination_file or disk

Here, do the following to add a mirror file.

ocrconfig -replace ocrmirror destination_file or disk

Replacing OCR works the same way. Do however check that the file to be replaced is online, the Clusterware is running on that particular node and that if it is on a CFS or on the network.

To replace OCR do the following:

ocrconfig -replace ocr destination_file or disk

and to replace the OCR mirror:

ocrconfig -replace ocrmirror destination_file or disk

Repairing the OCR comes in handy if the nodes where you may have been working are shutdown. Typing ocrconfig -repair brings those nodes back on line. More specifically, you can do this (to repair your OCR mirror):

ocrconfig –repair ocrmirror device_name 

This command repairs the OCR configuration locally. Also, note that you cannot do this on running Clusterware daemons.

To remove an OCR, you need to have at least one OCR online. You may want to do this to reduce overhead or for other storage reasons, such as stopping a mirror to move it to SAN, RAID etc. Carry out the following steps :

• Check if at least one OCR is online
• Remove the OCR or OCR mirror

ocrconfig -replace ocr OR ocrconfig -replace ocrmirror

Using the –backuploc option allows you to save/move the OCR file to a safe location. Type ocrconfig –option to see all the commands.

Conclusion:

The Oracle Clusterware has an HA (High Availability) framework that provides a robust infrastructure to manage any application. The Oracle Clusterware daemons make sure that all applications startup during system startup. Failed applications are started automatically to maintain the HA aspect of the RAC cluster. It is possible to configure all of the administrative aspects (like monitoring frequencies, startup, shutdown) of the RAC cluster.

This article examined the two most important components of Oracle Clusterware. In our next article, we will cover the backup and recovery administration of OCR using OCR backup files.

» See All Articles by Columnist Tarry Singh