Upgrade Oracle 9i RAC to Oracle 10g RAC

January 31, 2006

This article provides the procedures for converting Oracle 9i (9.2.0.4) RAC to Oracle 10g (10.2.0.1) RAC on Red Hat Enterprise Linux 3 (RHEL3).

Oracle Database 10g and Oracle Real Application Cluster (RAC) 10g itself, boast many new and exciting features that simplify database manageability and provide higher availability and scalability. Compared to its predecessors, Oracle 10g RAC is much easier to install, use and manage, thereby reducing the total cost of ownership.

Overview

The RAC cluster comprises two Intel x86 servers running on RHEL3 (Kernel 2.4.21-27). Each node has access to a shared storage and connectivity to the public and private network.

This article is structured into the following steps:

Unless otherwise specified, you should execute all steps on both nodes.

Here's an overview of our Oracle 9i RAC environment:

And an overview of the Oracle 10g RAC environment:

Step 1: Preliminary Installation

1a. Verify software package versions

Install the required packages. Additional information can be obtained from http://download-east.oracle.com/docs/cd/B19306_01/install.102/b14203/prelinux.htm#sthref380

• binutils-2.14
• compat-db-4.0.14-5
• compat-gcc-7.3-2.96.128
• compat-gcc-c++-7.3-2.96.128
• compat-libstdc++-7.3-2.96.128
• compat-libstdc++-devel-7.3-2.96.128
• gcc-3.2
• glibc-2.3.2-95.27
• make-3.79
• openmotif-2.2.3
• setarch-1.3-1

1b. Verify kernel parameters

Verify the following kernel parameters. Additional information can be obtained from http://download-east.oracle.com/docs/cd/B19306_01/install.102/b14203/prelinux.htm#sthref418

[root@salmon1]# sysctl -a | grep shm
kernel.shmmni = 4096
kernel.shmall = 2097152
kernel.shmmax = 2147483648
[root@salmon1]# sysctl -a | grep sem
kernel.sem = 250        32000   100     128
[root@salmon1]# sysctl -a | grep -i ip_local
net.ipv4.ip_local_port_range = 1024     65000
[root@salmon1]# sysctl -a | grep -i file-max
fs.file-max = 65536
[root@salmon1]# sysctl -a | egrep "rmem_default|rmem_max|wmem_default| wmem_max"
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.core.rmem_max = 262144
net.core.wmem_max = 262144

1c. Verify users and groups

Verify that the oracle user primary group is oinstall and the secondary group is dba.

[root@salmon1]# id oracle
uid=500(oracle) gid=501(oinstall) groups=501(oinstall),500(dba)
Verify that the user nobody exists on the node. 
[root@salmon1]# id nobody
uid=99(nobody) gid=99(nobody) groups=99(nobody)

1d. Edit the oracle user environment file

Verify that the oracle user primary group is oinstall and the secondary group is dba.

[oracle@salmon1]$ more .bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi
export PATH=$PATH:$HOME/bin
export ORACLE_SID=prod1a
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/crs
export ORA_CRS_HOME=$ORACLE_BASE/product/crs
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export PATH=$ORACLE_HOME/bin:$PATH
umask 022

1e. Configure the oracle user shell limits

[root@salmon1]# more /etc/security/limits.conf
*                soft    nproc            2047
*                hard    nproc            16384
*                soft    nofile           1024
*                hard    nofile           65536
[root@salmon1]# grep pam_limits /etc/pam.d/login
 session    required     /lib/security/pam_limits.so

1f. Configure public and private network

Using the information below, make the necessary changes to network interface devices eth0 (public) and eth3 (private).

[root@salmon1]# redhat-config-network

1g. Edit the /etc/hosts file

Add the entries for Virtual IP (VIP) addresses in /etc/hosts. VIP is required in Oracle 10g RAC to increase availability by eliminating the need to wait on network timeout which may take as long as 10 minutes. Refer to http://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=264847.1 for additional information on VIP configuration.

127.0.0.1   localhost.localdomain  localhost
10.10.10.84  salmon1-priv.dbsconsult.com        salmon1-priv
10.10.10.85  salmon2-priv.dbsconsult.com        salmon2-priv
192.168.0.184  salmon1.dbsconsult.com           salmon1
192.168.0.185  salmon2.dbsconsult.com           salmon2
192.168.0.186  salmon1-vip.dbsconsult.com     salmon1-vip
192.168.0.187  salmon2-vip.dbsconsult.com     salmon2-vip

Verify the hostname and the configured network interface devices.

[root@salmon1]# hostname
salmon1.dbsconsult.com
[root@salmon1]# /sbin/ifconfig

1h. Establish user equivalence with SSH

During the Oracle Clusterware and RAC installation, the Oracle Universal Installer (OUI) has to be able to copy the software as oracle to all RAC nodes without being prompted for a password. This can be accomplished using ssh instead of rsh.

To establish user equivalence, generate the user's public and private keys as the oracle user on both nodes.

[oracle@salmon1]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_dsa):
Created directory '/home/oracle/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_dsa.
Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.
The key fingerprint is:
5d:8c:42:97:eb:42:ae:52:52:e9:59:20:2a:d3:6f:59 oracle@salmon1.dbsconsult.com
The public key on each node is copied to both nodes. Execute the following on each node.    
[oracle@salmon1]$ ssh salmon1 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
The authenticity of host 'salmon1 (192.168.0.184)' can't be established.
RSA key fingerprint is 00:d9:70:08:bc:fd:b5:e4:e3:df:a3:c7:d8:46:1e:a5.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'salmon1,192.168.0.184' (RSA) to the list of known hosts.
oracle@salmon1's password:
[oracle@salmon1]$ ssh salmon2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
The authenticity of host 'salmon2 (192.168.0.185)' can't be established.
RSA key fingerprint is 00:d9:70:08:bc:fd:b5:e4:e3:df:a3:c7:d8:46:1e:a5.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'salmon2,192.168.0.185' (RSA) to the list of known hosts.
oracle@salmon2's password:

Test the connection on every node. Verify that you are not prompted for password when you run the following the second time.

• ssh salmon1 date
• ssh salmon2 date
• ssh salmon1-priv date
• ssh salmon2-priv date
• ssh salmon1.dbsconsult.com date
• ssh salmon2.dbsconsult.com date
• ssh salmon1-priv.dbsconsult.com date
• ssh salmon2-priv.dbsconsult.com date

1i. Configure hangcheck timer kernel module

The hangcheck timer kernel module monitors the system's health and restarts a failing RAC node. It uses two parameters, hangcheck_tick (defines the system checks frequency) and hangcheck_margin (defines the maximum hang delay before a RAC node is reset), to determine if a node is failing.

The hangcheck timer should have already been configured for Oracle 9i (9.2.0.4) RAC. Verify that the following line is in /etc/rc.d/rc.local.

[root@salmon1]# grep insmod /etc/rc.d/rc.local
insmod hangcheck-timer hangcheck_tick=30 hangcheck_margin=180

1j. Create pfile

Save a copy of the pfile for the database upgrade.

SQL> create pfile='/tmp/pfile_singleinst' from spfile;