Adding a Node to Oracle RAC 11g Release 2

Oracle Real Application Clusters (Oracle RAC) delivers the ability to horizontally scale out your database system to meet increasing business demands by utilizing inexpensive commodity server architectures. Nodes can be added to an existing pool of RAC nodes to balance computing resources and to achieve Service Level Agreements (SLAs).

In this guide, we will demonstrate a step-by-step procedure of adding a third node to our existing two-node Oracle RAC 11g.

Downloads for this guide are available at My Oracle Support:

  • Oracle Database 11g Release 2 (11.2.0.2.0) for Linux x86-64 (p10098816_112020_Linux-x86-64_1of7 and p10098816_112020_Linux-x86-64_2of7)
  • Oracle Database 11g Release 2 Grid Infrastructure (11.2.0.2.0) for Linux x86-64 (p10098816_112020_Linux-x86-64_3of7)

Overview

Our RAC environment set up comprises of two Intel x86 servers running on Oracle Enterprise Linux 5. The Operating System has been installed and patched on both nodes and each has access to a shared storage and connectivity to the public and private network.

This guide is structured into the following steps:

  1. Preliminary Installation
  2. Install the Oracle Database 11g Release 2 Grid Infrastructure software on the third node
  3. Install the Oracle Database 11g Release 2 software on the third node
  4. Add a third database instance on the third node

An overview of the two-node Oracle RAC environment:

Software owner accounts and locations:

Shared File storage:

IP Addresses:

And an overview of the third node:

Software owner accounts and locations:

Shared File storage:

IP Addresses:

For educational purposes, we will use the same oracle Unix account and group to install the Grid Infrastructure and database software. Oracle recommends using different Unix users and groups for separation of duties.

1. Preliminary Installation

This section lists the setup necessary before starting the Grid Infrastructure software installation.

1.1 Software and Hardware requirements

The required OS patches have been applied to the third node and connectivity to the database shared storage, public and private network have been configured. Review the software and hardware requirements listed here.

1.2 Configure GNS (Grid Naming Service) Virtual IP Address

The Grid Naming Service (GNS) is a part of the Grid Plug and Play feature (GPnP) which allows the Oracle Clusterware to dynamically allocate virtual IP address for each node through DHCP service running on the public network. All name resolution requests for the cluster within a subdomain delegated by the DNS are handed off to GNS using multicast Domain Name Service (mDNS) included within the Oracle Clusterware. Using GNS eliminates the need for managing IP addresses and name resolution and is especially advantageous in a dynamic cluster environment where nodes are often added or removed.

Prior to the Grid Infrastructure software installation, we have configured a static IP address for the GNS virtual IP and defined ayu.vcconsult.com as the subdomain to delegate in DNS.

Refer to the Oracle Grid Infrastructure Installation Guidefor additional information about GNS.

1.3 SCAN (Single Client Access Name)

SCAN simplifies client management by providing a single point of access for clients to connect to any database in the cluster. In the event when nodes were added or removed from the cluster, clients that use SCAN to access the cluster do not have to make any explicit changes to the client settings.

SCAN is resolvable by either of the following methods:

  1. Domain Name Service (DNS): At least one IP address has to be defined in DNS for the SCAN. The recommended approach is to create a round robin SCAN resolvable to a maximum of three different addresses for high availability and scalability.
  2. Grid Naming Service (GNS): Three IP addresses are automatically acquired from the DHCP service and the GNS provides name resolution for the SCAN.

SCAN addresses, virtual IP addresses, and public IP addresses must all be on the same subnet.

Refer to the following whitepaperfor additional information on SCAN.

In this installment, we will use GNS to resolve the SCAN addresses.


oracle@ayu1-> dig ayu-cluster-scan.ayu.vcconsult.com

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5 <<>> ayu-cluster-scan.ayu.vcconsult.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode:="" query="" status:="" noerror="" id:="" 37381="" flags:="" qr="" aa="" rd="" ra="" query:="" 1="" answer:="" 3="" authority:="" additional:="" 0="" question="" section:="" ayu-cluster-scan="" ayu="" vcconsult="" com="" in="" a="" answer="" 3600="" 192="" 168="" 235="" 234="" 233="" authority="" 86400="" ns="" gns="" time:="" msec="" server:="" 13="" 53="" when:="" sun="" dec="" 12="" 14:18:44="" 2010="" msg="" size="" rcvd:="" 126="" code="">

1.4 Configure Oracle ASM (Automatic Storage Management) shared storage

In our environment, we have created four ASM disks; three of which were dedicated for storing the OCR, Voting disk, datafiles and online redo logs and one for the Fast Recovery Area. Verify that the ASM disks are visible on the new node.


[root@ayu1 ~]# oracleasm listdisks
VOL1
VOL2
VOL3
VOL4

[root@ayu2 ~]# oracleasm listdisks
VOL1
VOL2
VOL3
VOL4

[root@ayu3 ~]# oracleasm listdisks
VOL1
VOL2
VOL3
VOL4

Oracle ASM storage configuration is described here.

1.5 CTSS (Cluster Time Synchronization Service)

CTSS is configured automatically as a part of the Grid Infrastructure installation to synchronize the clocks on all the nodes. As an alternative, you can manually configure NTP (Network Time Protocol) on all the nodes. We will synchronize the time across the cluster nodes using CTSS.

1.6 Modify /etc/hosts on all nodes

Add the public and private IP addresses of the third node in /etc/hosts. Verify that the /etc/hosts file on each node has the same entries.


oracle@ayu1-> more /etc/hosts
127.0.0.1       localhost.localdomain   	 localhost
192.168.1.121   ayu1.vcconsult.com              ayu1
192.168.1.122   ayu2.vcconsult.com              ayu2
192.168.1.123   ayu3.vcconsult.com              ayu3
10.10.10.121    ayu1-priv.vcconsult.com         ayu1-priv
10.10.10.122    ayu2-priv.vcconsult.com         ayu2-priv
10.10.10.123    ayu3-priv.vcconsult.com         ayu3-priv

1.7 Establish user equivalence with SSH


oracle@ayu3-> pwd
/home/oracle/.ssh
oracle@ayu3-> ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_rsa.
Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
f1:64:97:c2:ae:d2:12:e8:82:3a:07:34:a0:ab:71:01 oracle@ayu3

oracle@ayu3-> ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_dsa.
Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.
The key fingerprint is:
33:e7:43:9b:b8:76:f0:9e:7a:e3:f0:43:c8:04:af:53 oracle@ayu3

On ayu1: 
oracle@ayu1-> pwd
/home/oracle/.ssh
oracle@ayu1-> ssh ayu3 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
The authenticity of host 'ayu3 (192.168.1.123)' can't be established.
RSA key fingerprint is 24:04:fc:51:ed:cb:03:44:2d:90:8f:91:4a:7b:7e:4f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ayu3,192.168.1.123' (RSA) to the list of known hosts.
oracle@ayu3's password:

oracle@ayu1-> scp ~/.ssh/authorized_keys ayu2:~/.ssh/authorized_keys
authorized_keys                                                                     100% 3760     3.7KB/s   00:00

oracle@ayu1-> scp ~/.ssh/authorized_keys ayu3:~/.ssh/authorized_keys
oracle@ayu3's password:
authorized_keys                                                                     100% 3760     3.7KB/s   00:00

Test the connections on each node. Verify that you are not prompted for password.

1.8 Verify the integrity of the cluster and the new node


oracle@ayu1-> more .profile_grid
export EDITOR=vi
export ORACLE_UNQNAME=qdb
export ORACLE_SID=+ASM1
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/11.2.0.2/grid
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export PATH=$ORACLE_HOME/bin:/bin:/usr/bin:/usr/sbin:/usr/local/bin
umask 022	

oracle@ayu1-> . ./.profile_grid
oracle@ayu1-> which cluvfy
/u01/app/11.2.0.2/grid/bin/cluvfy
oracle@ayu1-> cluvfy stage -pre nodeadd -n ayu3 -verbose

Performing pre-checks for node addition

Checking node reachability...

Check: Node reachability from node "ayu1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  ayu3                                  yes
Result: Node reachability check passed from node "ayu1"


Checking user equivalence...

Check: User equivalence for user "oracle"
  Node Name                             Comment
  ------------------------------------  ------------------------
  ayu3                                  passed
Result: User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  ayu3          passed
  ayu2          passed
  ayu1          passed

Verification of the hosts config file successful
Code

<< snipped for clarity >>



Checking OCR integrity...

OCR integrity check passed

Checking Oracle Cluster Voting Disk configuration...

ERROR:
PRVF-5449 : Check of Voting Disk location "ORCL:VOL1(ORCL:VOL1)" failed on the following nodes:

        ayu3:No such file or directory

ERROR:
PRVF-5449 : Check of Voting Disk location "ORCL:VOL2(ORCL:VOL2)" failed on the following nodes:

        ayu3:No such file or directory

ERROR:
PRVF-5449 : Check of Voting Disk location "ORCL:VOL3(ORCL:VOL3)" failed on the following nodes:

        ayu3:No such file or directory

PRVF-5431 : Oracle Cluster Voting Disk configuration check failed
Check: Time zone consistency
Result: Time zone consistency check passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes
No NTP Daemons or Services were found to be running

Result: Clock synchronization check using Network Time Protocol(NTP) passed


Checking to make sure user "oracle" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  ayu3          does not exist            passed
  ayu1          does not exist            passed
Result: User "oracle" is not part of "root" group. Check passed
Checking consistency of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
File "/etc/resolv.conf" does not have both domain and search entries defined
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
search entry in file "/etc/resolv.conf" is consistent across nodes
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
All nodes have one search entry defined in file "/etc/resolv.conf"
Checking all nodes to make sure that search entry is "vcconsult.com" as found on node "ayu1"
All nodes of the cluster have same value for 'search'
Checking DNS response time for an unreachable node
  Node Name                             Status
  ------------------------------------  ------------------------
  ayu1                                  passed
  ayu3                                  passed
The DNS response time for an unreachable node is within acceptable limit on all nodes

File "/etc/resolv.conf" is consistent across nodes


Checking GNS integrity...
Checking if the GNS subdomain name is valid...
The GNS subdomain name "ayu.vcconsult.com" is a valid domain name
Checking if the GNS VIP is a valid address...
GNS VIP "192.168.1.200" resolves to a valid IP address
Checking the status of GNS VIP...
PRVF-5229 : GNS VIP is active before Clusterware installation
Checking if FDQN names for domain "ayu.vcconsult.com" are reachable

GNS integrity check passed

Pre-check for node addition was unsuccessful on all the nodes.

You can ignore the PRVF-5449 message if you have already configured and verified that the ASM shared disks are visible on the third node. The error was a result of having the voting disks stored in Oracle ASM storage, a new feature of Oracle 11g Release 2.

2. Install the Oracle Database 11g Release 2 Grid Infrastructure software on the third node

You will execute the addNode.sh script to install the Oracle Grid Infrastructure software on the new node. The script cannot proceed unless all the pre-requisites checks are successful. In our case, since we are getting the PRVF-5449 messages, we will set the IGNORE_PREADDNODE_CHECKS variable to bypass the node addition pre-check.


Execute the addNode.sh script on ayu1 as oracle:
oracle@ayu1-> export IGNORE_PREADDNODE_CHECKS=Y
oracle@ayu1-> . ./.profile_grid
oracle@ayu1-> cd $ORACLE_HOME/oui/bin
oracle@ayu1-> pwd
/u01/app/11.2.0.2/grid/oui/bin
oracle@ayu1-> ./addNode.sh "CLUSTER_NEW_NODES={ayu3}" 
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 1154 MB    Passed
Checking monitor: must be configured to display at least 256 colors.    Actual 16777216    Passed
Oracle Universal Installer, Version 11.2.0.2.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.


Performing tests to see whether nodes ayu2,ayu3 are available
............................................................... 100% Done.

.
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
   Source: /u01/app/11.2.0.2/grid
   New Nodes
Space Requirements
   New Nodes
      ayu3
         /u01: Required 3.55GB : Available 16.25GB
Installed Products
   Product Names
      Oracle Grid Infrastructure 11.2.0.2.0
      Sun JDK 1.5.0.24.08
      Installer SDK Component 11.2.0.2.0
      Oracle One-Off Patch Installer 11.2.0.0.2
      Oracle Universal Installer 11.2.0.2.0
      Oracle USM Deconfiguration 11.2.0.2.0
      Oracle Configuration Manager Deconfiguration 10.3.1.0.0
      Enterprise Manager Common Core Files 10.2.0.4.3
      Oracle DBCA Deconfiguration 11.2.0.2.0
      Oracle RAC Deconfiguration 11.2.0.2.0
      Oracle Quality of Service Management (Server) 11.2.0.2.0
      Installation Plugin Files 11.2.0.2.0
      Universal Storage Manager Files 11.2.0.2.0
      Oracle Text Required Support Files 11.2.0.2.0
      Automatic Storage Management Assistant 11.2.0.2.0
      Oracle Database 11g Multimedia Files 11.2.0.2.0
      Oracle Multimedia Java Advanced Imaging 11.2.0.2.0
      Oracle Globalization Support 11.2.0.2.0
      Oracle Multimedia Locator RDBMS Files 11.2.0.2.0
      Oracle Core Required Support Files 11.2.0.2.0
      Bali Share 1.1.18.0.0
      Oracle Database Deconfiguration 11.2.0.2.0
      Oracle Quality of Service Management (Client) 11.2.0.2.0
      Expat libraries 2.0.1.0.1
      Oracle Containers for Java 11.2.0.2.0
      Perl Modules 5.10.0.0.1
      Secure Socket Layer 11.2.0.2.0
      Oracle JDBC/OCI Instant Client 11.2.0.2.0
      Oracle Multimedia Client Option 11.2.0.2.0
      LDAP Required Support Files 11.2.0.2.0
      Character Set Migration Utility 11.2.0.2.0
      Perl Interpreter 5.10.0.0.1
      PL/SQL Embedded Gateway 11.2.0.2.0
      OLAP SQL Scripts 11.2.0.2.0
      Database SQL Scripts 11.2.0.2.0
      Oracle Extended Windowing Toolkit 3.4.47.0.0
      SSL Required Support Files for InstantClient 11.2.0.2.0
      SQL*Plus Files for Instant Client 11.2.0.2.0
      Oracle Net Required Support Files 11.2.0.2.0
      Oracle Database User Interface 2.2.13.0.0
      RDBMS Required Support Files for Instant Client 11.2.0.2.0
      RDBMS Required Support Files Runtime 11.2.0.2.0
      XML Parser for Java 11.2.0.2.0
      Oracle Security Developer Tools 11.2.0.2.0
      Oracle Wallet Manager 11.2.0.2.0
      Enterprise Manager plugin Common Files 11.2.0.2.0
      Platform Required Support Files 11.2.0.2.0
      Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
      RDBMS Required Support Files 11.2.0.2.0
      Oracle Ice Browser 5.2.3.6.0
      Oracle Help For Java 4.2.9.0.0
      Enterprise Manager Common Files 10.2.0.4.3
      Deinstallation Tool 11.2.0.2.0
      Oracle Java Client 11.2.0.2.0
      Cluster Verification Utility Files 11.2.0.2.0
      Oracle Notification Service (eONS) 11.2.0.2.0
      Oracle LDAP administration 11.2.0.2.0
      Cluster Verification Utility Common Files 11.2.0.2.0
      Oracle Clusterware RDBMS Files 11.2.0.2.0
      Oracle Locale Builder 11.2.0.2.0
      Oracle Globalization Support 11.2.0.2.0
      Buildtools Common Files 11.2.0.2.0
      Oracle RAC Required Support Files-HAS 11.2.0.2.0
      SQL*Plus Required Support Files 11.2.0.2.0
      XDK Required Support Files 11.2.0.2.0
      Agent Required Support Files 10.2.0.4.3
      Parser Generator Required Support Files 11.2.0.2.0
      Precompiler Required Support Files 11.2.0.2.0
      Installation Common Files 11.2.0.2.0
      Required Support Files 11.2.0.2.0
      Oracle JDBC/THIN Interfaces 11.2.0.2.0
      Oracle Multimedia Locator 11.2.0.2.0
      Oracle Multimedia 11.2.0.2.0
      HAS Common Files 11.2.0.2.0
      Assistant Common Files 11.2.0.2.0
      PL/SQL 11.2.0.2.0
      HAS Files for DB 11.2.0.2.0
      Oracle Recovery Manager 11.2.0.2.0
      Oracle Database Utilities 11.2.0.2.0
      Oracle Notification Service 11.2.0.2.0
      SQL*Plus 11.2.0.2.0
      Oracle Netca Client 11.2.0.2.0
      Oracle Net 11.2.0.2.0
      Oracle JVM 11.2.0.2.0
      Oracle Internet Directory Client 11.2.0.2.0
      Oracle Net Listener 11.2.0.2.0
      Cluster Ready Services Files 11.2.0.2.0
      Oracle Database 11g 11.2.0.2.0
-----------------------------------------------------------------------------


Instantiating scripts for add node (Sunday, December 12, 2010 11:48:21 AM EST)
.                                                                 1% Done.
Instantiation of add node scripts complete

Copying to remote nodes (Sunday, December 12, 2010 11:48:30 AM EST)
...............................................................................................                                 96% Done.
Home copied to new nodes

Saving inventory on nodes (Sunday, December 12, 2010 11:53:16 AM EST)
.                                                               100% Done.
Save inventory complete
WARNING:A new inventory has been created on one or more nodes in this session. However, it has not yet been registered as the central inventory of this system.
To register the new inventory please run the script at '/u01/app/oraInventory/orainstRoot.sh' with root privileges on nodes 'ayu3'.
If you do not register the inventory, you may not be able to update or patch the products you installed.
The following configuration scripts need to be executed as the "root" user in each cluster node.
/u01/app/oraInventory/orainstRoot.sh #On nodes ayu3
/u01/app/11.2.0.2/grid/root.sh #On nodes ayu3
To execute the configuration scripts:
    1. Open a terminal window
    2. Log in as "root"
    3. Run the scripts in each cluster node

The Cluster Node Addition of /u01/app/11.2.0.2/grid was successful.
Please check '/tmp/silentInstall.log' for more details.

Execute the orainstRoot.sh script on the third node as root:
[root@ayu3 ~]# /u01/app/oraInventory/orainstRoot.sh
Creating the Oracle inventory pointer file (/etc/oraInst.loc)
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

Execute the root.sh script on the third node as root:
[root@ayu3 ~]# /u01/app/11.2.0.2/grid/root.sh
Running Oracle 11g root script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/11.2.0.2/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0.2/grid/crs/install/crsconfig            _params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
OLR initialization - successful
Adding daemon to inittab
ACFS-9200: Supported
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node ayu2, number 2, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Preparing packages for installation...
cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Perform post-installation checks
oracle@ayu3-> pwd
/u01/app/11.2.0.2/grid/bin
oracle@ayu3-> ./crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.DATA.dg    ora....up.type ONLINE    ONLINE    ayu1
ora....ER.lsnr ora....er.type ONLINE    ONLINE    ayu1
ora....N1.lsnr ora....er.type ONLINE    ONLINE    ayu2
ora....N2.lsnr ora....er.type ONLINE    ONLINE    ayu3
ora....N3.lsnr ora....er.type ONLINE    ONLINE    ayu1
ora.asm        ora.asm.type   ONLINE    ONLINE    ayu1
ora....SM1.asm application    ONLINE    ONLINE    ayu1
ora....U1.lsnr application    ONLINE    ONLINE    ayu1
ora.ayu1.gsd   application    OFFLINE   OFFLINE
ora.ayu1.ons   application    ONLINE    ONLINE    ayu1
ora.ayu1.vip   ora....t1.type ONLINE    ONLINE    ayu1
ora....SM2.asm application    ONLINE    ONLINE    ayu2
ora....U2.lsnr application    ONLINE    ONLINE    ayu2
ora.ayu2.gsd   application    OFFLINE   OFFLINE
ora.ayu2.ons   application    ONLINE    ONLINE    ayu2
ora.ayu2.vip   ora....t1.type ONLINE    ONLINE    ayu2
ora....SM3.asm application    ONLINE    ONLINE    ayu3
ora....U3.lsnr application    ONLINE    ONLINE    ayu3
ora.ayu3.gsd   application    OFFLINE   OFFLINE
ora.ayu3.ons   application    ONLINE    ONLINE    ayu3
ora.ayu3.vip   ora....t1.type ONLINE    ONLINE    ayu3
ora.cvu        ora.cvu.type   ONLINE    ONLINE    ayu1
ora.gns        ora.gns.type   ONLINE    ONLINE    ayu1
ora.gns.vip    ora....ip.type ONLINE    ONLINE    ayu1
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora....network ora....rk.type ONLINE    ONLINE    ayu1
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    ayu1
ora.ons        ora.ons.type   ONLINE    ONLINE    ayu1
ora.qdb.db     ora....se.type ONLINE    ONLINE    ayu1
ora....ry.acfs ora....fs.type ONLINE    ONLINE    ayu1
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    ayu2
ora.scan2.vip  ora....ip.type ONLINE    ONLINE    ayu3
ora.scan3.vip  ora....ip.type ONLINE    ONLINE    ayu1



oracle@ayu3-> ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       ayu1
               ONLINE  ONLINE       ayu2
               ONLINE  ONLINE       ayu3
ora.LISTENER.lsnr
               ONLINE  ONLINE       ayu1
               ONLINE  ONLINE       ayu2
               ONLINE  ONLINE       ayu3
ora.asm
               ONLINE  ONLINE       ayu1                     Started
               ONLINE  ONLINE       ayu2
               ONLINE  ONLINE       ayu3
ora.gsd
               OFFLINE OFFLINE      ayu1
               OFFLINE OFFLINE      ayu2
               OFFLINE OFFLINE      ayu3
ora.net1.network
               ONLINE  ONLINE       ayu1
               ONLINE  ONLINE       ayu2
               ONLINE  ONLINE       ayu3
ora.ons
               ONLINE  ONLINE       ayu1
               ONLINE  ONLINE       ayu2
               ONLINE  ONLINE       ayu3
ora.registry.acfs
               ONLINE  ONLINE       ayu1
               ONLINE  ONLINE       ayu2
               ONLINE  ONLINE       ayu3
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       ayu2
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       ayu3
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       ayu1
ora.ayu1.vip
      1        ONLINE  ONLINE       ayu1
ora.ayu2.vip
      1        ONLINE  ONLINE       ayu2
ora.ayu3.vip
      1        ONLINE  ONLINE       ayu3
ora.cvu
      1        ONLINE  ONLINE       ayu1
ora.gns
      1        ONLINE  ONLINE       ayu1
ora.gns.vip
      1        ONLINE  ONLINE       ayu1
ora.oc4j
      1        ONLINE  ONLINE       ayu1
ora.qdb.db
      1        ONLINE  ONLINE       ayu1                     Open
      2        ONLINE  ONLINE       ayu2                     Open
ora.scan1.vip
      1        ONLINE  ONLINE       ayu2
ora.scan2.vip
      1        ONLINE  ONLINE       ayu3
ora.scan3.vip
      1        ONLINE  ONLINE       ayu1
                             

oracle@ayu1-> srvctl status nodeapps
VIP 192.168.1.232 is enabled
VIP 192.168.1.232 is running on node: ayu1
VIP 192.168.1.236 is enabled
VIP 192.168.1.236 is running on node: ayu2
VIP 192.168.1.237 is enabled
VIP 192.168.1.237 is running on node: ayu3
Network is enabled
Network is running on node: ayu1
Network is running on node: ayu2
Network is running on node: ayu3
GSD is disabled
GSD is not running on node: ayu1
GSD is not running on node: ayu2
GSD is not running on node: ayu3
ONS is enabled
ONS daemon is running on node: ayu1
ONS daemon is running on node: ayu2
ONS daemon is running on node: ayu3


oracle@ayu1-> srvctl config nodeapps
Network exists: 1/192.168.1.0/255.255.255.0/eth0, type dhcp
VIP exists: /192.168.1.232/192.168.1.232/192.168.1.0/255.255.255.0/eth0, hosting node ayu1
VIP exists: /192.168.1.236/192.168.1.236/192.168.1.0/255.255.255.0/eth0, hosting node ayu2
VIP exists: /192.168.1.237/192.168.1.237/192.168.1.0/255.255.255.0/eth0, hosting node ayu3
GSD exists
ONS exists: Local port 6100, remote port 6200, EM port 2016


oracle@ayu3-> crsctl check cluster -all
**************************************************************
ayu1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
ayu2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
ayu3:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************


oracle@ayu3-> srvctl config gns
GNS is enabled.
oracle@ayu3->
oracle@ayu3-> srvctl status gns
GNS is running on node ayu1.
GNS is enabled on node ayu1.

oracle@ayu3-> srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is running on node ayu2
SCAN VIP scan2 is enabled
SCAN VIP scan2 is running on node ayu3
SCAN VIP scan3 is enabled
SCAN VIP scan3 is running on node ayu1

oracle@ayu3-> srvctl config scan
SCAN name: ayu-cluster-scan.ayu.vcconsult.com, Network: 1/192.168.1.0/255.255.255.0/eth0
SCAN VIP name: scan1, IP: /192.168.1.233/192.168.1.233
SCAN VIP name: scan2, IP: /192.168.1.234/192.168.1.234
SCAN VIP name: scan3, IP: /192.168.1.235/192.168.1.235

oracle@ayu3-> srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node ayu2
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node ayu3
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node ayu1

oracle@ayu3-> srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521


oracle@ayu1-> srvctl status database -d qdb
Instance qdb1 is running on node ayu1
Instance qdb2 is running on node ayu2        

oracle@ayu1-> srvctl config database -d qdb
Database unique name: qdb
Database name: qdb
Oracle home: /u01/app/oracle/product/11.2.0.2/dbhome_1
Oracle user: oracle
Spfile: +DATA/qdb/spfileqdb.ora
Domain: vcconsult.com
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: qdb
Database instances: qdb1,qdb2
Disk Groups: DATA
Mount point paths:
Services:
Type: RAC
Database is administrator managed

You have completed the installation of the Oracle Grid Infrastructure on the new node!

3. Install the Oracle Database 11g Release 2 software on the third node

Similarly, you will set the IGNORE_PREADDNODE_CHECKS and execute the addNode.sh script to install the Oracle Database software on the new node.


Execute the addNode.sh script on ayu1 as oracle:
oracle@ayu1-> more .profile
export EDITOR=vi
export ORACLE_UNQNAME=qdb
export ORACLE_SID=qdb1
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0.2/dbhome_1
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export PATH=$ORACLE_HOME/bin:/bin:/usr/bin:/usr/sbin:/usr/local/bin
umask 022	

oracle@ayu1-> export IGNORE_PREADDNODE_CHECKS=Y
oracle@ayu1-> . ./.profile
oracle@ayu1-> cd $ORACLE_HOME/oui/bin
oracle@ayu1-> pwd
/u01/app/oracle/product/11.2.0.2/dbhome_1/oui/bin
oracle@ayu1-> ./addNode.sh "CLUSTER_NEW_NODES={ayu3}"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 1175 MB    Passed
Checking monitor: must be configured to display at least 256 colors.    Actual 16777216    Passed
Oracle Universal Installer, Version 11.2.0.2.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.


Performing tests to see whether nodes ayu2,ayu3 are available
............................................................... 100% Done.

.
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
   Source: /u01/app/oracle/product/11.2.0.2/dbhome_1
   New Nodes
Space Requirements
   New Nodes
      ayu3
         /u01: Required 3.96GB : Available 13.45GB
Installed Products
   Product Names
      Oracle Database 11g 11.2.0.2.0
      Sun JDK 1.5.0.24.08
      Installer SDK Component 11.2.0.2.0
      Oracle One-Off Patch Installer 11.2.0.0.2
      Oracle Universal Installer 11.2.0.2.0
      Oracle USM Deconfiguration 11.2.0.2.0
      Oracle Configuration Manager Deconfiguration 10.3.1.0.0
      Oracle DBCA Deconfiguration 11.2.0.2.0
      Oracle RAC Deconfiguration 11.2.0.2.0
      Oracle Database Deconfiguration 11.2.0.2.0
      Oracle Configuration Manager Client 10.3.2.1.0
      Oracle Configuration Manager 10.3.3.1.1
      Oracle ODBC Driverfor Instant Client 11.2.0.2.0
      LDAP Required Support Files 11.2.0.2.0
      SSL Required Support Files for InstantClient 11.2.0.2.0
      Bali Share 1.1.18.0.0
      Oracle Extended Windowing Toolkit 3.4.47.0.0
      Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
      Oracle Real Application Testing 11.2.0.2.0
      Oracle Database Vault J2EE Application 11.2.0.2.0
      Oracle Label Security 11.2.0.2.0
      Oracle Data Mining RDBMS Files 11.2.0.2.0
      Oracle OLAP RDBMS Files 11.2.0.2.0
      Oracle OLAP API 11.2.0.2.0
      Platform Required Support Files 11.2.0.2.0
      Oracle Database Vault option 11.2.0.2.0
      Oracle RAC Required Support Files-HAS 11.2.0.2.0
      SQL*Plus Required Support Files 11.2.0.2.0
      Oracle Display Fonts 9.0.2.0.0
      Oracle Ice Browser 5.2.3.6.0
      Oracle JDBC Server Support Package 11.2.0.2.0
      Oracle SQL Developer 11.2.0.2.0
      Oracle Application Express 11.2.0.2.0
      XDK Required Support Files 11.2.0.2.0
      RDBMS Required Support Files for Instant Client 11.2.0.2.0
      SQLJ Runtime 11.2.0.2.0
      Database Workspace Manager 11.2.0.2.0
      RDBMS Required Support Files Runtime 11.2.0.2.0
      Oracle Globalization Support 11.2.0.2.0
      Exadata Storage Server 11.2.0.1.0
      Provisioning Advisor Framework 10.2.0.4.3
      Enterprise Manager Database Plugin -- Repository Support 11.2.0.2.0
      Enterprise Manager Repository Core Files 10.2.0.4.3
      Enterprise Manager Database Plugin -- Agent Support 11.2.0.2.0
      Enterprise Manager Grid Control Core Files 10.2.0.4.3
      Enterprise Manager Common Core Files 10.2.0.4.3
      Enterprise Manager Agent Core Files 10.2.0.4.3
      RDBMS Required Support Files 11.2.0.2.0
      regexp 2.1.9.0.0
      Agent Required Support Files 10.2.0.4.3
      Oracle 11g Warehouse Builder Required Files 11.2.0.2.0
      Oracle Notification Service (eONS) 11.2.0.2.0
      Oracle Text Required Support Files 11.2.0.2.0
      Parser Generator Required Support Files 11.2.0.2.0
      Oracle Database 11g Multimedia Files 11.2.0.2.0
      Oracle Multimedia Java Advanced Imaging 11.2.0.2.0
      Oracle Multimedia Annotator 11.2.0.2.0
      Oracle JDBC/OCI Instant Client 11.2.0.2.0
      Oracle Multimedia Locator RDBMS Files 11.2.0.2.0
      Precompiler Required Support Files 11.2.0.2.0
      Oracle Core Required Support Files 11.2.0.2.0
      Sample Schema Data 11.2.0.2.0
      Oracle Starter Database 11.2.0.2.0
      Oracle Message Gateway Common Files 11.2.0.2.0
      Oracle XML Query 11.2.0.2.0
      XML Parser for Oracle JVM 11.2.0.2.0
      Oracle Help For Java 4.2.9.0.0
      Installation Plugin Files 11.2.0.2.0
      Enterprise Manager Common Files 10.2.0.4.3
      Expat libraries 2.0.1.0.1
      Deinstallation Tool 11.2.0.2.0
      Oracle Quality of Service Management (Client) 11.2.0.2.0
      Perl Modules 5.10.0.0.1
      JAccelerator (COMPANION) 11.2.0.2.0
      Oracle Containers for Java 11.2.0.2.0
      Perl Interpreter 5.10.0.0.1
      Oracle Net Required Support Files 11.2.0.2.0
      Secure Socket Layer 11.2.0.2.0
      Oracle Universal Connection Pool 11.2.0.2.0
      Oracle JDBC/THIN Interfaces 11.2.0.2.0
      Oracle Multimedia Client Option 11.2.0.2.0
      Oracle Java Client 11.2.0.2.0
      Character Set Migration Utility 11.2.0.2.0
      Oracle Code Editor 1.2.1.0.0I
      PL/SQL Embedded Gateway 11.2.0.2.0
      OLAP SQL Scripts 11.2.0.2.0
      Database SQL Scripts 11.2.0.2.0
      Oracle Locale Builder 11.2.0.2.0
      Oracle Globalization Support 11.2.0.2.0
      SQL*Plus Files for Instant Client 11.2.0.2.0
      Required Support Files 11.2.0.2.0
      Oracle Database User Interface 2.2.13.0.0
      Oracle ODBC Driver 11.2.0.2.0
      Oracle Notification Service 11.2.0.2.0
      XML Parser for Java 11.2.0.2.0
      Oracle Security Developer Tools 11.2.0.2.0
      Oracle Wallet Manager 11.2.0.2.0
      Cluster Verification Utility Common Files 11.2.0.2.0
      Oracle Clusterware RDBMS Files 11.2.0.2.0
      Oracle UIX 2.2.24.6.0
      Enterprise Manager plugin Common Files 11.2.0.2.0
      HAS Common Files 11.2.0.2.0
      Precompiler Common Files 11.2.0.2.0
      Installation Common Files 11.2.0.2.0
      Oracle Help for the  Web 2.0.14.0.0
      Oracle LDAP administration 11.2.0.2.0
      Buildtools Common Files 11.2.0.2.0
      Assistant Common Files 11.2.0.2.0
      Oracle Recovery Manager 11.2.0.2.0
      PL/SQL 11.2.0.2.0
      Generic Connectivity Common Files 11.2.0.2.0
      Oracle Database Gateway for ODBC 11.2.0.2.0
      Oracle Programmer 11.2.0.2.0
      Oracle Database Utilities 11.2.0.2.0
      Enterprise Manager Agent 10.2.0.4.3
      SQL*Plus 11.2.0.2.0
      Oracle Netca Client 11.2.0.2.0
      Oracle Multimedia Locator 11.2.0.2.0
      Oracle Call Interface (OCI) 11.2.0.2.0
      Oracle Multimedia 11.2.0.2.0
      Oracle Net 11.2.0.2.0
      Oracle XML Development Kit 11.2.0.2.0
      Database Configuration and Upgrade Assistants 11.2.0.2.0
      Oracle JVM 11.2.0.2.0
      Oracle Advanced Security 11.2.0.2.0
      Oracle Internet Directory Client 11.2.0.2.0
      Oracle Enterprise Manager Console DB 11.2.0.2.0
      HAS Files for DB 11.2.0.2.0
      Oracle Net Listener 11.2.0.2.0
      Oracle Text 11.2.0.2.0
      Oracle Net Services 11.2.0.2.0
      Oracle Database 11g 11.2.0.2.0
      Oracle OLAP 11.2.0.2.0
      Oracle Spatial 11.2.0.2.0
      Oracle Partitioning 11.2.0.2.0
      Enterprise Edition Options 11.2.0.2.0
-----------------------------------------------------------------------------


Instantiating scripts for add node (Sunday, December 12, 2010 12:27:57 PM EST)
.                                                                 1% Done.
Instantiation of add node scripts complete

Copying to remote nodes (Sunday, December 12, 2010 12:28:04 PM EST)
...............................................................................................                                 96% Done.
Home copied to new nodes

Saving inventory on nodes (Sunday, December 12, 2010 12:38:21 PM EST)
.                                                               100% Done.
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each cluster node.
/u01/app/oracle/product/11.2.0.2/dbhome_1/root.sh #On nodes ayu3
To execute the configuration scripts:
    1. Open a terminal window
    2. Log in as "root"
    3. Run the scripts in each cluster node

The Cluster Node Addition of /u01/app/oracle/product/11.2.0.2/dbhome_1 was successful.
Please check '/tmp/silentInstall.log' for more details.

Execute the root.sh script on the third node as root:
[root@ayu3 ~]# /u01/app/oracle/product/11.2.0.2/dbhome_1/root.sh
Running Oracle 11g root script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/11.2.0.2/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.

Execute the rootcrs.pl script on the third node as root:
[root@ayu3 ~]# /u01/app/11.2.0.2/grid/crs/install/rootcrs.pl
Using configuration parameter file: /u01/app/11.2.0.2/grid/crs/install/crsconfig_params
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Perform post-installation checks
oracle@ayu3-> . ./.profile_grid
oracle@ayu3-> which cluvfy
/u01/app/11.2.0.2/grid/bin/cluvfy
oracle@ayu3-> cluvfy stage -post nodeadd -n ayu3 -verbose
	
Performing post-checks for node addition

Checking node reachability...

Check: Node reachability from node "ayu3"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  ayu3                                  yes
Result: Node reachability check passed from node "ayu3"


Checking user equivalence...

Check: User equivalence for user "oracle"
  Node Name                             Comment
  ------------------------------------  ------------------------
  ayu3                                  passed
Result: User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  ayu3          passed

Verification of the hosts config file successful

<< snipped for clarity >>

Check CTSS state started...
Check: CTSS state
  Node Name                             State
  ------------------------------------  ------------------------
  ayu3                                  Active
CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...
Reference Time Offset Limit: 1000.0 msecs
Check: Reference Time Offset
  Node Name     Time Offset               Status
  ------------  ------------------------  ------------------------
  ayu3          0.0                       passed

Time offset is within the specified limits on the following set of nodes:
"[ayu3]"
Result: Check of clock time offsets passed


Oracle Cluster Time Synchronization Services check passed

Post-check for node addition was successful.

You have just completed the database software installation on the new node. At this stage, you should take a back up of your voting disk and Oracle Cluster Registry file. In the next section, we will create a third database instance on the new node.

4. Add a third database instance on the third node

We will use the Database Configuration Assistant (DBCA) to add the third instance, qdb3.


Execute dbca on ayu3 as oracle:
oracle@ayu1-> . ./.profile
oracle@ayu3-> which dbca
/u01/app/oracle/product/11.2.0.2/dbhome_1/bin/dbca
oracle@ayu3-> dbca
  1. Welcome: Select Oracle Real Application Clusters (RAC) database as the database type that you would like to create or administer.
  2. Operations: Select Instance Management.
  3. Instance Management: Select Add an instance.
  4. List of cluster databases: Enter the sys username and password.
  5. List of cluster database instances: Verify the existing database instances.
  6. Instance naming and node selection: Verify the instance name (qdb3) to be added and the node (ayu3) on which to add the instance.
  7. Instance Storage: Verify the storage configuration.
  8. Summary: Verify the set up and click on OK.

1. Database Configuration Assistant: Click on No.


oracle@ayu3-> ps -ef|grep dbw
oracle   13721     1  0 12:09 ?        00:00:00 asm_dbw0_+ASM3
oracle   21113     1  0 13:03 ?        00:00:00 ora_dbw0_qdb3
oracle   23785  2580  0 13:13 pts/0    00:00:00 grep dbw




oracle@ayu3-> crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.DATA.dg    ora....up.type ONLINE    ONLINE    ayu1
ora....ER.lsnr ora....er.type ONLINE    ONLINE    ayu1
ora....N1.lsnr ora....er.type ONLINE    ONLINE    ayu2
ora....N2.lsnr ora....er.type ONLINE    ONLINE    ayu3
ora....N3.lsnr ora....er.type ONLINE    ONLINE    ayu1
ora.asm        ora.asm.type   ONLINE    ONLINE    ayu1
ora....SM1.asm application    ONLINE    ONLINE    ayu1
ora....U1.lsnr application    ONLINE    ONLINE    ayu1
ora.ayu1.gsd   application    OFFLINE   OFFLINE
ora.ayu1.ons   application    ONLINE    ONLINE    ayu1
ora.ayu1.vip   ora....t1.type ONLINE    ONLINE    ayu1
ora....SM2.asm application    ONLINE    ONLINE    ayu2
ora....U2.lsnr application    ONLINE    ONLINE    ayu2
ora.ayu2.gsd   application    OFFLINE   OFFLINE
ora.ayu2.ons   application    ONLINE    ONLINE    ayu2
ora.ayu2.vip   ora....t1.type ONLINE    ONLINE    ayu2
ora....SM3.asm application    ONLINE    ONLINE    ayu3
ora....U3.lsnr application    ONLINE    ONLINE    ayu3
ora.ayu3.gsd   application    OFFLINE   OFFLINE
ora.ayu3.ons   application    ONLINE    ONLINE    ayu3
ora.ayu3.vip   ora....t1.type ONLINE    ONLINE    ayu3
ora.cvu        ora.cvu.type   ONLINE    ONLINE    ayu1
ora.gns        ora.gns.type   ONLINE    ONLINE    ayu1
ora.gns.vip    ora....ip.type ONLINE    ONLINE    ayu1
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora....network ora....rk.type ONLINE    ONLINE    ayu1
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    ayu1
ora.ons        ora.ons.type   ONLINE    ONLINE    ayu1
ora.qdb.db     ora....se.type ONLINE    ONLINE    ayu1
ora....ry.acfs ora....fs.type ONLINE    ONLINE    ayu1
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    ayu2
ora.scan2.vip  ora....ip.type ONLINE    ONLINE    ayu3
ora.scan3.vip  ora....ip.type ONLINE    ONLINE    ayu1



oracle@ayu3-> crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       ayu1
               ONLINE  ONLINE       ayu2
               ONLINE  ONLINE       ayu3
ora.LISTENER.lsnr
               ONLINE  ONLINE       ayu1
               ONLINE  ONLINE       ayu2
               ONLINE  ONLINE       ayu3
ora.asm
               ONLINE  ONLINE       ayu1                     Started
               ONLINE  ONLINE       ayu2
               ONLINE  ONLINE       ayu3
ora.gsd
               OFFLINE OFFLINE      ayu1
               OFFLINE OFFLINE      ayu2
               OFFLINE OFFLINE      ayu3
ora.net1.network
               ONLINE  ONLINE       ayu1
               ONLINE  ONLINE       ayu2
               ONLINE  ONLINE       ayu3
ora.ons
               ONLINE  ONLINE       ayu1
               ONLINE  ONLINE       ayu2
               ONLINE  ONLINE       ayu3
ora.registry.acfs
               ONLINE  ONLINE       ayu1
               ONLINE  ONLINE       ayu2
               ONLINE  ONLINE       ayu3
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       ayu2
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       ayu3
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       ayu1
ora.ayu1.vip
      1        ONLINE  ONLINE       ayu1
ora.ayu2.vip
      1        ONLINE  ONLINE       ayu2
ora.ayu3.vip
      1        ONLINE  ONLINE       ayu3
ora.cvu
      1        ONLINE  ONLINE       ayu1
ora.gns
      1        ONLINE  ONLINE       ayu1
ora.gns.vip
      1        ONLINE  ONLINE       ayu1
ora.oc4j
      1        ONLINE  ONLINE       ayu1
ora.qdb.db
      1        ONLINE  ONLINE       ayu1                     Open
      2        ONLINE  ONLINE       ayu2                     Open
      3        ONLINE  ONLINE       ayu3                     Open
ora.scan1.vip
      1        ONLINE  ONLINE       ayu2
ora.scan2.vip
      1        ONLINE  ONLINE       ayu3
ora.scan3.vip
      1        ONLINE  ONLINE       ayu1
                                   


oracle@ayu3-> srvctl status database -d qdb
Instance qdb1 is running on node ayu1
Instance qdb2 is running on node ayu2
Instance qdb3 is running on node ayu3


oracle@ayu3-> srvctl config database -d qdb
Database unique name: qdb
Database name: qdb
Oracle home: /u01/app/oracle/product/11.2.0.2/dbhome_1
Oracle user: oracle
Spfile: +DATA/qdb/spfileqdb.ora
Domain: vcconsult.com
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: qdb
Database instances: qdb1,qdb2,qdb3
Disk Groups: DATA
Mount point paths:
Services:
Type: RAC
Database is administrator managed


oracle@ayu3-> srvctl status asm
ASM is running on ayu3,ayu1,ayu2
oracle@ayu3->
oracle@ayu3-> srvctl config asm
ASM home: /u01/app/11.2.0.2/grid
ASM listener: LISTENER



SQL> select thread#,instance,enabled from v$thread;

   THREAD# INSTANCE   ENABLED
---------- ---------- --------
         1 qdb1       PUBLIC
         2 qdb2       PUBLIC
         3 qdb3       PUBLIC


SQL> select * from v$logfile order by 1;

    GROUP# STATUS  TYPE    MEMBER                                        IS_
---------- ------- ------- --------------------------------------------- ---
         1         ONLINE  +DATA/qdb/onlinelog/group_1.261.737743115     NO
         2         ONLINE  +DATA/qdb/onlinelog/group_2.262.737743119     NO
         3         ONLINE  +DATA/qdb/onlinelog/group_3.266.737743701     NO
         4         ONLINE  +DATA/qdb/onlinelog/group_4.267.737743703     NO
         5         ONLINE  +DATA/qdb/onlinelog/group_5.270.737989275     NO
         6         ONLINE  +DATA/qdb/onlinelog/group_6.271.737989279     NO

6 rows selected.



SQL> select file_name from dba_data_files where tablespace_name='UNDOTBS3';

FILE_NAME
--------------------------------------------------------
+DATA/qdb/datafile/undotbs3.269.737989267

Congratulations, you have successfully added a third database instance!

Conclusion

With a few simple steps, a node can be easily added to an existing Oracle RAC 11g Release 2 cluster. Hopefully, this guide has provided a clear and concise method of performing a cluster node addition.

For more information about adding a RAC node, please refer to Oracle Real Application Clusters Administration and Deployment Guide.

Vincent Chan (vkchan99@yahoo.com) is a Senior Computer Systems Architect at Lockheed Martin Corporation. He is an Oracle Certified Master DBA with more than 15 years of experience architecting and implementing Oracle solutions for various clients.

Vincent Chan
Vincent Chan
Vincent Chan is a Senior Consultant at MSD Inc. He is an Oracle Certified Master DBA with more than ten years of experience architecting and implementing Oracle solutions for various clients.

Get the Free Newsletter!

Subscribe to Cloud Insider for top news, trends & analysis

Latest Articles