Oracle RAC Administration - Part 9: Hands on administration

November 16, 2006

Brief intro

As a continuing part of the hands-on articles, we will take a deeper look at such things as errors in the installation (which are actually not errors), ESX host tuning for time synchronization (without which the whole RHEL RAC installation means nothing) and some SRVCTL commands.

Errors come and errors go

Let's take a look at a couple of screen shots and some of the typical Cluster Ready Services (CRS) errors and tricks to bring your RAC services and applications online.

This is a typical placement error that I get on every installation. I think it may have to do with the time issue; we will come to that. If you are using an ESX server to test/develop your RAC, then the information to test and fix your time synchronization issues will certainly come very handy.

It really doesn't mean a thing. I do run into this every time I do my virtual machine restart.

[oracle@vm1rh4 ~]$ $ORA_CRS_HOME/bin/crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....SM1.asm application    ONLINE    UNKNOWN   vm1rh4
ora....H4.lsnr application    ONLINE    UNKNOWN   vm1rh4
ora.vm1rh4.gsd application    ONLINE    UNKNOWN   vm1rh4
ora.vm1rh4.ons application    ONLINE    UNKNOWN   vm1rh4
ora.vm1rh4.vip application    ONLINE    ONLINE    vm1rh4
ora....SM2.asm application    ONLINE    UNKNOWN   vm2rh4
ora....H4.lsnr application    ONLINE    UNKNOWN   vm2rh4
ora.vm2rh4.gsd application    ONLINE    UNKNOWN   vm2rh4
ora.vm2rh4.ons application    ONLINE    UNKNOWN   vm2rh4
ora.vm2rh4.vip application    ONLINE    ONLINE    vm2rh4
ora....SM3.asm application    ONLINE    OFFLINE
ora....H4.lsnr application    ONLINE    OFFLINE
ora.vm3rh4.gsd application    ONLINE    UNKNOWN   vm3rh4
ora.vm3rh4.ons application    ONLINE    UNKNOWN   vm3rh4
ora.vm3rh4.vip application    ONLINE    UNKNOWN   vm1rh4
ora....SM4.asm application    ONLINE    OFFLINE
ora....H4.lsnr application    ONLINE    OFFLINE
ora.vm4rh4.gsd application    ONLINE    ONLINE    vm4rh4
ora.vm4rh4.ons application    ONLINE    ONLINE    vm4rh4
ora.vm4rh4.vip application    ONLINE    UNKNOWN   vm4rh4

This is really a VM issue. I am using a 1.2 G vMEM, 2vCPUs, and decent SCSI Virtual Disks (VMFS-2 file system on a VMDK format). Restarting the services does not go as expected. Let's see what happens if you try to do a crs_stat –stop and then a crs_stop –start.

[oracle@vm1rh4 ~]$ $ORA_CRS_HOME/bin/crs_stop -all
Attempting to stop `ora.vm4rh4.gsd` on member `vm4rh4`
Attempting to stop `ora.vm4rh4.ons` on member `vm4rh4`
Stop of `ora.vm4rh4.gsd` on member `vm4rh4` succeeded.
Stop of `ora.vm4rh4.ons` on member `vm4rh4` succeeded.
Attempting to stop `ora.vm2rh4.LISTENER_VM2RH4.lsnr` on member `vm2rh4`
Attempting to stop `ora.vm1rh4.LISTENER_VM1RH4.lsnr` on member `vm1rh4`
Stop of `ora.vm2rh4.LISTENER_VM2RH4.lsnr` on member `vm2rh4` succeeded.
Attempting to stop `ora.vm2rh4.ASM2.asm` on member `vm2rh4`
Stop of `ora.vm1rh4.LISTENER_VM1RH4.lsnr` on member `vm1rh4` succeeded.
Attempting to stop `ora.vm1rh4.ASM1.asm` on member `vm1rh4`
Stop of `ora.vm2rh4.ASM2.asm` on member `vm2rh4` succeeded.
Attempting to stop `ora.vm2rh4.vip` on member `vm2rh4`
Stop of `ora.vm2rh4.vip` on member `vm2rh4` succeeded.
Stop of `ora.vm1rh4.ASM1.asm` on member `vm1rh4` succeeded.
Attempting to stop `ora.vm1rh4.vip` on member `vm1rh4`
Stop of `ora.vm1rh4.vip` on member `vm1rh4` succeeded.

As you can see, it just does not restart all of the services when we do the crs_stat –t.

[oracle@vm1rh4 ~]$ $ORA_CRS_HOME/bin/crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....SM1.asm application    OFFLINE   OFFLINE
ora....H4.lsnr application    OFFLINE   OFFLINE
ora.vm1rh4.gsd application    ONLINE    UNKNOWN   vm1rh4
ora.vm1rh4.ons application    ONLINE    UNKNOWN   vm1rh4
ora.vm1rh4.vip application    OFFLINE   OFFLINE
ora....SM2.asm application    OFFLINE   OFFLINE
ora....H4.lsnr application    OFFLINE   OFFLINE
ora.vm2rh4.gsd application    ONLINE    UNKNOWN   vm2rh4
ora.vm2rh4.ons application    ONLINE    UNKNOWN   vm2rh4
ora.vm2rh4.vip application    OFFLINE   OFFLINE
ora....SM3.asm application    ONLINE    OFFLINE
ora....H4.lsnr application    ONLINE    OFFLINE
ora.vm3rh4.gsd application    ONLINE    UNKNOWN   vm3rh4
ora.vm3rh4.ons application    ONLINE    UNKNOWN   vm3rh4
ora.vm3rh4.vip application    ONLINE    UNKNOWN   vm1rh4
ora....SM4.asm application    ONLINE    OFFLINE
ora....H4.lsnr application    ONLINE    OFFLINE
ora.vm4rh4.gsd application    OFFLINE   OFFLINE
ora.vm4rh4.ons application    OFFLINE   OFFLINE
ora.vm4rh4.vip application    ONLINE    UNKNOWN   vm4rh4

And this is bizarre , of course, I will test it on a ESX 3.0 with more capacity and see if it vanishes, but now the task is to start all the services one by one, which is not easy when you have “ora....H4.lsnr” names. So, when you do the following, you get full names of the services.

[oracle@vm1rh4 ~]$ $ORA_CRS_HOME/bin/crs_stat

In my case, it gave these and now you have the full names.

NAME=ora.brianic.brianic1.inst
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.brianic.brianic2.inst
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.brianic.brianic3.inst
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.brianic.brianic4.inst
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.brianic.db
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.brianic.fokeserv.brianic1.srv
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm1rh4
NAME=ora.brianic.fokeserv.brianic2.srv
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm2rh4
NAME=ora.brianic.fokeserv.brianic3.srv
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm3rh4
NAME=ora.brianic.fokeserv.brianic4.srv
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm4rh4
NAME=ora.brianic.fokeserv.cs
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm4rh4
NAME=ora.vm1rh4.ASM1.asm
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm1rh4.LISTENER_VM1RH4.lsnr
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm1rh4.gsd
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm1rh4
NAME=ora.vm1rh4.ons
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm1rh4
NAME=ora.vm1rh4.vip
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm2rh4.ASM2.asm
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm2rh4.LISTENER_VM2RH4.lsnr
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm2rh4.gsd
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm2rh4
NAME=ora.vm2rh4.ons
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm2rh4
NAME=ora.vm2rh4.vip
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm3rh4.ASM3.asm
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm3rh4.LISTENER_VM3RH4.lsnr
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm3rh4.gsd
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm3rh4
NAME=ora.vm3rh4.ons
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm3rh4
NAME=ora.vm3rh4.vip
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm4rh4.ASM4.asm
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm4rh4.LISTENER_VM4RH4.lsnr
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm4rh4.gsd
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm4rh4
NAME=ora.vm4rh4.ons
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm4rh4
NAME=ora.vm4rh4.vip
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE